cmd_queue.airflow_queue module

UNFINISHED - NOT FUNCTIONAL

Airflow backend

Requires:

pip install apache-airflow pip install apache-airflow[cncf.kubernetes]

class cmd_queue.airflow_queue.AirflowJob(command, name=None, output_fpath=None, depends=None, partition=None, cpus=None, gpus=None, mem=None, begin=None, shell=None, **kwargs)

Bases: Job

Represents a airflow job that hasn’t been executed yet

finalize_text()

class cmd_queue.airflow_queue.AirflowQueue(name=None, shell=None, **kwargs)

Bases: Queue

Example

>>> # xdoctest: +REQUIRES(module:airflow)
>>> # xdoctest: +SKIP
>>> from cmd_queue.airflow_queue import *  # NOQA
>>> self = AirflowQueue('zzz_mydemo')
>>> job1 = self.submit('echo hi 1 && true')
>>> job2 = self.submit('echo hi 2 && true')
>>> job3 = self.submit('echo hi 3 && true', depends=job1)
>>> self.print_commands()
>>> self.write()
>>> self.run()
>>> #self.run()
>>> # self.read_state()

classmethod is_available()

Determines if we can run the tmux queue or not.

run(block=True, system=False)

finalize_text()

submit(command, **kwargs)

print_commands(with_status=False, with_gaurds=False, with_locks=1, exclude_tags=None, style='auto', with_rich=None, colors=1, **kwargs)

Print info about the commands, optionally with rich

Example

>>> # xdoctest: +SKIP
>>> from cmd_queue.airflow_queue import *  # NOQA
>>> self = AirflowQueue()
>>> self.submit('date')
>>> self.print_commands()
>>> self.run()

rprint(with_status=False, with_gaurds=False, with_locks=1, exclude_tags=None, style='auto', with_rich=None, colors=1, **kwargs)

Print info about the commands, optionally with rich

Example

>>> # xdoctest: +SKIP
>>> from cmd_queue.airflow_queue import *  # NOQA
>>> self = AirflowQueue()
>>> self.submit('date')
>>> self.print_commands()
>>> self.run()

cmd_queue.airflow_queue.demo()

Airflow requires initialization:

airflow db init