yellowdog_provider.sensors package

Submodules

yellowdog_provider.sensors.yellowdog_sensors module

YellowDog Airflow sensors for work requirements, worker pools and compute requirements. The sensors test for when one of a specified set of states is reached.

class yellowdog_provider.sensors.yellowdog_sensors.ComputeRequirementStateSensor(task_id: str, connection_id: Callable[[Context, Environment], str] | str, target_states: list[ComputeRequirementStatus], compute_requirement_id: Callable[[Context, Environment], str] | str | None = None, namespace: Callable[[Context, Environment], str] | str | None = None, compute_requirement_name: Callable[[Context, Environment], str] | str | None = None, poke_interval: float = 60, **kwargs)

Bases: YellowDogSensor

Sensor that tests if a compute requirement has reached one of a specified set of states. Either the compute_requirement_id or both the namespace and compute_requirement_name must be supplied.

Emits the compute requirement ID to XCom using key ‘return_value’.

Parameters:

• task_id (str) – the Airflow task ID

• connection_id (str | Callable) – connection to run the operator with (templated) or a Callable that generates the connection ID

• target_states (list[ComputeRequirementStatus]) – the list of ComputeRequirementStatus states to test for

• compute_requirement_id (str | Callable | None) – the ID of the compute requirement (templated) or a Callable that returns the ID

• namespace (str | Callable | None) – the namespace of the work requirement (templated) or a Callable that returns the namespace

• compute_requirement_name (str | Callable | None) – the name of the compute requirement (templated) or a Callable that returns the compute requirement name

• poke_interval (float) – the time between sensor checks in seconds (default = 60.0)

poke(context) → bool

Tests the compute requirement’s status.

template_fields: Sequence[str] = ('connection_id', 'compute_requirement_id', 'namespace', 'compute_requirement_name')

class yellowdog_provider.sensors.yellowdog_sensors.ObjectName(*values)

Bases: Enum

Object naming.

COMPUTE_REQUIREMENT = 'compute requirement'

WORKER_POOL = 'worker pool'

WORK_REQUIREMENT = 'work requirement'

class yellowdog_provider.sensors.yellowdog_sensors.WorkRequirementStateSensor(task_id: str, connection_id: Callable[[Context, Environment], str] | str, target_states: list[WorkRequirementStatus], work_requirement_id: Callable[[Context, Environment], str] | str | None = None, namespace: Callable[[Context, Environment], str] | str | None = None, work_requirement_name: Callable[[Context, Environment], str] | str | None = None, poke_interval: float = 60, **kwargs)

Bases: YellowDogSensor

Sensor that tests if a work requirement has reached one of a specified set of states. Either the work_requirement_id or both namespace and work_requirement_name must be supplied.

Emits the work requirement ID to XCom using key ‘return_value’.

Parameters:

• task_id (str) – the Airflow task ID

• connection_id (str | Callable) – connection to run the operator with (templated) or a Callable that generates the connection ID

• target_states (list[WorkRequirementStatus]) – the list of WorkRequirementStatus states to test for

• work_requirement_id (str | Callable | None) – the ID of the work requirement (templated) or a Callable that generates the ID

• namespace (str | Callable | None) – the namespace of the work requirement (templated) or a Callable that returns the namespace

• work_requirement_name (str | Callable | None) – the name of a work requirement (templated) or a Callable that generates a work requirement name

• poke_interval (float) – the time between sensor checks in seconds (default = 60.0)

poke(context) → bool

Tests the work requirement’s status.

template_fields: Sequence[str] = ('connection_id', 'work_requirement_id', 'namespace', 'work_requirement_name')

class yellowdog_provider.sensors.yellowdog_sensors.WorkerPoolStateSensor(task_id: str, connection_id: Callable[[Context, Environment], str] | str, target_states: list[WorkerPoolStatus], worker_pool_id: Callable[[Context, Environment], str] | str | None = None, namespace: Callable[[Context, Environment], str] | str | None = None, worker_pool_name: Callable[[Context, Environment], str] | str | None = None, poke_interval: float = 60, **kwargs)

Bases: YellowDogSensor

Sensor that tests if a worker pool has reached one of a specified set of states. Either the worker_pool_id or both the namespace and worker_pool_name must be supplied.

Emits the worker pool ID to XCom using key ‘return_value’.

Parameters:

• task_id (str) – the Airflow task ID

• connection_id (str | Callable) – connection to run the operator with (templated) or a Callable that generates the connection ID

• target_states (list[WorkerPoolStatus]) – the list of WorkerPoolStatus states to test for

• worker_pool_id (str | Callable | None) – the ID of the worker pool (templated) or a Callable that generates the ID

• namespace (str | Callable | None) – the namespace of the worker pool (templated) or a Callable that returns the namespace

• worker_pool_name (str | Callable | None) – the name of a worker pool (templated) or a Callable that generates a worker pool name

• poke_interval (float) – the time between sensor checks in seconds (default = 60.0)

poke(context) → bool

Tests the worker pool’s status.

template_fields: Sequence[str] = ('connection_id', 'worker_pool_id', 'namespace', 'worker_pool_name')

class yellowdog_provider.sensors.yellowdog_sensors.YellowDogSensor(connection_id: Callable[[Context, Environment], str] | str, **kwargs)

Bases: BaseSensorOperator

Base class for YellowDog sensors.

Parameters:

connection_id (str | Callable) – connection to run the operator with (templated) or a Callable that generates the connection ID

check_status(context: Context, object_id: str, object_type: ObjectName, current_state: WorkRequirementStatus | WorkerPoolStatus | ComputeRequirementStatus, target_states: list[WorkRequirementStatus] | list[WorkerPoolStatus] | list[ComputeRequirementStatus], target_states_names: list[str]) → bool

Checks current state against the target state(s) and returns true if the condition is met.

template_fields: Sequence[str] = ('connection_id',)

Module contents