corelay.pipeline.base

Base classes Task and Pipeline.

Classes

Pipeline	Abstract base class for all pipelines using MetaPipeline's tracked Task attributes
Task	A single item in a Pipeline task scheme.
TaskPlug	Plug to ensure all contained objects are Processors

class corelay.pipeline.base.Pipeline(*args, **kwargs)

Bases: Processor

Abstract base class for all pipelines using MetaPipeline’s tracked Task attributes

task_scheme

OrderedDict of Tasks which can be filled with Processor.

Type:

collections.OrderedDict

processes

OrderedDict of the Processor that filled the task_scheme.

Type:

collections.OrderedDict

checkpoint_processes()

Find the checkpoint Processor closest to output and return an collections.OrderedDict of that and all following Processor.

Returns:

The Processor that is the checkpoint closest to output and all its following Processor in an OrderedDict.

Return type:

collections.OrderedDict

Raises:

RuntimeError – If there is not a single checkpoint.

from_checkpoint()

Re-evaluate from last checkpointed Processor using its respective output.

Returns:

Output of the whole pipeline, starting from checkpointed Processor closest to output.

Return type:

object

Raises:

RuntimeError – If the checkpointed Processor closest to output does not have any checkpoint_data store, i.e. the Processor was never called once after being declared a checkpoint.

function(data)

Propagate data through the whole pipeline from front to back, calling all Processors in series.

data

The pipeline input passed to the first Processor in self.processors. Type depends on first Processor.

Returns:

Output of all Processor that are flagged as pipeline outputs. If no processors are flagged as outputs, return the output of the last processor.

Return type:

object

class corelay.pipeline.base.Task(proc_type=<class 'corelay.processor.base.Processor'>, default=<function Task.<lambda>>, **kwargs)

Bases: Slot

A single item in a Pipeline task scheme. Tasks are slots that ensure all contained objects in Plugs and own default values are Processors.

proc_type

Class of the Processor allowed for this Task.

Type:

type

default

Default Processor to use if no Processor is assigned. If a types.FunctionType or types.MethodType, an appropriate FunctionProcessor will be created.

Type:

Processor or types.FunctionType or types.MethodType

proc_kwargs

Keyword arguments to overwrite on the supplied Processor.

Type:

dict

class corelay.pipeline.base.TaskPlug(slot, obj=None, default=None, **kwargs)

Bases: Plug

Plug to ensure all contained objects are Processors