A trivial Example#

This is a very simple example that shows how to set up the pipeline.

The project folder contains the following files:

📂 examples/trivial
├── 📂 steps
│   ├── __init__.py     # Empty file marking directory as Python package.
│   └── 00_start.py     # Implementation of the first step(s).
│   └── 01_continue.py  # Implementation of the next step.
└── config.py           # Configuration file for the pipeline.

The config file can be used to define custom variables.

config.py#
variable_a = 1

In the context of a Config instance config, such a variable can be accessed as:

config.variable_a

Next we define some steps, i.e. subclasses of Pipeline_Step that implement the step() method. The steps are defined in the steps/ directory.

steps/00_start.py#
from lw_pipeline import Pipeline_Step


class A_First_Start_Pipeline_Step(Pipeline_Step):
    def __init__(self, config):
        super().__init__("This is a description of a first step.", config)

    def step(self, data):
        print(f"Here data is '{data}', we set it to variable_a from the config.")
        data = self.config.variable_a
        return data


class A_Second_Start_Pipeline_Step(Pipeline_Step):
    def __init__(self, config):
        super().__init__("This is a description of a second step.", config)

    def step(self, data):
        print(f"Here data is '{data}'. We will add 1 to it.")
        data += 1
        return data
steps/01_continue.py#
from lw_pipeline import Pipeline_Step


class A_Continued_Pipeline_Step(Pipeline_Step):
    def __init__(self, config):
        super().__init__("This is a description of a continued step.", config)

    def step(self, data):
        print(f"Here data is '{data}'. We will subtract 2.")
        if type(data) is not int:
            data = 0
        data -= 2
        return data

Already with this setups we can run the pipeline with the following command:

$ lw_pipeline -c config.py --run
Output#
Using configuration file: config.py.
Running entire pipeline
--------------Step 1: steps.00_start / A_First_Start_Pipeline_Step--------------
ℹ This is a description of a first step.
Here data is 'None', we set it to variable_a from the config.
-------------Step 2: steps.00_start / A_Second_Start_Pipeline_Step--------------
ℹ This is a description of a second step.
Here data is '1'. We will add 1 to it.
-------------Step 3: steps.01_continue / A_Continued_Pipeline_Step--------------
ℹ This is a description of a continued step.
Here data is '2'. We will subtract 2.
--------------------Pipeline finished with following output:--------------------
0

This Page