A trivial Example#

This is a very simple example that shows how to set up the pipeline.

The project folder contains the following files:

📂 examples/trivial
├── 📂 steps
│   ├── __init__.py     # Empty file marking directory as Python package.
│   └── 00_start.py     # Implementation of the first step(s).
│   └── 01_continue.py  # Implementation of the next step.
└── config.py           # Configuration file for the pipeline.

The config file can be used to define custom variables.

config.py#
variable_a = 1

In the context of a Config instance config, such a variable can be accessed as:

config.variable_a

Next we define some steps, i.e. subclasses of Pipeline_Step that implement the step() method. The steps are defined in the steps/ directory.

steps/00_start.py#
from lw_pipeline import Pipeline_Step


class A_First_Start_Pipeline_Step(Pipeline_Step):
    def __init__(self, config):
        super().__init__("This is a description of a first step.", config)

    def step(self, data):
        print(f"Here data is '{data}', we set it to variable_a from the config.")
        data = self.config.variable_a
        return data


class A_Second_Start_Pipeline_Step(Pipeline_Step):
    def __init__(self, config):
        super().__init__("This is a description of a second step.", config)

    def step(self, data):
        print(f"Here data is '{data}'. We will add 1 to it.")
        data += 1
        return data
steps/01_continue.py#
from lw_pipeline import Pipeline_Step


class A_Continued_Pipeline_Step(Pipeline_Step):
    def __init__(self, config):
        super().__init__("This is a description of a continued step.", config)

    def step(self, data):
        print(f"Here data is '{data}'. We will subtract 2.")
        if type(data) is not int:
            data = 0
        data -= 2
        return data

Already with this setups we can run the pipeline with the following command:

$ lw_pipeline -c config.py --run
Output#
Logging to file: .../pipeline.log
Using configuration file (specified): .../config.py
Running entire pipeline
-------------- Step 1: steps.00_start / A_First_Start_Pipeline_Step --------------
- This is a description of a first step.
Here data is 'None', we set it to variable_a from the config.
------------- Step 2: steps.00_start / A_Second_Start_Pipeline_Step --------------
- This is a description of a second step.
Here data is '1'. We will add 1 to it.
------------- Step 3: steps.01_continue / A_Continued_Pipeline_Step --------------
- This is a description of a continued step.
Here data is '2'. We will subtract 2.
-------------------- Pipeline finished with following output: --------------------
0

Note

In fact, running lw_pipeline --run (without specifying a config file) will also work, since the pipeline will look for a config.py in the current directory by default.

This Page