Output Management#

The output management system provides consistent saving of pipeline outputs with automatic path generation, metadata, and flexible control.

Quick Start#

Save outputs using the output_manager available in every Pipeline_Step:

from lw_pipeline import Pipeline_Step

class MyStep(Pipeline_Step):
    def step(self, data):
        # Create a plot
        fig, ax = plt.subplots()
        ax.plot(data)

        # Save it
        self.output_manager.save_figure(fig, "analysis_plot", format="pdf")

        return data

The output manager handles path generation, overwrite checking, and creates sidecar metadata automatically.

Core Components#

Output_Manager#

The Output_Manager class handles saving various output types:

Figures: save_figure(fig, name, format="pdf", ...)
DataFrames: save_dataframe(df, name, format="csv", ...)
Arrays: save_numpy(arr, name, format="npy", ...)
JSON: save_json(data, name, ...)
Text: save_text(text, name, ...)
MNE objects: save_mne_object(raw, name, ...)

All methods automatically:

Prefix filenames with the step ID
Check overwrite settings
Create sidecar JSON with provenance metadata
Handle directory creation

Output Registration#

Use @register_output to declare outputs with automatic existence checking and default parameters:

from lw_pipeline import Pipeline_Step, register_output

class Analysis(Pipeline_Step):

    @register_output(
        "quick_plot",
        "Overview visualization",
        check_exists=True,    # Skip if file exists
        extension=".png",     # Default extension
        suffix="overview"     # Default BIDS suffix
    )
    def create_plot(self):
        # Skipped automatically if:
        # 1. Output disabled in config/CLI
        # 2. File exists and overwrite_mode='never'

        fig = self._make_plot()
        # extension and suffix automatically used from decorator
        self.output_manager.save_figure(fig, "overview")

    @register_output(
        "detailed",
        "Expensive analysis",
        enabled_by_default=False,
        check_exists=True,
        extension=".csv"
    )
    def detailed_analysis(self):
        # Only runs when explicitly requested AND file doesn't exist
        result = expensive_computation()
        self.output_manager.save_dataframe(result, "detailed_results")

The function @register_output has two main functions: 1. Register outputs to manage generation via config/CLI

enabled_by_default controls whether output is generated by default

see below for config/CLI usage

Register the actual output path and check if it exists (depending on overwrite_mode config variable)
- check_exists=True automatically checks if output file exists before running expensive code

Configuration#

Add to your config.py:

# Where to save outputs (defaults to deriv_root)
output_root = "/path/to/outputs"

# Overwrite behavior: "never", "always", "ask", or "ifnewer"
overwrite_mode = "never"

# Select which outputs to generate
outputs_to_generate = ["plot", "stats"]  # or use wildcards: ["*plot*"]

# Skip specific outputs
outputs_to_skip = ["expensive_*"]  # Takes precedence over outputs_to_generate

# Optional: enable performance tracking
output_profiling = True

# Optional: disable automatic sidecar JSON
sidecar_auto_generate = False

Command Line Usage#

List available outputs:

python -m lw_pipeline -c config.py --list-outputs

Generate specific outputs:

# Specific outputs
python -m lw_pipeline -c config.py --run --outputs "plot,stats"

# Wildcards
python -m lw_pipeline -c config.py --run --outputs "*plot*"

# Step-scoped (only from step 01)
python -m lw_pipeline -c config.py --run --outputs "01:*"

Skip outputs:

python -m lw_pipeline -c config.py --run --skip-outputs "expensive_*"

BIDS Path Handling#

By default, outputs use simple paths. To use BIDS structure with MNE-BIDS:

from mne_bids import BIDSPath

def step(self, data):
    # Assume data contains a BIDSPath
    bids_path = data.bids_path

    # Save with BIDS structure
    self.output_manager.save_figure(
        fig,
        name="channel_analysis",
        bids_path=bids_path,
        suffix="plot",
        extension=".pdf"
    )

The output will use MNE-BIDS to generate proper BIDS-compliant paths. The bids_path is updated with description, suffix, and extension parameters.

Examples#

Saving Multiple Output Types#

def step(self, data):
    # Save figure
    self.output_manager.save_figure(fig, "analysis", format="pdf", dpi=300)

    # Save statistics table
    self.output_manager.save_dataframe(stats_df, "statistics", format="csv")

    # Save processed array
    self.output_manager.save_numpy(features, "features", format="npy")

    # Save metadata
    self.output_manager.save_json(params, "parameters")

    return data

Custom Metadata#

metadata = {
    "FilterSettings": {"highpass": 1.0, "lowpass": 40.0},
    "Method": "Butterworth",
    "Order": 4
}

self.output_manager.save_figure(
    fig, "filtered_signal",
    metadata=metadata,
    format="pdf"
)

Source File Tracking#

# Only regenerate if source is newer than output
self.output_manager.save_figure(
    fig, "analysis",
    source_file="/path/to/source/data.fif",
    format="pdf"
)

When overwrite_mode="ifnewer", output is only regenerated if the source file is newer.