🏗️ Pipeline Architecture

This document provides a deeper look at the internal architecture of the Energy Measurement Pipeline, explaining its stage-based design, execution flow, and extensibility mechanisms.

⚙️ Modular Stage Pipeline

The pipeline is implemented as a pipe-and-filter model composed of modular units called stages. Each stage performs a focused task and passes control to the next stage via a shared context.

Each stage implements the same interface:

class PipelineStage(ABC):
    @abstractmethod
    def run(self, context: dict[str, Any]) -> None:
        ...

🧱 Stage Categories

Stages are grouped by when and how often they are executed:

Stage Type

Frequency

Example Tasks

Pre-Stages

Once per batch

Check RAPL/perf access

Pre-Test Stages

Once per unique commit

Checkout, compile, setup

Batch Stages

Repeated for every test run

Measure energy, cleanup

🔁 Execution Flow

Pipeline (per batch)
├── Pre-Stages (1x)
│   └── e.g. VerifyPerfStage
├── Pre-Test Stages (1x per commit, parallelized)
│   ├── CheckoutStage
│   ├── BuildStage
│   └── JavaSetupStage
└── Batch Stages (N x per commit)
    ├── TemperatureCheckStage
    ├── MeasureEnergyStage
    └── PostTestStage

This model enables:

  • ✅ Pre-building commits once and reusing them

  • ✅ Concurrent stage execution where safe

  • ✅ Fine-grained extensibility per stage group

🧠 Shared Context

All stages receive a context: dict[str, Any] which allows:

  • Passing commit information

  • Communicating control signals (abort_pipeline, build_failed, etc.)

  • Sharing paths, results, and state between stages

Example usage:

context["build_failed"] = True
context["abort_pipeline"] = True

🔌 Plugin System for Stages

Each stage is a self-contained Python class and can be loaded dynamically from user-defined files.

Requirements for a Custom Stage

  • Inherits from PipelineStage

  • Implements run(context: dict) method

  • Exposes get_stage() function (used for dynamic loading)

Example

# modules/python_env_stage.py
class PythonEnvStage(PipelineStage):
    def run(self, context: dict[str, Any]) -> None:
        os.system("pip install -r requirements.txt")

def get_stage():
    return PythonEnvStage()

Then include it in your config:

"modules_enabled": ["python_env_stage.py"]

📁 Directory Layout

.
├── main.py               # CLI entry point
├── pipeline/             # Pipeline engine and interfaces
│   ├── core_stages/      # Built-in stages (checkout, measure, etc.)
│   ├── pipeline.py       # Orchestrator for stages
│   └── stage_interface.py
├── modules/              # Optional user-defined custom stages
├── config/               # Config models (Pydantic)
├── plot.py               # Plotting results
├── sort.py               # Sort results by Git history
├── system_setup.sh       # System preparation script

🛠 Example Execution Flow (High-Level)

main.py measure → load config → prepare repo
        ↓
   gather commits + batch them
        ↓
Run pre-stages (e.g. perf check)
        ↓
Run pre-test stages (in parallel):
    - Checkout → Build → JavaSetup
        ↓
For each commit:
    Repeat batch stages (MeasureEnergy, PostTest) N times
        ↓
Restore repo HEAD

💡 Design Principles

Principle

Implementation

Modularity

Each stage is an isolated Python class

Extensibility

Users can add their own stages dynamically

Separation

Config-driven behavior, no hardcoded logic

Reproducibility

Deterministic commit batching + reuse

Minimal coupling

Context dictionary avoids global state

🔄 Parallelism Strategy

  • Pre-Test stages for different commits are executed in parallel using ProcessPoolExecutor

  • Batch stages are run sequentially per commit to preserve measurement integrity

🚀 Optimization Ideas (Planned)

  • [ ] Detect and skip already measured commits

  • [ ] Smart batching based on CPU temperature

  • [ ] Reuse compiled artifacts across sessions

  • [ ] Live log dashboards

  • [ ] Advanced scheduling policies