Ploomber

Ploomber is a Python framework that turns loose collections of Jupyter notebooks and scripts into structured, dependency-aware data pipelines, letting data scientists prototype in their favorite editor and deploy without rewriting code. Pipelines are declared in a simple pipeline.yaml file describing tasks and their upstream dependencies, and Ploomber automatically caches previous results so re-running a pipeline only recomputes the steps affected by a code change, saving significant time during iterative experimentation. Once ready, the same pipeline can be exported to run distributedly on Kubernetes, Apache Airflow, AWS Batch, or SLURM clusters without altering the underlying task code, and a companion feature can automatically refactor a single monolithic notebook into a modular, maintainable pipeline with one command.