Ziggy, a Pipeline Management System for Data Analysis Pipelines(ARC-16332-1B)

Ziggy is a software tool that manages complex data analysis pipelines for science missions. It provides marshaling and persisting services for each algorithm step, sequencing for algorithm steps, monitoring, logging, resource management, and exception handling for all stages of data processing. Ziggy provides the ability to select either local processing or processing on the NASA Advanced Supercomputer (NAS); pipelines can contain a mix of algorithms that run locally and others that run on NAS. Data accountability is provided at all stages of processing: for any given data product produced at any stage, a permanent record is saved regarding which software version was used, what parameters and inputs were provided to the algorithm, and much more. Ziggy can run on computers running macOS or Linux. It supports algorithms written in Python, C++, MATLAB, or Java, and additional language support can be added with just a few lines of code.