Towards validating and debugging scientific data analysis workflows
Options
BORIS DOI
Subtitle
irregularity localization and contract-driven design
Abstract
The paradigm shift towards data-driven science is massively transforming the
scientific process. Scientific data analysis workflows have become crucial in various domains such as genome sequencing, geographic land classification or materials discovery. These data analysis workflows which are labeled directed graphs of data processing nodes, are used to model complex experiments by connecting numerous scientific software packages. The workflows rapidly become very complex, as the interplay of the various modules to process huge amounts of data also requires sophisticated compute infrastructure with components to manage resources and coordinate execution schedules. Unexpected behaviour and crashes are inevitable in such environments. Because it is often assumed that errors originate from infrastructure failures like node failures, current workflow management systems usually include mechanisms to provide fault tolerance, for example, through re-execution, checkpointing, and over-provisioning of resources, as well as extensive performance monitoring capabilities. Such mechanisms do not address the fundamental problem of dependability though: A workflow may fail since it has never been designed to be applied in contexts other than the one during development, so that validity assumptions on the input data, the task parameters, the applied data partitioning, or the configurations of infrastructure components are violated. When workflows show irregular behavior that not necessarily affect common performance indicators, the human developers and operators have to diagnose the root cause, at present unstructured and manually.
This thesis seeks to contribute to and progress the state of the art in debugging and validating of scientific data analysis workflows. In particular, it explores frameworks and techniques that support users to validate their scientific data analysis workflows and locate causes of observed irregular results in a systematic manner.
In total, three core contributions were made towards the above stated objective. First, to get a better picture of how scientists perform and evaluate debugging of problems in data analysis, a user study was performed where researchers were surveyed on the role of debugging in their data analysis work. The participants noted that debugging makes up a substantial amount of time in the activities around developing data analysis workflows. Answers on the approaches of debugging their problems indicated ad-hoc and manual methods, lacking structure and automation support.
Second, a framework was proposed to automatically reduce a workflow’s input data while still observing some outcome of interest, thereby computing a minimal reproducible example to support workflow debugging. In essence, the problem is reduced to the determination of the input that is relevant to reproducing the irregularity. To that end, a novel reduction strategy is presented that is tailored to data analysis workflows that operate on tabular data. This new strategy is evaluated together with three other from software engineering adapted strategies on their feasibility in terms of input reduction, and their effectiveness and efficiency within three characteristic cases.
And third, contract-driven design of data analysis workflow is introduced, where workflow developers can specify contracts in the form of requirements and promises for each task of a workflow, thus, enabling automated workflow validation. These contracts serve as guards to ensure that tasks run in a proper environment and produce correct results. The work contains a first formal definition of contracts for data analysis workflows and shows how they are connected to workflow scheduling and execution. Furthermore, the feasibility of automatically generating such contracts from logical workflow descriptions and natural language documentation is demonstrated with an LLM-based pipeline called SANA. SANA was shown to be able to generate a substantial amount of high quality contracts. The approach demonstrates potential in assisting domain scientists with contract-driven design of data analysis workflows, laying the groundwork for its future adoption.
scientific process. Scientific data analysis workflows have become crucial in various domains such as genome sequencing, geographic land classification or materials discovery. These data analysis workflows which are labeled directed graphs of data processing nodes, are used to model complex experiments by connecting numerous scientific software packages. The workflows rapidly become very complex, as the interplay of the various modules to process huge amounts of data also requires sophisticated compute infrastructure with components to manage resources and coordinate execution schedules. Unexpected behaviour and crashes are inevitable in such environments. Because it is often assumed that errors originate from infrastructure failures like node failures, current workflow management systems usually include mechanisms to provide fault tolerance, for example, through re-execution, checkpointing, and over-provisioning of resources, as well as extensive performance monitoring capabilities. Such mechanisms do not address the fundamental problem of dependability though: A workflow may fail since it has never been designed to be applied in contexts other than the one during development, so that validity assumptions on the input data, the task parameters, the applied data partitioning, or the configurations of infrastructure components are violated. When workflows show irregular behavior that not necessarily affect common performance indicators, the human developers and operators have to diagnose the root cause, at present unstructured and manually.
This thesis seeks to contribute to and progress the state of the art in debugging and validating of scientific data analysis workflows. In particular, it explores frameworks and techniques that support users to validate their scientific data analysis workflows and locate causes of observed irregular results in a systematic manner.
In total, three core contributions were made towards the above stated objective. First, to get a better picture of how scientists perform and evaluate debugging of problems in data analysis, a user study was performed where researchers were surveyed on the role of debugging in their data analysis work. The participants noted that debugging makes up a substantial amount of time in the activities around developing data analysis workflows. Answers on the approaches of debugging their problems indicated ad-hoc and manual methods, lacking structure and automation support.
Second, a framework was proposed to automatically reduce a workflow’s input data while still observing some outcome of interest, thereby computing a minimal reproducible example to support workflow debugging. In essence, the problem is reduced to the determination of the input that is relevant to reproducing the irregularity. To that end, a novel reduction strategy is presented that is tailored to data analysis workflows that operate on tabular data. This new strategy is evaluated together with three other from software engineering adapted strategies on their feasibility in terms of input reduction, and their effectiveness and efficiency within three characteristic cases.
And third, contract-driven design of data analysis workflow is introduced, where workflow developers can specify contracts in the form of requirements and promises for each task of a workflow, thus, enabling automated workflow validation. These contracts serve as guards to ensure that tasks run in a proper environment and produce correct results. The work contains a first formal definition of contracts for data analysis workflows and shows how they are connected to workflow scheduling and execution. Furthermore, the feasibility of automatically generating such contracts from logical workflow descriptions and natural language documentation is demonstrated with an LLM-based pipeline called SANA. SANA was shown to be able to generate a substantial amount of high quality contracts. The approach demonstrates potential in assisting domain scientists with contract-driven design of data analysis workflows, laying the groundwork for its future adoption.
Date of Publication
2026
Year of graduation
2026
Theses Type
dissertation
Keyword(s)
workflows
•
debugging
•
validity
Language(s)
en
Author(s)
Faculty/Graduate School
Access(Rights)
open.access
Primary OA Publication
true