The Computing Innovation Fellows Project

Posted on: Monday, June 8th, 2009
Broad Research Area: Numerical/Scientific Computing / HPC / Data-Intensive Scalable Computing, Other

Research Interests:

My research focuses mainly on enabling the efficient and reliable execution of scientific workflows in distributed environments. In my group at ISI, we have been working with a number researchers in variety of science domains, including astronomy, biology, earthquake science, physics, and others and developing technologies to support their large-scale science. As a result we built a workflow management system, Pegasus: http://pegasus.isi.edu and related technologies including resource provisioning capabilities.
Many scientific disciplines are using workflow technologies to manage their complex analysis. Workflows provide a representation of complex analyses composed of heterogeneous models designed by groups of scientists. At the same time, workflows have also become a useful representation that is used to manage the execution of large-scale computations. This representation not only facilitates overall creation and management of the computation but also builds a foundation upon which results can be validated and shared. Since workflows formally describe the sequence of computational and data management tasks, it is easy to trace back how particular results were derived by examining the associated provenance trail.
In order to use the system, the scientist needs to describe the workflow at a high-level, capturing the computations involved and the data that need to be processed. This description is usually independent of the underlying cyberinfrastructure. The workflow along with information about the execution environment (including data and computational resources) is used by Pegasus to generate an executable workflow and to run it in an efficient and reliable fashion.
Within Pegasus, the Pegasus mapper finds the appropriate software and computational resources where the execution can take place as well as finding copies of the data indicated in the workflow instance. The mapping process can also involve workflow restructuring geared towards optimizing the overall workflow performance as well as workflow transformation geared towards data management and provenance information generation. The result of the mapping process is an executable workflow, which can be executed by a workflow engine that follows the dependencies defined in the workflow and executes the activities defined in the workflow nodes. DAGMan, the workflow engine of Pegasus relies on the resources (compute, storage, and network) defined in the workflow to perform the necessary actions. As part of the execution, data are generated along with their associated metadata and any provenance information that is collected.
A part of our system development we conduct research in the area of workflow scheduling, resource provisioning, data management, cloud computing, and others.
We develop novel algorithms to schedule jobs onto distributed resources, to estimate the resource needs of applications and provision resources accordingly. Through simulation and experimentation we evaluate the applicability and costs of cloud computing for science applications.
Much of our research focus is also on scalability and robustness of our system. In one scientific workflow in earthquake science, we need to manage approximately a million of individual jobs.

Contact Information:

email obfuscated - click to reveal, www.isi.edu/~deelman