Title
Applying frequency analysis techniques to dag-based workflows to benchmark and predict resource behavior on non-dedicated clusters
Document Type
Conference Presentation
Department
Electrical and Computer Engineering
Conference Title
2014 IEEE International Conference on Cluster Computing, CLUSTER 2014
Date of Presentation
11-26-2014
Abstract
Today, scientific workflows on high-end nondedicated clusters increasingly resemble directed acyclic graphs (DAGs). The execution trace analysis of the associated DAG-based workflows can provide valuable insights into the system behavior in general, and the occurrences of events like idle times in particular, thereby opening avenues for optimized resource utilization. In this paper, we propose a bipartite tool that uses frequency analysis techniques to benchmark and predict event occurrences in DAG-based workflows; highlighting the system behavior for a given cluster configuration. Using an empirically determined prediction window, the tool parses real-time traces to generate the cumulative distribution function (CDF) of the event occurrences. The CDF is then queried to predict the likelihood of a given number of event instances on the cluster resources in a future time frame. Our results yield average prediction hit-rates as high as 94%. The proposed research enables a runtime system to identify unfavorable event occurrences, thereby allowing for preventive scheduling strategies that maximize system utilization.
First Page
29
Last Page
37
DOI
10.1109/CLUSTER.2014.6968734
Recommended Citation
Pallipuram, V. K.,
Dimarco, J.,
&
Taufer, M.
(2014).
Applying frequency analysis techniques to dag-based workflows to benchmark and predict resource behavior on non-dedicated clusters.
Paper presented at 2014 IEEE International Conference on Cluster Computing, CLUSTER 2014.
https://scholarlycommons.pacific.edu/soecs-facpres/476
