Applying frequency analysis techniques to dag-based workflows to benchmark and predict resource behavior on non-dedicated clusters

Document Type

Conference Presentation

Department

Electrical and Computer Engineering

Conference Title

2014 IEEE International Conference on Cluster Computing, CLUSTER 2014

Date of Presentation

11-26-2014

Abstract

Today, scientific workflows on high-end nondedicated clusters increasingly resemble directed acyclic graphs (DAGs). The execution trace analysis of the associated DAG-based workflows can provide valuable insights into the system behavior in general, and the occurrences of events like idle times in particular, thereby opening avenues for optimized resource utilization. In this paper, we propose a bipartite tool that uses frequency analysis techniques to benchmark and predict event occurrences in DAG-based workflows; highlighting the system behavior for a given cluster configuration. Using an empirically determined prediction window, the tool parses real-time traces to generate the cumulative distribution function (CDF) of the event occurrences. The CDF is then queried to predict the likelihood of a given number of event instances on the cluster resources in a future time frame. Our results yield average prediction hit-rates as high as 94%. The proposed research enables a runtime system to identify unfavorable event occurrences, thereby allowing for preventive scheduling strategies that maximize system utilization.

First Page

29

Last Page

37

DOI

10.1109/CLUSTER.2014.6968734

Link to Full Text

Share

COinS