A Machine Learning and Constraint Programming Approach to Power-Aware Job Dispatching in HPC Systems

The demand for more powerful supercomputers continues to increase along with the types of applications submitted by researchers. Recent developments in the analysis of big data and new paradigms of programming as the map-reduce, put further emphasis on short jobs. Hence, HPC job dispatcher have to rapidly process a large number of short jobs in real time taking into account their energy needs. Since these can be similar, it is effective to exploit past executions information quickly. The existing Machine Learning methods need to be trained on new data, so we build a very simple data-driven model of power consumption that can be updated online. Using data from the Eurora’s monitoring framework we built a data set to develop our ML-based model. We integrated our job power predictor in an existing CP-based dispatcher to obtain a power-aware job dispatcher. We achieved excellent results without affect the HPC systems' quality-of-Service (QoS).