Hierarchical Reinforcement Learning under hyperparameter variations and uncertainties in the control loop

Hierarchical Reinforcement Learning (HRL) is an extension of traditional Reinforcement Learning (RL) that breaks down a complex task into smaller subtasks at multiple levels of abstraction. It leads to several benefits such as (i) temporal abstraction, (ii) state abstraction, (iii) improved exploration, (iv) long-horizon planning, (v) transfer learning, and (vi) safe and explainable RL, among others. As robots and AI agents are taking on more and more complex tasks, HRL algorithms are becoming increasingly relevant. Although numerous HRL algorithms have demonstrated superior performance on challenging sequential decision-making tasks, comparatively little attention has been given to their robustness under realistic operating conditions such as sensor noise, actuator disturbances, communication delays, and imperfect hyperparameter tuning. These factors are particularly important for real-world robotic and autonomous systems, where uncertainty and latency are unavoidable. The goal of this thesis is to provide a reusable experimental framework for robustness studies in the presence of disturbances. It outlines eight implemented algorithms from both flat-RL and HRL to compare robustness to hyperparameter variations as well as to delay and noise injection in the actions and observations during testing. The results also provide an exploratory comparison among the eight algorithms in the face of increasing levels of disturbances for two Gymnasium-based environments. In the tested settings, all four HRL algorithms were more sensitive to hyperparameter variations than the four flat-RL algorithms, whereas robustness to delays and noise depended more on the individual algorithm and environment than on the presence of a hierarchy. The framework also provides a starting point for benchmarking robustness properties of other algorithms and can be extended to more Gymnasium-based environments.