Ciaramella, Lorenzo
(2026)
Scalability Analysis of Mixture of Experts models on HPC
architectures.
[Laurea magistrale], Università di Bologna, Corso di Studio in
Matematica [LM-DM270]
Documenti full-text disponibili:
Abstract
The increasing size of Large Language Models (LLMs) has led to growing computational and memory requirements, making their efficient training a challenge. Mixture of Experts (MoE) architectures address this issue through conditional computation, allowing model capacity to increase while activating only a subset of parameters for each input. This thesis investigates the scalability of MoE models on High-Performance Computing (HPC) architectures, with a focus on parallelization strategies and training configurations. Experiments are conducted on the Leonardo supercomputer, analyzing the computational efficiency, memory requirements, and scaling performance of dense and MoE models. The results show that there is no universally optimal configuration, as the optimal choice depends on both the model and the available hardware resources. They also show that MoE models can achieve effective scaling, with their advantages over dense models being particularly evident in terms of token per second (TPS).
Since efficiency alone does not determine model quality, the thesis also explores the integration of MoE architectures with the Joint-Embedding Predictive Architecture (JEPA) paradigm. Experiments are conducted in both fine-tuning and pre-training settings to compare the effect of the JEPA objective on MoE and dense models. The results show that JEPA can be effectively integrated with MoE models, although pre-training reveals a decrease in accuracy not observed in the dense case.
Abstract
The increasing size of Large Language Models (LLMs) has led to growing computational and memory requirements, making their efficient training a challenge. Mixture of Experts (MoE) architectures address this issue through conditional computation, allowing model capacity to increase while activating only a subset of parameters for each input. This thesis investigates the scalability of MoE models on High-Performance Computing (HPC) architectures, with a focus on parallelization strategies and training configurations. Experiments are conducted on the Leonardo supercomputer, analyzing the computational efficiency, memory requirements, and scaling performance of dense and MoE models. The results show that there is no universally optimal configuration, as the optimal choice depends on both the model and the available hardware resources. They also show that MoE models can achieve effective scaling, with their advantages over dense models being particularly evident in terms of token per second (TPS).
Since efficiency alone does not determine model quality, the thesis also explores the integration of MoE architectures with the Joint-Embedding Predictive Architecture (JEPA) paradigm. Experiments are conducted in both fine-tuning and pre-training settings to compare the effect of the JEPA objective on MoE and dense models. The results show that JEPA can be effectively integrated with MoE models, although pre-training reveals a decrease in accuracy not observed in the dense case.
Tipologia del documento
Tesi di laurea
(Laurea magistrale)
Autore della tesi
Ciaramella, Lorenzo
Relatore della tesi
Correlatore della tesi
Scuola
Corso di studio
Indirizzo
Curriculum Generale
Ordinamento Cds
DM270
Parole chiave
Large Language Models (LLMs),Mixture of Experts (MoE),High-Performance Computing (HPC),Distributed Training,Joint-Embedding Predictive Architecture (JEPA),Scalability
Data di discussione della Tesi
30 Ottobre 2026
URI
Altri metadati
Tipologia del documento
Tesi di laurea
(NON SPECIFICATO)
Autore della tesi
Ciaramella, Lorenzo
Relatore della tesi
Correlatore della tesi
Scuola
Corso di studio
Indirizzo
Curriculum Generale
Ordinamento Cds
DM270
Parole chiave
Large Language Models (LLMs),Mixture of Experts (MoE),High-Performance Computing (HPC),Distributed Training,Joint-Embedding Predictive Architecture (JEPA),Scalability
Data di discussione della Tesi
30 Ottobre 2026
URI
Statistica sui download
Gestione del documento: