DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models
Paper · https://arxiv.org/abs/2603.26164
Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters but also the selection, composition, and weighting of training data during optimization. However, existing approaches to data selection, data mixture optimization, and data reweighting are often developed in isolated codebases with inconsistent interfaces, hindering reproducibility, fair comparison, and practical integration. In this paper, we present DataFlex, a unified data-centric dynamic training framework built upon LLaMA-Factory. DataFlex supports three major paradigms of dynamic data optimization: sample selection, domain mixture adjustment, and sample reweighting, while remaining fully compatible with the original training workflow. It provides extensible trainer abstractions and modular components, enabling a drop-in replacement for standard LLM training, and unifies key model-dependent operations such as embedding extraction, inference, and gradient computation, with support for large-scale settings including DeepSpeed ZeRO-3. We conduct comprehensive experiments across multiple data-centric methods. Dynamic data selection consistently outperforms static full-data training on MMLU across both Mistral-7B and Llama-3.2-3B. For data mixture, DoReMi and ODM improve both MMLU accuracy and corpus-level perplexity over default proportions when pretraining Qwen2.5-1.5B on SlimPajama at 6B and 30B token scales. DataFlex also achieves consistent runtime improvements over original implementations. These results demonstrate that DataFlex provides an effective, efficient, and reproducible infrastructure for data-centric dynamic training of LLMs.
First seen by Oriel on .
AI summary
DataFlex is a unified LLaMA-Factory framework for reproducible dynamic optimization of LLM training data through selection, mixture adjustment, and reweighting.
- The framework provides modular trainer abstractions and drop-in compatibility with standard LLM training workflows.
- It unifies embedding extraction, inference, and gradient computation, including large-scale training with DeepSpeed ZeRO-3.
- Dynamic data selection outperformed static full-data training on MMLU for Mistral-7B and Llama-3.2-3B.
- DoReMi and ODM improved MMLU accuracy and corpus-level perplexity when training Qwen2.5-1.5B on SlimPajama at 6B and 30B token scales.
- DataFlex also delivered consistent runtime improvements over the original implementations, targeting reproducible comparison and practical integration of data-centric methods.
#llm training #data-centric ai #llama-factory #deepspeed
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hugging Face Papers: 248, latest observation ; 0 over observed period since
Comments
Comment count reported by the source.
- Hugging Face Papers: 4, latest observation ; 0 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Papers | 2603.26164 | https://arxiv.org/abs/2603.26164 Read discussion |