🔍 Read the full analysis: Why FLOPs Underestimate How Long Fine-Tuning Takes on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Hugging Face reports that a CatBoost model predicted runtime for 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP-based estimate had a 12.41-hour error, but the results come from one production pool and do not show that a live scheduler saved time or GPU capacity.
Hugging Face says a CatBoost model predicted the runtime of 272 supervised fine-tuning jobs with a mean absolute error of 1.71 hours in five-fold cross-validation, compared with 12.41 hours for the best tested FLOP-based estimate. The result suggests that compute-only formulas can miss important parts of end-to-end job duration, but it comes from a single production environment and does not establish that using the model improves scheduling in practice.
The jobs ran on shared NVIDIA H200 GPUs in a corporate production environment and accounted for about 7,382 GPU-hours. Their durations ranged from 1.6 to 172.4 hours. The data covered model families up to 15 billion parameters, LoRA and full fine-tuning, different dataset sizes, and jobs assigned one, two or four GPUs. The pool contained 8 to 32 H200s across one to four nodes, with other training and inference workloads running alongside them.
Hugging Face defines runtime as wall-clock time from GPU allocation to job completion; time spent waiting in a queue is excluded. The predictor used information available before dispatch, including configuration and model metadata such as evaluation cadence, checkpoint settings, per-GPU token volumes and fine-tuning method. It also used a FLOP-based duration estimate as one of its inputs. After feature selection and tuning, CatBoost used 24 features.
In the reported five-fold cross-validation, CatBoost had an R² of 0.98, a root mean squared error of 3.52 hours and a median error of 0.51 hours, alongside the 1.71-hour mean absolute error. The FLOP baseline’s best tested setting assumed 30% Model FLOPs Utilization (MFU) and had a 12.41-hour mean absolute error. Its error was 13.01 hours at 45% MFU and 14.62 hours at 60%. The preview says testing a wider range of MFU assumptions, from 25% to 65%, did not resolve the baseline’s shortcomings.
Why Fine-Tuning Takes Longer Than FLOPs Say
Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.
What the estimates capture
The target is time from GPU allocation to job completion; queue time is excluded.
The training loop
FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.
The whole job
CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.
Other runtime costs
Setup, data handling, network and storage can add time alongside training.
Prediction error in the tested data
Lower error means predicted duration was closer to observed duration on average.
What this could mean for GPU scheduling
More useful duration estimates could help teams plan scarce GPU capacity.
- GPUs may sit idle until another job starts.
- Operators can misjudge when capacity will be free.
- Other work can be delayed.
- Delivery dates may shift.
How far the result reaches
The preview reports cross-validation on one corporate production environment.
Runs used shared NVIDIA H200 GPUs and varied configurations.
Accuracy on other clusters, software stacks and workload mixes is not established.
The preview reports prediction error, not savings in GPU time or scheduling delays.
More Reliable GPU Job Scheduling
Runtime predictions help cluster operators decide when to dispatch work and how to allocate scarce GPUs. If a job finishes earlier than expected, its GPUs may sit idle until another job starts; if it runs longer, it can hold resources and delay other jobs or delivery plans. A more accurate estimate could help operators plan allocations around expected completion times.
The reported error gap is large within this dataset, but the study measured prediction accuracy, not operational savings. It does not report a controlled scheduler deployment showing reduced GPU idle time, lower costs or fewer delays. Whether the model helps in practice will depend on how well it predicts other workloads and whether operators can act on its estimates.
Products Worth Considering

Jackery HomePower 3600 Plus Portable Power Station, 3584Wh, 3600W

DJI Mini 4K Drone Combo with 4K Camera, 3-Axis Gimbal, and Two Batteries

LG UltraGear 27-inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDR True Black 400
As an affiliate, we earn on qualifying purchases.
Why FLOPs Miss Job Overheads
A common estimate for dense Transformer training approximates compute as six times the model parameter count and training-token count, then divides by GPU count, peak throughput and an assumed MFU. MFU describes the share of theoretical peak throughput achieved during actual work; the preview describes typical values as spanning roughly 30% to 60%.
That calculation focuses on training compute, while this study measures a job’s full runtime after GPU allocation. Setup, model downloads and loading, tokenization, scheduled evaluations, checkpoint uploads and final storage can all add time. Hugging Face says these tasks vary with job configuration and can compete for network and storage resources on a shared cluster. The researchers included scheduled evaluation tokens in the baseline comparison and tested multiple MFU assumptions, rather than relying on one setting.
“Everything is known before dispatch.”
— Hugging Face study preview
How Far the Results Extend
The reported scores come from cross-validation within one production pool, not testing across independently collected clusters or sites. The preview does not establish how well the predictor would work with other GPU types, software stacks, workload mixes or changing infrastructure conditions. It also does not report a test on a later, unseen period or a live scheduling trial.
The full paper has not yet appeared, and its publication date is unspecified. Further details about the dataset and evaluation design may clarify how robust the reported scores are. It is also not clear how frequently operators would need to retrain the model as job configurations and shared-cluster conditions change.
Full Paper and External Tests
Hugging Face says it plans to post the full paper on arXiv, but has not given a date. That paper may provide more information on the data, feature selection and evaluation design. A next test would be to evaluate the predictor on jobs collected at different times or in other environments, rather than relying only on cross-validation within this production pool.
To show operational value, researchers or cluster operators would also need to test the predictions in a scheduler and measure whether they reduce idle GPU intervals, delays or costs. Until such results are reported, the preview supports a comparison of prediction errors on the study’s dataset, not a claim of proven scheduling gains.
Key Questions
What did Hugging Face report?
Hugging Face reported that CatBoost predicted the end-to-end runtime of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation.
How did the FLOP-based estimate compare?
The best tested FLOP-based setting, at 30% MFU, had a 12.41-hour mean absolute error. The preview also reports errors of 13.01 hours at 45% MFU and 14.62 hours at 60% MFU.
What runtime did the study measure?
It measured wall-clock time from when a job received its GPUs until it finished. Queue time was excluded, while job activities such as setup, evaluation and checkpointing could contribute to the measured duration.
Does the study prove that CatBoost saves GPU time?
No. The preview reports prediction errors from one production environment; it does not describe a controlled live deployment or quantify GPU time, cost or delays saved by changing a scheduler.
When will the full paper be available?
Hugging Face says the full version will be posted on arXiv, but the preview gives no publication date.
Primary source: Hugging Face · via ThorstenMeyerAI.com
