📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva-3B, a sovereign LLM trained entirely from scratch with half Italian data, underperforms on Italian academic benchmarks. This challenges assumptions about data size, language specialization, and model scaling in national AI projects.
Italy’s Minerva-3B, a large language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, a result that raises questions about the effectiveness of large-scale native-language investment in sovereign AI projects.
Minerva-3B was developed by Sapienza University of Rome and its partners, utilizing Italy’s national supercomputing infrastructure and open data. Despite the substantial training dataset and significant institutional backing, the model’s performance on complex academic content was near chance, contradicting expectations that larger, native-language models would excel in country-specific benchmarks.
The evaluation, conducted by researchers from the Minerva project, concluded that while dataset composition is important, the overall size and parameter count of the model are more critical for handling complex language tasks. The results suggest that even large investments may not suffice without appropriate scaling, challenging the prevailing assumptions in the European sovereign-LLM movement.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Scaling AI: The AI Governance and Security Playbook for Executives
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Doom's Benchmark: The Game That Measures Machines (Prompt Engineering with AI)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
SO-ARM101 Robotic Arm 3D Printed Parts Only (Leader)
Fully compatible with the official Hugging Face LeRobot library and TheRobotStudio SO-ARM100/SO-101 GitHub repository for seamless integration, tutorials,…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign AI Strategies
The findings from Minerva-3B highlight that significant native-language data and large models alone may not guarantee high performance on complex tasks. This raises questions about the scale of investment needed for truly effective country-specific AI, suggesting a potential reevaluation of current approaches within Europe’s sovereign AI initiatives.
For policymakers and researchers, the results underscore the importance of balancing data quantity, model size, and architectural design. The case demonstrates that substantial technical and infrastructural investments must be matched with realistic expectations about what scale can achieve, especially in languages with complex academic and societal content.
European Sovereign-LLM Development and the Scaling Debate
Italy’s Minerva project represents a deliberate effort to build a sovereign language model from scratch, contrasting with approaches like Portugal’s AMÁLIA, which relied on continuation training of multilingual models with limited native data. Minerva’s development involved 15 researchers, extensive computational resources, and a focus on Italian content, resulting in a model that outperforms multilingual counterparts on Italian benchmarks but performs poorly on academic tests.
This outcome feeds into broader debates about the optimal strategies for national AI development, especially the trade-offs between data size, model scale, and specialization. The European sovereign-LLM movement is grappling with whether current scaling efforts are sufficient or whether more aggressive investment is necessary to achieve meaningful country-specific AI capabilities.
“Despite training on 2.5 trillion tokens, Minerva-3B’s performance on academic benchmarks was near chance, highlighting the limits of current scaling assumptions.”
— Research team member
Unresolved Questions About Model Scaling and Effectiveness
It remains unclear whether further increasing model size or dataset scale will significantly improve performance on complex, country-specific tasks. The long-term effectiveness of the current approach is still under evaluation, and ongoing iterations may alter conclusions.
Additionally, how these findings translate to other languages and national contexts is not yet confirmed, and the optimal balance between data, model size, and architecture remains an open question.
Next Steps for Minerva and European Sovereign AI Projects
The Minerva team plans to continue refining their models, including ongoing experiments with continual training and larger architectures. Further benchmarking and performance evaluations are expected to clarify whether scale alone can overcome current limitations.
European policymakers and AI strategists may reassess funding and development priorities, emphasizing not just data and size but also architectural innovation and task-specific tuning.
Key Questions
Why did Minerva-3B perform poorly on Italian academic tests?
Despite extensive training on a large dataset, the model’s architecture and the scale of parameters may not be sufficient to handle complex, academic content, indicating that data size alone isn’t enough.
Does this mean large native-language models are not worth the investment?
Not necessarily; the results suggest that scale and data are crucial, but effective architecture and task-specific tuning are also essential for high performance.
How does Minerva compare to other European sovereign models?
Minerva’s approach of training from scratch with large native data sets contrasts with models like Portugal’s AMÁLIA, which rely on continuation training of multilingual models. Performance differences highlight the importance of scale and architecture choices.
What does this mean for future European AI initiatives?
It indicates a need to reevaluate assumptions about data size and scale, potentially increasing investment in model architecture, compute resources, and targeted data collection to achieve desired country-specific AI capabilities.
Source: ThorstenMeyerAI.com
