Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation
Paper
โข 2602.17316 โข Published
deepset
company
Verified
AI & ML interests
Semantic Search, Language models, Domain adaptation, Question Answering
Recent Activity
View all activity
bogdankosticย
authored a paper 8 days ago
bilgeyucelย
updated a Space 19 days ago
Post
347
๐ญ Do thinking traces make Language Models learn better? Curious what others think
๐ฆ๐ฐ๐ฒ๐ป๐ฎ๐ฟ๐ถ๐ผ
You take an instruction-following LM.
You want to train it with a GRPO-style RL algorithm on a task like Tic Tac Toe.
Rewards are outcome-based, applied only at the end of each episode: win/loss/draw, format adherence...
During training, the model could just output answers, but a common choice is to make it also output thinking traces.
๐ง๐ต๐ฒ ๐พ๐๐ฒ๐๐๐ถ๐ผ๐ป
Does forcing the model to produce thinking traces during training actually improve learningโ
๐ฌ I'd like to hear your thoughts. Share ideas and links to relevant papers and resources.
From what I've understood so far, the answer seems to be ๐๐ฒ๐.
1๏ธโฃ If you force the model to think during training, it becomes a model that thinks at inference time. It naturally allocates more budget (tokens) to a problem, which tends to improve performance.
2๏ธโฃ While the model's "reasoning" already exists in its activation space, using explicit thinking traces as a scratchpad allows training to steer and shape that reasoning.
3๏ธโฃ As the model produces more traces during training, the RL algorithm can progressively give higher rewards to the reasoning patterns that lead to better outcomes.
๐ฆ๐ฐ๐ฒ๐ป๐ฎ๐ฟ๐ถ๐ผ
You take an instruction-following LM.
You want to train it with a GRPO-style RL algorithm on a task like Tic Tac Toe.
Rewards are outcome-based, applied only at the end of each episode: win/loss/draw, format adherence...
During training, the model could just output answers, but a common choice is to make it also output thinking traces.
๐ง๐ต๐ฒ ๐พ๐๐ฒ๐๐๐ถ๐ผ๐ป
Does forcing the model to produce thinking traces during training actually improve learningโ
๐ฌ I'd like to hear your thoughts. Share ideas and links to relevant papers and resources.
From what I've understood so far, the answer seems to be ๐๐ฒ๐.
1๏ธโฃ If you force the model to think during training, it becomes a model that thinks at inference time. It naturally allocates more budget (tokens) to a problem, which tends to improve performance.
2๏ธโฃ While the model's "reasoning" already exists in its activation space, using explicit thinking traces as a scratchpad allows training to steer and shape that reasoning.
3๏ธโฃ As the model produces more traces during training, the RL algorithm can progressively give higher rewards to the reasoning patterns that lead to better outcomes.
Post
460
I made a visualization based on the Prime Intellect INTELLECT-3 technical report.
Wild to see how far they pushed GLM-4.5-Air-Base with SFT + RL.
SOTA for its size and competitive with models 3x larger.
All open.
Congrats on the release!
Model: PrimeIntellect/INTELLECT-3
Technical report: https://storage.googleapis.com/intellect-3-paper/INTELLECT_3_Technical_Report.pdf
Chat: https://chat.primeintellect.ai/
Wild to see how far they pushed GLM-4.5-Air-Base with SFT + RL.
SOTA for its size and competitive with models 3x larger.
All open.
Congrats on the release!
Model: PrimeIntellect/INTELLECT-3
Technical report: https://storage.googleapis.com/intellect-3-paper/INTELLECT_3_Technical_Report.pdf
Chat: https://chat.primeintellect.ai/
Post
2888
LLMs can leak their post-training data (RL included) ๐ง
New interesting paper on this topic from Google DeepMind: Extracting alignment data in open models (2510.18554)
It's known that Language Models memorize data that can be extracted via prompting.
In this paper, the authors investigate this aspect:
- using open models, where prompting can be fully customized by the user, including special tokens.
- focusing on open-source models like Olmo, where full training data is available.
๐ค How do they extract data?
During post-training (like SFT), new tokens such as <|user|> are introduced.
The authors hypothesize prompting the model with these tokens can make it output its alignment data (remember Magpie?).
For example, for SFT, their extraction prompt is <|endoftext|><|user|>.
๐ Evaluating memorization
The authors compare each sampled example with the original data using vector search with embedding similarity.
They find that many outputs are semantically very similar to the original data, even if the exact words differ.
Traditional string-matching algorithms underestimate memorization by 10x.
๐ What about RL?
Surprisingly, the same technique works to extract data from Reinforcement Learning (PPO/GRPO) phases.
This is counter-intuitive because the RL objective is not designed to increase sequence likelihoods (unlike SFT).
Practical limitation: in this case, extraction relies on using the initial part of the training prompt, which is not generally public.
๐ Are the extracted data effective for post-training?
Both in SFT and RL, the extracted data can be used to fine-tune models to similar performance to the originals.
The authors suggest that model distillation, where a stronger model is used to drive the training of a weaker one, may be a form of indirect training on the original dataset.
New interesting paper on this topic from Google DeepMind: Extracting alignment data in open models (2510.18554)
It's known that Language Models memorize data that can be extracted via prompting.
In this paper, the authors investigate this aspect:
- using open models, where prompting can be fully customized by the user, including special tokens.
- focusing on open-source models like Olmo, where full training data is available.
๐ค How do they extract data?
During post-training (like SFT), new tokens such as <|user|> are introduced.
The authors hypothesize prompting the model with these tokens can make it output its alignment data (remember Magpie?).
For example, for SFT, their extraction prompt is <|endoftext|><|user|>.
๐ Evaluating memorization
The authors compare each sampled example with the original data using vector search with embedding similarity.
They find that many outputs are semantically very similar to the original data, even if the exact words differ.
Traditional string-matching algorithms underestimate memorization by 10x.
๐ What about RL?
Surprisingly, the same technique works to extract data from Reinforcement Learning (PPO/GRPO) phases.
This is counter-intuitive because the RL objective is not designed to increase sequence likelihoods (unlike SFT).
Practical limitation: in this case, extraction relies on using the initial part of the training prompt, which is not generally public.
๐ Are the extracted data effective for post-training?
Both in SFT and RL, the extracted data can be used to fine-tune models to similar performance to the originals.
The authors suggest that model distillation, where a stronger model is used to drive the training of a weaker one, may be a form of indirect training on the original dataset.
Post
498
Your Language Model needs better (open) environments to learn ๐
๐ https://huggingface.co/blog/anakin87/environments-hub
RL environments help LLMs practice, reason, and improve.
I explored the Environments Hub and wrote a walkthrough showing how to train and evaluate models using these open environments.
1๏ธโฃ ๐ช๐ต๐ ๐ฅ๐ ๐บ๐ฎ๐๐๐ฒ๐ฟ๐ ๐ณ๐ผ๐ฟ ๐๐๐ ๐
DeepSeek-R1 made clear that Reinforcement Learning can be used to incentivize reasoning in LLMs.
In GRPO, the model generates multiple answers and learns to prefer the better ones from rewards.
2๏ธโฃ ๐ช๐ต๐ฎ๐ ๐ฒ๐ป๐๐ถ๐ฟ๐ผ๐ป๐บ๐ฒ๐ป๐๐ ๐ฎ๐ฟ๐ฒ
In classic RL, the environment is the world where the Agent lives, interacts, and get rewards to learn.
We can also think of them as software packages, containing data, harness and scoring rules - for the model
to learn and be evaluated.
Nowadays, the Agent is not just the LLM. It can use tools, from a weather API to a terminal.
This makes environments for training and evaluation more complex and critical.
3๏ธโฃ ๐๐ก๐ ๐จ๐ฉ๐๐ง ๐๐ก๐๐ฅ๐ฅ๐๐ง๐ ๐
Big labs are advancing, but open models and the community still face a fragmented ecosystem.
We risk becoming users of systems built with tools we can't access or fully understand.
4๏ธโฃ ๐๐ง๐ฏ๐ข๐ซ๐จ๐ง๐ฆ๐๐ง๐ญ๐ฌ ๐๐ฎ๐
That's why, I was excited when Prime Intellect released the Environments Hub.
It's a place where people share RL environments: tasks you can use to train LLMs with RL (GRPO-style) or evaluate Agents.
Plus, the Verifiers library ( @willcb ) standardizes the creation of RL environments and evaluations.
They can help to keep science and experimentation open. ๐ฌ
I explored the Hub and wrote a hands-on walkthrough ๐
- RL + LLMs basics
- Environments Hub navigation
- Evaluating models/Agents
- GRPO Training a tiny model on an alphabetical sort task
Take a look!
๐ https://huggingface.co/blog/anakin87/environments-hub
๐ https://huggingface.co/blog/anakin87/environments-hub
RL environments help LLMs practice, reason, and improve.
I explored the Environments Hub and wrote a walkthrough showing how to train and evaluate models using these open environments.
1๏ธโฃ ๐ช๐ต๐ ๐ฅ๐ ๐บ๐ฎ๐๐๐ฒ๐ฟ๐ ๐ณ๐ผ๐ฟ ๐๐๐ ๐
DeepSeek-R1 made clear that Reinforcement Learning can be used to incentivize reasoning in LLMs.
In GRPO, the model generates multiple answers and learns to prefer the better ones from rewards.
2๏ธโฃ ๐ช๐ต๐ฎ๐ ๐ฒ๐ป๐๐ถ๐ฟ๐ผ๐ป๐บ๐ฒ๐ป๐๐ ๐ฎ๐ฟ๐ฒ
In classic RL, the environment is the world where the Agent lives, interacts, and get rewards to learn.
We can also think of them as software packages, containing data, harness and scoring rules - for the model
to learn and be evaluated.
Nowadays, the Agent is not just the LLM. It can use tools, from a weather API to a terminal.
This makes environments for training and evaluation more complex and critical.
3๏ธโฃ ๐๐ก๐ ๐จ๐ฉ๐๐ง ๐๐ก๐๐ฅ๐ฅ๐๐ง๐ ๐
Big labs are advancing, but open models and the community still face a fragmented ecosystem.
We risk becoming users of systems built with tools we can't access or fully understand.
4๏ธโฃ ๐๐ง๐ฏ๐ข๐ซ๐จ๐ง๐ฆ๐๐ง๐ญ๐ฌ ๐๐ฎ๐
That's why, I was excited when Prime Intellect released the Environments Hub.
It's a place where people share RL environments: tasks you can use to train LLMs with RL (GRPO-style) or evaluate Agents.
Plus, the Verifiers library ( @willcb ) standardizes the creation of RL environments and evaluations.
They can help to keep science and experimentation open. ๐ฌ
I explored the Hub and wrote a hands-on walkthrough ๐
- RL + LLMs basics
- Environments Hub navigation
- Evaluating models/Agents
- GRPO Training a tiny model on an alphabetical sort task
Take a look!
๐ https://huggingface.co/blog/anakin87/environments-hub
Post
4741
Want to quickly try Gemma 3 270m? ๐๐ฌ
I made a simple Space to do that: anakin87/gemma-3-270m-it
โก Fast: Flash Attention, Zero GPU
โ๏ธ Configurable
I made a simple Space to do that: anakin87/gemma-3-270m-it
โก Fast: Flash Attention, Zero GPU
โ๏ธ Configurable
Post
399
๐ต๏ธ๐ Building Browser Agents - notebook
No API? No problem.
Browser Agents can use websites like you do: click, type, wait, read.
๐ Step-by-step notebook: https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/browser_agents.ipynb
๐ฅ In the video, the Agent:
- Goes to Hugging Face Spaces
- Finds black-forest-labs/FLUX.1-schnell
- Expands a short prompt ("my holiday on Lake Como") into a detailed image generation prompt
- Waits for the image
- Returns the image URL
## What else can it do?
Great for information gathering and summarization
๐๏ธ๐๏ธ Compare news websites and create a table of shared stories with links
โถ๏ธ Find content creator social profiles from YouTube videos
๐๏ธ Find a product's price range on Amazon
๐ ๐ Gather public transportation travel options
## How is it built?
๐๏ธ Haystack โ Agent execution logic
๐ง Google Gemini 2.5 Flash โ Good and fast LLM with a generous free tier
๐ ๏ธ Playwright MCP server โ Browser automation tools: navigate, click, type, wait...
Even without vision capabilities, this setup can get quite far.
## Next steps
- Try a local open model
- Move from notebook to real deployment
- Incorporate vision
And you? Have you built something similar? What's in your stack?
No API? No problem.
Browser Agents can use websites like you do: click, type, wait, read.
๐ Step-by-step notebook: https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/browser_agents.ipynb
๐ฅ In the video, the Agent:
- Goes to Hugging Face Spaces
- Finds black-forest-labs/FLUX.1-schnell
- Expands a short prompt ("my holiday on Lake Como") into a detailed image generation prompt
- Waits for the image
- Returns the image URL
## What else can it do?
Great for information gathering and summarization
๐๏ธ๐๏ธ Compare news websites and create a table of shared stories with links
โถ๏ธ Find content creator social profiles from YouTube videos
๐๏ธ Find a product's price range on Amazon
๐ ๐ Gather public transportation travel options
## How is it built?
๐๏ธ Haystack โ Agent execution logic
๐ง Google Gemini 2.5 Flash โ Good and fast LLM with a generous free tier
๐ ๏ธ Playwright MCP server โ Browser automation tools: navigate, click, type, wait...
Even without vision capabilities, this setup can get quite far.
## Next steps
- Try a local open model
- Move from notebook to real deployment
- Incorporate vision
And you? Have you built something similar? What's in your stack?
Post
1092
Haystack can now see ๐
The latest release of the Haystack OSS LLM framework adds a long-requested feature: image support!
๐ Notebooks below
This isn't just about passing images to an LLM. We built several features to enable practical multimodal use cases.
What's new?
๐ง Support for multiple LLM providers: OpenAI, Amazon Bedrock, Google Gemini, Mistral, NVIDIA, OpenRouter, Ollama and more (support for Hugging Face API coming ๐)
๐๏ธ Prompt template language to handle structured inputs, including images
๐ PDF and image converters
๐ Image embedders using CLIP-like models
๐งพ LLM-based extractor to pull text from images
๐งฉ Components to build multimodal RAG pipelines and Agents
I had the chance of leading this effort with @sjrhuschlee (great collab).
๐ Below you can find two notebooks to explore the new features:
๓ ฏโข๓ ๓ Introduction to Multimodal Text Generation https://haystack.deepset.ai/cookbook/multimodal_intro
๓ ฏโข๓ ๓ Creating Vision+Text RAG Pipelines https://haystack.deepset.ai/tutorials/46_multimodal_rag
(๐ผ๏ธ image by @bilgeyucel )
The latest release of the Haystack OSS LLM framework adds a long-requested feature: image support!
๐ Notebooks below
This isn't just about passing images to an LLM. We built several features to enable practical multimodal use cases.
What's new?
๐ง Support for multiple LLM providers: OpenAI, Amazon Bedrock, Google Gemini, Mistral, NVIDIA, OpenRouter, Ollama and more (support for Hugging Face API coming ๐)
๐๏ธ Prompt template language to handle structured inputs, including images
๐ PDF and image converters
๐ Image embedders using CLIP-like models
๐งพ LLM-based extractor to pull text from images
๐งฉ Components to build multimodal RAG pipelines and Agents
I had the chance of leading this effort with @sjrhuschlee (great collab).
๐ Below you can find two notebooks to explore the new features:
๓ ฏโข๓ ๓ Introduction to Multimodal Text Generation https://haystack.deepset.ai/cookbook/multimodal_intro
๓ ฏโข๓ ๓ Creating Vision+Text RAG Pipelines https://haystack.deepset.ai/tutorials/46_multimodal_rag
(๐ผ๏ธ image by @bilgeyucel )
Post
449
๐ก๏ธ AI Guardrails with Open Language Models - Tutorial
๐ https://haystack.deepset.ai/cookbook/safety_moderation_open_lms
How do you ensure your AI application is safe from harmful or inappropriate user inputs?
This is a core requirement for real-world AI deployments. Luckily, several open Language Models are built specifically for safety moderation.
I've been exploring them and put together a hands-on tutorial using the Haystack framework to build your own AI guardrails.
In the notebook, you'll learn how to use and customize:
๐น Meta Llama Guard (via Hugging Face API)
๐น IBM Granite Guardian (via Ollama), which can also evaluate RAG specific risk dimensions
๐น Google ShieldGemma (via Ollama)
๐น Nvidia NemoGuard models family, including a model for topic control
You'll also see how to integrate content moderation into a ๐ RAG pipeline.
๐ https://haystack.deepset.ai/cookbook/safety_moderation_open_lms
How do you ensure your AI application is safe from harmful or inappropriate user inputs?
This is a core requirement for real-world AI deployments. Luckily, several open Language Models are built specifically for safety moderation.
I've been exploring them and put together a hands-on tutorial using the Haystack framework to build your own AI guardrails.
In the notebook, you'll learn how to use and customize:
๐น Meta Llama Guard (via Hugging Face API)
๐น IBM Granite Guardian (via Ollama), which can also evaluate RAG specific risk dimensions
๐น Google ShieldGemma (via Ollama)
๐น Nvidia NemoGuard models family, including a model for topic control
You'll also see how to integrate content moderation into a ๐ RAG pipeline.
Post
1213
๐งฐ Free up space on the Hub with
As you may know, Hugging Face Hub has storage limits on private repos (100 GB for free users, 1 TB for PROs).
This weekend I did some cleanup on my private repos
I went 1.58 TB down to 1 GB. ๐
Besides deleting old, unused models, the main tool I used was a lesser-known command:
When you train a model, you often push multiple checkpoints to the Hub.
Each checkpoint = a commit.
A 2.6B model in BF16 is ~5 GB.
So 10 checkpoints = 50 GB. That adds up fast.
While full commit history can be useful for rollbacks, it's often unnecessary for older experiments where only the final model matters.
In these cases, you can use
https://huggingface.co/docs/huggingface_hub/main/en/package_reference/hf_api#huggingface_hub.HfApi.super_squash_history
โ ๏ธ super_squash_history is a non-revertible operation. Once squashed, the commit history cannot be retrieved.
Hope this is useful to others.
super_squash_history ๐งนAs you may know, Hugging Face Hub has storage limits on private repos (100 GB for free users, 1 TB for PROs).
This weekend I did some cleanup on my private repos
I went 1.58 TB down to 1 GB. ๐
Besides deleting old, unused models, the main tool I used was a lesser-known command:
super_squash_history.When you train a model, you often push multiple checkpoints to the Hub.
Each checkpoint = a commit.
A 2.6B model in BF16 is ~5 GB.
So 10 checkpoints = 50 GB. That adds up fast.
While full commit history can be useful for rollbacks, it's often unnecessary for older experiments where only the final model matters.
In these cases, you can use
super_squash_history: it reduces your entire repo history to a single commit.https://huggingface.co/docs/huggingface_hub/main/en/package_reference/hf_api#huggingface_hub.HfApi.super_squash_history
โ ๏ธ super_squash_history is a non-revertible operation. Once squashed, the commit history cannot be retrieved.
Hope this is useful to others.
- 2 replies
anakin87ย
updated a Space 9 months ago
robert
#30 opened 10 months ago
by
arvindparihar
Post
3517
๐ ๐๐ฟ๐ฎ๐ถ๐ป๐ฒ๐ฑ ๐ฎ ๐๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐ ๐ผ๐ฑ๐ฒ๐น ๐๐ผ ๐๐ฐ๐ต๐ฒ๐ฑ๐๐น๐ฒ ๐ฒ๐๐ฒ๐ป๐๐ ๐๐ถ๐๐ต ๐๐ฅ๐ฃ๐ข! ๐ ๐๏ธ
โ๏ธ Blog post: https://huggingface.co/blog/anakin87/qwen-scheduler-grpo
I experimented with GRPO lately.
I am fascinated by models learning from prompts and rewards - no example answers needed like in Supervised Fine-Tuning.
After the DeepSeek boom, everyone is trying GRPO with GSM8K or the Countdown Game...
I wanted a different challenge, like ๐๐ฒ๐ฎ๐ฐ๐ต๐ถ๐ป๐ด ๐ฎ ๐บ๐ผ๐ฑ๐ฒ๐น ๐๐ผ ๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ฎ ๐๐ฐ๐ต๐ฒ๐ฑ๐๐น๐ฒ ๐ณ๐ฟ๐ผ๐บ ๐ฎ ๐น๐ถ๐๐ ๐ผ๐ณ ๐ฒ๐๐ฒ๐ป๐๐ ๐ฎ๐ป๐ฑ ๐ฝ๐ฟ๐ถ๐ผ๐ฟ๐ถ๐๐ถ๐ฒ๐.
Choosing an original problem forced me to:
๐ค Think about the problem setting
๐งฌ Generate data
๐ค Choose the right base model
๐ Design reward functions (and experiencing reward hacking)
๐ Run multiple rounds of training, hoping that my model would learn something.
A fun and rewarding ๐ experience.
I learned a lot of things, that I want to share with you. ๐
โ๏ธ Blog post: https://huggingface.co/blog/anakin87/qwen-scheduler-grpo
๐ป Code: https://github.com/anakin87/qwen-scheduler-grpo
๐ค Hugging Face collection (dataset and model): anakin87/qwen-scheduler-grpo-680bcc583e817390525a8837
โ๏ธ Blog post: https://huggingface.co/blog/anakin87/qwen-scheduler-grpo
I experimented with GRPO lately.
I am fascinated by models learning from prompts and rewards - no example answers needed like in Supervised Fine-Tuning.
After the DeepSeek boom, everyone is trying GRPO with GSM8K or the Countdown Game...
I wanted a different challenge, like ๐๐ฒ๐ฎ๐ฐ๐ต๐ถ๐ป๐ด ๐ฎ ๐บ๐ผ๐ฑ๐ฒ๐น ๐๐ผ ๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ฎ ๐๐ฐ๐ต๐ฒ๐ฑ๐๐น๐ฒ ๐ณ๐ฟ๐ผ๐บ ๐ฎ ๐น๐ถ๐๐ ๐ผ๐ณ ๐ฒ๐๐ฒ๐ป๐๐ ๐ฎ๐ป๐ฑ ๐ฝ๐ฟ๐ถ๐ผ๐ฟ๐ถ๐๐ถ๐ฒ๐.
Choosing an original problem forced me to:
๐ค Think about the problem setting
๐งฌ Generate data
๐ค Choose the right base model
๐ Design reward functions (and experiencing reward hacking)
๐ Run multiple rounds of training, hoping that my model would learn something.
A fun and rewarding ๐ experience.
I learned a lot of things, that I want to share with you. ๐
โ๏ธ Blog post: https://huggingface.co/blog/anakin87/qwen-scheduler-grpo
๐ป Code: https://github.com/anakin87/qwen-scheduler-grpo
๐ค Hugging Face collection (dataset and model): anakin87/qwen-scheduler-grpo-680bcc583e817390525a8837
- 2 replies
Post
1753
๐๐๐ฐ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐ฆ๐๐ฅ๐ฅ ๐๐๐ง๐ ๐ฎ๐๐ ๐ ๐๐จ๐๐๐ฅ๐ฌ: ๐๐๐ฆ๐ฆ๐ ๐๐๐จ๐ ๐๐ง๐๐ฌ๐ข๐ฌ ๐๐จ๐ฅ๐ฅ๐๐๐ญ๐ข๐จ๐ง ๐๐๐ฎ๐น
I am happy to release two new language models for the Italian Language!
๐ช Gemma 2 9B Neogenesis ITA
anakin87/gemma-2-9b-neogenesis-ita
Building on the impressive work by VAGO Solutions, I applied Direct Preference Optimization with a mix of Italian and English data.
Using Spectrum, I trained 20% of model layers.
๐ Evaluated on the Open ITA LLM leaderboard ( mii-llm/open_ita_llm_leaderboard), this model achieves strong performance.
To beat it on this benchmark, you'd need a 27B model ๐
๐ค Gemma 2 2B Neogenesis ITA
anakin87/gemma-2-2b-neogenesis-ita
This smaller variant is fine-tuned from the original Gemma 2 2B it by Google.
Through a combination of Supervised Fine-Tuning and Direct Preference Optimization, I trained 25% of the layers using Spectrum.
๐ Compared to the original model, it shows improved Italian proficiency, good for its small size.
Both models were developed during the recent #gemma competition on Kaggle.
๐ Training code: https://www.kaggle.com/code/anakin87/post-training-gemma-for-italian-and-beyond
๐ Thanks @FinancialSupport and mii-llm for the help during evaluation.
I am happy to release two new language models for the Italian Language!
๐ช Gemma 2 9B Neogenesis ITA
anakin87/gemma-2-9b-neogenesis-ita
Building on the impressive work by VAGO Solutions, I applied Direct Preference Optimization with a mix of Italian and English data.
Using Spectrum, I trained 20% of model layers.
๐ Evaluated on the Open ITA LLM leaderboard ( mii-llm/open_ita_llm_leaderboard), this model achieves strong performance.
To beat it on this benchmark, you'd need a 27B model ๐
๐ค Gemma 2 2B Neogenesis ITA
anakin87/gemma-2-2b-neogenesis-ita
This smaller variant is fine-tuned from the original Gemma 2 2B it by Google.
Through a combination of Supervised Fine-Tuning and Direct Preference Optimization, I trained 25% of the layers using Spectrum.
๐ Compared to the original model, it shows improved Italian proficiency, good for its small size.
Both models were developed during the recent #gemma competition on Kaggle.
๐ Training code: https://www.kaggle.com/code/anakin87/post-training-gemma-for-italian-and-beyond
๐ Thanks @FinancialSupport and mii-llm for the help during evaluation.
-
- 3 replies
Post
574
Hey, it has been a while... I was busy participating in ๐ ๐๐๐ฆ๐ฆ๐ ๐๐จ๐ฆ๐ฉ๐๐ญ๐ข๐ญ๐ข๐จ๐ง!
Here's the idea: Gemma open models have a large vocabulary size (256K), so improving them for a specific language or cultural context should be pretty affordable - no need for continued pre-training.
My submission: ๐๐๐ฎ๐น ๐๐๐จ๐ ๐๐ง๐๐ฌ๐ข๐ฌ - ๐๐จ๐ฌ๐ญ-๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐๐ฆ๐ฆ๐ ๐๐จ๐ซ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐ง๐ ๐๐๐ฒ๐จ๐ง๐
๐ Kaggle notebook: https://www.kaggle.com/code/anakin87/post-training-gemma-for-italian-and-beyond
In this notebook, I show how I improve the performance of Gemma 2 2B on Italian via Post-Training.
I believe this method is adaptable to other languages and model sizes.
๐๐ฆ๐บ ๐๐ต๐ฆ๐ฑ๐ด
๐ Choose reference metrics
๐งโ๐ฌ Data curation for Instruction Fine Tuning: identify existing datasets + generate synthetic data
๐๏ธโโ๏ธ Efficient Instruction Fine Tuning with Spectrum
๐งโ๐ฌ Data curation for Preference Tuning: identify existing datasets + generate synthetic data
๐๐ Efficient Direct Preference Optimization with Spectrum
๐ Evaluation
๐ค Hugging Face collection (with models and datasets): anakin87/gemma-neogenesis-67824b7bf13ac9cfe091fe2e
I'm also planning a ๐ Gemma Giveaway (on LinkedIn - https://www.linkedin.com/in/stefano-fiorucci) in the next few days - sharing techniques, datasets, and models I used for my project... so stay tuned! ๐ป
Here's the idea: Gemma open models have a large vocabulary size (256K), so improving them for a specific language or cultural context should be pretty affordable - no need for continued pre-training.
My submission: ๐๐๐ฎ๐น ๐๐๐จ๐ ๐๐ง๐๐ฌ๐ข๐ฌ - ๐๐จ๐ฌ๐ญ-๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐๐ฆ๐ฆ๐ ๐๐จ๐ซ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐ง๐ ๐๐๐ฒ๐จ๐ง๐
๐ Kaggle notebook: https://www.kaggle.com/code/anakin87/post-training-gemma-for-italian-and-beyond
In this notebook, I show how I improve the performance of Gemma 2 2B on Italian via Post-Training.
I believe this method is adaptable to other languages and model sizes.
๐๐ฆ๐บ ๐๐ต๐ฆ๐ฑ๐ด
๐ Choose reference metrics
๐งโ๐ฌ Data curation for Instruction Fine Tuning: identify existing datasets + generate synthetic data
๐๏ธโโ๏ธ Efficient Instruction Fine Tuning with Spectrum
๐งโ๐ฌ Data curation for Preference Tuning: identify existing datasets + generate synthetic data
๐๐ Efficient Direct Preference Optimization with Spectrum
๐ Evaluation
๐ค Hugging Face collection (with models and datasets): anakin87/gemma-neogenesis-67824b7bf13ac9cfe091fe2e
I'm also planning a ๐ Gemma Giveaway (on LinkedIn - https://www.linkedin.com/in/stefano-fiorucci) in the next few days - sharing techniques, datasets, and models I used for my project... so stay tuned! ๐ป
Post
1675
Tulu 3 SFT Mixture by AllenAI is a massive, good, multilingual dataset for fine-tuning Language Models.
Unfortunately, it was missing the "language" column.
I added it using the good old fastText.
Check out the dataset here ๐ anakin87/tulu-3-sft-mixture-with-language
Unfortunately, it was missing the "language" column.
I added it using the good old fastText.
Check out the dataset here ๐ anakin87/tulu-3-sft-mixture-with-language
-
- 1 reply
Post
444
๐๐๐ ๐ ๐๐ฐ๐๐ซ๐ฆ ๐จ๐ ๐๐ ๐๐ง๐ญ๐ฌ ๐ฐ๐ข๐ญ๐ก ๐๐ฅ๐๐ฆ๐ 3.2, ๐๐๐-4๐จ ๐ฆ๐ข๐ง๐ข ๐๐ง๐ ๐๐ฅ๐๐ฎ๐๐ 3.5 ๐๐จ๐ง๐ง๐๐ญ
๐๐;๐๐: I reimplemented the Swarm concept using Haystack, but made it work with both open and proprietary models ๐ซ
โ๏ธ blog article: https://haystack.deepset.ai/blog/swarm-of-agents
๐ notebook: https://haystack.deepset.ai/cookbook/swarm
Some time ago OpenAI published Swarm: an educational framework for building multi-agent systems.
Their approach focuses on two main concepts:
ใป ๐๐จ๐ฎ๐ญ๐ข๐ง๐๐ฌ: Each agent follows specific ๐ instructions and uses ๐ ๏ธ tools to execute them.
ใป ๐๐๐ง๐๐จ๐๐๐ฌ ๐ค: Agents can transfer control to one another using tool/function calling.
When I first read these ideas, I thought: ๐ด๐ช๐ฎ๐ฑ๐ญ๐ฆ ๐ฃ๐ถ๐ต ๐ฑ๐ฐ๐ธ๐ฆ๐ณ๐ง๐ถ๐ญ! And they pair well with the recent unified tool support in Haystack.
๐งโ๐ป So, I decided to re-implement these concepts using Haystack, and in just a few lines of code, I had a working prototype.
๐ Bonus feature: this implementation isn't tied to a single model provider - different agents can be powered by different models!
I replicated the ACME customer service example from the original article, with 3 Agents:
๐ Triage Agent - Llama 3.2 running on Ollama
๐ Sales Agent - Anthropic Claude 3.5 Sonnet
๐ Issues and Repairs Agent - OpenAI GPT-4o mini
Want to see the full implementation and give it a try? Check out the blog post and notebook! โจ
๐๐;๐๐: I reimplemented the Swarm concept using Haystack, but made it work with both open and proprietary models ๐ซ
โ๏ธ blog article: https://haystack.deepset.ai/blog/swarm-of-agents
๐ notebook: https://haystack.deepset.ai/cookbook/swarm
Some time ago OpenAI published Swarm: an educational framework for building multi-agent systems.
Their approach focuses on two main concepts:
ใป ๐๐จ๐ฎ๐ญ๐ข๐ง๐๐ฌ: Each agent follows specific ๐ instructions and uses ๐ ๏ธ tools to execute them.
ใป ๐๐๐ง๐๐จ๐๐๐ฌ ๐ค: Agents can transfer control to one another using tool/function calling.
When I first read these ideas, I thought: ๐ด๐ช๐ฎ๐ฑ๐ญ๐ฆ ๐ฃ๐ถ๐ต ๐ฑ๐ฐ๐ธ๐ฆ๐ณ๐ง๐ถ๐ญ! And they pair well with the recent unified tool support in Haystack.
๐งโ๐ป So, I decided to re-implement these concepts using Haystack, and in just a few lines of code, I had a working prototype.
๐ Bonus feature: this implementation isn't tied to a single model provider - different agents can be powered by different models!
I replicated the ACME customer service example from the original article, with 3 Agents:
๐ Triage Agent - Llama 3.2 running on Ollama
๐ Sales Agent - Anthropic Claude 3.5 Sonnet
๐ Issues and Repairs Agent - OpenAI GPT-4o mini
Want to see the full implementation and give it a try? Check out the blog post and notebook! โจ
Update README.md
1
#29 opened over 1 year ago
by
Piterasny
Post
1116
Ok, you're finally convinced that synthetic data works... โ๏ธ
๐๐จ๐ฐ ๐ฒ๐จ๐ฎ ๐ฐ๐๐ง๐ญ ๐ญ๐จ ๐ ๐๐ง๐๐ซ๐๐ญ๐ ๐๐ง ๐ข๐ง๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ข๐จ๐ง ๐๐๐ญ๐๐ฌ๐๐ญ ๐๐จ๐ซ ๐๐ข๐ง๐-๐ญ๐ฎ๐ง๐ข๐ง๐ ๐ข๐ง ๐ ๐ฅ๐๐ง๐ ๐ฎ๐๐ ๐ ๐จ๐ญ๐ก๐๐ซ ๐ญ๐ก๐๐ง ๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก.
But how do you get started?
I explore how to do this with Magpie in my new article
https://huggingface.co/blog/anakin87/multilingual-magpie
---
๐ฆโโฌ ๐๐ก๐๐ญ ๐ข๐ฌ ๐๐๐ ๐ฉ๐ข๐?
It's a recent technique for creating synthetic instruction datasets.
Magpie is based on a simple but ingenious idea ๐
if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction
Here's an example:
model: Llama-3-8B-Instruct
pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
generated user instruction: "What are some of the responsibilities of a commercial pilot?"
You can then feed this instruction back into the same model to get the assistant response.
By repeating this process, it's possible to generate large synthetic datasets with relatively little effort.
๐ช The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model.
๐ง๐๐๐ง๐๐ซ๐๐ญ๐ข๐ง๐ ๐ง๐จ๐ง-๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก ๐๐๐ญ๐
Most Language Models are primarily trained on English texts, so they tend to produce data in English.
How can we overcome this?
Earlier approaches were complex or costly.
Then @mrm8488 found a simple solution: add the target language to the pre-query template.
For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:".
This method works for Spanish and German!
โ Unfortunately, it does not work well for other languages (๐ฎ๐น, ๐ณ๐ฑ, ...)
๐
๐๐จ๐ฐ ๐ฒ๐จ๐ฎ ๐ฐ๐๐ง๐ญ ๐ญ๐จ ๐ ๐๐ง๐๐ซ๐๐ญ๐ ๐๐ง ๐ข๐ง๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ข๐จ๐ง ๐๐๐ญ๐๐ฌ๐๐ญ ๐๐จ๐ซ ๐๐ข๐ง๐-๐ญ๐ฎ๐ง๐ข๐ง๐ ๐ข๐ง ๐ ๐ฅ๐๐ง๐ ๐ฎ๐๐ ๐ ๐จ๐ญ๐ก๐๐ซ ๐ญ๐ก๐๐ง ๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก.
But how do you get started?
I explore how to do this with Magpie in my new article
https://huggingface.co/blog/anakin87/multilingual-magpie
---
๐ฆโโฌ ๐๐ก๐๐ญ ๐ข๐ฌ ๐๐๐ ๐ฉ๐ข๐?
It's a recent technique for creating synthetic instruction datasets.
Magpie is based on a simple but ingenious idea ๐
if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction
Here's an example:
model: Llama-3-8B-Instruct
pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
generated user instruction: "What are some of the responsibilities of a commercial pilot?"
You can then feed this instruction back into the same model to get the assistant response.
By repeating this process, it's possible to generate large synthetic datasets with relatively little effort.
๐ช The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model.
๐ง๐๐๐ง๐๐ซ๐๐ญ๐ข๐ง๐ ๐ง๐จ๐ง-๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก ๐๐๐ญ๐
Most Language Models are primarily trained on English texts, so they tend to produce data in English.
How can we overcome this?
Earlier approaches were complex or costly.
Then @mrm8488 found a simple solution: add the target language to the pre-query template.
For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:".
This method works for Spanish and German!
โ Unfortunately, it does not work well for other languages (๐ฎ๐น, ๐ณ๐ฑ, ...)
๐
-
- 1 reply