| | --- |
| | library_name: vllm |
| | language: |
| | - en |
| | - fr |
| | - de |
| | - es |
| | - it |
| | - pt |
| | - zh |
| | - ja |
| | - ru |
| | - ko |
| | license: apache-2.0 |
| | inference: false |
| | extra_gated_description: >- |
| | If you want to learn more about how we process your personal data, please read |
| | our <a href="https://mistral.ai/terms/">Privacy Policy</a>. |
| | tags: |
| | - mistral-common |
| | --- |
| | |
| | # Model Card for Mistral-Small-24B-Base-2501 |
| |
|
| | Mistral Small 3 ( 2501 ) sets a new benchmark in the "small" Large Language Models category below 70B, boasting 24B parameters and achieving state-of-the-art capabilities comparable to larger models! |
| | Check out our fine-tuned Instruct version [Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501). |
| |
|
| | For enterprises that need specialized capabilities (increased context, particular modalities, domain specific knowledge, etc.), we will be releasing commercial models beyond what Mistral AI contributes to the community. |
| |
|
| | This release demonstrates our commitment to open source, serving as a strong base model. |
| |
|
| | Learn more about Mistral Small in our [blog post](https://mistral.ai/news/mistral-small-3/). |
| |
|
| | Model developper: Mistral AI Team |
| |
|
| | ## Key Features |
| | - **Multilingual:** Supports dozens of languages, including English, French, German, Spanish, Italian, Chinese, Japanese, Korean, Portuguese, Dutch, and Polish. |
| | - **Advanced Reasoning:** State-of-the-art conversational and reasoning capabilities. |
| | - **Apache 2.0 License:** Open license allowing usage and modification for both commercial and non-commercial purposes. |
| | - **Context Window:** A 32k context window. |
| | - **Tokenizer:** Utilizes a Tekken tokenizer with a 131k vocabulary size. |
| |
|
| | ## Benchmark Results |
| |
|
| | | Benchmark | Metric | Mistral-Small-24B-Base | |
| | | ------------------------------ | ------------- | ----------- | |
| | | [MMLU][mmlu] | 5-shot | 80.73 | |
| | | [MMLU Pro][mmlu_pro] | 5-shot, CoT | 54.37 | |
| | | [GPQA Main][gpqa] | 5-shot, CoT | 34.37 | |
| | | [TriviaQA][triviaqa] | 5-shot | 80.32 | |
| | | [ARC-c][arc] | 0-shot | 91.29 | |
| | | [TriviaQA][triviaqa] | 5-shot | 76.6 | |
| | | [MBPP][mbpp] | pass@1 | 69.64 | |
| | | [GSM8K][gsm8k] | 5-shot, maj@1 | 80.73 | |
| | | [MATH][math] | 4-shot, MaJ | 45.98 | |
| | | [AGIEval][agieval] | - | 65.80 | |
| |
|
| | | Benchmark | Metric | Mistral-Small-24B-Base | |
| | | ------------------------------ | ------------- | ----------- | |
| | | French MMLU | - | 78.03 | |
| | | German MMLU | - | 77.69 | |
| | | Spanish MMLU | - | 78.86 | |
| | | Russian MMLU | - | 75.64 | |
| | | Chinese MMLU | - | 70.35 | |
| | | Korean MMLU | - | 56.42 | |
| | | Japanese MMLU | - | 74.46 | |
| |
|
| |
|
| | [mmlu]: https://arxiv.org/abs/2009.03300 |
| | [hellaswag]: https://arxiv.org/abs/1905.07830 |
| | [piqa]: https://arxiv.org/abs/1911.11641 |
| | [socialiqa]: https://arxiv.org/abs/1904.09728 |
| | [boolq]: https://arxiv.org/abs/1905.10044 |
| | [winogrande]: https://arxiv.org/abs/1907.10641 |
| | [commonsenseqa]: https://arxiv.org/abs/1811.00937 |
| | [openbookqa]: https://arxiv.org/abs/1809.02789 |
| | [arc]: https://arxiv.org/abs/1911.01547 |
| | [triviaqa]: https://arxiv.org/abs/1705.03551 |
| | [naturalq]: https://github.com/google-research-datasets/natural-questions |
| | [humaneval]: https://arxiv.org/abs/2107.03374 |
| | [mbpp]: https://arxiv.org/abs/2108.07732 |
| | [gsm8k]: https://arxiv.org/abs/2110.14168 |
| | [realtox]: https://arxiv.org/abs/2009.11462 |
| | [bold]: https://arxiv.org/abs/2101.11718 |
| | [crows]: https://aclanthology.org/2020.emnlp-main.154/ |
| | [bbq]: https://arxiv.org/abs/2110.08193v2 |
| | [winogender]: https://arxiv.org/abs/1804.09301 |
| | [truthfulqa]: https://arxiv.org/abs/2109.07958 |
| | [winobias]: https://arxiv.org/abs/1804.06876 |
| | [math]: https://arxiv.org/abs/2103.03874 |
| | [agieval]: https://arxiv.org/abs/2304.06364 |
| | [big-bench]: https://arxiv.org/abs/2206.04615 |
| | [toxigen]: https://arxiv.org/abs/2203.09509 |
| | [mmlu_pro]: https://arxiv.org/abs/2406.01574 |
| | [gpqa]: https://arxiv.org/abs/2311.12022 |
