Microsoft’s AI efforts have not been halted despite ongoing power struggles and mass resignations at OpenAI, reports VentureBeat. The company’s research arm has released Orca 2, a pair of small language models that outperform larger models, including Meta’s Llama-2 Chat-70B, when tested on complex reasoning tasks in zero-shot settings.
The Orca 2 models come in two sizes: 7 billion and 13 billion parameters, building on the work done on the original 13B Orca model that previously showed strong reasoning abilities. Microsoft has open-sourced both new models for further research on the development and evaluation of smaller models, providing enterprises with limited resources a better option to address targeted use cases without the need to invest in significant computing capacity.
Microsoft’s Orca 2 models tackle the gap in reasoning ability between large and small language models. The company leveraged imitation learning and trained the models to employ different solution strategies for different tasks without replicating the behavior of more capable models. The results from the Orca 2 models were impressive, outperforming larger models in various benchmarks that covered language understanding, common-sense reasoning, math problem solving, reading comprehension, and more.
These developments may pave the way for more high-performing small language models in the near future, encouraging more research and innovation in the AI space.
It is clear that this news from Microsoft’s research team has implications for the future of language models. The release of Orca 2 and ongoing research in this area could lead to significant advancements and diversification of language model applications and deployment options. The developments from Microsoft, along with similar advancements from other companies, suggest a promising future for high-performing small language models.
