Microsoft, in collaboration with OpenAI, has been steadily integrating AI capabilities into its products and services while also developing case-specific models. Recently, Microsoft Research introduced Orca, a new AI model that learns by imitating large language models. The aim of Orca is to address the limitations of smaller models by emulating the reasoning processes found in substantial foundation models like GPT-4.
Language models such as Orca can be optimized for specific tasks and trained using large language models like GPT-4. The advantage of Orca lies in its smaller size, which requires fewer computing resources to run and operate. Researchers have the flexibility to fine-tune their models according to their needs and run them independently, without relying on a large-scale data center.
Orca, a 13 billion parameter-powered AI model, is built upon the foundations of Vicuna and can learn from and imitate large language models like GPT-4. It is capable of acquiring knowledge, understanding step-by-step thought processes, and comprehending complex instructions with the assistance of GPT-4, which is rumored to boast over one trillion parameters.
Microsoft leverages extensive and diverse imitation data to facilitate progressive learning with Orca, leading to significant advancements. Notably, Orca has already outperformed Vicuna by 100% on challenging zero-shot reasoning benchmarks like Big-Bench Hard (BBH). Furthermore, this new model demonstrates a 42% increase in speed compared to conventional AI models on AGIEval.
Despite being a smaller model, Orca exhibits comparable reasoning capabilities to ChatGPT on benchmarks like BBH. It also demonstrates competitive performance on academic exams such as SAT, LSAT, GRE, and GMAT, although it falls short of GPT-4. The Microsoft research team emphasizes that Orca can learn from step-by-step explanations created by humans and more advanced language models, with the expectation of further enhancing its skills and capabilities over time.
