Together AI’s Mixture of Agents: A Leap Forward in AI Innovation

In a groundbreaking move that sets a new benchmark in the AI realm, Together AI has unveiled the Mixture of Agents (MoA) framework. This innovative approach cleverly combines the strengths of multiple large language models (LLMs) to significantly enhance the quality and performance of AI outputs, overshadowing previous leaders in the field.

Understanding Mixture of Agents (MoA)

Together’s MoA leverages a layered architecture where each layer is populated with several LLM agents. Each agent draws information from the outputs generated by its predecessors to provide refined responses. By merging diverse insights and capabilities from various models, MoA achieves a more robust and versatile AI solution. Impressively, the implementation of MoA has achieved a score of 65.1% on the AlpacaEval 2.0 benchmark, surpassing the earlier benchmark holder, GPT-4o, which scored 57.5%.

An illustration of the layered architecture in MoA!

At the heart of MoA’s success is the principle of collaborativeness among LLMs. Research suggests that LLMs produce higher-quality outputs when they can reference responses generated by other models—even those of lesser caliber. This led to the innovative classification of models within the MoA framework into two categories: proposers and aggregators. Proposers generate initial responses rich in nuance, while aggregators synthesize these responses into consolidated high-quality outputs. This collaborative process continues across multiple layers, culminating in an accurate and comprehensive final response.

Testing and Performance Metrics

The MoA framework has undergone rigorous testing against multiple prominent benchmarks, namely AlpacaEval 2.0, MT-Bench, and FLASK. Its performance has been nothing short of remarkable, attaining top spots within the leaderboards of these benchmarks. Specifically, on AlpacaEval 2.0, MoA not only eclipsed GPT-4o with a 7.6% improvement but achieved this feat by exclusively utilizing open-source models, showcasing its superiority against closed-source alternatives.

Graphical representation of MoA’s performance metrics compared to competitors

Moreover, the architecture of Together MoA has been optimized for cost-effectiveness. Preliminary analyses illustrate that its design offers the finest balance between quality and cost-efficiency, particularly highlighted by the MoA-Lite configuration, which provides equivalent quality to GPT-4o but at a more attractive price point due to its fewer layers.

Collaboration and Community Impact

The progress of Together MoA is the result of fruitful collaborations among various players in the open-source AI landscape. Notable contributors include Meta AI, Mistral AI, Microsoft, Alibaba Cloud, and DataBricks, whose combined efforts have seeded advancements in models such as Meta Llama 3, Mixtral, WizardLM, Qwen, and DBRX. Similarly, performance benchmarks like AlpacaEval, MT-Bench, and FLASK, developed by entities such as Tatsu Labs, LMSYS, and KAIST AI, have been pivotal in evaluating and validating MoA’s capabilities.

The ecosystem of collaboration driving advancements in AI

The Future of MoA

Looking towards the future, Together AI is committed to continuously optimizing the MoA architecture by investigating various configurations, prompts, and model choices. A particular focus lies in minimizing latency, especially regarding the time to the first token. This forward-thinking will boost MoA’s proficiency in reasoning-intensive tasks, further cementing its position as a trailblazer in AI innovation.

Conclusion

In summation, Together AI’s Mixture of Agents signifies a transformative step in harnessing the collaborative prowess of open-source models. By employing a sophisticated layered approach and fostering a collaborative culture, MoA exemplifies the potential for enhancing AI systems to be more capable, robust, and aligned with human reasoning. The AI community is keenly watching this cutting-edge technology as it evolves and finds new applications in various domains.

For more insights, refer to the Paper, explore our GitHub, and check out our Blog. Stay connected for further updates on social media outlets like Twitter, and enrich your engagement through our Telegram Channel or LinkedIn Group. And don’t forget to join our vibrant community on Reddit with over 44k members and subscribe to our newsletter for the latest news!