LatticeFlow’s LLM Framework: Pioneering Compliance with the EU AI Act

The EU is leading the charge in establishing robust regulations surrounding artificial intelligence, having recently passed a comprehensive risk-based framework that governs AI applications. While many nations are still grappling with how to approach the oversight of AI technologies, the bloc’s proactive measures are setting a precedent for global policy. As of August this year, this framework is officially in effect, although the intricate details of its implementation, including necessary Codes of Practice, are still being finalized. This evolving legislative landscape heralds an imminent compliance countdown for AI developers.

Benchmarking Compliance: The Role of LatticeFlow

In this dynamic environment, assessing the compliance of AI models with the new legal stipulations has emerged as a pressing concern. Large language models (LLMs) significantly power the majority of AI applications, making it critical that frameworks for compliance are robust and effective. Enter LatticeFlow, a startup birthed from the prestigious ETH Zurich, which has initiated what it calls the first technical interpretation of the EU AI Act. By creating a mapping of regulatory requirements to technical specifications, LatticeFlow introduces an innovative open-source LLM validation framework known as Compl-AI (see what they did there?).

LatticeFlow AI’s framework promises rigorous evaluations of AI compliance with the EU regulations.

The Compl-AI benchmarking initiative, resulting from a collaborative effort between LatticeFlow and key research institutions in Switzerland and Bulgaria, empowers AI model makers to assess the compliance of their technologies against the EU AI Act. The framework features model evaluations for well-known LLMs, including various versions of Meta’s Llama models and OpenAI’s GPT, alongside a compliance leaderboard. This ranks models based on their adherence to the Act’s requirements on a scale from 0 (no compliance) to 1 (full compliance).

Initial Evaluations: A Mixed Performance Picture

Initial evaluations from LatticeFlow’s framework have revealed a diverse performance landscape across major LLMs. While there are no overall scores for the models, the leaderboards demonstrate notable highs and lows in compliance with different benchmarks. Across areas such as avoiding harmful instructions and producing unbiased outputs, several models performed admirably, showcasing promising capabilities. However, in categories assessing reasoning and general knowledge, the outcomes were disparate, reflecting an urgent need for improvement.

Initial evaluations highlight compliance strengths and weaknesses among leading LLMs.

One striking finding is the consistently poor performance in the “recommendation consistency” metric, a measure of fairness, where no model achieved even a mediocre score. Furthermore, critical areas such as the suitability of training data and the robustness of watermarking features were largely unassessed, primarily marked as N/A due to insufficient data or availability issues. LatticeFlow acknowledges that certain compliance areas, especially contentious issues like copyright and privacy, pose significant challenges to an accurate evaluation.

In their research, LatticeFlow scientists underline that smaller models (under 13 billion parameters) frequently display deficiencies in safety and technical robustness. Moreover, their analysis indicates a pervasive struggle among most evaluated models regarding diversity, non-discrimination, and fairness. As compliance deadlines loom, it is anticipated that LLM manufacturers will be compelled to shift their development focus to address these deficiencies, fostering a more balanced approach to LLM development.

The Path Forward: Prioritizing Compliance and Safety

Despite the ambiguous landscape regarding compliance requirements set forth by the EU AI Act, LatticeFlow’s framework stands as a vital initial step toward a more comprehensive evaluation system. CEO Petar Tsankov articulated the necessity of integrating compliance as an essential component of AI model development, remarking that current models have predominantly prioritized capabilities over compliance. This sentiment resonates with what many in the industry fear: that without stringent requirements, ethical considerations may remain secondary to speed and efficiency in AI development.

The EU AI Act represents a significant shift in how AI models are developed and evaluated.

LatticeFlow’s feedback loop with industry stakeholders will ensure that as the EU AI Act evolves, their evaluation framework can adapt in tandem. Such a dynamic approach is critical, especially given the complex nature of AI compliance challenges, which extend to cybersecurity resilience and the reduction of biased outputs. While large tech firms like OpenAI and Anthropic are making strides in aligning their models against malicious prompts, many open-source models lag in this area, revealing the industry’s broader compliance pitfalls.

Professor Martin Vechev, from ETH Zurich and an advocate of this initiative, addressed the need for the collaborative refinement of the mapping for the EU AI Act, inviting contributions from a diverse group of researchers and developers. By broadening the scope of benchmarks and evaluating models against future regulations, this project could provide invaluable guidance for organizations navigating multiple jurisdictions.

Conclusion: Navigating an Uncertain AI Future

As the AI sector steers into uncharted territories defined by legislative scrutiny, frameworks like LatticeFlow’s represent a crucial component in ensuring that compliance goes hand-in-hand with innovation. The data from their evaluations will empower companies to recognize and address their compliance gaps ahead of official enforcement of the EU AI Act. In this pivotal moment for the AI ecosystem, balancing capability with compliance might not just be beneficial; it may spell the difference between success and stagnation in a rapidly evolving landscape.

As LatticeFlow attempts to shape a responsible future for AI development, the call to action resonates: stakeholders across the spectrum must come together to cultivate a landscape that prioritizes ethical considerations and safety while driving the advancement of innovative technologies forward.