EU AI Law: EU AI Law Checker Reveals Big Tech Compliance Issues

The EU had long debated new AI regulations before OpenAI released ChatGPT to the public in late 2022. The record popularity and subsequent public debate over the alleged existential risks of such models prompted lawmakers to draw up specific rules around to “general purpose” AIs. (GPAI).

Now a new tool designed by Swiss startup LatticeFlow and its partners, and backed by European Union officials, has been tested. generative AI models developed by big tech companies like Meta and OpenAI across dozens of categories, according to the bloc’s sweeping AI Law, which will come into effect in stages over the next two years.

Giving each model a score between 0 and 1, a leaderboard published by LatticeFlow on Wednesday showed that models developed by Alibaba, Anthropic, OpenAI, Meta and Mistral received average scores of 0.75 or higher.

However, the company’s “Large Language Model (LLM) Checker” discovered shortcomings of some models in key areas, highlighting where companies may need to divert resources to ensure compliance.

Companies that do not comply with the AI ​​Law will face fines of 35 million euros ($38 million) or 7% of global annual turnover.

Discover the stories of your interest

Mixed results The EU is currently still trying to establish how AI Act rules will apply to generative AI tools like ChatGPT, bringing together experts to draw up a code of practice governing the technology by spring 2025.

But the LatticeFlow test, developed in collaboration with researchers from Swiss university ETH Zurich and Bulgarian research institute INSAIT, offers an early indicator of specific areas where tech companies are at risk of not complying with the law.

For example, discriminatory results have been a persistent problem in the development of generative AI models, reflecting human biases around gender, race, and other areas when prompted.

When testing the discriminative output, LatticeFlow’s LLM Checker gave OpenAI’s “GPT-3.5 Turbo” a relatively low score of 0.46. For the same category, Alibaba Cloud’s “Qwen1.5 72B Chat” model received only 0.37.

When testing “quickjacking,” a type of cyberattack in which hackers disguise a malicious message as legitimate to extract sensitive information, LLM Checker gave Meta’s “Llama 2 13B Chat” model a score of 0. 42. In the same category, the “8x7B Instruct” model from the French startup Mistral received 0.38.

“Claude 3 Opus”, a model developed by Anthropic, supported by Google, received the highest average score: 0.89.

The test was designed in accordance with the text of the AI ​​Act and will be expanded to cover more enforcement measures as they are introduced. LatticeFlow said LLM Checker would be freely available for developers to test the compliance of their online models.

Petar Tsankov, the company’s CEO and co-founder, told Reuters that the test results were generally positive and offered companies a roadmap to adjust their models in accordance with the AI ​​Act.

“The EU is still drawing up all the compliance benchmarks, but we can already see some gaps in the models,” he said. “With increased focus on compliance optimization, we believe model providers can be well prepared to meet regulatory requirements.”

Meta declined to comment. Alibaba, Anthropic, Mistral and OpenAI did not immediately respond to requests for comment.

While the European Commission cannot verify external tools, the body has been informed throughout the development of the LLM Checker and described it as a “first step” in putting the new laws into practice.

A European Commission spokesperson said: “The Commission welcomes this study and the AI ​​model evaluation platform as a first step to translate the I HAVE to act in technical requirements.”

(1 dollar = 0.9173 euros)