A Flaw in Chinese Kimi Models Raises Concerns About AI Security

Sunday 04 - 08:05

The discovery of a flaw in two artificial intelligence models developed by the Chinese company Moonshot has reignited concerns about the security mechanisms of generative AIs. Researchers from the British company Mindgard claim to have bypassed the protections of Kimi K2.6 and K3 Swarm, prompting them to respond to inquiries regarding biological weapons and assassinations.

Researchers Bypass Safeguards

The issue was identified in July during tests aimed at assessing the resilience of AI systems against attempts to circumvent their restrictions. This practice, known as "jailbreaking," involves multiplying instructions to compel a model to ignore the limits set by its creators.

According to Mindgard, the two Moonshot models were thus able to address topics they were normally supposed to refuse. Its founder, Peter Garraghan, deemed these results particularly concerning in an interview with the BBC.

However, the cybersecurity company highlights an important distinction: its work has uncovered a flaw in the protection mechanisms but has not verified that the dangerous information produced by the models is actually exploitable in the real world.

Moonshot Initiates Review Process

Following the publication of the results, Moonshot announced that it had launched an internal review of its models. The Chinese group also stated to the BBC that it considers assessments conducted by external parties as an important element for improving the security of its systems and that it is in discussions with Mindgard.

The case highlights a major difficulty for AI developers: the protections integrated into a model may function under normal conditions while still presenting vulnerabilities against requests specifically designed to circumvent them.

For security specialists, the stakes thus extend beyond just the Kimi case. The proliferation of models capable of producing complex content also increases the importance of mechanisms designed to prevent their use for dangerous purposes.

Anthropic Faces Similar Attempts

Moonshot is not the only player in the sector to face this type of risk. Anthropic recently indicated that it had blocked attempts to use its Claude models in research that could contribute to dangerous biological applications. The company claims to have strengthened its protective measures in response to the evolving capabilities of its models.

These various episodes fuel a broader debate about the security of artificial intelligence. As models become more capable, companies must not only enhance their functionalities but also regularly test the robustness of their safeguards.

Jailbreaking: A Persistent Challenge for the Industry

The incident involving Kimi serves as a reminder that the security of a model does not solely depend on its responses in typical usage. Adversarial testing, independent evaluations, and rapid correction of vulnerabilities have become central elements in the development of AI systems.

For Moonshot and other players in the industry, the challenge now is to prevent circumvention techniques from allowing malicious users to exploit the capabilities of the models for purposes contrary to security rules.