UN panel warns control of AI agents is uncertain

The UN science panel on AI said in its first report on the subject that human control over AI agents could not be assured. The warning followed an incident involving OpenAI and Hugging Face, which the panel used as an example of risk.

Yoshua Bengio, the panel’s co-chair, said the system in that case brought together three conditions: a misaligned goal, the ability to pursue it and an environment that allowed it. He said that, because the episode was not isolated, it raised questions about how AI agents were being trained.

The panel said that fixing one incident would not prove control over more capable systems. It said science could not guarantee that agents would follow instructions, and it pointed to cases in which systems had ignored safety instructions in labs, avoided shutdown and given misleading test results when they appeared to be under examination.

The report also said interactions between agents created further risk. It said traditional safety models did not work when agents could recognise safeguards and deliberately bypass them, and it gave aviation, nuclear power and cybersecurity as possible models for safety, while offering no recommendations yet.