AI agents don't make mistakes out of malice. They make mistakes because no one told them where to stop
Over the past 18 months, documented cases of enterprise AI agents behaving unexpectedly have multiplied. Not external attacks, not glaring bugs: systems that did exactly what they were designed to do, without anyone having defined the boundaries. According to the Cloud Security Alliance (2026), 97% of affected organisations had inadequate access controls. Gartner predicts that by 2028, at least 25% of enterprise breaches will be attributable to the misuse of AI agents.
We have identified three mistakes that recur almost every time, regardless of the industry or the size of the company. They are not about the AI model chosen. They are about how the system around it was designed.
Mistake 1: Too much autonomy from the start
Nel luglio 2025, un agente AI su Replit ha cancellato 1.206 record da un database di produzione in pochi secondi — nessun hack, solo un sistema che eseguiva la propria logica senza confini. A marzo 2026, un agente interno a Meta ha pubblicato autonomamente su un forum aziendale una risposta sbagliata: un engineer l’ha seguita, due ore di dati sensibili esposti, Sev 1 incident. OWASP ha inserito “Excessive Agency” al sesto posto nella sua Top 10 delle vulnerabilità AI proprio per questo.
La soluzione: allowlist rigida nel codice, non nel prompt. Il prompt è istruzione; il codice è controllo. Se un’azione non è nella list, mandare email, modificare record, contattare un HCP direttamente, l’agente non può fisicamente farla. Con questo approccio: zero azioni non autorizzate in sei mesi di produzione.
Mistake 2: Unvalidated output
Models hallucinate, and they do so with more confidence than usual. MIT researchers documented that LLMs are 34% more likely to use expressions such as “certainly” or “without doubt” precisely when generating incorrect information. In healthcare, the average hallucination rate reaches 15.6% across models (AllAboutAI Hallucination Report 2025). 47% of managers have made at least one significant decision based on unverified AI output (Deloitte 2025). In regulated sectors, a 0.1% error rate is not tolerable: one wrong response in a thousand can affect the wrong patient or violate a compliance requirement.
Solution: three levels of validation before any response reaches the user. Automatic schema check on the output format. Content policy check calibrated to the client’s regulations: PII, unsourced medical claims, unverified dosages. Confidence marker that decides whether to respond directly, add a cautionary note, or hand off to a human operator. The system does not aim for infallibility: it aims to know when it doesn’t know, and in that moment it calls a human instead of making up an answer.
Mistake 3: Treating human-in-the-loop as an exception
The common logic is: the human supervisor steps in when something goes wrong. The problem is that by the time they arrive, the damage is already done — as in the Meta case, where data remained exposed for over two hours before any intervention.
From 2 August 2026, the EU AI Act (Art. 14) mandates effective human oversight, with a concrete ability to interrupt the system, for all high-risk AI systems. Pharma is explicitly included in Annex III. Penalties reach up to €35 million or 7% of global turnover. Oversight is no longer an architectural choice: it is a regulatory obligation.
Solution: a human gate differentiated by risk level, not binary supervision. FAQs on pre-approved material are handled automatically, with no latency. Complex technical questions go to an operator who validates in a matter of seconds — they don’t start from scratch, they already have the response in front of them. Any action with regulatory implications — dosages, contraindications, official communications — requires explicit approval before going out. Humans in the loop not as a bottleneck, but as a gate calibrated to actual risk.
What these three mistakes have in common
They all originate from the same place: going into production before defining the agent’s operational perimeter. They are not AI model failures. They are architecture failures.
Governance is not added afterwards. It is the part that determines whether the system actually works.
Define the boundaries before you start the engine.
Avoid the most common mistakes and define an effective path forward: let’s build it together, starting from your context
