OPINION: Agentic AI’s Big Risk Is Data With No Boundaries
Ask anyone in finance what agentic AI can do, and the answer is usually a clear-cut list: flag fraud before it clears, resolve a ticket without a human’s involvement, and process a borrower’s entire profile in seconds for a loan application. What’s not being asked, however, is whether the agent is actually right more often than the human.
Beneath this ongoing question is the governance issue. The simple fact of the matter is that a lot of what these agents are trained on isn’t real. For the majority of financial institutions, synthetic data is the easy, privacy-safe choice for getting a model off the ground without compromising on stringent compliance demands and cost. Yet it’s a thin foundation for decisions that affect real borrowers and real fraud losses. Yes, it indicates what could be plausible, but not actually what happens in outlier situations, which makes banks and other financial institutions vulnerable to credit and fraud risk.
In fact, researchers are increasingly probing the risks and vulnerabilities surrounding how the finance sector leans on data, with one study calling synthetic data “a double-edged sword.” A highly respected computer scientist, Richard Sutton, takes a harder stance, calling the heavy reliance on synthetic data “a big mistake.”
Now, put these two problems together, synthetic data and no meaningful limits on agentic AI, and there’s a third issue that is also alarming and more latent than what the IAPP’s prognosis recognizes. Even the best-governed, best-trained agent is still just a hypothetical tool until someone proves it’s making better decisions than a human would.
The Synthetic Data Trap
Synthetic data has become a popular workaround for all the privacy concerns that exist in the finance world. It’s also incredibly useful for testing and makes new technologies more accessible. But synthetic data, like any kind of data, is not without its vulnerabilities. It’s still at risk of hallucinating and subpar performance; these risks are especially troubling when it’s used as the foundation for systems that act on real money and people’s lives.
AI agents solely trained on synthetic data are also ultimately not as adept at handling outliers or inconsistencies. For instance, in the case of evaluating credit risk, an agent that was trained on synthetic data might only feature steady-state income data. But in the real world, gig workers, freelancers, and small business owners with irregular incomes don’t perfectly match the pattern.
Worse, whatever bias or hallucination existed in a synthetic dataset gets inherited. Banks end up building decisioning architectures on a blueprint or foundation they can’t actually explain. Racial discrimination is one of the biggest consequences here, especially in loan and credit assessments. A model trained on narrow synthetic data that excludes or dismisses certain demographics only magnifies that discrimination. Of course, the algorithm doesn’t bear the consequence of an accusation of racism; the people who signed off on the technology without the right guardrails to keep it in check do. “We can’t say why exactly” is not an acceptable response for a regulator, auditor or customer.
Agents Running Amok
Thin, synthetic data is just one-half of the story. The second is how agents behave. Agentic systems can drift past their original mandate, simply because the data needed to do so is sitting right there. In credit scoring, for instance, that can look like a credit decisioning agent (that’s only supposed to make one decision) pulling third-party or behavioral insights beyond the original design. The agent uses and holds on to that data not just in one session, but across multiple, potentially skewing future decisions.
By the time you notice an agent has drifted from its remit, it’s already acted, and the damage is done. In the worst-case scenario, this drift can manifest as leaked highly sensitive personal data that an agent keeps using without permission (a legal, reputational, and security nightmare for any bank). In financial services, drifts can easily become massive security threats that beckon hackers. This is an increasingly alarming risk as a fifth of firms cannot even pinpoint when misconfigured AI leads to hacks. An agent that’s not told where its workflow ends widens security gaps.
Agents trained on synthetic data are generally worse at recognizing when they’ve moved beyond their designated lane because they’ve never seen the edge cases that inspire hesitation or further escalation.
Prove First, Trust Later
The IAPP stresses that purpose limitation (what an agent is allowed to do and touch) and data minimization (how much they can do) need to move to the center of agent design. Yet neither actually tells us whether the agents are making better decisions than before. This fix actually isn’t complicated, but it does take a little bit more time.
Agents’ decisions must be run in parallel with the existing human-led process, on the same portfolios/accounts before giving any reins to agentic AI. Getting this piloting stage right is crucial for succeeding with agentic AI. Compare approval rates, default rates, false positives, and false negatives to understand how trustworthy the tools actually are.
This is an important step to incrementally building trust among teams and builds evidence for regulators who are more concerned with explainability than with AI, especially in Europe where the EU AI Act explicitly classifies agentic AI in credit decisioning as high-risk. By running parallel tests, you’re giving direct proof and justification that the technology improves on previous processes, as well as explainability on precisely how you measured that.
Moreover, a human-in-the-loop here is much more than a safety net. An analyst flagging where the agent got a call wrong, or right for the wrong reason, critically improves a deployed system over time.
The Bigger Challenge of Minimization
Restricting what an agent can see and access can be a challenge in and of itself. The answer is feature-importance analysis to find which inputs actually drive decision quality.
Monitoring has a version of the same problem: an agent that only sees the current session can’t catch a slow-building issue like cash-flow deterioration that only shows up after six months, so the fix is an explicit retention window, such as a rolling twelve months for portfolio health and purged beyond that (rather than unlimited memory or none at all).
As banks and financial institutions move to comply with a new wave of AI rules, they contend with a backdrop of intense regulatory demand and fast-evolving rules. The majority of the EU AI Act’s rules have been enforced, even as the European Data Protection Board (EDPB) has clarified that purpose limitation and data minimization apply to every step of an agent’s decision workflow. Meanwhile, GDPR-style rules are pushing for less retention, but audit requirements under DORA (which also includes third-party ICT risk rules applied directly to agents) and EBA guidance demand more for longer.
In the U.S., the National Institute of Standards and Technology (NIST) has likewise developed an AI Risk Management Framework (RMF), to encourage building AI trustworthiness.
It’s tricky navigating all these requirements, particularly while they’re actively evolving. This is why financial organizations should prepare a structured decision audit trail and minimize or time-lock the raw data feeding it. This ensures explainability without keeping everything forever.
Banks need to design for this now rather than do an overhaul further down the line. Delayed reactions are always far more painful than simply being proactive from day one. Consider this, though, very carefully: GDPR raises a problem few have solved yet. An agent that retains context across sessions is effectively building a new dataset that was never disclosed, and if a customer exercises their right to erasure after that data has already shaped the agent’s memory, undoing that influence has to be solved in the architecture itself, not in the policy playing catch-up.
Better data hygiene is just one part of the answer for becoming agentic AI-ready. The other, equally vital, part is demonstrating that agentic AI earns the title of trustworthy by proving it against real decisions first.