Remember the viral case of an airline forced to honor incorrect fare terms that a chatbot gave a customer? As AI agents step into bigger roles, automating and carrying out complex tasks, accountability for mistakes like this is a real concern.
When AI makes decisions on its own, who’s on the hook if things go sideways? How can you ensure AI accountability — and prevent mistakes from happening in the first place?
Artificial intelligence (AI) accountability is a major concern for both employees and executives. In fact, it’s such a pressing issue that The Wharton School recently launched the Accountable AI Lab, a research initiative focused on AI’s ethical, regulatory, and governance challenges.
For businesses, AI accountability is critical: It builds trust, mitigates risk, and ensures compliance. Companies must be able to explain and justify AI’s decisions, and if those decisions are incorrect, rectify the outcomes. Without clear accountability, businesses face not only legal exposure, but reputational damage and loss of customer confidence.
“Companies are already held accountable for what their AI does,“ said Jason Ross, product security principal at Salesforce. “But there are legal, ethical, and social issues coming together in a way with agentic AI that hasn’t happened with other technology, even cloud and mobile.”
Unlike conventional software, which follows predefined rules, agentic AI learns, adapts, and generates responses dynamically, making its decision-making process less predictable. This autonomy creates challenges in pinpointing responsibility when mistakes occur.
How to ensure AI accountability
Businesses need a multifaceted approach to AI accountability grounded in, among other things, AI-specific technology safeguards, quality data, and new organizational governance.
Silvio Savarese, Salesforce’s chief scientist, wrote recently of the importance of establishing clear oversight frameworks to ensure a comprehensive approach to AI accountability. Consider adding these five frameworks to your AI implementation.
Set up clear chains of responsibility for an AI agent’s decisions
To ensure accountability, businesses should establish unambiguous chains of responsibility for AI decisions. This involves identifying who is accountable for each step the AI takes, from initial deployment to final output. You might need to create new roles to oversee AI functions, such as a Chief AI Officer (CAIO) or an AI Ethics Manager, who would monitor, review, and be accountable for the performance of AI systems.
A CAIO would make sure AI systems follow company guidelines, and in the event of an error, be the first point of contact for rectifying the situation. Similarly, teams formed to audit and track AI decision-making processes would make sure decisions align with corporate values and ethical standards.
According to a March 2024 study, roughly 15% of companies in the S&P 500 already provide some degree of board-level oversight of AI.
Create systems that detect and correct incomplete, incorrect or toxic outputs
AI models don’t always get things right. They hallucinate, misinterpret context, or reinforce biases. To minimize harm, businesses need real-time monitoring systems, robust audit trails, and the ability to intervene fast in the event of an error.
One example is the Salesforce research team’s advancements in retrieval-augmented generation (RAG), which improves how AI systems access and verify information. This enables rapid evaluation and course-correction, ensuring that AI systems deliver accurate, reliable results you can trust.
Similarly, human-in-the-loop monitoring systems allow for continuous oversight of AI outputs, flagging and correcting issues before they escalate. A framework might include:
- Automated dashboards that flag potentially harmful or incorrect outputs,
- Fallback mechanisms that default to a human if an AI response is flagged as problematic, and
- Regular audits and bias evaluations to continually assess AI accuracy and validity over time.
Define processes that balance agent autonomy with human control
AI agents need clearly defined boundaries. Not everything should be left to automation, especially high-stakes decisions that require human judgment. Organizations should build structured intervention frameworks to determine when and how humans should step in. In finance, for example, agents might recommend investment strategies, but would require human input for any investments over a certain threshold. In healthcare, doctors could require human review of a high-risk diagnosis or decision.
Develop approaches for making things right when mistakes occur
AI failures don’t just impact internal operations; they can erode trust with customers and employees. Businesses need structured plans for remediation, communication, and systematic improvement when things go wrong. Consider an ecommerce platform using AI for customer support. If the AI makes a wrong decision about, say, a refund, the remediation blueprint would include:
- Immediate rollback procedures to correct the error,
- Proactive customer communication, acknowledging the mistake and outlining next steps,
- Compensation guidelines, like offering credits, and
- Long-term corrective actions, such as retraining the AI model to prevent future errors.
Establish new legal and compliance frameworks that explicitly address AI agent accountability
The regulatory landscape is still evolving, and many existing laws don’t account for autonomous AI decision-making. So, companies need to develop their own AI-specific governance structures that combine legal, ethical, compliance, and operational expertise to make sure AI is used responsibly. This could include establishing a cross-functional AI Center of Excellence (CoE), where different teams work together to continually assess AI systems against evolving legal requirements and ethical standards.
The CoE could oversee independent audits that verify compliance with industry standards, as well as its own internal standards. It could also create transparency reports that disclose how AI models make decisions.
How Salesforce advances AI accountability
What does AI accountability look like? Consider Salesforce’s approach to AI safety, trust, and ethics, which can provide a foundation for your organization.
- The Einstein Trust Layer, a set of features that protects the security of your data and improves safety and accuracy, can evaluate and score content based on its toxicity. Is the content biased or hateful? The scores are logged and stored in Data Cloud, a platform for unifying and harmonizing data from across your company, as part of an audit trail. Data Cloud can generate reports on audit trail data and user feedback.
- The trust layer also verifies the safety and accuracy of responses generated by your large language model (LLM), drastically reducing the likelihood of a bad response.
- Prompt Builder, a tool for building generative AI prompt templates, uses system policies to decrease the risk of the LLM generating something inaccurate or harmful. The policies are a set of instructions to the LLM for how to behave. For example, you can tell it to not generate answers when it lacks information about a subject.
- Dynamic grounding improves the accuracy and relevancy of your AI’s results by combining your structured and unstructured data with an AI prompt, thereby adding more context and accuracy to the response.
- AI systems should be designed to provide clear and understandable explanations for their decisions. Salesforce’s Einstein platform offers explainability tools that help users understand the factors that influence AI outputs.
AI accountability is critical for business
AI isn’t just another tool. It’s a decision-maker that shapes business outcomes in real time — which means the old rules around safety and governance don’t apply. Companies that fail to build AI accountability into their foundations may face a range of negative consequences. It’s time for new governance structures, oversight mechanisms, and AI-specific safeguards that match the power of the technology itself. This is the new backbone of trust in the AI age.
How digital labor will reshape the enterprise
Agentic AI is a new labor model, new productivity model, and a new economic model. Its impact on work will be permanent and profound.


