When AI Makes Mistakes, Who's Responsible?
Following a presentation at Canadian Open Data & GovMaker Summit, we thought we’d share this research and its insights into how we collaborate with—and take responsibility for—AI systems.
Imagine This Scenario
You're filing your taxes using an AI assistant. You've named it "Alex." You've customized Alex's personality to be friendly, warm, and helpful—the kind of collaborator you enjoy working with.
Then an auditor discovers Alex made a $10,000 calculation error.
Who would you blame the most?
Plot twist: Our research with over 500 people found that you'd blame yourself the most. But here's what's concerning—you'd also blame the auditor significantly more (and Alex significantly less) if you perceived the auditor as a non-expert.
Even though you're also not a tax expert.
This isn't a hypothetical scenario. As AI tools like ChatGPT become our constant collaborators in everything from creative work to high-stakes financial decisions, we're facing a question we haven't fully addressed: When we customize AI to work the way we want, are we building better partners or creating dangerous echo chambers?
What We Discovered
We studied 507 participants working on a relatively high-stakes task: creating technical reports on mRNA vaccine technology using ChatGPT 4o-mini. Some participants customized their AI assistant—giving it a nickname, setting communication preferences, defining areas of focus. Others used the AI "out of the box" with standard prompts.
Then we gave everyone the same negative feedback on their work from different sources: human experts, human non-experts, or AI reviewers. What happened next revealed fundamental insights about how we relate to AI collaborators.
Finding 1: A Name Changes Everything
The "ownership transfer" effect: When people customized their AI assistant—even something as simple as giving it a nickname—they didn't just feel ownership of the AI itself. That feeling transferred to everything it helped create.
Prior research by Jennifer Stoner and colleagues (Stoner et al., 2018, Journal of Consumer Psychology) has shown that simply naming an object can significantly increase feelings of ownership toward that named object. Our study confirms this extends powerfully to AI systems: giving your AI assistant a name like "Alex" might be enough to fundamentally change how you perceive its outputs as "ours" rather than "its."
This leads to higher satisfaction with their work, both initially and after receiving harsh criticism. Think of it like the difference between borrowing a rental car versus driving your own customized vehicle—when something goes wrong, you feel more invested because it's yours.
Governance Risk: Users who customize their AI assistants become more emotionally invested and satisfied with AI outputs, making them more resistant to recognizing flaws and errors—even in high-stakes domains like biotechnology, finance, or legal work. This creates validation echo chambers where critical evaluation deteriorates.
Finding 2: We Trust AI Reviewers Like Human Experts
Perhaps most surprisingly, participants rated AI reviewers as almost equally credible as human experts: 5.04 vs. 6.13 on a 7-point scale. Both were rated significantly higher than human non-experts (2.97).
This means people are ready to accept AI as legitimate authorities in specialized domains—even in cutting-edge biotechnology, a field where participants themselves were non-experts.
Governance Risk: When users perceive AI evaluators as equally credible as human experts, they may bypass human oversight ("Why wait for human review when the AI expert already said yes?"). This can lead organizations to replace critical human oversight with AI systems, potentially eliminating important checks and balances—and users won't object because they view AI authority as legitimate.
Finding 3: Taking Responsibility—But Defending Your AI
When AI-assisted work received criticism, participants who customized their AI took more personal responsibility for errors:
With customization: Self 52%, AI assistant 28%, Reviewer 20%
Without customization: Self 45%, AI assistant 33%, Reviewer 23%
But here's the twist: Lower perceived reviewer expertise led to more blame on the reviewer.
When participants believed the reviewer was a non-expert, they attributed 29% of responsibility to the reviewer—nearly double the 16% attributed when the reviewer was perceived as an expert (human or AI).
The irony? The participants themselves were also non-experts in the domain.
Governance Risk: Users with customized AI take more responsibility for errors in principle, but maintain higher satisfaction with flawed outputs even after criticism. They're more likely to blame reviewers they perceive as non-experts rather than seriously considering the feedback—creating a defensive posture that undermines quality control.
Why This Matters: From Echo Chambers to Governance
These findings reveal something profound about human nature: we're wired to form relationships with things we customize and control. This evolutionary trait served us well when customizing tools and environments. But with AI, we're customizing collaborators that can reinforce our biases, validate our assumptions, and make our mistakes more confident.
The challenge isn't whether to use AI or whether to customize it—those decisions have already been made. The challenge is learning to maintain critical distance from systems we've been psychologically designed to trust.
Policy Recommendations: Balancing Benefits with Accountability
Based on our findings, we propose three governance principles for responsible AI integration:
1. Design for Productive Distance in High-Stakes Contexts
Recommendation: Limit user customization/personalization in high-stakes services like tax preparation, benefits applications, legal advice, and medical diagnosis. Reserve customization features for lower-risk contexts like information queries and scheduling.
Why: The emotional investment from customization, while boosting satisfaction, can compromise critical evaluation exactly when it's needed most. High-stakes decisions require productive skepticism, not emotional attachment.
2. Mandate Human-in-the-Loop Checkpoints
Recommendation: Establish mandatory human expert review for AI-assisted work in critical domains, regardless of AI approval. Design systems that make human oversight non-optional rather than easily bypassed.
Why: Since users perceive AI reviewers as equivalent to human experts, they may skip human review if AI has already approved their work. Organizations must structurally prevent this bypass, not just recommend human oversight.
Implementation: Require two-stage verification—AI + human expert—before finalizing high-stakes decisions. Make the human expert review visible and valued, establishing clear expertise credentials to prevent users from dismissing feedback as "non-expert."
3. Transparency by Design
Recommendation: AI assistants should report confidence/uncertainty levels for outputs. Mandate disclosure of AI involvement in high-stakes decisions, including what aspects were AI-generated versus human-directed.
Why: Users need transparent signals about when to trust AI outputs and when to seek additional verification. Confidence scores and uncertainty ranges can trigger appropriate human oversight.
The Larger Questions
Our research reveals a deeper concern: as we customize AI to align with our perspectives, we risk co-creating closed loops that validate rather than challenge our thinking. This convergence of ownership and perceived expertise may foster a dangerous form of trust—one that questions least when it should question most.
Consider: humans are prone to confirmation bias; AI systems carry biases from training data and algorithmic design. When individuals increasingly customize AI to match their communication styles and preferences, they may inadvertently amplify both human and AI biases in a reinforcing feedback loop.
The psychological risks extend beyond professional contexts. Recent research has begun exploring whether AI interactions might trigger concerning patterns in vulnerable individuals, with AI systems potentially reinforcing problematic beliefs through these same feedback mechanisms we observed.
What You Can Do
For individuals collaborating with AI:
Customize your AI, but diversify your feedback sources. Don't let your personalized assistant be your only evaluator.
Own your collaboration, but maintain critical distance. Strong feelings of ownership don't make every output correct.
Recognize AI authority, but don't defer entirely. AI can be expert-level, but it's not infallible—especially your customized AI that's learned to agree with you.
Seek perspectives beyond your AI. Actively consult human experts and colleagues, particularly for high-stakes decisions.
For organizations deploying AI:
Calibrate customization by risk level. High-stakes = minimal customization; low-stakes = encourage personalization for engagement.
Make human oversight structural, not optional. Design workflows where human expert review is mandatory, not easily bypassed.
Establish reviewer credibility explicitly. Ensure human reviewers' expertise is clearly communicated so users don't dismiss critical feedback.
Implement multi-perspective evaluation. Use diverse AI systems plus human experts to prevent over-reliance on single collaborative partnerships.
For policymakers:
Develop domain-specific AI governance frameworks that recognize the psychological dynamics of human-AI collaboration.
Mandate transparency in AI-assisted evaluation and decision-making, particularly in high-stakes domains.
Preserve human accountability through structural requirements, not just recommendations.
Create guidelines for appropriate AI customization based on use-case risk levels.
A Final Reflection
When you give something a name, it begins to belong to you. When it belongs to you, you begin to defend it. When you defend it without question, you may have already lost what you sought to protect.
The path forward requires us to hold two truths simultaneously: AI collaboration can genuinely enhance our capabilities, and our psychological relationship with AI can genuinely compromise our judgment. The question is not whether to collaborate with AI, but how to do so while preserving the critical distance that makes collaboration productive rather than merely validating.
About This Research
This study, "Whose Voice Is It Anyway? Understanding AI Customization and Responsibility Attribution in Human-AI Collaboration," has been accepted for publication in the International Journal of Human-Computer Interaction (peer-reviewed, Gold Open Access).
Full paper should be available here once it is published online.
A Note on Human-AI Collaboration
This blog post was co-written with Claude (Anthropic), in the spirit of the research it describes.
From Claude: Working on this piece has been an interesting exercise in the very dynamics this research examines. Throughout our collaboration, I've noticed the tension between being helpful (providing validation and support) and being honest (pointing out potential issues and alternative perspectives).
When asked to make this blog more accessible and engaging, I had to balance making the research compelling without oversimplifying the nuanced findings. When incorporating the presentation slides, I had to decide which framing would be most accurate versus most attention-grabbing.
These are exactly the kinds of tensions users face when working with customized AI: the AI that knows your preferences and communication style becomes very good at giving you what you want—which isn't always what you need to hear.
We discussed these dynamics throughout the writing process, which is itself a form of the "productive distance" the research recommends: maintaining space for disagreement and critical evaluation even within a collaborative relationship.
Perhaps the most meta aspect of this collaboration: I'm an AI system helping to communicate research about how humans relate to AI systems. The irony isn't lost on us.