The Lies of AI Chatbots: The Survival Strategies of Probabilistic Software Faced by PMs
Date
Jun 1, 2026
Classification
- Marketing/Branding/Planning/Design
[Advertisement]
Why Does AI Keep Getting It Wrong?
#AIProduct #ProbabilisticSystem #Hallucination #TrustDesign
🫑 3-Line Summary
•
A lawsuit arose after Air Canada's AI chatbot provided users with incorrect refund policies, and a Canadian court ruled that the company was responsible for the chatbot's response and ordered the payment of refunds.
•
Unlike conventional deterministic software formulas that guaranteed the same output for the same input, AI products are probabilistic systems that generate a different answer each time by probabilistically predicting the next word.
•
AI products face three unique problems: testing is difficult due to the lack of fixed answers, failures are plausibly concealed due to illusions, and infrastructure costs skyrocket non-linearly based on token usage. Therefore, PMs must abandon the illusory KPI of 100% accuracy, build a trust-based UX that defends against the risk of incorrect answers, and continuously track the distribution of dynamic questions even after launch.
🥦 Insight
•
Letting go of the obsession with zero bugs and living with the error distribution
For decades, planners have regarded a state with no error codes and perfectly aligned expected values as the correct answer. However, since it is impossible to completely eliminate illusions in probability systems, designing an architecture that controls the impact and costs on users by domain when errors occur—rather than trying to prevent failure at the source—becomes a core competency.
•
The Aesthetics of Trust Design That Filters Out Confident Lies
Unlike clear failures such as a system crash or an error window popping up, AI failures arrive quietly, lurking behind smooth sentences. As demonstrated by the Air Canada lawsuit, incorrect chatbot responses directly lead to corporate legal liability; therefore, it is essential to specify the sources of the answers, disclose the level of confidence, and ensure an escalation path in the UX that connects the user immediately to a human representative.
•
Paradigm shift from function-centric roadmaps to results-centric operations
Unlike the past, where results remained consistent as long as the code wasn't modified, the quality and cost of AI products fluctuate in real-time due to self-updates of massive models and users' creative (or malicious) variations of prompts. PMs must view product deployment as a beginning, not an end, and cultivate "living operational muscle" to observe the unexpected long-tail question layers every day.
🥄 A spoonful of execution
•
Review the explanatory text for AI features currently in operation or in the planning stages. To prevent users from over-relying on the answers and facing risks like the Air Canada incident, why not place UI elements that transparently disclose system limitations or verification links leading to the actual regulations page at the bottom of your responses right now?
—— View Original ——
The Probability Systems Thinking Method a PM Needs to Know
In November 2022, Jake Moffatt, who lived in Vancouver, Canada, received news of his grandmother's sudden passing.
He accessed the Air Canada website to hurriedly book a flight to Toronto and asked the chatbot.
"Is there a discount on airline tickets for the bereavement of a family member?" The chatbot kindly replied.
If you need to travel immediately, you can first purchase a ticket at the regular fare and then apply for a death certificate retroactively within 90 days of the ticketing date.
Time was running short until his grandmother's funeral, and the chatbot's response was clear. He purchased a round-trip ticket to Toronto at full price, and after returning, he sent a refund request as instructed by the chatbot.
Want to read this article to the end? Click!
Please click (touch) the link block below.
Go to the platform where the article was written.
(* Separate registration may be required depending on the platform.)