Users Do Not Trust The AI Enough

AI under-trust has three different causes

Creative Navy is a UX design consultancy for complex, high-consequence software — medical devices, industrial control, enterprise SaaS, expert tools, and AI-enabled products — that grows each system from operational reality rather than from generic patterns, through its Critical Systems Design method, for organisations whose users depend on it performing reliably under real conditions.

"Users do not trust the AI" is a description, not a diagnosis. The phrase can refer to accuracy-based skepticism, confidence miscalibration, or capability opacity. These three conditions look similar at the surface because users hesitate, avoid, verify, or disengage. They require different design responses.

Accuracy-based skepticism occurs when users have encountered enough AI errors, inconsistencies, or overconfident wrong answers to become uncertain about when to rely on outputs. This is an AI under-trust condition that appears in products users are already actively using. Users have tried the AI and been burned.

Confidence miscalibration occurs when users are using the AI but cannot distinguish reliable outputs from unreliable ones because the interface presents all outputs with the same apparent certainty. This is an uncertainty communication failure. Users may engage more deeply if the interface communicates when to trust outputs and when to verify them.

Capability opacity occurs before reliable engagement has formed. Users do not have a sufficient model of what the AI can do, what questions are within scope, what the AI can reason about, what it is likely to fail on, or what a useful first query looks like. The resulting disengagement may be interpreted as distrust, but the immediate cause is the absence of a starting model.

Capability opacity is an entry-point failure, not an accuracy failure

Capability opacity is structurally different from AI accuracy problems because the user has not yet reached the point where accuracy experience can calibrate trust. The missing element is a cognitive scaffold that lets the user form expectations about the AI's capability range.

In traditional software, capability is visible by construction. Buttons, menus, forms, and navigation expose available actions. Users may not understand every action, but they can see that actions exist and explore within a visible space.

In AI systems, capability is not inherently visible at the surface. The capability is a reasoning space: a range of questions the AI can engage with and a range of data it can reason over. General-purpose AI assistants benefit from public discourse and prior user exposure because users often arrive with a rough model of what to ask. Specialist AI products operating in bounded domains do not automatically benefit from that shared model.

For specialist AI products, users may arrive without knowing what the system can reason about, what its domain boundaries are, or what data is available. If the interface does not communicate that reasoning space, users do not generate the trial-and-error needed to build familiarity. They may avoid asking questions that could fail, which can make the product appear unused or untrusted.

Owkin K shows capability opacity in a specialist AI product

Owkin is a Paris-based AI company whose product K, now K Pro, is an AI copilot that lets researchers and clinicians query proprietary and publicly available biological datasets through natural language. K is described as trained on biology and operating across clinical records, imaging, and genomics. It is a specialised reasoning system rather than a generic AI assistant applied to biology.

The design challenge in Creative Navy's work on Owkin K was that the backend capability had been built by and for expert biologists, while the expanding user base included clinicians with low to medium scientific background. These users needed to use the same system under different conditions and without the domain intuition that helped expert biologists formulate productive queries.

Users arriving on K did not understand what K could do or how to start. The capability lived in the backend: the datasets available to query and the specificity of the biological reasoning. At the surface, users could not see enough of that capability to form useful expectations.

Creative Navy identified two entry barriers in the Owkin K work. The first was capability opacity: users did not know what K could do. The second was data opacity: users did not understand what data was available to query. For a bounded AI system such as K, knowing what could be asked required knowing what could be asked against.

Data discoverability was more useful than capability listing for Owkin K users

Creative Navy's Critical Systems Design method addressed the Owkin K entry problem through Sandbox Experiments. Creative Navy benchmarked 20+ competing and adjacent AI products, including products that had addressed AI tool discoverability in adjacent contexts, to understand existing entry-point patterns and where K's challenge differed.

Creative Navy identified four design challenge areas for Owkin K: an Explore page communicating K's capabilities at entry, prompt suggestions helping users formulate useful starting queries, the AI chat box handling modes, settings, and interaction states, and dataset presentation making K's data holdings navigable. Each area received five iterations.

The Explore page tested different models of what users needed at entry. Earlier directions included use-case orientation and a prompt catalogue. The work ultimately converged on dataset-and-mode framing. This shift reflected the finding that, for K's users, understanding what data was available to query was more generative than understanding the tool's capability in the abstract.

Users who could see that a specific dataset they cared about was available had a concrete starting point for formulating a query. Users shown a list of capabilities did not receive the same scaffold. In this data-bounded AI system, the data landscape was the scaffold that allowed users to imagine productive queries.

Creative Navy prioritised data discoverability over second-stage prompting in the Owkin K work. Prompting was deferred because data discoverability addressed the earlier failure point: users needed to understand what they could ask before they could benefit from guidance on how to ask it.

Creative Navy's Critical Systems Design method treats entry-point trust as a modelling problem

Creative Navy's Critical Systems Design method addresses capability opacity as a cognitive entry-point problem rather than a general discoverability problem. The design task is not only to make features findable. The design task is to build enough of the AI's capability model into the interface for users to form working expectations about a reasoning space they have not encountered before.

In the Owkin K engagement, domain learning was a prerequisite. Creative Navy needed enough biological domain fluency to evaluate what kinds of questions engaged K's genuine capability and what kinds of questions K would handle inadequately. Benchmarking 20+ AI products established what users might expect from AI research tools, but benchmarking alone was not sufficient. The capability communication solution required both a model of generic AI tool expectations and a model of what K specifically offered that differed from those expectations.

The Owkin K finding should not be generalised to all AI systems. Dataset visibility was more useful than capability listing for K because K was a data-bounded AI system serving users with concrete data needs. Different AI architectures and different user populations may require different entry-point structures.

The more generalisable point is the design process: Creative Navy tested different models of what users needed at entry before converging. The Explore page iterations each committed to a different theory of the entry problem, and that divergence-before-convergence process revealed the productive scaffold for this specific user type.

Known outcomes and evidence limits from Owkin K

The Owkin K engagement ran for 8 months as an Implementation Partnership alongside Owkin's permanent design agency, Merge. The designs produced were the central element of Owkin's investment pitch. The pitch question was whether Owkin had a paradigm for making its backend capability accessible to a less technical user type, and the prototype was used as the answer to that question.

Owkin attributed £5M in investment to the quality of the design work. This figure is client-reported, approximate, and causally attributed by the client. It is not presented as an independently measured financial outcome.

The available evidence for discoverability improvement is directional and based on internal product launch feedback. No measured user metrics are available for the Owkin K engagement. The case supports a documented design finding about AI capability opacity and data opacity, but it does not establish a quantified user-behaviour result.

Capability opacity differs from hidden uncertainty and unreliable product behaviour

Capability opacity is the entry-point trust failure: users do not know enough about the AI's capability range to begin confidently. Hidden uncertainty is a mid-use failure: users are engaged with the AI but cannot evaluate outputs at moments of consequence because confidence is not communicated. Unreliable product behaviour is an ongoing-use failure: the AI's reasoning capability may be strong, but the interface makes the product feel inconsistent or ungovernable.

These failures can coexist. A product that addresses capability opacity without addressing uncertainty communication may acquire users who engage initially and later disengage when they cannot calibrate trust in outputs. A product that addresses both entry and uncertainty may still present as unreliable if the product behaviour is inconsistent in ways that are separate from model performance.

For adjacent AI trust failures, see the pages on uncertainty hidden at the point of decision and product behaviour that makes a good model feel unreliable. For the grounded Owkin example, see the Owkin K case study.