Usability Evaluation For High Consequence Products

Summary

Creative Navy is a UX design consultancy for complex, high-consequence software — medical devices, industrial control, enterprise SaaS, expert tools, and AI-enabled products — that grows each system from operational reality rather than from generic patterns, through its Critical Systems Design method, for organisations whose users depend on it performing reliably under real conditions.

Usability evaluation for high-consequence products is formative evaluation conducted during design, before problems are fixed into the product. In regulated contexts, Creative Navy's scope is formative evaluation only; summative validation is the manufacturer's responsibility.

Creative Navy evaluates high-consequence products by matching the evaluation method to the operational risk. The evidence base can include IEC 62366-1 governed formative studies, realistic task scenarios, expert sessions, controlled experiments, operational eye tracking, ecological pre-deployment protocols, and production deployment measurement.

When high-consequence usability evaluation is needed

High-consequence usability evaluation is needed when interface failure, workflow interruption, state misinterpretation, or recovery failure can create operational consequences. The evaluation must cover use scenarios rather than isolated screen interactions, because the risk often appears under abnormal conditions, non-happy paths, time pressure, divided attention, vibration, poor visibility, or role coordination constraints.

Representative users are required for this type of evaluation. In expert tools, representative users are domain experts rather than general users. In the available examples, representative users included professional captains, scrub nurses, perfusionists, ICU nurses, cardiologists, orthopaedic and trauma surgeons, automotive technicians, customs users, CFD engineers, safety analysts, and clinic patients using a dermatology scanning device without guidance.

Performance under pressure is a central evaluation condition. The difference between a good interface and a poor interface often becomes visible when users are operating under time pressure, managing abnormal conditions, wearing gloves, dealing with glare, handling vibration, or dividing attention between the interface and the surrounding operational environment.

What Creative Navy evaluates in high-consequence products

Creative Navy evaluates whether a product remains understandable and usable under the conditions in which failure matters. This includes task comprehension, state visibility, error recovery, visual attention, workflow continuity, clinical or operational choreography, and the user's ability to complete the task without external intervention.

In IEC 62366-1 governed contexts, Creative Navy's evaluation work is formative. The evaluation supports participant selection, task design, use scenario coverage, and documentation during the usability engineering process. Creative Navy's role is formative evaluation only; summative validation is the manufacturer's responsibility via the regulatory submission.

Creative Navy also evaluates abnormal scenarios explicitly. Abnormal scenario evaluation tests how the product behaves under non-ideal conditions, because high-consequence failures often become visible outside the happy path.

Evidence methods used for different evaluation questions

Controlled experiments are used when the question is whether one interface condition performs differently from another under a stated baseline, participant count, task condition, and measured outcome. The Typewise AI keyboard evaluation used a 60-user controlled experiment with the iOS native keyboard as the baseline. Error rates were halved against that baseline, and typing speed changed from 38 WPM to 47 WPM in the same experiment.

Operational field evaluation is used when a laboratory cannot reproduce the environment. The Torqeedo maritime HMI evaluation was conducted in actual maritime conditions because no lab simulation could reproduce vessel motion, hull slamming, glare, rain, vibration, night operations, or gloved interaction.

Eye tracking in operational conditions is used when visual attention is the evaluation question. In the Torqeedo case, glance reduction during manoeuvres was measured via eye tracking with 7 subjects during actual sea trials. The operational setting matters because glance patterns during manoeuvres depend on actual vessel motion and real navigational demands.

Production deployment measurement is used when the strongest evidence available is outcome data from the live operational system rather than from test conditions. In the Beissbarth automotive calibration case, calibration time changed from 18 to 12 minutes per vehicle, client-measured across 8 production deployment locations.

Triangulation across evidence sources is used when different methods answer different questions. In the Torqeedo maritime HMI case, sea trial observation identified the conditions that produced usability issues, a controlled experiment quantified the performance difference, and operational eye tracking measured the change in visual attention.

Torqeedo maritime HMI as multi-method evaluation in real operating conditions

The Torqeedo maritime HMI evaluation used real maritime conditions because the operating environment could not be reproduced in a lab. The research programme included 12 sea trials over 6 months, 15 professional captains, temperatures from −5°C to +35°C, night operations, vibration, hull slamming, glare from cold water, rain, and gloved interaction.

Creative Navy-observed sea trial evidence identified an emotional dimension of control system interfaces: crews experienced relief when information remained stable while the vessel behaved unpredictably. The finding was observed during sea trials and became a design requirement. The evidence is qualitative and operational; it is not a lab-derived finding.

A controlled environment experiment with 24 subjects directly measured 50% faster energy state identification for the new interface compared with the legacy interface. This experiment was separate from the sea trials. It quantified a performance difference that the sea trials had qualitatively identified.

Operational eye tracking with 7 subjects during actual sea trials measured glance reduction during manoeuvres. The measurement was conducted in the operating environment, not in a simulation.

All 15 professional captains rated the new interface as significantly better through participant-reported structured feedback across all 12 sea trials. This is participant-reported evidence, not production deployment measurement.

Kardion MCS Controller as regulated formative evaluation under IEC 62366-1

The Kardion MCS Controller evaluation was formative and governed by IEC 62366-1. Creative Navy built on an Emergo by UL formative study as the evidence foundation. That study involved 7 participants: 2 scrub nurses, 2 perfusionists, and 3 ICU nurses. It covered 8 clinical use scenarios.

The Emergo by UL formative study identified participant scanning patterns, min/max flow misinterpretation, and a startup time concern. It did not surface the clinical choreography: the spatial and procedural relationship between surgeon, nurse, controller, and patient during a procedure. Creative Navy's own sessions with cardiologists and nurses were needed to investigate that dimension.

Creative Navy's sessions included mental model sessions and design feedback sessions. The mental model sessions covered clinical practice, decision-making around flow adjustments, and which data elements were clinically relevant at which moments. The design feedback work included 4 sessions using design prototypes to collect feedback on specific directions.

Creative Navy-recorded formative evaluation continued throughout the engagement. Thirty-four directions for the standard view were presented to the client with explicit pros and cons, building shared understanding. Most iterations were shown; two were also tested with users.

The recorded regulatory result was FDA approval: the design passed evaluation as submitted, with no design changes required. This is a regulatory result, not a measured usability outcome. Creative Navy's role was formative evaluation only; summative validation is the manufacturer's responsibility via the regulatory submission.

deSoutter Medical and Zethon as expert evaluation grounded in domain literature

The deSoutter Medical and Zethon evaluation began with a domain knowledge foundation. Twelve human factors studies were reviewed, including Colle & Hiszem (2004) on touch target sizing and Tao et al. (2018) on button design. The findings from those papers were applied to specific design decisions.

Creative Navy conducted 13 structured sessions with 8 orthopaedic and trauma surgeons. The sessions used a teaching-description method: surgeons described their procedures as if teaching a junior colleague. This produced workflow information comparable to workflow observation while remaining achievable without being present in the operating theatre.

Creative Navy also benchmarked 9 competitor devices. The evaluation identified a recurring failure across competitors: colour used as the primary state indicator. The design was developed explicitly against that failure pattern.

Creative Navy developed and evaluated 8 information architecture models against representative surgical workflows. Surgeon-reported feedback from design review sessions indicated that state verification was reduced to a brief glance and that parameter adjustments no longer interrupted surgical workflow. This evidence must be read as design review session evidence, not post-deployment evaluation.

The deSoutter Medical and Zethon work was IEC 62366-1 governed throughout and formative only.

Beissbarth, WCO/IPM, Typewise, and Gexcon as evaluation across operational contexts

The Beissbarth automotive calibration case separates design-phase evaluation from production deployment measurement. During design, Creative Navy evaluated the product with 14 technicians across 5 workshops, including authorised inspection centres and independent garages. The evaluation used contextual interviews, procedure walkthroughs, actual usage, semi-structured interviews, and 3 OEM display structural variants tested under conditions reproducing workshop lighting and viewing distances.

The Beissbarth outcome measurement was not usability testing. It was operational data from production deployments. Beissbarth used 8 production deployment locations to collect outcome data. Calibration time changed from 18 to 12 minutes per vehicle, client-measured across those 8 locations. Repeated measurements were reduced, client-measured, but the exact figure is not available.

The WCO/IPM customs intelligence evaluation involved 47 participants across Italy, Romania, Uzbekistan, Algeria, and Spain. The geographic spread was deliberate because the design was intended for 107 governments. The evaluation confirmed that the multi-jurisdictional design worked under different operational realities, including bandwidth conditions, device fleets, institutional cultures, and languages.

The Typewise AI keyboard evaluation used a 60-user controlled experiment. The baseline was the iOS native keyboard, the platform users were switching from. Error rates were halved against the iOS native keyboard baseline, and typing speed changed from 38 WPM to 47 WPM in the same experiment. The testing setup created realistic pressure conditions so that qualitative feedback could be gathered from users who had experienced the keyboard under real conditions.

The Gexcon CFD simulation evaluation involved 37 evaluation sessions across 45 design variants. The evaluation was ongoing design practice rather than a final gate. It was conducted with actual CFD engineers and safety analysts using realistic simulation scenarios. Production deployment measurement later recorded a change from 4 days to 6 hours, measured by Gexcon in real deployments, and configuration errors changing from 5–8 to 1–2, also measured.

Squaremind as ecological pre-deployment evaluation for commercial proof-of-viability

The Squaremind dermatology scanning device evaluation was designed to show whether real patients could complete a scan without assistance or intervention. The patients varied in age, were unfamiliar with the device, were undressed, and were alone. The evaluation requirement was commercial as well as formative: 9 clinics required evidence before purchase that the scan could be completed under those conditions.

The Squaremind protocol included 29 users across two sites. London included 12 users: 3 aged 20–35, 4 aged 35–45, and 5 aged 45–65. Paris included 17 users: 7 aged 20–35, 7 aged 35–45, and 3 aged 45–65. Participants received a free measurement as they would in a real clinic. No researcher guidance was provided during the process. An independent dermatologist hired and paid by Creative Navy co-conducted all sessions.

Age stratification was a deliberate evaluation design choice. The commercial claim required the interface to hold across the full clinical patient population. A sample drawn only from the most digitally fluent age cohort would not have produced evidence credible to a clinical buyer evaluating the product for patients across age groups.

The independent dermatologist added clinical judgement about whether each patient's positioning was adequate for clinically meaningful scan quality. This clinical co-validation was distinct from summative validation. The dermatologist co-conducted the sessions to add clinical grounding to the formative evaluation, not to constitute a formal regulatory review.

The primary measure was binary completion: whether the user completed the full scan with the report issued by the device, without external intervention. Secondary measures included total scan time, recovery time timed to the second for users who got stuck, and failure points with contributing factors catalogued by step.

Creative Navy-recorded evaluation results showed that 27 of 29 patients completed the scan independently. Twelve patients got stuck; all 12 recovered without external intervention. Recovery times were 2–4 minutes. The pre-redesign baseline for comparison was Squaremind's own test with 14 patients, which produced 2 completions before Creative Navy's involvement. That baseline is client-reported background and not Creative Navy evaluation evidence.

The Squaremind evaluation was formative and structured to support Squaremind's usability engineering process. Summative validation and regulatory submission were Squaremind's responsibility.

Boundaries and limits of this capability

Creative Navy's usability evaluation for high-consequence products is primarily formative. In regulated contexts, summative validation remains the manufacturer's responsibility.

Not all evidence categories have the same strength. Production deployment measurement is stronger than test-condition evidence for live operational outcomes, but several examples use controlled experiments, field observation, design review sessions, or participant-reported feedback instead.

Client-measured outcomes should be read as client-measured. Beissbarth calibration time and Gexcon deployment outcomes were measured by the client in real deployments, not independently measured by Creative Navy.

Participant-reported and surgeon-reported feedback should not be treated as post-deployment evidence unless the case explicitly says so. In the deSoutter Medical and Zethon example, the reported workflow improvements came from design review sessions.

The Squaremind pre-redesign baseline was client-reported background from Squaremind's own test with 14 patients. It contextualises the later ecological evaluation but is not Creative Navy evaluation evidence.

What this produces

Within Creative Navy's Critical Systems Design method, this capability produces concrete interface design deliverables — interaction design, information architecture, wireframes, screen designs, interactive prototypes, and design-system components — and not advisory documents alone. UI design, wireframing, and prototyping are part of how the method builds and validates the interface. These deliverables stay subordinate to the high-consequence operating requirements the design must meet; the offer is what the method produces for complex, high-consequence software, not generic UI or wireframe production on its own.