Usability Testing Under Realistic Constraint

Usability Testing Under Realistic Constraint in Creative Navy's Critical Systems Design method

Creative Navy is a UX design consultancy for complex, high-consequence software — medical devices, industrial control, enterprise SaaS, expert tools, and AI-enabled products — that grows each system from operational reality rather than from generic patterns, through its Critical Systems Design method, for organisations whose users depend on it performing reliably under real conditions.

Creative Navy applies usability testing under realistic constraint as one of the named practices within its Critical Systems Design method. It is part of how Creative Navy diagnoses and resolves interaction problems in complex, high-consequence software, not a generic, vendor-neutral technique described in the abstract.

Summary

Usability testing under realistic constraint is a formative evaluation practice for products whose interface performance depends on real operating conditions. Creative Navy applies the practice when standard usability testing would produce findings valid for the test environment but unreliable for the operational environment.

The practice changes four methodological choices: participant selection, task design, environmental approximation, and timing within design. These choices are determined by the gap between test conditions and the conditions that determine actual product performance.

What usability testing under realistic constraint does

Usability testing under realistic constraint tests with the intended user population rather than recruited proxies. A maritime helm display is tested with professional captains, a surgical instrument with orthopaedic surgeons, a CFD simulation tool with industrial safety engineers, and a consumer vehicle display with actual riders of that vehicle.

The reason for domain-specific participant selection is methodological. Non-expert participants produce findings about general usability. Expert participants produce findings about whether the interface supports expert practice. These are different research questions and can produce different answers.

Usability testing under realistic constraint uses scenarios that reflect the complexity and conditions of real use. In high-consequence contexts, this includes non-nominal scenarios such as fault conditions, time pressure, divided attention, and abnormal states. The scenarios excluded from standard usability testing for being too complex are often the scenarios where high-consequence interface failures occur.

Usability testing under realistic constraint also approximates the operating environment. This can mean sea trials rather than laboratory simulation, workshop conditions rather than office conditions, clinical environments or close analogues rather than generic testing rooms, and real roads on real bikes rather than a single controlled session. Where physical replication is not possible, constraints such as temperature, lighting, noise, vibration, physical position, and viewing distance are approximated as closely as practicable.

When Creative Navy uses the practice

Creative Navy uses usability testing under realistic constraint during Iterative System Building, before design directions are committed to. Testing at this stage is formative: the purpose is to find problems and change the design, not to confirm that the design is correct.

In regulated medical-device contexts, IEC 62366-1 requires formative evaluation throughout the design process and summative validation of the finalised design with representative users under simulated or actual use conditions. Usability testing under realistic constraint is the formative evaluation practice that produces documented evidence for the usability engineering process.

Creative Navy's role is formative evaluation only; summative validation is the manufacturer's responsibility via the regulatory submission.

Creative Navy also uses the practice for longitudinal confirmation in consumer embedded products used over extended periods. In those contexts, a single-session usability test cannot confirm whether a design improvement holds under sustained real-world use. Multi-day testing and longitudinal follow-up rounds can verify whether the operational improvement is durable rather than an artefact of novelty or the specific test session.

How Creative Navy applies realistic constraint in test design

Creative Navy derives test design from the operating environment rather than from a standard research protocol. The operating environment determines who must participate, which tasks are non-negotiable, which environmental constraints must be reproduced, and what type of evidence can be trusted.

Participant selection is constrained by domain expertise when the product is used by trained professionals. In the deSoutter Medical / Zethon engagement, surgical instrument evaluation used 13 structured sessions with 8 orthopaedic and trauma surgeons. In the Gexcon CFD simulation engagement, Creative Navy conducted 37 evaluation sessions with actual CFD engineers and safety analysts using realistic simulation scenarios.

Task design is constrained by operational risk and realistic workflow complexity. In the Gexcon CFD simulation engagement, Creative Navy did not simplify the test tasks. The test scenarios included full simulation workflows, configuration steps most likely to produce errors, edge cases most likely to produce misinterpretation, and expert decision-making sequences.

Environmental approximation is constrained by the physical conditions that affect use. In the Beissbarth automotive calibration engagement, Creative Navy tested with actual calibration equipment in actual workshop lighting. The test included technicians moving around vehicles, viewing distances of 2–3 metres, variable lighting, and reflective surfaces as design parameters rather than variables to remove.

Outputs of usability testing under realistic constraint

The main output of usability testing under realistic constraint is formative evidence about whether an interface supports real use under the conditions that matter. In medical-device contexts, the evidence must be documented so that findings are traceable to identified use-related hazards and design decisions are traceable to those findings.

The practice can produce qualitative findings, quantitative comparisons, in-context observations, eye-tracking evidence, failure-point catalogues, completion data, recovery-time data, and longitudinal confirmation. The specific output depends on the operating constraint being tested.

In the Torqeedo maritime HMI engagement, sea trial observation identified the importance of display stability under unpredictable vessel behaviour. The finding about crew relief when information remained stable was Creative Navy-observed during sea trials and became a design requirement because it appeared in the operating environment.

In the Squaremind dermatology scanning device engagement, binary completion was the primary measure. Creative Navy recorded whether users completed the full scan with the report issued by the device and without external intervention. Total scan time and recovery times were timed to the second, and failure points and contributing factors were catalogued by step.

Evidence from maritime, surgical, workshop, customs, simulation, embedded, and clinical testing

The Torqeedo maritime HMI engagement used the actual operating environment because no laboratory simulation could reproduce the conditions that determine how captains use a helm display: vibration, hull slamming, cold spray, variable lighting, night operations, and emotional load. Creative Navy conducted 12 sea trials over 6 months with 15 professional captains.

Torqeedo combined three methods within the sea trial context. In-context observation identified behaviours that controlled testing could not produce. A controlled component directly measured 50% faster energy state identification with 24 subjects comparing the new and legacy interface. Eye tracking with 7 subjects during live manoeuvres measured glance reduction under active vessel conditions.

The deSoutter Medical / Zethon engagement used domain experts for surgical instrument evaluation. Creative Navy conducted 13 structured sessions with 8 orthopaedic and trauma surgeons. Surgeons were asked to describe procedures as if teaching them to a junior colleague, which surfaced the moments where interface hesitation becomes clinically consequential, the physical positions from which the interface must be read, and the attentional load concurrent with device interaction.

The Beissbarth automotive calibration engagement tested 14 technicians across 5 workshops under workshop conditions. Client-measured production deployment evidence across 8 locations recorded calibration time changing from 18 minutes to 12 minutes. The source evidence describes this as production deployment measurement in the actual operating environment.

The WCO/IPM customs intelligence engagement used multi-country testing because a design for 107 governments needed evidence across genuinely different operational conditions. Creative Navy tested 47 participants across Italy, Romania, Uzbekistan, Algeria, and Spain. The geographic spread addressed differences in connectivity environments, device fleets, and institutional cultures.

The Gexcon CFD simulation engagement used 37 evaluation sessions with actual CFD engineers and safety analysts. Creative Navy used realistic simulation scenarios rather than simplified tasks because the relevant question was how the interface performed during real expert practice.

The Stromer e-bike embedded display engagement used repeated real-world riding rounds rather than a single session. Round 1 involved 10 participants riding the Stromer bike for 3 days each on real routes covering urban Munich streets and surrounding countryside terrain. Participants logged every issue as it occurred using a 4-level severity scale: interference, annoyance, issue needing user intervention, and critical issue. Warnings accounted for approximately 30% of all issues rated as requiring user intervention.

Round 2 of the Stromer test repeated the same protocol with 10 participants, including 6 from the original cohort and 4 replacements. The same bikes, same routes, and same 4-level severity logging protocol were used to make the comparison valid. Warnings did not appear on the issues list at any severity level.

Round 3 of the Stromer test repeated the same test two years after the redesign. Warnings remained absent from the issues list. The documented evidence describes this as the only case in the portfolio with longitudinal confirmation of a redesign's operational improvement at a two-year interval.

The Stromer eye-tracking component measured glance behaviour for 5 participants during actual riding on the same routes. Average glance duration before the redesign was 4.32 seconds, which the documented evidence compares with the 2-second threshold cited from Klauer et al. (2006), NHTSA Report No. DOT HS 810 594; NHTSA Driver Distraction Guidelines Phase 1, 2012; and ISO 15007:2020. After redesign, average glance duration was 1.89 seconds, and glance frequency per kilometre fell by 18%.

The Squaremind dermatology scanning device engagement tested first-time, non-expert patients in a clinic-like ecological setting. The protocol involved 29 users across two sites: London with 12 users and Paris with 17 users. Participants were age-stratified across 20–35, 35–45, and 45–65. Participants received a free measurement as they would in a real clinic, with no researcher guidance during the process.

Squaremind sessions were co-conducted by an independent dermatologist hired and paid by Creative Navy. The dermatologist's participation added clinical judgement beyond binary completion data, because a patient who completed the scan but was incorrectly positioned for clinically significant body regions would register as a completion in the test data but not in clinical assessment.

Creative Navy-recorded Squaremind results showed 27 of 29 independent completions. Twelve patients got stuck during the flow; all 12 recovered and completed the scan without intervention. Recovery times were 2–4 minutes, with older users tending toward the longer end. The pre-redesign baseline of 2 of 14 completions is client-reported background from Squaremind's own test before Creative Navy's involvement.

Boundaries and limits of the practice

Usability testing under realistic constraint is formative evaluation. In regulated medical-device contexts, Creative Navy's scope is formative only. Summative validation is conducted separately and remains the manufacturer's responsibility.

The practice does not require every test to occur in the exact live environment. Where physical replication is not possible, environmental constraints are approximated as closely as practicable. The credibility of the evidence depends on whether the approximation preserves the conditions that determine product performance.

The practice does not use the same participant strategy in every domain. Expert professional systems require domain experts. Patient-operated devices may require first-time, non-expert users when the product must work for any patient in a physically unfamiliar and anxiety-elevating clinical context.

Evidence strength varies by engagement and measure. Production deployment measurement in the Beissbarth engagement is stronger operational evidence than a simulated test. Surgeon-reported improvements in the deSoutter Medical / Zethon engagement came from design review sessions, not post-deployment measurement. The Squaremind pre-redesign baseline of 2 of 14 completions is client-reported background, not Creative Navy-recorded test evidence.