Validity in Design Science

Design science creates artefacts: methods, models, algorithms, machine-learning systems. But when may we claim that an artefact works, that it is useful, or that it is better than an alternative?

With Kai R. Larsen, Roman Lukyanenko, Veda C. Storey, Jeffrey Parsons, Debra VanderMeer and Dirk Hovorka, I co-developed a framework that specifies which validity types support which claims about artefacts, including machine-learning systems. The conference version received the Herbert A. Simon Award at DESRIST 2020. The full article appeared in MIS Quarterly (2025, VHB A+). Translations are available in Chinese, Spanish, French and Russian, and there is an annotated, simplified version of the paper.

Animated summary of Validity in Design Science (MIS Quarterly, 2025), 12 minutes, with English captions. Also on YouTube.

A jungle of validities

Word cloud of validity types: theoretical, pragmatic, face, internal, measurement, external, construct, statistical and ecological validity

Internal, external, construct, ecological, face, pragmatic: research uses a bewildering number of validity concepts. In a multi-year review of about 7,500 articles and books, our team of seven found more than 441 distinct validities across technical, behavioural and social disciplines, of which 77 are relevant to design science. In the top IS journals, regular expressions located candidate sentences in 6,083 articles, and more than 10,000 sentences were read manually. The resulting framework was validated against all design validity definitions found, a sample of published design science evaluations, and with experts.

Three insights shaped the framework:

One does not validate an artefact; one validates claims about an artefact.
Validities are always comparisons between the focal artefact and another entity: an existing artefact, process or human performance, or a manipulated version of the same artefact with some part removed.
Validity is a matter of degree, not of kind. It is up to the community, initially the reviewers, to decide whether the degree is high enough.

From claims to validity types

The framework distinguishes three kinds of knowledge claims, which are typically established in sequence:

Process: establish criterion claim (Is it better?), then causal claim (How does it work?), then context claim (Where does it work?) — Criterion claims ask whether an artefact outperforms existing solutions, causal claims which design features produce the performance, and context claims in which settings it performs.

Each claim type is grounded in specific validity types, from criterion efficacy and characteristic validity to causal, ecological and external validity:

Taxonomy linking criterion, causal and context claims to validity types such as criterion efficacy, criterion characteristic, causal efficacy, causal characteristic, ecological and external validity, with sub-types such as predictive, concurrent, instance, theory, model and method validity — Knowledge claims and the validity types that support them.

The Design Validity Envelope

To make the framework usable in practice, we developed the Design Validity Envelope, a one-page worksheet (© Kai R. Larsen, Roland M. Mueller, Katja Thoring, Phil Cash, Roman Lukyanenko, 2026). Researchers start with their artefact (1), state one clear claim about it (2), define the reference entity they compare against (3) and the validation approach that produces evidence (4), and finally select the matching validity type (5). The envelope is shown at the top of this page; you can download the worksheet as a PDF and use it in your own research, teaching or reviews.

We introduced it in the workshop Mapping Validity in Design Research at the International Design Conference (DESIGN 2026) in Cavtat, Croatia, together with Philip Cash (Northumbria University), Katja Thoring (Technical University of Munich) and Kai R. Larsen (University of Colorado Boulder) for the Design Research Quality SIG. Working in pairs, participants mapped their own design research, from products and spaces to services, conceptual models and digital tools.

Roland M. Mueller presenting the slide The Validity Envelope, a model representing the set of concepts and their relationships relevant to design science validity, on a projection screen — Introducing the Validity Envelope at DESIGN 2026 in Cavtat.

Three workshop participants around a small round table discussing a filled-in Design Validity Envelope worksheet with sticky notes — Mapping a design research project with the worksheet.

Why it matters

Higher-quality scholarship: the link between design science claims and their validation becomes explicit and open to scrutiny.
A shared vocabulary for evaluation and validation practices unique to design science.
A clear validation argument that helps authors publish their papers.
Protection against excessive reviewer requests: only validate what you claim.

In 2026 I extended this line with Fruitfulness in Design Science, presented at DESRIST in Münster, which evaluates contributions by what they make possible rather than only by their fitness for a given problem. Keynotes in China, New Zealand and Germany carried the validity framework further into design research.

Validity was also at the heart of my work as Transparency Editor of MIS Quarterly, and it remains central to the research agenda on uncertainty-aware AI.