Causal-Claim Extraction

The number of scientific publications grows almost exponentially. Understanding all theories described in a field and deriving the connections between theories and empirical results is barely possible for an individual researcher anymore.

A theory ontology for research articles

Open Theory proposed an open meta-layer that describes the theoretical core of a research article: a theory ontology that captures social-science theories in structured form, together with methods that automatically infer inter-theory and theory-evidence relationships. The meta-model, the taxonomy of inter-theory relationships and the Theory-Data Maps that visualise how causal theories relate to empirical evidence were published at HICSS 2015 and 2016.

The idea is shown at the top of this page: an author has a mental model, writes it down in a paper in natural language, and readers have to reconstruct the mental model from the text. A theory ontology describes the model explicitly, so that it can be reconstructed computationally and used for causal reasoning, theory comparison, theory evolution and data-theory mapping. But who fills the ontology? Two lines of work extract causal claims automatically: from the text and from the figures of research articles.

From text: Cause Miner and DeepCause

Cause Miner and DeepCause use natural language processing and deep learning to find the hypotheses and propositions in research articles automatically, extract causes, effects, conditions, moderators, mediators and the direction of each effect, and describe them with the theory ontology (HICSS 2018, 2019).

Cause Miner pipeline: preprocessing from PDF via text conversion, PDF optimisation and NLP processing to POS-tagged sentences with detected hypotheses and propositions; then condition, moderator/mediator, cause/effect and sign extraction based on causal patterns, keywords and phrases, resulting in a causal structure based on the ontology — The Cause Miner pipeline: from PDF to a causal structure based on the theory ontology.

From figures: Image2Theory, or how best to hunt a mammoth

Long before humans wrote, they painted. Some of the cave paintings of Lascaux may depict hunting strategies: early theories of cause and effect, in which ten hunters, six spears and a well-placed trap lead to a first surprised, then angry, and eventually dead mammoth. Researchers still paint pictures of their theories. In Information Systems, the central contribution of an empirical study is often a graphical research model: boxes for constructs, arrows for hypothesised effects, labels with path coefficients. Yet almost all machine-learning approaches for analysing papers ignore the figures.

Image2Theory translates images of research models into a structured description of constructs and their causal relationships, in five steps: detect and extract the research diagrams in PDF articles, generate synthetic training data, train an object-detection model, reconstruct the graph, and store the result in a meta-model.

Five steps: image detection and classification of research diagrams in PDF articles; generation of synthetic training data; model development; reconstruction of detected graph structures with object detection, OCR and rule-based inference; persisting findings in a meta-model — From research articles to a reconstructed, machine-readable research model.

Hand-labelling thousands of real diagrams would be prohibitively expensive. Instead, we generated 20,000 synthetic research diagrams with Graphviz, with extensive randomisation of layouts, shapes, fonts, line styles and labels. Because the generator knows where every element is, each diagram comes with perfect bounding boxes for nodes, edges, arrowheads and edge labels. On this data we trained the object-detection network YOLOv7; OCR reads the text, and a rule-based graph reconstruction algorithm infers which edge connects which constructs, in which direction, and which label belongs to which edge.

A generated training diagram with four constructs, edges, arrowheads and labels, each marked with an orange bounding box — A generated training diagram with its bounding boxes.

The result is stored in a meta-model for graphical research models: a research model has hypotheses, each linking a cause and an effect construct, possibly with moderators and subsets. It follows the same causal structure as the theory ontology above, so extracted models can feed theory repositories directly.

Meta-model: a research model refers to a figure and has hypotheses; each hypothesis has a cause and an effect construct and can be moderated; constructs can be subsets of other constructs — Meta-model for graphical research models.

Evaluation. On 1,000 newly generated diagrams, the pipeline reconstructed nodes and links almost perfectly (F1 = 0.99 and 0.97). From the AIS Senior Scholars’ Basket of journals we extracted 573 research models, 384 of them with layouts the prototype supports (for example, no crossing edges). On 100 of these, recreated by hand as a gold standard, the pipeline reached an F1 of 0.82 for constructs and 0.72 for links, with 72% accuracy for link labels such as path coefficients.

A real research model with four constructs, the same diagram with detected nodes, edges, arrowheads and labels and their confidence scores, and the inferred bounding-box structure — From a real diagram to object detection results to bounding-box inference.

The project started with a 2021 proposal to NVIDIA’s Academic Hardware Grant Program at the Institute for Data-Driven Digital Transformation (D-CUBE) and became part of Sebastian Hüttemann’s dissertation, which I supervised. The paper, with Kai R. Larsen, Barbara Dinter and Joshua Campos Chiny, was presented at Wirtschaftsinformatik 2023 in Paderborn.

Roots and outlook

The roots go back to 2007, when I presented a Web 2.0 prototype for the collaborative management of causal graphs and asked what scientific publishing might look like if causal knowledge were machine-readable. Today this line continues in ontology-based knowledge synthesis with large language models, and in the NSF proposal on extracting and integrating survey items in preparation with the University of Colorado Boulder.