SemanticIS

Standing on the shoulders of giants requires knowing what the giants did. Yet information overload is a daily reality in research: millions of papers are published every year, spread across many databases. Google Scholar or Web of Science offer limited filters and return thousands of results that must be screened by hand; methods, topics and technologies are often not mentioned in titles and abstracts; and generative AI search tools hallucinate and do not reveal their knowledge base. Literature reviews therefore remain limited to small samples and offer a static view of a small section of a field.

With Sebastian Hüttemann, Kai R. Larsen and Barbara Dinter (TU Chemnitz), I developed two connected tools that address this:

SemanticIS, a search engine that combines ontologies with large language models and indexes more than 57,000 articles from the major journals and conferences of Information Systems, from MIS Quarterly and Information Systems Research to ICIS, ECIS, HICSS and DESRIST (International Journal of Information Management, 2025);
the Knowledge Synthesis Dashboard, which turns the same annotations into syntheses of whole research fields (ICIS 2024, Best Design Science Paper, second runner-up).

SemanticIS helps find the relevant articles; the dashboard helps make sense of them as a body of knowledge.

Grounded in information foraging theory

Researchers search for literature much like animals forage for food: they move between “patches” of search results, follow the information scent of promising hits, and weigh the information gain against the interaction cost in time and effort. A good search system should strengthen the scent, increase the gain and reduce the cost.

Information foraging in search engines: user activities such as keyword search and filtering lead to information patches; information scent shapes the perceived value and cost; enrichments such as brainstorming keywords, exploring filters and page parking increase information gain and interaction cost — Information foraging theory applied to academic search engines.

The design requirements came from a review of literature-review guidelines in IS, an analysis of existing search systems, the typical results of published literature reviews (concept matrices, summary tables, trend charts), and interviews with master’s students, doctoral students and professors. Participants typically used three or more search engines and named searching across databases and extracting methods, topics and technologies from articles as the most time-consuming steps.

How it works

Research articles are extracted from PDF, split into sentences and annotated with the terms of the Information Systems Ontology (2,725 entities and 383,101 synonyms). Ontological annotation works like a highlighter: when the word survey appears, the system knows it is a data collection method and therefore a research method; UTAUT is a theory; 234 senior executives are participants.

But context matters. “A case study can be used to address such research questions” is background information. “Smith et al. (2020) performed a case study” refers to related work. Only “In this paper, we conducted a case study” belongs to the article itself. A transformer-based sentence classifier therefore keeps only the terms that describe the focal article.

Data extraction and preprocessing: research articles in PDF format, text extraction, sentence classification, ontological annotation and population of the data model; data model linking papers, sentences, entity annotations and IS Ontology entities such as theory, topic and research method — From PDF to a semantic data model.

Searching: three semantic features

Semantic filters. Users choose from thousands of filters, for example all articles that conducted a case study, used a specific theory or studied a particular IS topic, and combine them into a filter funnel. A browser shows the ontology hierarchy with article counts and all synonyms of a term.

Semantic filters: a searchable tree of ontology entities with article counts, such as research method, literature study and design science with sub-terms like design principle and design theory, and a list of synonyms — Browsing the ontology to choose semantic filters, with article counts and synonyms.

Semantic search results. For each hit, SemanticIS shows the most common topics, methods and theories, including the exact sentences in which they appear, so relevance can be judged at a glance (shown at the top of this page). Analytical charts show the results by year, outlet, topic and method, and each bar can become a further filter.

Semantic analysis. Saved collections of articles are grouped automatically by their methods, theories and topics, and can be exported to Excel.

Semantic analysis: saved articles grouped by theories, with a table of articles, their topics, methods and theories — Semantic analysis of a saved collection of articles.

Synthesising: the Knowledge Synthesis Dashboard

The Knowledge Synthesis Dashboard implements our framework for ontology-based knowledge synthesis. It consists of a process model (how to extract and annotate data), a data model (how to represent it) and an operations model: a grammar for composing syntheses from filters, focal aspects, numeric operations and presentation types, so that researchers can assemble concept matrices, trend charts or gap matrices for any ontological key term.

Operations model: options for filtering and grouping, focal aspect, numeric operation, presentation type and semantic operation, combined into an operation such as counting papers by research method per year as a bar chart — The operations model: a grammar for composing syntheses.

We applied it to 6,575 full-text articles of the International Conference on Information Systems from 1994 to 2023. Overviews that would take weeks of manual literature review can be derived in minutes:

Three bar charts: the top ten theories (led by the technology acceptance model and the theory of affordance), topics (led by IS use and database systems) and research methods (led by survey and qualitative interview) in ICIS articles — The ten most frequent theories, topics and research methods in ICIS articles.

Trends become visible: the technology acceptance model peaked around 2010, and affordance theory began to become the most prominent theory in ICIS articles around 2013. Research gaps appear in a co-occurrence matrix: case studies, for example, have rarely been combined with e-commerce in thirty years of ICIS.

Line chart of the top ten theories in ICIS by article count and year, 1994 to 2023 — Top ten theories in ICIS by year.

Heat map of research methods (case study, experiment, survey and others) against topics (database systems, e-commerce, IS use and others) by article count — Research gap matrix of methods and topics.

The framework was presented at ICIS 2024 in Bangkok, where it received the Best Design Science Paper award (second runner-up).

Roland M. Mueller presenting the paper A Framework for Ontology-based Knowledge Synthesis from Research Articles at ICIS 2024, with the title slide and the logos of the University of Colorado Boulder, TU Chemnitz and HWR Berlin — Presenting the framework at ICIS 2024.

Award ceremony on the ICIS 2024 stage: seven people, three of them holding award certificates, Roland M. Mueller in the centre — Best Design Science Paper award, second runner-up, at ICIS 2024 (Roland M. Mueller in the centre).

A related method for ontology-based computational literature reviews, exemplified for design science research, was published at WI 2024.

Evaluation

Annotation quality. Two annotators labelled the key terms in 100 abstracts (Cohen’s kappa 0.77). The pipeline reached an F1 score of 0.87 for exact matches (precision 0.92, recall 0.83) and 0.90 for partial matches.

Against LLM-based tools. In August 2024 we compared the pipeline with ChatGPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro on extracting theories, methods, topics and technologies from ten articles of the Senior Scholars’ list of premier IS journals. It extracted on average 3.1 times as many terms with by far the lowest error rate, and unlike the LLM tools it also organises the terms in the ontology’s hierarchy.

Tool	Terms found per article (avg.)	Error rate
Our pipeline	73.7	0.2%
ChatGPT-4o	29.6	17.6%
Claude 3.5 Sonnet	22.5	18.5%
Gemini 1.5 Pro	19.1	11.6%

Search. We compared SemanticIS with the metasearch engine LitSonar on a published benchmark: re-identifying the 121 case studies of a systematic literature review among 774 articles in six leading IS journals (2001–2005). Searching with ontology terms outperformed the keyword-based baseline, and sentence classification improved precision and recall further. A closer look at the “false positives” showed that 59 of 88 articles explicitly described a case study, which points to the not fully transparent filter criteria of the original review.

Users. In one-hour sessions with six experienced IS researchers, five found SemanticIS useful or very useful and would consider using it for their next project. In an applicability check, eleven mostly early-career IS researchers used it in their own projects for three weeks and rated it as important, accessible and suitable for literature reviews.

Semantic analysis, semantic filters and semantic search results increase information scent and information gain and decrease interaction cost — How the semantic features act on information foraging: more scent and gain, less interaction cost.

Background and next steps

Both tools build on Sebastian Hüttemann’s dissertation Domain-Specific Knowledge Extraction and Synthesis for Literature Reviews (2025, summa cum laude), which I supervised. The ontology itself is kept up to date with language models (HICSS 2025), and Image2Theory extracts knowledge from research-model figures (WI 2023). Further work classifies the research methods of IS papers with deep transfer learning for semi-automatic meta-analysis (HICSS 2021) and answers questions about long research articles with transformer models (HICSS 2023, Best Paper nomination).

Barbara Dinter, Sebastian Hüttemann in cap and gown, and Roland M. Mueller — Sebastian Hüttemann (centre) after his doctoral defence at TU Chemnitz, 2025, with Barbara Dinter and Roland M. Mueller.

The next step combines LLMs with ontological constraints, so that each extracted element carries its provenance and a calibrated confidence: living evidence maps and semi-automated meta-analyses that can be audited.