The Problem
Asynchronous Online Focus Groups (AOFGs) are a widely used tool to collect feedback at scale, while addressing key geographic barriers and participant time constraints; however, the feedback collected via AOFGs is often too vague or ambiguous to act on.
"Q: Was this helpful? A: Kind of" gives a designer nothing to act on.
ClarifAI intercepts vague feedback before it becomes unusable. A four-module LLM pipeline filters out irrelevant responses, flags vague or ambiguous comments, and conducts a targeted follow-up dialogue to turn a thin comment into richer, more granular data for design and evaluation teams.
AI
Human
AI
Human
AI
Problem Clarified
Raw response: "somewhat useful"
Granular data captured: useful for orientation, but missing stopped/new medications and rationale needed for medication reconciliation.
How It Works
The Telemetry module classifies whether each piece of feedback (e.g., an individual answer to a feedback question/prompt) is addressing the informational intent of the question/prompt. Off-topic or tangential responses are filtered out before they enter the pipeline.
The Flight module further filters relevant feedback for responses that are not specific enough to act on (i.e., contain any vagueness or ambiguity). Only feedback that contains vagueness or ambiguity is escalated to the clarification dialogue, keeping the experience lightweight for users who already gave clear and relevant responses.
The CapCom module engages the user in a short, targeted follow-up conversation, for each vague or ambiguous topic identified by Flight. The LLM interviewer only asks questions needed to resolve the vagueness or ambiguity, then sends the transcript of the conversation to the Payload module for final processing.
The Payload module extracts the relevant and granular information elicited from the user by the CapCom module, and injects the new granular information into the original feedback response, resulting in a structured, machine-readable, and specific insight that designers can act on directly.
System Architecture
The complete ClarifAI pipeline, from AOFG collection through four LLM modules to structured, actionable output.
Stage 1
Prerequisite Task
Stage 2
Project & Tasks
Stage 3
Discussion Board
Contextual
Relevance
Requires
Clarification
Clarification
Dialogue
Summarisation
& Refactor
Why It Matters
Actionability of Feedback by Condition
100%
Telemetry precision
94%+
Flight accuracy
4
LLM pipeline modules
AOFG
Feedback context
Team
James Mitchell
Assistant Professor · Principal Investigator
Department of Biomedical Informatics · CU Anschutz
Alexander Kotz
PhD Student · CPBS Program
Clinical NLP and LLM evaluation
Manjushree B. Aithal
Project Team
Postdoctoral Fellow · CU Anschutz
Faisal Alquaddoomi
Project Team
Software engineering and research infrastructure
