Abstract
Large language models are increasingly used to scale codebook-based annotation in scientific research, but existing workflows provide limited support for translating domain experts' codebooks into reliable, revisable, and auditable prompts. Prompts are often treated as fixed instructions and hidden from annotators, making it difficult for non-technical domain experts to diagnose and correct model behavior when outputs violate codebook guidelines. We present CALICO, a human-centered, codebook-aligned annotation workbench that treats prompts as editable, versioned, and optimizable artifacts. CALICO integrates codebook parsing, prompt generation, result inspection, prompt versioning, natural-language human feedback, and label-supervised prompt optimization through existing optimizers such as GEPA, MIPROv2, and OPRO, together with our reflection-based optimizer, ReflectAgent.
Empirically, we evaluate CALICO on a domain-specific AI-companion chatbot conversation codebook where two trained coders agree on only 34.8% of items for one central dimension. Across evaluated dimensions, CALICO improves mean held-out performance by +13.0 and +7.4 points for the two coders, respectively, and a coder-specificity analysis suggests the optimized prompts capture coder-specific interpretations rather than only generic codebook clarification. CALICO runs as a web application that takes users from raw codebook materials to inspectable, exportable labels.
What the system does
1Codebook-aligned
Structured parsing keeps every prompt tied to your codebook's dimensions, labels, and definitions.
2Human-centered
Inspect draft labels, give feedback, and let calibration turn your corrections into better prompts.
3Versioned prompts
Every prompt revision is tracked, so you can compare rounds and roll back with confidence.
4Easy export
Run the final annotation pass and export labels as CSV or JSON.
BibTeX
@misc{yuan2026calico,
title = {{CALICO}: A Human-Centered, Codebook-Aligned System for Annotation},
author = {Yuan, Boqin and Gu, Xiaoyi and Li, Fiona and Wan, Chang and Hwang, Angel Hsing-Chi and Zhao, Jieyu},
year = {2026},
url = {https://calico-annotation.github.io}
}