TopicLLM

A human-in-the-loop instrument for public health surveillance

Turn social text into
interpretable evidence.

A five-stage pipeline that turns large volumes of social media text into a small set of interpretable themes, and those themes into prevalence estimates you can test. Use the visual demonstration to understand every step, then fork the repository or download the Python package for a reproducible, agent-assisted research workflow.

DISCOVER · NAME · SYNTHESIZE · CLASSIFY · ESTIMATE · IIIIIIIVV
I

The method

Five stages, from raw text to a test statistic

Machines do what machines do well: search a large model space, read every document, and count. People do what only people can: decide what a pattern means. The pipeline keeps the two in their lanes and writes down everything that passes between them.

I

Discover

BERTopic (MiniLM embeddings, UMAP, HDBSCAN, c-TF-IDF) under randomized hyperparameter search. The model with the highest cv coherence wins.

50 draws × 24 solutions1,200 planned candidate models

II

Name

A language model names each topic thousands of times, independently. The modal label is adopted, and a fragmented vote flags an ambiguous topic.

Votes per topic5,000 · temperature 0.5

III

Synthesize

Two researchers group topics into themes under the Theme and Topic system, reading against the literature. Deliberately not automated.

ProtocolPrimary + senior, consensus

IV

Classify

Boolean and LLM classifiers are piloted against human-labeled documents. The pipeline refuses to scale a classifier below the agreement threshold.

Benchmark · gate250 documents · >85%

V

Estimate

Every document carries a label, so prevalence is computable in any slice. Proportional z-tests compare periods under Bonferroni correction.

OutputPrevalence + z-tests

Pipeline: documents flow through five stages, each emitting an audit artifact. Topics c_v argmax of 1,200 all_iteration_results.json I Labels modal vote of 5,000 vote_tallies.csv II Themes two-person consensus theme_map.csv III Labeled corpus gated classifier benchmark_report.json IV Prevalence z-tests, Bonferroni ztest_results.csv V Each stage writes an artifact that can be published with the findings.
Plate IThe five stages and their audit artifacts. The prompts are released verbatim, and a new domain requires revisiting corpus design, stopwords, context, theme definitions, human benchmarks, time periods and statistical assumptions.
II

The demonstration

Alcohol marketing on Twitter during COVID-19

The pipeline was built for, and first applied in, the author's dissertation: a study of what eleven alcohol companies said on Twitter before, during, and after the first pandemic lockdowns, and of whether that messaging moved the public conversation.

5,872brand tweets from 11 companies, Jan 2019 – Jul 2021
0.768cv coherence of the winning model, out of 1,200
7 → 5topics synthesized into five marketing themes
3 of 5themes rose significantly during the lockdown
486,786user-generated tweets classified in the second study
View as a table
ThemePre-pandemic (%)Peri-pandemic (%)Post-pandemic (%)
Alcohol Delivery and Isolation Drinking9.1*17.4†11.9*†
Restaurant Support1.9*5.5†1.7*
Social Media Promotions6.6*16.5†10.6*†
Sports8.211.48.9
Gaming1.12.01.6

* differs from peri-pandemic; † differs from pre-pandemic (p < 0.0033, Bonferroni-corrected).

Plate IITheme prevalence across the three response periods, as published (Thomas, 2025, Study 1). Alcohol delivery and isolation drinking, restaurant support, and social media promotions all rose in the lockdown months. Two of them stayed above pre-pandemic levels afterwards. Sports and gaming held steady.
III

The preprints

One method, two studies

The methods paper describes the instrument. The original studies are available directly from the unchanged dissertation. Existing reflowed PDF and Word editions are also preserved. The methods paper has a separately identified audited revision.

Methods · 2026

Coherence-Optimized Topic Modeling with Democratic Large Language Model Interpretation

Thomas, Murthy, Pinedo, Loukas & Pasch

A transparent chain from documents to topics, human-reviewed themes, validated labels and prevalence estimates.

Study 1 · 2026

Alcohol Marketing on Twitter During the COVID-19 Pandemic: Theme Identification, Prevalence Comparison, and Sentiment Analysis

Thomas, Murthy, Pinedo, Loukas & Pasch

“This study aimed to identify themes used in tweets by alcohol brands early in the pandemic, determine if the prevalence of these themes differed across time, and characterize the sentiment used in the themes.”

Study 2 · 2026

Alcohol Information Dynamics During Crisis: Bidirectional Influence Between Alcohol Industry Marketing and User-Generated Content on Twitter

Thomas, Murthy, Pinedo, Loukas & Pasch

“Early potential exposure to alcohol brand tweets significantly predicted subsequent user-generated content about alcohol delivery and isolation drinking … Conversely, early user-generated content did not significantly predict subsequent potential exposure.”

The full dissertation, Alcohol Marketing on Social Media During the COVID-19 Pandemic: Historical Perspectives, Modern Evidence, and Future Regulation (The University of Texas at Austin, 2025), is available here unchanged.

IV

The instrument

A guided study. An inspectable workbench.

Start with a guided, seven-topic synthetic reconstruction of the dissertation. Recalculate the published prevalence pattern, inspect the source, and download the evidence. Try fixed live OpenAI examples, or explore local discovery with the teaching mock. For your own research corpus, fork the repository or install the Python package.

What it does

Every computation, in the open

The guided study explains each stage and exports its synthetic calculation. Live examples return model identifiers and token usage. The local sandbox exposes the mechanics and a validation gate. The Python workflow retains the complete research audit trail and requires actual researcher review.

Explore the study

When to use the Python pipeline instead

Publication-grade runs

Browser discovery is a distinct approximation with different JavaScript algorithms and preprocessing; it does not establish numerical parity with Python BERTopic. For a study you intend to publish, use the reference implementation in the code section. It provides seeded search, checkpoint guards and validation artifacts. Retain and validate the exact environment used for your analysis.

V

The data

An open corpus built to be run

The dissertation's data cannot be posted openly. In their place is a synthetic demonstration corpus that has the same shape as the original and contains no real posts, accounts, or people.

Built to mirror the original

The corpus follows the reported total and brand structure; rounded theme prevalence matches the published table. Period sizes are synthetic calibration choices, not recovered historical denominators. Feed the ground-truth labels to Stage 5 and you get back Plate II's prevalence table and its significance pattern. Run the full pipeline and you see how close a classifier comes. The campaigns are the same kinds the dissertation found: a delivery-app promotion, stay-home sipping, a restaurant relief fund, living-room concerts, giveaways, a virtual draft party, home sports challenges, and a charity gaming tournament, all set among ordinary brand chatter.

Released under CC BY 4.0. The generator is dtm_pipeline/make_demo_data.py, which is seeded and reproduces the files byte for byte.

Access to the dissertation data

The source data are brand tweets recorded by the Prevention Research Lab Twitter Marketing Study, and user tweets from the Computational Media Lab COVID-19 Twitter Dataset. They include account-level information and content governed by the platform's terms and by the agreements under which they were collected, so they are not posted publicly. The original records are not distributed with this tool because of privacy and research-governance constraints. Methodological questions may be directed to: jethomasphd@gmail.com.

VI

The code

Fork it. Give your agent a method.

A maintained Python pipeline written against the current OpenAI SDK, with documented prompts and search spaces. Maintained changes, including a larger classification output limit, are distinguished from the historical protocol. It adds a seeded search, an enforced benchmark gate, checkpointing for corpus-scale runs, and a test suite. Out of the box it runs on the demonstration corpus.

Start with the reproduction guide. The repository includes an agent contract, installable Python package, synthetic acceptance checks, protected source hashes and explicit researcher review gates.

git clone https://github.com/jethomasphd/TopicLLM.git
cd TopicLLM
python -m pip install ".[dev]"
topicllm --profile quickcheck --output-dir runs/quickcheck

# Then read AGENTS.md and REPRODUCE.md with your agent.
# Install ".[ml,dev]" for discovery and declare a budget before API calls.

Start a reproducible workflow Download Python source + data

ComponentWhereWhat it is
Reference pipelinedtm_pipeline/Five stage scripts, orchestrator, configuration, demonstration-corpus generator, tests
Browser instrumentdocs/app/Visual synthetic walkthrough, capped live examples, and a local teaching sandbox; research runs use Python
Supplement S1S1_bertopic_optimization.txtArchival Stage 1 code, transcribed from the dissertation appendices
Supplement S2S2_democratic_topic_naming.txtArchival democratic naming (openai==0.27.0)
Supplement S3S3_llm_classification.txtArchival corpus-scale classification (openai==0.27.0)
§

Citation

How to cite

Cite the methods preprint for the pipeline and the software for the code. DOIs will be added here when the preprints are deposited. The repository also carries a CITATION.cff that GitHub reads.

Thomas, J. E., Murthy, D., Pinedo, M., Loukas, A., & Pasch, K. E. (2026). Coherence-optimized topic modeling with democratic large language model interpretation: A human-in-the-loop pipeline for public health surveillance of social media [Preprint].

Thomas, J. E., Murthy, D., Pinedo, M., Loukas, A., & Pasch, K. E. (2026). Alcohol marketing on Twitter during the COVID-19 pandemic: Theme identification, prevalence comparison, and sentiment analysis [Preprint].

Thomas, J. E., Murthy, D., Pinedo, M., Loukas, A., & Pasch, K. E. (2026). Alcohol information dynamics during crisis: A longitudinal analysis of bidirectional influence between alcohol industry marketing and user-generated content on Twitter during the COVID-19 pandemic [Preprint].

Thomas, J. E. (2026). TopicLLM: Coherence-optimized topic modeling with democratic large language model interpretation [Computer software]. https://github.com/jethomasphd/TopicLLM

Thomas, J. E. (2025). Alcohol marketing on social media during the COVID-19 pandemic: Historical perspectives, modern evidence, and future regulation [Doctoral dissertation, The University of Texas at Austin].