TopicLLM

Measured software validation · 16 November 2026

Run the method.
Keep the evidence.

All five stages completed on 5,872 synthetic records. These measurements establish how the software behaved on generated data. They do not reproduce private historical observations or establish research validity.

Download the run evidenceMachine-readable resultsReproduce in Python
1,200discovery candidates recorded
93.6%LLM benchmark agreement with generator labels
$2.48conservatively accounted API cost · $5 ceiling

01 / Discovery and interpretation

Twenty-one topics were found.

The 50-draw search over ceilings 2–25 selected 21 observed topics at ceiling 22, iteration 16; cv = 0.7962. Candidate records: 50 insufficient topics, 1,150 scored. The saved model reloaded with identical assignments. The seven-topic guided reconstruction is a distinct teaching artifact with explicitly assigned categories.

The run collected 420 live naming ballots: twenty per topic, versus the historical 5,000 per topic. A disclosed generator-majority mapping exercised the synthesis step. No human review was claimed. Read the vote log, topic map and review record in the evidence download.

02 / Benchmark and scaling

A comparison you can inspect.

Both classifiers were evaluated on 250 generator-labeled corpus examples. OpenAI agreement was 93.6% (κ = 0.9096), compared with 92.0% for Boolean matching. OpenAI passed the configured gate and classified all 5,872 records with their IDs and order preserved.

OpenAI · 93.6%

Boolean · 92.0%

Live benchmark performance against generator labels
ThemeSupportPrecisionSensitivity
Alcohol Delivery and Isolation Drinking4989.4%85.7%
Restaurant Support10100.0%100.0%
Social Media Promotions4794.0%100.0%
Sports28100.0%89.3%
Gaming360.0%100.0%
Residual / no theme11394.7%94.7%

The benchmark is sampled from the corpus, and all records share generator templates. Gaming has only three benchmark examples. The synthetic mapping uses generator labels across the corpus. These are software checks, not independent human validation.

Agreement across all corpus predictions was 95.8%; excluding the 250 benchmark texts, agreement on 5,622 remaining records was 95.8%. This additional comparison still shares the same generator and mapping limitations.

03 / Inference

Show the differences.

The inference stage produced five theme rows and fifteen period comparisons. The guided reconstruction matches all reported one-decimal prevalence values by construction. Newly generated live predictions are shown below without forcing agreement. Differences are percentage points, based on displayed one-decimal values.

Reported study prevalence versus new synthetic live predictions
ThemePeriodReportedSynthetic liveDifference
Alcohol Delivery and Isolation Drinkingpre9.1%8.8%-0.3 pp
Alcohol Delivery and Isolation Drinkingperi17.4%16.0%-1.4 pp
Alcohol Delivery and Isolation Drinkingpost11.9%10.9%-1.0 pp
Restaurant Supportpre1.9%1.9%+0.0 pp
Restaurant Supportperi5.5%5.5%+0.0 pp
Restaurant Supportpost1.7%1.7%+0.0 pp
Social Media Promotionspre6.6%6.8%+0.2 pp
Social Media Promotionsperi16.5%16.0%-0.5 pp
Social Media Promotionspost10.6%11.0%+0.4 pp
Sportspre8.2%7.7%-0.5 pp
Sportsperi11.4%10.4%-1.0 pp
Sportspost8.9%8.5%-0.4 pp
Gamingpre1.1%1.4%+0.3 pp
Gamingperi2.0%2.7%+0.7 pp
Gamingpost1.6%1.7%+0.1 pp

Period denominators and exclusions, raw counts, later-minus-earlier z statistics, p values and Bonferroni thresholds are in the evidence archive. Synthetic statistics are not historical statistics.

04 / Cost and provenance

Every attempt counts.

6,542 successful calls; 6,547 attempts recorded. Returned usage: 757,189 input tokens and 56,958 output tokens. Known-usage token estimate: $2.462553. Including conservative reservations for uncertain failures: $2.477165 of the authorized $5. The dated price schedule ignores caching discounts and is not a billing invoice.

Model: gpt-4o-2024-08-06. The archive contains candidate records, environment, input/model hashes, ballots, raw API response metadata, review record, benchmark predictions, corpus labels and inference outputs. Machine-local paths are redacted and JSON is reformatted; original and public hashes identify those changes. Executable model serialization and SQLite are omitted; the full JSON response ledger is included.

Archive SHA-256: 9aeb4de8a4e01fcf1cbbe55048c7277ffabe8f2d6fca353378a20f2c387f9282

The public site's separate allowance is $1/day and $5/month, with fixed examples and durable controls. Data handling and service limits.

Bring this workflow to your research.

Fork the repository or install the Python package. Have an agent verify sources and the environment, prepare a costed plan, retain evidence and assemble the review packet. Actual researchers approve the theme map and assess independent benchmark evidence before scaling. The package does not implement LIWC or Study 2 structural equation modeling.

Fork + reproduce