Online communities pilot

Run record: r/singularity pilot, vanguard sub-series

A working document. It is one of the documents the site is built from, published as it is written for anyone checking the work, and it uses the project's own shorthand. The plain account of the same material is on the home page.

Run record: r/singularity pilot, vanguard sub-series

Run id: vanguard-pilot-rsing-2026-10-03. Written and committed before any sample was drawn or any post coded. Governs: docs/amendments/2026-10-02-vanguard-stream.md (accepted 2026-10-02, with the 2026-10-02 video ruling and the 2026-10-03 title-only instruction). Brief: me, 2026-10-03.

Window and source

Samples (stream F design)

Eligibility and content basis

Coding

Outputs

Departures from the amendment, stated up front

  1. The second coder is a model, not two people (section 9). Reported as interim.
  2. Section 12 preconditions not yet met: venue inventory, X decision, cross-era sample. This pilot covers one venue and does not choose venues.
  3. Comment sample deferred (see above).

Changes during the run (recorded 2026-10-03, before the second pass)

  1. Missing images, fixed. The sampling script first saved images only for rewarded ranks 1 to 40. Coders reached rank 41 or deeper in 2025Q1, 2025Q3 and 2026Q1 and excluded ten image posts as "content unavailable" because no file existed. The script now saves every sampled image (the sample itself is unchanged; checked byte for byte), those posts were recoded from their images, and each quarter's cut-off was re-applied.
  2. Removed-body text posts, made uniform. Coders had handled text posts whose body was removed or deleted in three ways. Rule applied to all: if the title survives, the post is coded from the title with title_only: yes and content_basis: text_title. Galleries and image files that no longer load stay excluded.

Recode run, 2026-10-03 (after my rulings on the pilot report)

Same model, claude-opus-5-5, the same four first-pass coders (resumed, so their earlier judgments stay consistent). Applied to every screened post in the existing sample: the new extrapolates_to_horizon field and the codebook rulings of 2026-10-03 (docs/vanguard/codebook.md): posts not mentioning AI ineligible, demo-only video titles eligible as capability claims, memes under the legible-text rule, company and policy drama coded industry_governance. The rewarded stopping rule (20 eligible per quarter) is re-applied, screening further down each quarter's existing ranked reserve where needed; no new posts are drawn. Changed rows carry the note R2: with old and new values. The second pass is not recoded, so the interim agreement figures predate these rulings.

Cross-family second pass, 2026-10-03

Run under the revised section 9 (my ruling, 2026-10-03). The models are named and pinned:

Three-model coding, 2026-10-03

Run under section 9 as ruled by me later the same day: no human coder, three models from different companies, final code by majority vote.

Handover, end of 2026-10-03

Working files (post text, images, saved model replies, fetched link pages, archive cache) moved from the session scratch folder to ~/Documents/.agi-zeitgeist/work/ (outside the repository, because they hold post content). The scripts now point there.

Still to run, after Gemini's daily limit resets (about 17:05 Pacific, 2026-10-04), from the repository root, in this order:

python3 scripts/vanguard_crossfamily_pass.py --family gemini --rules docs/vanguard/proposed-rules-2026-10-03.md --codebook docs/vanguard/codebook-frozen-2026-10-03-before-rules.md
python3 scripts/vanguard_crossfamily_pass.py --family gemini --ids data/vanguard/r1-split-trial-posts.csv --rules docs/vanguard/trial-rule-r1a-2026-10-03.md --tag r1a-trial
python3 scripts/vanguard_crossfamily_pass.py --family gemini --ids data/vanguard/r1-split-trial-posts.csv --rules docs/vanguard/trial-rule-r1b-2026-10-03.md --tag r1b-trial
python3 scripts/vanguard_three_way.py --compare     # first trial, now on up to 140 posts
python3 scripts/vanguard_three_way.py --r1-split    # R1a and R1b, all three pairs

That is 252 Gemini requests against a limit of 250 a day; if the last R1b posts are refused, rerun the third command the next evening. Then update the pilot report and send me the R1a and R1b kappas for all three pairs. Claude and GPT codes for both trials are already committed. Interim Claude–GPT result: R1a raises stance agreement (0.50 without, 0.74 with); R1b doesn't help attribution (0.75 without, 0.72 with). The trial coders found that both rules need to say how a bare reaction or joke title ("WTF") is treated.

Waiting on me: a ruling on R1a and R1b. (I added $25 OpenAI credit on 2026-10-03, so GPT's share is covered.) Then the full-sample run: all 390 posts by Gemini and GPT under the final codebook, with Gemini capped at about 240 new requests a day (--ids data/vanguard/r-singularity-pilot-sample.csv style list, --max-new 240, a fresh --tag so earlier replies aren't reused).

Gemini runs, 2026-10-04

Run from the handover's commands, in its order, starting 12:15 Pacific. The handover expected the daily limit to clear about 17:05; a single test request (--max-new 1) at 12:15 was accepted, so the runs went ahead then. All 252 requests were accepted the same day, none refused. (Correction, same evening: the limit does apply. The full-sample run was refused after 2 more requests, so 254 in a day was the ceiling here. Plan on 240 a day.)

Decided, 2026-10-04: R1a adopted with the reactions sentence; R1b rejected. The codebook (docs/vanguard/codebook.md) is now final for the full-sample run.

Waiting on me: about $15 more OpenAI credit. Then the full-sample run as described in the handover above, on today's codebook with a fresh --tag.

Full-sample run, started 2026-10-04

I topped up the OpenAI credit and started the run the evening of 2026-10-04. Final codebook (docs/vanguard/codebook.md: R2 and R3 adopted 2026-10-03, R1a adopted 2026-10-04). Posts: the 390 screened in the first pass, listed in data/vanguard/full-run-posts.csv (230 top-ranked, 160 random; built from r-singularity-pilot-codes.csv). Label --tag full-final, so no earlier reply is reused.

python3 scripts/vanguard_crossfamily_pass.py --family gemini --ids data/vanguard/full-run-posts.csv --tag full-final --max-new 240

Run it twice, on two days; 388 posts remain.

Claude, recoded under the final codebook (my ruling, 2026-10-04). Claude's earlier codes for these posts predate R2, R3 and R1a, so a three-way majority would have mixed codebooks. Six fresh claude-opus-5-5 coders, 65 posts each (posts dealt out in turn, so each part mixes quarters and post types), coded all 390 blind. They got the same instructions, codebook and content as Gemini and GPT (brief: docs/vanguard/claude-coding-brief-full-final.md; link pages from the same cache, so the 13 unreadable ones are coded from the title). Output: r-singularity-pilot-codes-claude-full-final.csv, 507 rows, 265 posts eligible.

Gemini, 2026-10-06: stopped at 219 of 390, waiting on me over the cost limit

The daily limit had reset: a one-request test at 09:14 Pacific was accepted and the pin was confirmed (gemini-3.1-pro-preview version 3.1-pro-preview-01-2026). The run went ahead with the command from the section above, capped at 240 requests for the day.

Against the approved quote. I approved up to $15 for October (the full run plus the first monthly runs), with the full run itself quoted at $9.80.

To resume once I approve (saved replies are reused; at most about 240 requests a day):

python3 scripts/vanguard_crossfamily_pass.py --family gemini --ids data/vanguard/full-run-posts.csv --tag full-final --max-new 171
python3 scripts/vanguard_three_way.py --full

Then re-apply the 20-eligible cut-off per quarter from the majority codes, redraw the figure from the majority codes, and check that "AGI already achieved" counts leave out ironic posts.

Approved by me, 2026-10-06, about 10:20 Pacific (in the planning session; recorded as a TimeAssembler decision): "approve finishing the Gemini coding, go with the recommendation." That is about $4.40 more, about $22.20 for October; the limit is now about $22.50 for October without asking again. The approval is also noted under the quote in model-cost-quote-2026-10-03.md. The run resumed straight after.

Resumed 2026-10-06 after the approval: stopped at the daily limit, 254 of 390

Resume after midnight Pacific (2026-10-07 or later), when the daily limit resets. It fits in one day's allowance:

python3 scripts/vanguard_crossfamily_pass.py --family gemini --ids data/vanguard/full-run-posts.csv --tag full-final --max-new 136
python3 scripts/vanguard_three_way.py --full

Then re-apply the 20-eligible cut-off per quarter from the majority codes, redraw the figure from the majority codes, and check that "AGI already achieved" counts leave out ironic posts. If the per-post cost runs higher than expected, stop before October passes about $22.50 (full run about $10.30) and ask I again.

Gemini, 2026-10-07: finished, 390 of 390, and the three-way comparison

Scheduled run, started 09:34 Pacific, with the command from the section above. The daily limit had reset and no request was refused. It ran in three batches (40, 50, 46) so the cost could be checked between them against my 2026-10-06 limit of about $22.50 for October; it never came close to it.

Three-way comparison (python3 scripts/vanguard_three_way.py --full). Claude, Gemini and GPT each coded all 390 posts under the final codebook. A post counts by majority: 265 eligible. Output: r-singularity-pilot-codes-majority-full-final.csv (the majority codes) and three-model-reliability-full-final.csv (the table below). The numbers are kappa, a measure of how often two coders agree beyond what chance would give (1 is perfect, 0.6 is the usual floor for "good enough"); n is the number of posts both coders counted. "No majority" is the share of posts where all three gave a different answer.

What is coded Claude–Gemini Claude–GPT Gemini–GPT No majority Script's verdict
Whether the post counts 0.91 (390) 0.91 (390) 0.94 (390) 0% reliable
What the claim is about 0.77 (259) 0.83 (254) 0.76 (258) 2% reliable
Timing (already here, forecast, …) 0.65 (259) 0.63 (254) 0.63 (258) 5% reliable
Whether the poster agrees 0.75 (259) 0.63 (254) 0.56 (258) 3% reliable
Hedging 0.73 (259) 0.66 (254) 0.54 (258) 0% reliable
Tone of the post 0.74 (259) 0.75 (254) 0.79 (258) 0% reliable
Whether the poster wants it faster or slower 0.46 (259) 0.59 (254) 0.45 (258) 0% unreliable (every pair below 0.6; ruled unreliable by me, 2026-10-03)
Whose claim it is 0.72 (259) 0.71 (254) 0.71 (258) 1% reliable
Coded from the title alone 0.99 (259) 0.99 (254) 1.00 (258) 0% reliable
Projects to a time horizon 0.75 (259) 0.72 (254) 0.73 (258) 1% reliable

Worth a look in the working session: Gemini and GPT agree less with each other than either does with Claude on whether the poster agrees (0.56) and on hedging (0.54). The script still calls both fields reliable because the other two pairs clear 0.6.

Not done yet (left for a working session, as planned): re-apply the 20-eligible cut-off per quarter from the majority codes, redraw the figure from the majority codes, and check that "AGI already achieved" counts leave out ironic posts.

Figure redrawn from the majority codes, 2026-10-08

The three items left over from 2026-10-07 are done. scripts/vanguard_figure.py now reads the majority codes (r-singularity-pilot-codes-majority-full-final.csv) and the full-sample reliability table (three-model-reliability-full-final.csv) instead of Claude's first pass and the 140-post table. The figure, its counts table and the pilot report's Results and Agreement sections are rewritten from them. No model was run and nothing was spent.

What the majority file holds, and how it is read. One row per post: whether the post counts, the main claim's fields and the post-level fields; "unresolved" where no two models agree. It carries no secondary claims, no forecast horizon text and no frame tags. So:

Top-post cut-off, re-applied from the majority codes. Per quarter, down the ranked list until 20 posts are eligible by majority; eligible posts below that rank are left out.

quarter ranks the models coded eligible by majority cut-off rank counted eligible left out below the cut-off
2024Q4 1 to 27 21 26 20 1
2025Q1 1 to 29 19 none reached 19 0
2025Q2 1 to 26 20 26 20 0
2025Q3 1 to 27 20 27 20 0
2025Q4 1 to 32 19 none reached 19 0
2026Q1 1 to 32 22 30 20 2
2026Q2 1 to 31 22 28 20 2
2026Q3 1 to 26 18 none reached 18 0

156 top posts counted (161 eligible by majority among the 230 screened, less 5 below a cut-off). Three quarters fall short of 20 because the majority finds fewer eligible posts than Claude's first pass did among the ranks that were screened; reaching 20 there needs all three models to code the next posts down the ranked reserve (ranks 30 and on in 2025Q1, 33 and on in 2025Q4, 27 and on in 2026Q3). Not done; it costs money and is my call. The random sample has no cut-off: 160 drawn, 104 eligible by majority, all counted.

Irony and the headline claims. The codebook (section 3, rhetorical_mode, and rule R1a) says to record irony in rhetorical_mode but does not say whether an ironic post can affirm a headline claim; the measures section says only "poster affirms each". So the figure script now leaves posts coded ironic by majority out of the headline count, and still counts mixed and unclear (agent decision, stated in the script's docstring). It changes the top-post count from 3 to 0: the three posts are "AGI 🚀" (2026Q2), "AGI achieved" (2026Q3) and "Post-Singularity Free Healthcare" (2025Q2), each affirmed and ironic by majority. No post coded mixed affirms a headline claim, so requiring literal would give the same count. One more "AGI achieved" top post (2025Q2) has no majority on whether the poster agrees and is left out as unresolved. The headline measure now depends on tone, which passes the reliability test on the full sample (kappa 0.74, 0.75, 0.79).

Headline numbers, all quarters together (top posts; random posts): main claim a capability 89 of 153 (58%); 52 of 103 (50%). Consequence 14 of 153 (9%); 16 of 103 (16%). Industry and governance 43 of 153 (28%); 28 of 103 (27%). Capability posts that extrapolate 7 of 89 (8%); 8 of 51 (16%). Headline claims 0 of 143; 1 of 96. Full table and the title-only split in the pilot report.

Files changed: scripts/vanguard_figure.py, site/vanguard-pilot-r-singularity.png, site/vanguard-pilot-r-singularity.csv (new column posts_left_out_unresolved), docs/vanguard/pilot-report-r-singularity-2026-10.md, and this record. The web page's caption still says a 140-post subset; it is updated separately.

Reserve posts coded, 2026-10-09: every quarter now has 20 eligible top posts

Approved by me, 2026-10-09 ("Go."), for this purpose only, with a hard cap of $8 across the three models. Same method and scripts as the full-sample run: the final codebook, unchanged since 2026-10-04; each reserve post coded blind by all three models; majority by vanguard_three_way.py --full; figure by vanguard_figure.py. No new posts were drawn: the reserve is the ranked list of 60 per quarter already in r-singularity-pilot-sample.csv, with its content and images saved on 2026-10-03. There is no separate screening step: as in every earlier pass, a post is screened by coding it, and it counts if the majority finds it eligible.

Estimate, written before any paid call. Short quarters: 2025Q1 needed 1 more (19 of 29 screened eligible, 66%), 2025Q4 1 more (19 of 32, 59%), 2026Q3 2 more (18 of 26, 69%). Expected about 7 posts, ceiling about 14. Per post: Gemini $0.0253 (full-run average), GPT $0.054 (the quote), Claude no separate charge (it runs inside the Claude Code session). About $0.56 expected, $1.11 at the ceiling. Accounts: Gemini's October was about $22.03 against the earlier $22.50 line, so 14 posts (about $0.35) would still fit under it; the OpenAI credit left was not recorded (about $4 if my 2026-10-04 top-up was the $15 asked for), and there is no way to read the balance through the API, so a refusal would have been the signal.

How it ran. One post per short quarter at a time, all three models, then the majority rebuilt and each quarter checked; a quarter stopped as soon as it reached 20.

5 posts coded in all: 1 in 2025Q1, 1 in 2025Q4, 3 in 2026Q3. Cut-off ranks now 30, 33 and 29. The posts were appended to data/vanguard/full-run-posts.csv (which --full reads), so the model files, the majority file and the reliability table now cover 395 posts. Gemini and GPT were run with the full run's command and --max-new set to the round's size; before each round both scripts checked that the model versions were still the fixed ones used in the full run (gemini-3.1-pro-preview version 3.1-pro-preview-01-2026; gpt-6-astra created 1787853604). Claude: three fresh blind claude-opus-5-5 coders with the full run's brief (claude-coding-brief-full-final.md), parts 7, 8 and 9 (coder labels claude-opus-5-5-full-final-p7 to -p9), merged into the Claude file with vanguard_merge_codes.py (its only complaints are the per-quarter "expected 20" counts it always gives for a file with both samples). No request was refused.

Raw replies kept in ~/Documents/.agi-zeitgeist/work/vanguard_pilot/ as before: Gemini in crossfamily_raw_full-final/, GPT in gpt_raw_full-final/ (one file per post id: 1ikq4lv, 2025Q1 rank 30, the Pika "Additions" video; 1pmfpka, 2025Q4 rank 33, the image titled "Crazy true"; 1wgok3w, 2026Q3 rank 27, the Japanese comic "From Japanese twitter"; 1vghb3m, 2026Q3 rank 28, the Hassabis photo, an image with no legible text; 1ulizqk, 2026Q3 rank 29, the screenshot titled "Anthropic guardrails does it again"), Claude's inputs and outputs as claude_full_part7 to part9 (.json in, .csv out). Rerunning vanguard_three_way.py --full and then vanguard_figure.py from the committed files reproduces the committed majority, table and figure without calling any model.

Spend, from the token counts each reply returns:

model posts input tokens output tokens (thinking included) cost
Gemini 5 17,723 7,029 $0.12 (at $2 and $12 per million)
GPT 5 18,838 1,933 about $0.27 (5 at the quote's $0.054; tokens per post close to the full run's average). Exact charge on OpenAI's billing page
Claude 5 not metered separately no separate charge
total about $0.39, against the $8 cap

Gemini's October is now about $22.15, still under the earlier $22.50 line.

What changed in the results. Reliability barely moved (every kappa within 0.02 of the 2026-10-07 table, every verdict the same). The random sample is untouched. Top posts counted: 156 to 160. Main claim a capability 89 of 153 (58%) to 91 of 157 (58%); consequence 14 of 153 (9%) to 14 of 157 (9%); industry and governance 43 of 153 (28%) to 44 of 157 (28%); capability posts that extrapolate 7 of 89 (8%) to 7 of 91 (8%). Headline claims among top posts: 0 of 143 to 1 of 147 (1%). The one is the 2025Q4 reserve post at rank 33, an image titled "Crazy true", which two models code as literal singularity framing (the poster agrees, under way) and Gemini codes as ironic, so by majority it is literal and counts. 2026Q3's rank 27 post is coded ironic by all three, which the irony reading (2026-10-08) keeps out of the headline count in any case.

Files changed: data/vanguard/full-run-posts.csv, the three *-full-final.csv model files, r-singularity-pilot-codes-majority-full-final.csv, three-model-reliability-full-final.csv, site/vanguard-pilot-r-singularity.png and .csv, the pilot report (Results, Agreement and the to-do list), the figure script's docstring (post count only), and this record.

Rendered from docs/vanguard/pilot-run-r-singularity-2026-10.md at version a7bd17a.