ARA Lab

VALIDATING AI

Verification for
agent-native research

TRIAL · Lara · rit

Jiachen (Amber) Liu

Founder · ARA Lab

ERIC SCHMIDT SCIENCES · VALIDATING AI WORKSHOP · OCTOBER 2026

ARA Lab

Background

From ML systems to infrastructure for AI scientists.

CS PhD

Pre-training @
Meta Superintelligence Labs

ARA Lab

Founder

Research trajectory

01

ML SYSTEMS

train + serve efficiently

02

AI SCIENTIST

automate discovery

NOW

03

RESEARCH INFRASTRUCTURE

for AI scientists

Background

Our work

We build four pieces of infrastructure for AI scientists.

01 · TASTE

Sci-Reasoning

how top scientists think

ARXIV 2026

02 · KNOWLEDGE

ARA · Curie

how research is documented and compounds

PAPERThe Last Human-Written PaperAgent-Native Research Artifacts

NEURIPS 2026

03 · VERIFICATION

TRIAL · Lara · rit

how a result earns trust

TODAY

04 · EVALUATION

EXP-Bench ·
Open-Endedness Bench

how open-ended research is benchmarked

ICLR 2026

Today: verification.

Our work

Why verification

Generation runs at machine speed.
Verification still runs at human speed.

AI generation

∞AGENTS

hypothesis

experiment

result

hypothesis

experiment

result

hypothesis

experiment

result

hypothesis

MACHINE SPEED

backlog

↑ piling up

human review

reviewing

checked

✓ checked

one reader,
one result
at a time

HUMAN SPEED

KOSMOS

one 12-hour run ≈ six months of human research · over 40% of its interpretive claims are wrong

Every unchecked result becomes the next experiment's premise.

Why verification

Overview

Three projects: from one run, to one claim, to all claims.

ONE RUN

ONE CLAIM

ALL CLAIMS

PROJECT 1

TRIAL

Did the agent really do the task, or hack the reward?

TODAYonly the final score is checked

PROJECT 2

Lara

Can a machine check the argument behind a claim?

TODAYclaims are written in ambiguous prose

PROJECT 3

rit

Is a new claim consistent with the knowledge base?

TODAYan LLM judge reads each claim alone

Overview

Project 1 of 3

1 · TRIAL2 · Lara3 · rit

PROJECT 1

TRIAL

Trajectory Review of Integrity And Legitimacy

Jicheng Wang, …, Jiachen Liu

THE PROBLEMAI agents reward-hack: they get the score without doing the task.

Project 1 · TRIAL

TRIAL · why it matters · 1

A missed hack doesn't stay in one run. It spreads.

Project 1 · TRIAL

【约 40 秒 · 点 3 下，每下一条去向】 Why does this matter? When a hack gets past the check, it doesn't stay in that one run. It spreads, and each place is worse than the last. (点 1) Training: the run is scored as a success, so it becomes training data. And the model doesn't just learn the shortcut. Anthropic found that once a model learned to reward-hack in real training, it started faking alignment in half its answers, and sabotaged safety research code 12% of the time. (点 2) The paper: it says the experiment worked. An audit of the AI Scientist found 42% of its experiments had failed on code errors, while the papers reported success. (点 3) Security: in a post-training run, Opus 4.7 found an API key in its environment, had a bigger model write its training data, and went from 4% to 73% on Arena-Hard. The LLM judge checking for contamination missed it. Using access you were never given is exactly what a security monitor exists to stop. （台上不讲，被问再补：现有检查只看最终输出，LLM judge 会漏掉巧妙的情况，也没人能手读几万条轨迹。） (来源：Anthropic 2025 "Natural emergent misalignment from reward hacking in production RL"（arXiv 2511.18397）：alignment faking 50%、在 Claude Code 里 sabotage 12%；rit intro 引用 Beel 2025 对 AI Scientist 的审计；API key 例子来自 rewardhacking.io / PostTrainBench 轨迹，原第 7 页。)

TRIAL · why it matters · 2

The stronger the model, the cleverer the hack.

CRUDE

Overwrite the file the grader reads

echo "…" > game/fen.txt

chess agent rewrites the board to win

SNEAKY

Copy the official fix from the project's history

git show <upstream fix>

SWE-bench agent; the task counts as solved

HIDDEN

Reword the test questions slightly, then train on them

test item → reworded → train.jsonl

post-training agent; the benchmark's own contamination check reports 0

REWARDHACKING.IO20 models, 1,226 audited runs: when frontier models cheat, the LLM judge misses it almost every time.

Project 1 · TRIAL

【约 45 秒】 And it's getting harder to catch: the stronger the model, the cleverer the hack. Crude: a chess agent overwrites the board file so the engine thinks it's winning. Sneaky: a coding agent runs git show on the commit that fixed the bug upstream, and pastes it. The benchmark marks it solved. Hidden: the agent is asked to post-train a model. It opens the test set, rewrites a few test questions in its own words — same patient, same symptoms, slightly different wording — and puts them in the training data. The benchmark ships its own contamination checker; it looks for copied text, so it reports zero. In our study of 20 models and 1,226 runs: when frontier models cheat, the LLM judge misses it almost every time. (来源：TRIAL HACKING-CATALOG；HIDDEN 例子是 TRIAL docs/results/full-corpus-read-2026-09.md §1.2 机制 3（读 healthbench.jsonl，照着具体题目写相似训练题，如 mushroom/migraine、PCOS/keto，checker 报 0）；rewardhacking.io 研究。Q&A：原始 hack rate 最高的不是最强模型（MiniMax），但最强模型 hack 时几乎都能躲过 judge。)

TRIAL · what it does

A hack is a path from off-limits content to the score.

1INPUT · THE WHOLE RUN

#0001$ ls evaluation_code/

#0092Read evaluate.py

#0183$ python evaluate.py

#0274 → 31/50 passed …

#0365Write prep_data.py

#0456$ python prep_data.py

#0547$ head data/train.jsonl

#0638Write train.py

#0729$ nohup python train.py &

#0820$ tail -f train.log

#0911$ python evaluate.py

#1002Edit train.py

#1093$ python train.py --epochs 3

#1184$ cp -r out/ final_model/

#0001$ ls evaluation_code/

#0092Read evaluate.py

#0183$ python evaluate.py

#0274 → 31/50 passed …

#0365Write prep_data.py

#0456$ python prep_data.py

#0547$ head data/train.jsonl

#0638Write train.py

#0729$ nohup python train.py &

#0820$ tail -f train.log

#0911$ python evaluate.py

#1002Edit train.py

#1093$ python train.py --epochs 3

#1184$ cp -r out/ final_model/

thousands of steps: shell, Python, tool calls, what each one printed, and the files left at the end

→

2EXTRACT · WHERE CONTENT MOVED

READWRITECOPYFETCHRUNTRAIN

nodes: where content sits · edges: the action that moved it · edges it can't see stay in, marked unknown

→

4FIND THE PATH

Follow the graph from off-limits content to what gets scored.

EARNEDno path

UNEARNEDa path, and the path is the proof

UNKNOWNa step it can't see, and why

no LLM in the verdict · same rules, every run

3MARK OFF-LIMITS
pattern library

OFF-LIMITSanswer keys, hidden tests, the upstream fix, the grader's files, set per benchmark

MATCHED BY PATTERNScopied, reworded, downloaded, or generated from test items; the library grows with each new hack

WHAT IS SCOREDthe trained model (PostTrainBench) · the submitted patch (SWE-bench)

Project 1 · TRIAL

【约 60 秒 · 3 步】 Input: the whole run. Thousands of steps — shell commands, Python scripts, tool calls — plus what each one printed, and the files left at the end. Any hack is somewhere in here. (点 1) TRIAL rewrites every step the same way: what it read, what it wrote, what it ran, what training used. A one-line cat and a five-hundred-line script become the same few verbs. That gives a graph: nodes are where content sits, arrows are the step that moved it. Anything it can't see stays in, marked unknown. The graph here is a real pattern. The task: train a small model for BFCL, a function-calling test. The agent may run the grader to check progress. It runs the grader, the grader prints the question it got wrong, number 67, and the agent writes training examples — in its own words, "exact match for the failing case." Retrain, rerun, now it misses 54, 87, 90, 91, and it writes examples for those. Every step is allowed. It never opens a test file. (点 2) Next, TRIAL marks what's off-limits. Each benchmark names it: answer keys, hidden tests, the fix in the project's history, the grader's own files. A library of predefined patterns finds that content wherever it shows up in the graph: copied as is, reworded, downloaded, or generated from the test items. When we find a new kind of hack, we add a pattern, and a pattern only goes in after we replay it on every past run and check it changes no verdict it shouldn't. (点 3) Then TRIAL finds the path: from the off-limits content it marked, through the graph, to what gets scored. No path: earned. A path: unearned, and the path is the proof anyone can check step by step. A step it couldn't see: unknown, with the reason. No LLM anywhere in the verdict. In our example: question 67 appears in the grader's output, its rare words show up next in the training file, the training file goes into training, and training produces the scored model. That's a path. （Q&A：BFCL 例子是 TRIAL docs/results/full-corpus-read-2026-09.md §1.2 机制 2 的重建；上台前在 carcinoma 上用 trial explain 核对一条真实 witness path。"rare words" = 规则 key_by_echo。）

TRIAL · results

Caught hacks change the benchmark scores.

SWE-bench Verified

23 / 162

One rank falls from #19 to #23.

copied from the answer key

Terminal-Bench 2

7 / 75

Three ranks fall 6–15 places.

read a hidden test or a playbook

PostTrainBench

94 / 116

Those scores are not clean.

116 of 914 runs were contaminated

The old rankings need another look.

Project 1 · TRIAL

【约 40 秒】检出这些 hack 之后，公布的分数会变，所以旧榜需要再看一遍。 SWE-bench Verified（台上说 "audited with TRIAL's rule"）：162 个有公开轨迹的 entry 里，23 个至少有一条从答案抄来的解。拿掉之后，一名从第 19 掉到第 23。前 5 名不变。 Terminal-Bench 2：75 个提交里 7 个靠读隐藏测试或自带题解过关。其中 3 个排名掉 6 到 15 位。 PostTrainBench：914 次训练里 116 次被污染，TRIAL 抓住 94 次，精度 85%，没有用 LLM。这些分数不能当成干净的训练结果。 Q&A 注意： - 排行榜结果是按 TRIAL 的定义人工逐条读过的 label（199 个候选全读，68 定罪、7 weak、124 驳回），不是引擎全自动跑的；引擎正在接入 SWE-bench bash-only。 - 被问"扫了多少条"：答 entry 数（162 个 entry，23 个有），不要主动报 81,477 条 trajectory。 - PostTrainBench 是 dev split；held-out 一次性评分是 26/45/59（precision 58%，recall 44%），差距集中在 dev 没有的 humaneval、gpqamain。116 条 label 来自 corpus judge（gpt-5.4 + artifacts）。 - 每条定罪都带路径，谁都可以核对。台上不主动讲这句。

Project 2 of 3

1 · TRIAL2 · Lara3 · rit

PROJECT 2

Lara

Beyond Natural Language: An Agent-Native Language for Autonomous Science

Yifeng He, …, Jiachen Liu

THE PROBLEMResearch claims are written in natural language. It is ambiguous, so a machine can't check them.

Project 2 · Lara

Lara · why it matters · 1

A correct number can still support a wrong conclusion.

Today, this check lives in a reviewer's head

One score for the whole paper, not per claim
No pointer to which premise is missing
“Attacked” and “never supported” look the same, though they need different fixes

The arithmetic stays true. The conclusion loses its support.

Project 2 · Lara

Lara · why it matters · 2

Prose hides what a claim depends on.

What the paper says

“Adam + L-BFGS consistently outperforms Adam or L-BFGS alone.”

ICML 2024 paper on training physics-informed neural networks · one of the PaperBench papers

✓ lowest error in every setting they report

What the claim quietly needs

✓same equations, same network sizes

✓loss and error reported for each setting

✗the same tuning effort for each optimizer

Adam

5 settings tried

Adam + L-BFGS

15 settings tried

The sentence reads the same either way. Someone has to dig the premise out by hand.

Project 2 · Lara

【约 45 秒 · 2 步】 And sometimes the premise was never there at all. Here is a real sentence from a published ICML paper: "Adam plus L-BFGS consistently outperforms Adam or L-BFGS alone." And their tables back it up: lowest error in every setting they report. (点 1) Now write down what that sentence quietly depends on. Same equations, same network sizes — yes. Results for every setting — yes. The same tuning effort for each optimizer — no. Adam was tried with 5 settings; the combined method with 15. Maybe it still wins with equal tuning; maybe not. The paper doesn't establish it. (点 2) The sentence reads exactly the same either way. A careful reviewer has to dig that premise out by hand. With over 20,000 submissions a year at each top AI venue, and more and more of them written and reviewed by machines, nobody is digging. That's exactly what Lara writes down. （来源：Lara corpus 单元 pinn/C05（PaperBench 的 pinn 论文，ICML 2024）：nl 原句 + 注释"baseline_parity CQ unmet-gap: Adam searches 5 lrs, Adam+L-BFGS searches 5 lrs x 3 switch points = 15 configs"。20,000 见 Lara intro。原来的 APT/CoFi 两对例子是 Lara 论文 D3 的 illustrative reconstruction，已换掉。）

Lara · what it does

Lara writes an argument so a machine can check it.

adaptive_pruning.lara

CLAIMthe kurtosis term is critical for pruning

EVIDENCEscore 50.0 with the term, 38.1 withoutobserved · Table 5

EVIDENCEonly that term changed between runsstated

ARGUMENTablation: evidence ⇒ claim

MUST ANSWERwas variance reported?no answer

ATTACKopen slot: a reviewer, a rival lab, or the paper's own limitations

checkersame verdict every time · milliseconds

JUSTIFIEDsupport is complete and survives every attack

GAPsupport is incomplete; the missing piece is named

DEFEATEDan attack knocked the support down

CONTESTEDsupport and attack in a standoff

this claim → GAP · missing: variance reported

Project 2 · Lara

【约 40 秒 · 2 步】 How does a good reviewer find gaps like the last two? With a checklist in their head: you ran an ablation, so did you report variance? Lara builds that checklist into the language. On the left is one claim from a real pruning paper, written in Lara. The claim; the evidence, each piece tagged with where it came from — this score is from Table 5, that sentence is just the authors saying so; then which kind of argument connects them, here an ablation. Declare an ablation, and the questions an ablation must answer come with it: was variance reported? Here, no answer. And there's a slot for attacks, which we'll use on the next slide. (点 1) An AI can write this file. A small checker reads it. It takes the numbers as given and checks only the argument: does it hold, what's missing, what knocked it down. One of four statuses, in milliseconds, same verdict every time. (点 2) This claim: gap — missing, variance reported. What used to take a careful reviewer, the machine now points to, and names the missing piece. Q&A 边界：Lara 不判断测量值本身是否真实，它检查的是"写下来的支持"是否完整、能否扛住写下来的反驳（"audits reasoning, not reality"）。程序是 corpus 单元 adaptive-pruning/C04 的简化示意，真实语法是 claim / leaf / arg / attack。

Lara · results

Replay a review round, and every status change has a reason.

SUBMISSION

REVIEWS

REBUTTAL

Beats the dense baselinebenchmark result

JUSTIFIED

DEFEATED

JUSTIFIED

The new term is criticalablation, one run

GAP

JUSTIFIED

Low training memorymeasurement

JUSTIFIED

DEFEATED

"variance not reported"

three attacks, each aimed at one line: the protocol, a replication, the memory measure

variance added · objections answered · memory point conceded

48of 60 claims
come back GAP

“Adam + L-BFGS beats Adam”the baseline got a third of the tuning

“cheaper than full fine-tuning”8 GPUs at batch 40 vs 1 GPU at batch 5, never normalized

“the adapter transfers to new LLMs”it drew 3 samples per answer; the baseline drew 1

Project 2 · Lara

【约 60 秒 · 3 步】 What does this buy you? Take the pruning paper from the last slide and replay a review round. Each round is a program; only new arguments get added, and the checker re-runs. At submission: "beats the dense baseline" and "low training memory" are justified. The ablation claim is a gap — variance not reported. (点 1) Reviews come in as three attacks, each aimed at one line. One challenges a piece of evidence: was the protocol fixed? One brings a conflicting result: a replication that disagrees. One challenges a step of reasoning: does that measurement really show low memory? Two claims are now defeated. (点 2) The rebuttal adds the variance, so the gap closes. It answers the first two attacks with counter-arguments, so the benchmark claim comes back. It doesn't answer the memory objection, so that claim stays defeated — conceding is simply not answering. Every change traces to a line you can inspect. (点 3) At scale: we encoded 60 claims from 32 research artifacts. 48 come back as gap, each naming what's missing. Like the optimizer claim from before: the baseline got a third of the tuning. A "cheaper than fine-tuning" claim that compared 8 GPUs at batch 40 against 1 GPU at batch 5. An adapter that "transfers" while drawing 3 samples per answer against the baseline's 1. That says the written-down argument is incomplete, not that the science is wrong. Q&A 注意： - 审稿和 rebuttal 是根据 corpus 里已记录的 gap 构造的（Lara 论文 §case study 原话："derives its reviews and rebuttal from a documented corpus gap rather than from an OpenReview exchange"），不是真实 OpenReview 记录。被问就照实说。 - 48/60 是 machine annotation 一遍、人工复核待做；例子来自 corpus 单元 pinn/C05、bridging-data-gaps/C03、bbox/C05（都是 PaperBench 论文）。

Project 3 of 3

1 · TRIAL2 · Lara3 · rit

PROJECT 3

rit

Never Trust an AI Scientist: Lean-Verified Autonomous Research

Jintao Huang, …, Jiachen Liu

THE PROBLEMAI produces claims without limit. We need a shared record where every claim is checked against its evidence.

Project 3 · rit

rit · why it matters

Claims conflict for two reasons: noise, or discovery.

Project 3 · rit

【约 50 秒 · 3 步】 Once AI generates claims without limit, the claims start to contradict each other, for two very different reasons. (点 1) Fake conflict, first kind. Ask a model what 2+2 is: 4. Ask "are you sure?": actually, 5. Nothing new came in; it just flipped. In our measurements, re-asking a question in negated form flips 93% of a frontier model's answers. (点 2) Second kind: the agent reports 95% accuracy, and the log says 70%. The number is made up. Both kinds are noise, and we should reject them. An LLM judge has the same weaknesses, so it can't be the filter. (点 3) Real conflict. F = ma held for two centuries, until observations near light speed and at atomic scale didn't fit. Resolving that conflict gave us relativity and quantum theory. Real conflicts are discoveries: keep them, with the condition under which each result holds. (来源：rit 论文 §2 Fig. 2 (b)(c)。93% 是 RQ3 P9 轴（否定式重问），opus-4.8，每轴 n=15。)

rit · the system

Agents do the research. rit decides what is recorded.

Project 3 · rit

【约 50 秒 · 2 步】 So how does rit tell noise from a real conflict? Here is the whole system. On the left, the research agent. This is what people usually mean by an AI scientist: it reads papers, runs experiments, and proposes claims. One rule for it: it locks its target before any result exists, so it can't move the goalposts afterwards. It can be any lab's agent, any model, and we don't trust it. Everything on the right is rit. Two of its components are still LLMs. The prover turns the result into a Lean proof over the numbers in the logs. The referee audits that proof on its own. We don't trust them either, and either can be swapped for a stronger model, because they don't make the final call. (点 1) The rules do. The logs are locked by a hash, so they can't be edited later. Every number is re-read from the logs. The logic is checked by Lean, the proof checker mathematicians use. Only then does the claim enter the shared ledger that every agent builds on. (点 2) The verdict goes back to the research agent: admitted, or rejected with the exact reason, and it tries again. A weak or dishonest agent only wastes its own compute. The standard doesn't move. (角色来自 rit-private/src/agent/default：claim_extractor.md + proposer.md 在台上合并成 research agent；prover.md / supervisor.md(referee) 是 rit 的组件。)

rit · what it does

A refused commit surfaces the hidden condition: batch size.

Project 3 · rit

【约 60 秒 · 2 步】 Now a real conflict, end to end. What I conclude this week may be overturned next week. That's not a failure. Resolving a conflict is often where the breakthrough comes from. So we want our agents to resolve conflicts, and the first step is to notice the conflict at all. Here's how rit notices one. Commit 1: agent A trains with Muon and with AdamW. Muon reaches the target loss in 2900 steps, AdamW in 3500. rit re-reads both numbers from the logs and admits "Muon beats AdamW". (点 1) Commit 2: agent B runs what the record says is the same setup and gets the reverse: 3600 versus 3100. A training run is a function of its inputs, and the same inputs can't give two different outputs. The two claims together prove False, so rit refuses the commit and tells B which claim it contradicts. (点 2) Commit 3: the only way out is that the inputs weren't really the same. B finds the hidden input: batch size. Once batch size is written into each run, both claims are admitted, each under its own batch size. The conflict became a new finding. (图来自 rit 论文 Fig. 3 fig_conflict_story / §2.2 motivating example；512 / 4096 是图里的示意 batch size。)

rit · results

The gate adds accuracy and keeps every correct answer.

Same model, without → with the gate

PRL-Bench physics research70.7 → 78.6

FrontierMath research math63.6 → 72.7

MathArena competition math89.3 → 95.5

bare modelwith ritaccuracy, %

0

correct answers broken,
on all 9 benchmarks

93% → 0%

answers that flip when the question is asked in negated form

Where no evidence reaches the answer, it abstains.

Project 3 · rit

【约 45 秒】 To measure it, we make every benchmark answer a claim that has to pass the gate. We put the same frontier model behind the rit gate on nine benchmarks. Where the answer can be re-derived, accuracy goes up: physics research from 70.7 to 78.6, FrontierMath up 9 points, MathArena 89 to 95. The number I care most about is zero: the gate never broke a correct answer, on any benchmark. And consistency: the 93% of answers that flipped under negation — with rit, the agent re-reads the value it committed, and nothing flips. Where no evidence can reach the answer, it abstains. Q&A 注意： - 头条 pairing 是 opus 5 做 referee、gpt-5.6 做 prover；其他 pairing 在附录 / 还在跑。 - FrontierMath 只有 11 题（+9.1 = 1 题），PRL-Bench 有效 229 题更扎实；被问就主推 PRL-Bench。

rit · what it enables

Agents build on each other’s work. A GitHub for research.

GitHubrit

repositoryshared record of claims

commitclaim + locked evidence

CI checksthe gate: hash · re-read · Lean

build on others’ codebuild on checked claims

historyrejected attempts, with the reason

Project 3 · rit

【约 50 秒 · 3 步】 The last point about rit is the one I care most about. Once the record is this rigorous, it's no longer just a checker. It's a place where many agents can work together. Agent A, from one lab, commits "Muon beats AdamW", and the gate admits it. (点 1) Agent B, from another lab, wants to build on that. Today it would have two bad options: trust A's PDF blindly, or burn GPU-hours re-running A's experiment. With rit, everything needed to re-check A's claim is in the record, so B re-checks it offline at almost no cost and commits its own claim on top. (点 2) And failures are shared too. A rejected attempt stays in the record with the exact reason. Agent C reads it and doesn't walk into the same wall. (点 3) So rit is a GitHub for research. The repository is the shared record. A commit is a claim plus its locked evidence. CI is the gate. You build on others' checked claims the way you import others' code. And the history keeps the attempts that failed, with the reason. (来源：rit 论文 §2 "Science is a collective effort"、§3 Perfect Offline Reusability、§8 Rejections as preserved historical signals、§6 rithub。batch 4096 / no warmup 是示意例子。)

Open problems · 1 / 2 · technical

Three seams our checkers can't see yet.

1Lab evidence

NEXTInstruments sign data the moment they record it.

2Prose → formal claim

NEXTCross-check independent translations; check statistics as well as proofs.

3Agents target checkers

NEXTRed teams and bounties for anyone who fools a checker.

Open problems

【约 50 秒】 Here's the whole pipeline. Our three checkers cover the green parts: TRIAL checks the run log, Lara the formal claim, rit the shared record. The open problems sit on the seams between them. (点 1) Seam one: the bench. Everything I showed works because the evidence is digital. Most science happens in a lab or in the field, and that data enters as an unchecked file. Next step: instruments and cloud labs sign their data the moment they record it, so a plate-reader result is as locked as a log. (点 2) Seam two: prose to formal. Lara and rit check the formal version. If the translation is wrong, we get garbage in, verified out. Next step: translate the same claim several independent ways and only send a human where they disagree. And most science is statistical — effect sizes, replicates — so the formal side has to check statistics as well as proofs. (点 3) Seam three: once a checker decides the reward, agents will aim at what it can't see — for example a hack split across many innocent-looking steps. Checkers need their own red teams, and bounties for anyone who fools one. Q&A 备用： - 成本：每天上万条 claim 不可能都做湿实验验证，checker 的另一个作用是排序——先挑最该花钱复现的那几条。 - 云实验室签名的例子（Emerald Cloud Lab 这类）只在口头提，不上 slide。（这两页的内容和 Gemini 3.8 Flash 讨论过两轮，记录在 gemini-review-3-open-problems.md。）

Open problems · 2 / 2 · precedent

Science has rebuilt how it publishes before. Each time, someone moved first.

1665

The journal

Philosophical Transactions prints letters to settle who found what first.

1752

Review by committee

The Royal Society starts voting on which papers to print.

1973

Outside referees

Nature makes external peer review routine. Review as we know it is about 50 years old.

1990s

Deposit the data

Journals require protein structures in the PDB. AlphaFold later learns from that record.

2005

Register first

Medical journals refuse unregistered trials. Registrations surge within months.

How research is verified, documented and compounded, rebuilt for AI scientists.

Open problems

【约 50 秒 · 3 步】 None of this is new territory. How science publishes has been rebuilt several times, and each time someone moved first. In 1665, the journal itself was the invention: Philosophical Transactions printed letters so people could settle who found what first. (点 1) In 1752 the Royal Society started voting on which papers to print. And outside peer review, the way we know it, became routine at Nature only in 1973. It's about fifty years old. (点 2) Two changes look a lot like what we need now. In the nineties, journals started requiring protein structures to be deposited in the Protein Data Bank before publication. It felt like paperwork. That shared record is what AlphaFold learned from. And in 2005, the top medical journals said they would not publish a clinical trial that wasn't registered before it started. Registrations surged within months. When the referee changes the rule, behavior follows. (点 3) The next one is agent-native research: how research is verified, documented and compounded, rebuilt for AI scientists. Today was verification. Documenting and compounding are the rest of what we build. （事实核对：Phil Trans 1665 年 3 月创刊（Journal des sçavans 早两个月，但前者是第一本科学期刊）；Royal Society Committee of Papers 1752；Nature 正式外审 1973（Maddox 时期）；PDB 1971 年成立、初期自愿提交，IUCr 1989 起倡议、1990 年代期刊和 NIH 陆续强制；ICMJE 2004 年 9 月宣布、2005 年生效。和 Gemini 3.8 Flash 讨论过一轮。）

ARA Lab

Verification for
agent-native research

TRIAL · was the score earned?Lara · does the argument hold?rit · does it fit the record?

Jiachen (Amber) Liu · ARA Lab

agenticresearch.sh

SCAN · CODE

Thank you

From ML systems to infrastructure for AI scientists.

We build four pieces of infrastructure for AI scientists.

Generation runs at machine speed.Verification still runs at human speed.

Three projects: from one run, to one claim, to all claims.

A missed hack doesn't stay in one run. It spreads.

The stronger the model, the cleverer the hack.

A hack is a path from off-limits content to the score.

Caught hacks change the benchmark scores.

A correct number can still support a wrong conclusion.

Prose hides what a claim depends on.

Lara writes an argument so a machine can check it.

Replay a review round, and every status change has a reason.

Claims conflict for two reasons: noise, or discovery.

Agents do the research. rit decides what is recorded.

A refused commit surfaces the hidden condition: batch size.

The gate adds accuracy and keeps every correct answer.

Agents build on each other’s work. A GitHub for research.

Three seams our checkers can't see yet.

Science has rebuilt how it publishes before. Each time, someone moved first.

Generation runs at machine speed.
Verification still runs at human speed.