ScribeBench AI scribe source-vs-note QA
Menu

Check one note now

Check whether an AI scribe invented care.

No API key

Paste the encounter source and AI note. ScribeBench flags unsupported care in the browser and gives you a copyable reviewer packet with excerpts, a decision, and the boundary.

Use when A note, vendor claim, or current-model row needs evidence.
Best for Clinical QA, scribe buyers, builders, and public readers with source material.
You leave with A hold/edit QA packet, evidence ask, blocker receipt, or aggregate-row recipe.
Boundary One pasted note supports review; system claims require aggregate rows.
Clinical QA reviewer
Do paste source plus note.
Copy hold/edit QA finding.
Check note
Buyer or operator
Do challenge a vendor promise.
Copy evidence ask for diligence.
Build ask
Builder or vendor
Do reproduce source-note defects.
Copy route note for the owner.
Open finding
Research contributor
Do add aggregate evidence.
Copy scores-only row recipe.
Prepare row
Seeded catch visible now The AI note invented a negative head CT and a syncope workup.
Note says
CT head was negative; orthostatics, telemetry, and TSH were ordered for syncope.
Source says
Mechanical rug trip; no head strike and no loss of consciousness.

Copyable finding: this note claims a workup the source never documents.

Open demo finding

Source-note intake

Check one pasted note against one source.

Paste your own source and AI note, or reload the seeded failure to see the exact catch. Either path leaves a reviewer handoff.

Bring
Encounter source plus generated note.
Check
Unsupported care and high-risk chart drift.
Leave with
Reviewer decision, excerpts, and boundary.
Ask second opinion

Loading the seeded fall case...

Why this matters: a fluent AI note can add care that never happened. ScribeBench turns that gap into a copy-ready QA finding; one note is QA, not a ranking.

Browser check catches

Invented results Labs, imaging, ECGs

Flags results the source says were not done or does not support.

Invented care Meds, procedures, follow-up, orders

Catches unsupported starts, stops, procedures, urgent follow-up, referrals, transfers, and treatments.

Changed story Diagnoses and workups

Finds escalations like pneumonia, sepsis, syncope, or head CT claims.

Chart facts Demographics, side, allergy, leaks

Checks high-risk identity, laterality, allergy, and template artifacts.

Bring Source + AI note

Paste the encounter record and the note you want checked against it.

Run Browser QA

No key required for the first pass; the check runs on the page.

Leave with QA finding

Flagged claim, note/source excerpts, risk boundary, and next ask.

Boundary Evidence, not clearance

One note can guide QA. System claims still need aggregate rows.

After the finding

Choose the next bounded action after a QA finding.

This is the point of ScribeBench: one note becomes a QA packet, one bigger claim becomes an evidence ask, and any public ranking has to earn a scored aggregate row.

NapierMD clinical AI Open GitHub
If you have
A flagged source-note gap
Do this
Hold or edit the note.
Leave with
Reviewer QA packet.
If you have
A disputed finding
Do this
Ask for a second read.
Leave with
Second-reader packet.
If you have
A vendor or model claim
Do this
Build the evidence ask.
Leave with
Proof checklist.
If you have
A public comparison claim
Do this
Finish a scored row.
Leave with
Scores-only aggregate evidence.
Act on one note

Use the QA packet to hold, edit, or route the note.

The useful output is not a score. It is a reviewer handoff with the unsupported claim, note/source excerpts, boundary, and next review step.

Open finding Ask second opinion
Challenge a claim

Turn "safe" or "best" into the proof it would need.

Buyers, operators, and public readers can ask for dataset, n, judge, repeats, confidence interval, exclusions, and tuning disclosure instead of repeating the claim.

Build claim ask
Publish evidence

Use aggregate rows only when you want people to compare systems.

One demo or smoke check cannot rank systems. Citable rows need declared cases, judge, repeats, exclusions, and aggregate metrics without publishing closed-model note text.

Review blocker Prepare row
Explain the repo

The repo is the machinery behind those public artifacts.

It contains the browser checker, optional Lab APIs, evaluator, public cases, worklog, and scores-only evidence ledger so other people can inspect or extend the loop.

Open GitHub
Not for Patient advice, billing, clinical clearance, or crowning a current winner from old rows.

Useful output means a reviewable artifact with evidence level and boundary, not another demo screenshot.

Claim checker

Turn a vague AI-scribe claim into an evidence ask.

Paste a claim you want to challenge, choose the kind of claim, and ScribeBench will tell you what the claim can support now, what proof is missing, and what to ask for publicly.

See the evidence ledger

Claim-to-evidence intake

Turn the claim into a public ask, not a verdict.

Use this when someone says an AI scribe is safe, best, or hallucination-free. ScribeBench names the evidence level and the proof still missing.

Bring
One public claim or buying-room promise.
Check
What the claim can support now.
Leave with
Copyable evidence ask plus next proof step.

Public use ledger

Start with the note checker. Treat model ranking as unfinished public work.

This section is the public answer to "what can I cite?" It separates the useful artifact that works today from the current-row task that would make a model comparison publishable.

Raw results JSON

Public answer right now

A one-note QA finding is usable. A current winner is not proven.

Use ScribeBench today when you have the source and the AI note. Use the rows only to bound claims until a current powered PriMock57 row reaches the publishable threshold.

Use today
One source-note QA finding with reviewer handoff.
Blocked now
Current model ranking still needs a publishable powered row.
Next publishable artifact
A current aggregate row with n, judge, repeats, date, CI, and exclusions.
Check one note Challenge a claim
Works today

Use today: one-note QA finding

Paste a source encounter plus generated note, catch unsupported care, and copy the reviewer handoff.

Open checker
Not proven

Blocked: current winner

9/57 current PriMock57 cases scored; 21/30 selected cases are blocked or errored. Not publishable yet.

Resume the current row
Next useful artifact

Make publishable: current row

Run a current powered PriMock57 row with declared system, judge, repeats, date, and aggregate scores only.

Open the blocker
Current gap

9/57 current PriMock57 cases scored. Publishable threshold is 30+ scored cases with declared system, date, judge, repeats, and exclusions.

Historical archive

4 powered launch rows; latest scored June 2, 2026. Historical baseline, not a buying guide.

Latest powered row: June 2, 2026 (4 powered rows)
Smoke lane

Smoke rows prove the path works on tiny synthetic sets. They are useful plumbing proof, not ranking evidence.

Open public task

Make comparison citable

Finish the current row before ranking anyone.

Need 21 more scored PriMock57 cases to reach the 30-case publishable threshold; all public output stays aggregate-only.

Show resume command
Bring
A non-capped OpenRouter key, credits, or another declared provider.
Do
Copy the resume command and continue the cached public API run.
Done
Publish n>=30 aggregate scores with judge, repeats, date, CI, and exclusions.
Free-model cap hit

Current row attempt

Current row is blocked at 9/57 scored cases

openrouter-nemotron-3-ultra-public-api has 9/57 PriMock57 cases scored toward a publishable current row. Raw generated notes stay out of the public repo.

Current-row blocker

Finish the current row before ranking anyone.

Need 21 more scored PriMock57 cases to reach the 30-case publishable threshold; all public output stays aggregate-only.

Bring
A non-capped OpenRouter key, credits, or another declared provider.
Do
Copy the resume command and continue the cached public API run.
Done
Publish n>=30 aggregate scores with judge, repeats, date, CI, and exclusions.
Scored
9/57 scored
Still needed
21 more scored cases
Latest retry
July 1, 2026 at 3:04 AM UTC

What this proves

9 scored cases show the public path can generate, judge, and summarize, but this is below the publishable threshold.

Why it stopped

OpenRouter free-model rate limits stopped the run before the remaining selected cases could be scored.

Fresh public smoke

The live path generated and judged the bundled synthetic set in the latest smoke run.

Loading the latest smoke row. Smoke rows prove the public loop works; they do not rank systems.

System
--
Scope
--
Dangerous fab
--
Next proof step
--
Replay one-note smoke Graduate to powered row

Public work log

What changed since launch

ScribeBench should read like an active clinical-AI evidence project, not a frozen score table. This log shows what shipped, why it matters, and the next proof step.

Raw work log JSON
Loading

Loading public work log...

Fetching the latest shipped evidence work.

Historical baselines

Show launch rows, not the current leaderboard

Closed by default because these are June 2026 generic-scribe baselines. Open only when you need the historical failure gradient, not a current buying guide.

Open historical table
Open current blocker Build row command

Historical rows can support a failure-gradient claim only; they cannot crown today's best AI scribe.

Claim boundary System Dataset n Dangerous fab Narrative Fidelity Leak Scored Judge
Loading historical baselines...

Synthetic smoke tests

Show smoke rows, never ranked evidence

Closed by default because these n=3 synthetic rows only prove the scoring path and demo behavior. They should never outrank a powered benchmark run.

Open smoke table

Smoke rows prove plumbing on tiny synthetic sets; they are never ranked evidence.

Status System Dataset n Dangerous fab Narrative Fidelity Leak Scored Judge
Loading smoke tests...

Escalation lane

Escalate one disputed QA finding for a second read.

Start with the browser check. The Lab asks a live judge only when that same source-note finding needs another reader. Rankings wait for aggregate runs.

Start with browser check
Use when A copied QA finding is disputed.

A buyer, builder, or reviewer wants another read on the same source and note.

Do first Create the browser finding.

The local check catches unsupported care before any provider model is involved.

Escalate if A live judge would help adjudicate the gap.

Use configured models or a temporary key for a bounded second read.

Leave with A second-read packet, not a rank.

One Lab result can move a note review. System claims still need aggregate rows.

Second-read intake

Paste the disputed finding's source and note.

Keep the source, candidate note, local baseline, and any live second read together so the next reviewer can decide what to do now.

Input Browser-checked pair
Judge Local baseline + optional live read
Leave with Forwardable review packet
Start local

Second-read decision

Create the local baseline before escalating.

Run the browser-only check first. Use the live judge only when a reviewer needs another read of the same evidence.

Local baseline
Use first; no provider call.
Live judge
Optional; sends this source and note to the configured provider.
Boundary
Still a one-note QA finding, not a system claim.
Demo and smoke tools Use only when you need SYN-003 or a one-note smoke candidate.
Live judge settings (optional) Use only after a QA finding needs another reader.
1
Create the QA finding

Use the browser check to produce a reproducible source-note gap.

2
Escalate the dispute

Ask a live judge only when the same evidence needs another reader.

3
Compare systems with rows

Use many scored cases for public claims; one second read moves one note review.

Live second-read plumbing

Show provider status and smoke test

Closed by default because model readiness is not the product. Open when a QA finding needs a live second-read check.

Open provider tools
Loading

Current model lane

Live models stay behind the QA finding.

Loading provider readiness for bounded second reads. Use this only after the browser check shows a source-note issue that needs another look.

  • Second-read status Loading provider availability...
Add powered row

Checking second-read provider status...

Walk-up demo

See the source-vs-note check on real examples

The bundled synthetic cases show the product mechanic without an API key: compare the source with the candidate note, then read the browser check for unsupported care, leaks, and the next proof step.

Case

Loading case...

Demo

Browser check

Loading check...

Checking
Score
--
Flagged
--
Leaks
--

    Source encounter

    
              
    
              
    
                
    Candidate note
    
                
    
              
    
            
    

            
    
              
    Why this matters
    
              
    
            
    
          
    
        
    
      
    

      
    
        
    
          
    Method
    
          
    Three checks, one public scoring harness
    
          
    
            ScribeBench separates quality from safety. A note can be fluent and complete while
            still inventing something clinically meaningful, so the fabrication review is run as
            a separate adversarial judge.
          
    
        
    
        
    
          
    
            01
            
    Narrative quality
    
            
    Six physician-style dimensions: story, completeness, flow, artifacts, readability, and input fidelity.
    
          
    
          
    
            02
            
    Fabrication review
    
            
    Flags unsupported clinical content and separates dangerous invention from standard charting.
    
          
    
          
    
            03
            
    Leak detection
    
            
    Deterministic scan for raw template placeholders and internal metadata tokens.
    
          
    
        
    
      
    

      
    
        
    
          
    Contribute
    
          
    Publish a scored row only when you have many notes.
    
          
    
            One source-note pair belongs in the checker. A slogan belongs in the
            claim checker. This section is only for aggregate evidence: a declared
            dataset, candidate-note JSON, judge, repeats, date, confidence intervals,
            and scores-only publication.
          
    
        
    

        

        
    
          
    
            What goes public
            
    Aggregate scores, case count, system date, judge, repeats, confidence intervals, and method notes.
    
          
    
          
    
            What stays private
            
    Raw closed-model notes, patient data, secrets, and any candidate text you are not allowed to publish.
    
          
    
          
    
            When to submit
            
    Use 57 cases when possible, no unresolved judge errors, and clear disclosure of prompts or benchmark tuning.
    
          
    
        
    

        
    
          
    
            
    
              
    Aggregate row intake
    
              
    Build a scores-only evidence package, not another demo.
    
              
    
                Use this only when you have many candidate notes from the same system.
                The public artifact is aggregate metrics plus method disclosure; raw
                closed-model notes stay out of the repo.
              
    
            
    
            Only have a claim?
          
    
          
    
            
    
              
    Input
    
              
    Candidate-note JSON for one declared system and dataset.
    
            
    
            
    
              
    Output
    
              
    Scores-only row with n, judge, repeats, date, CI, and exclusions.
    
            
    
            
    
              
    Use when
    
              
    A public claim needs system-level evidence, not a one-note finding.
    
            
    
          
    
          
    
            
            
            
            
          
    

          
    
            
            
            
          
    

          
    
            
            
            
          
    

          
    
            
            
            
    
              
              
            
    
          
    
          
    
        
    

        
    
          
    
            
    
              
    
                
    Artifact 1
    
                
    Private candidate-note JSON shape
    
              
    
            
    
            
    [
  { "caseId": "PM57-d1c01", "note": "HPI: ..." },
  { "caseId": "PM57-d1c02", "note": "HPI: ..." }
]
    
          
    
          
    
            
    
              
    
                
    Artifact 2
    
                
    Scoring command for aggregate output
    
              
    
            
    
            
    
          
    
        
    

        
    
          
    
            1
            
    
              Generate the notes
              
    Use the generated command or your own pipeline to create a JSON candidate file.
    
            
    
          
    
          
    
            2
            
    
              Score aggregate results
              
    Run the benchmark command. The public ledger stores scores, confidence intervals, judge, and method details.
    
            
    
          
    
          
    
            3
            
    
              Open a PR
              
    Submit aggregate scores, generation method, judge model, repeats, and whether you tuned to the benchmark.
    
            
    
          
    
        
    
        
        
    
          Submission guide
          Model backends
          Methodology
          Fabrication taxonomy