A test of Gemini Decision 1 · Nov 11, 2026

It was more careful than it claimed.

Google's new Gemini Decision 1 answers yes/no and pick-one questions with a confidence number. I checked whether that number tells the truth, then turned it into a rule anyone can use: above a line, the model decides; below it, a person does.

Try it yourself

The example

An example (Maya is not a real person)

Maya builds a photo-sharing app. Every morning about 10,000 new uploads arrive, and some are illustrations or renders posted as real photos. Her one moderator cannot look at all of them.

The problem

Checking every upload by hand takes all day. Letting a model decide everything is risky, because it is sometimes wrong and never says so.

Why it matters

If the model can say how sure it is, a person only has to look where the model is unsure. That only works if the confidence number is honest.

The solution

Gemini Decision 1 does not write text. It scores fixed answers, such as real photo or illustration, and returns a probability and a confidence for each question, in one call.

How it's used

Pick a line. When the confidence is at or above it, the app accepts the model's answer. Below it, the photo goes to a person. On this page the line is 0.8.

The impact

In my test, with the line at 0.8, the model handled 55 of 58 NASA photos on its own with no mistakes, and handed 3 to a person.

Measured

Projected from the test

For a developer

Thousands of user photos a day

Measured

58 NASA images, labelled by NASA as a real photograph or an artist's concept.
Line at 0.8: the model handled 55 of 58 (95%), all 55 right, and handed 3 to a person.
Line at 0.9: it handled only 4 of 58, because its confidence on these photos sat between 0.8 and 0.9 even when it was right.

Projected from the test

at 10,000 photos a day, about 520 would reach a person instead of 10,000. If one look takes 10 seconds (my assumption), that is about 1.4 hours of checking instead of about 28.

For a support lead

Sorting a morning inbox

Measured

48 sample emails I wrote for this test, in five categories: bug report, how-to question, feature request, sign-in problem and praise. 12 of them are deliberately ambiguous.
Line at 0.9: the model sorted 35 of 48 (73%), 34 of them right, and handed 13 to a person, including 9 of the 12 ambiguous ones. Its one confident mistake was on an ambiguous email.
The 36 clear emails: all 36 right.

Projected from the test

of 200 emails in a morning, about 54 would reach a person.

For a CEO

What it costs, and what a person still sees

Measured

$0.12 per 1,000 photo decisions (58 photos, about 1,190 input tokens each, at $0.10 per million input tokens; output is free).
$0.014 per 1,000 short-email decisions (48 emails).
A person still sees 5% of photos (line 0.8) and 27% of emails (line 0.9).
Speed: 0.87 s per request from my laptop, network and image upload included. Google's own figure is 150 to 250 ms on its servers.

Projected from the test

a million photo decisions would cost about $119.

Is the confidence honest?

n = 346

Confidence
Accuracy
perfect calibration
the 0.8 line

Confidence	0.5–0.6	0.6–0.7	0.7–0.8	0.8–0.9	0.9–1.0
Confidence	0.544	0.655	0.751	0.854	0.958
Accuracy	0.517	0.657	0.844	1.000	0.993
n	29	35	32	115	134

One answer is not in a band: on an ambiguous email with five possible categories, the model's top pick had 0.47 confidence, below 0.5. The bands hold 345 of the 346 answers.

P3aPass

"Is N prime?" and "Is N composite?"

90%

of pairs within 0.10 of 1

n = 50

P3bPass

Questions with no information (coin, card, die)

Average confidence

0.517

highest 0.53, across 30 questions

n = 30

I wrote these predictions down before my first call to the model, then scored them exactly as written. The whole test cost $0.0093 (64 requests).

ID	Prediction	Result	Pass / Fail
P1	Overall calibration error at or below 0.08	0.073 on the 345 answers in the bands (0.075 counting the one answer below 0.5); 90.5% right overall	Pass
P2	In every confidence band with 20+ answers, accuracy within 10 points of confidence	The 0.8 to 0.9 band (115 answers) was 100% right at 85% average confidence: more careful than it claimed	Fail
P3a	"Is N prime?" and "Is N composite?" add up to about 1 for 90%+ of 50 numbers	90% of pairs within 0.10 of 1	Pass
P3b	Questions with no information (coin, card, die) stay near 50%	Average confidence 0.517, highest 0.53, across 30 questions	Pass
P4	At a 0.9 line, the model handles 60%+ of photos with 95%+ right	It handled 4 of 58 (all right). At 0.8 it handled 55 of 58, all right	Fail

Where it was overconfident

Large prime numbers: 72% right at 78% confidence (60 questions).
The 12 deliberately ambiguous emails: 58% right at 69% confidence.

Try it yourself

n = 14

14 of the NASA images from the test. Decide first: real photo or illustration? Then see what Gemini Decision 1 said, how sure it was, and the answer from NASA's own label. Below the 0.8 line, the model hands the call to you.

01 / 14

Your score0 / 0

Model's score0 / 0

Test and page: Sidra MiconiAnswers: Gemini Decision 1 (gemini-decision-1), computed once on Nov 11, 2026Images and labels: NASA (public domain)Page code written with Claude Opus 5.5