AI benchmarks
We test each model inside NotesXML on three kinds of work, weighted equally: document processing, reading images (OCR), and chat knowledge and reasoning. Speed counts too. We tested 52 models; these 10 make up the NotesXML catalog.
Scores
| Model | Documents | OCR | Chat | Tokens/s | Overall |
|---|---|---|---|---|---|
| LFM3 VL 2B (free) | 0.86 | 1.00 | 0.84 | 329 | 0.89 |
| Gemma 4 E2B | 0.89 | 1.00 | 0.82 | 226 | 0.90 |
| Ministral 3 3B (free) | 0.95 | 1.00 | 0.75 | 194 | 0.88 |
| Gemma 4 E4B | 0.92 | 1.00 | 0.89 | 171 | 0.93 |
| Gemma 4 12B | 0.93 | 1.00 | 0.87 | 106 | 0.89 |
| GPT-OSS 20B | 0.95 | — | 0.81 | 148 | 0.90 |
| Devstral Small 2 24B | 0.94 | 1.00 | 0.86 | 47 | 0.78 |
| Gemma 4 26B-A4B | 0.95 | 1.00 | 0.93 | 122 | 0.93 |
| Gemma 4 31B | 0.97 | 1.00 | 0.92 | 44 | 0.79 |
| Laguna XS 2.1 33B-A3B | 0.89 | — | 0.86 | 133 | 0.89 |
Scores run from 0 to 1. Pass marks: documents 0.85, OCR 0.90, chat 0.70. — = text-only model. Tokens/s on the desktop below. Overall is the harmonic mean of documents, chat and a speed rating, 1 − e−tokens/s ÷ 50 (50 tokens/s rates 0.63, 150 rates 0.95), so a model must do well on every count. OCR is left out of Overall because every vision model scored 1.00.
Speed on other hardware
| Model | Desktop, CPU only | Laptop, Radeon 780M | Laptop, Intel 155H | Galaxy S24 Ultra |
|---|---|---|---|---|
| LFM3 VL 2B | — | 61 | 12 | 14 |
| Gemma 4 E2B | 32 | 75 | 10 | 8.7 |
| Ministral 3 3B | 15 | 30 | 8.5 | 5.3 |
| Gemma 4 E4B | 21 | 50 | 6.1 | 4.5 |
| Gemma 4 12B | 8.7 | 18 | 2.0 | desktop only |
| GPT-OSS 20B | — | 21 | 3.9 | desktop only |
| Devstral Small 2 24B | 2.6 | 5.0 | 2.4 | desktop only |
| Gemma 4 26B-A4B | 14 | 8.9 | 3.8 | desktop only |
| Gemma 4 31B | 4.0 | 3.3 | 0.9 | desktop only |
| Laguna XS 2.1 33B-A3B | — | 9.0 | 3.5 | desktop only |
Tokens per second. — = not measured.
Which model?
- Phone: LFM3 VL 2B for speed; Gemma 4 E4B for the highest score.
- Laptop without a graphics card: Gemma 4 E2B, or Gemma 4 E4B if it is fast enough for you.
- Desktop with 24 GB or more of graphics memory: Gemma 4 26B-A4B.
- Best document work, speed no object: Gemma 4 31B.
How we test
- Documents: the everyday note actions — summarize, title, polish, translate, pull out action items, format — plus every built-in AI prompt across the note types, and finding facts in notes of 4,000 to 32,000 tokens.
- OCR: a photo of mixed print and handwriting, a PDF with text, a handwritten PDF page and a screenshot, each read through the note action and through AI Chat.
- Chat: 100 knowledge questions from school to professional-exam level, answered with a reason; 100 logic puzzles built from templates so they cannot be memorized; saying “I don’t know” to made-up or false-premise questions; three eight-turn conversations about an attached memo; and using AI Chat’s tools correctly.
Each test scores 0 to 1, and each of the three scores is the average of its tests. The tests send the same prompts NotesXML itself sends.
Test machines
Desktop: AMD Ryzen 7 7800X3D, 32 GB RAM, NVIDIA RTX 5070 Ti 16 GB + RTX 5060 Ti 16 GB. Laptops: AMD Ryzen 9 7940HS with Radeon 780M graphics; Intel Core Ultra 7 155H with its built-in graphics. Phone: Samsung Galaxy S24 Ultra. Results as of October 2026.