NotesXML AI Model Benchmark Results & Recommendations

IWV Digital Solutions LLC | 10-model catalog v2026.07.26.01

Overview

NotesXML's Professional tier ships with 10 local AI models. They span every hardware tier, from phones and low-RAM devices to enthusiast workstations with discrete GPUs. This article gives the full benchmark results for all 10 catalog models, the Harmonic Mean Score (HMS) ranking, and the efficiency-frontier analysis.

All AI inference in NotesXML runs entirely on your device. No internet connection is required. No data is transmitted.

Benchmark Deep-Dives

Each benchmark in the suite has its own article. The article covers how the benchmark works, the full question or task list, how answers are scored, and the research behind its design. This page gives the scores; the deep-dive articles explain what the scores mean.

NotesXML Benchmark Suite

NotesXML uses eight purpose-designed benchmarks to evaluate models as they actually run on local hardware — testing the combination of model capability and local inference performance that users experience in practice.

Long Context Benchmark (100 points)

Evaluates a model's ability to process, retain, and reason over extended input sequences. Passages with embedded facts at varying context depths are presented; the model is queried on details requiring genuine retention rather than positional pattern-matching. Critical for note-taking applications where users ask AI to summarize or analyze multi-page documents.

Tool Calling Benchmark (100 points)

Evaluates a model's ability to correctly invoke structured function calls. Tests cover simple single-tool invocations, multi-step chains, parallel calls, and error recovery — all capabilities underpinning NotesXML's AI action system.

Model Benchmark — In-App Task Suite (Grade A–F + Harmonic)

Runs the seven tasks NotesXML performs in practice and grades each A–F: six text tasks — summarization, auto-titling, grammar and spelling correction, English→Spanish translation, action-item extraction, and Markdown/XML generation — plus a vision OCR task for vision-capable models. The per-task grades are combined into a normalized quality score (exported as the AvgScore field) and an overall letter grade; the Harmonic score pairs that quality with inference speed, rewarding models that are both correct and fast. Chat-template handling, stop-token behavior, and output-format integrity are scored within each task.

HEE — Human Education Evaluation (50 points)

Evaluates broad factual knowledge across mathematics, science, history, literature, geography, and reasoning — drawn from secondary school through postgraduate level, weighted toward the upper end. The closest NotesXML benchmark to the academic MMLU family.

PLE — Professional License Exam (200 points)

200 questions across ten professional fields: Law (MBE), Medicine (USMLE), Engineering (PE), Psychology (EPPP), Nursing (NCLEX-RN), Real Estate, Finance (CPA), Architecture (ARE), and Cybersecurity (CISSP). Tests not only factual recall but multi-step logical deduction and contextual synthesis under professional-exam constraints.

PhD Philosophy — PhD-Level Logic & Philosophy Comprehensive Exam (100 points)

100 questions spanning metalogic, modal logic, philosophy of language and mind, epistemology, metaphysics, and continental phenomenology. Designed so that correctly answering requires understanding how theories mechanically function, not merely pattern-matching to familiar names. The most intellectually demanding single benchmark in the suite.

SDB-100 — Synthetic Deduction Benchmark (100 points)

10 questions testing multi-step deductive reasoning using synthetic ontologies that cannot exist in any training corpus — novel self-contained universes governed by arbitrary but logically absolute rules. By stripping away semantic priors, the benchmark forces genuine structural reasoning rather than pattern completion. The most discriminating and contamination-resistant benchmark in the suite.

Full Results

NotesXML Benchmark Results

Full 8-benchmark sweep for all 10 catalog models, AI Benchmark Orchestrator v2.3 on Ryzen 7 7800X3D (Linux). The first eight models were run on Beta 5.141.38 (30 Jun 2026); Gemma 4 E2B and Ministral 3 14B were added and run on Beta 5.142.19 (9 Jul 2026) on the same rig. Rows are sorted by HMS (the speed/quality score). ★ marks the 8 models on the Pareto efficiency frontier; the two unmarked models (Gemma 4 E2B, Ministral 3 14B) are off the frontier on this hardware and are explained below.

Avg TPS = tokens per second (text tasks only). Grade = Model Benchmark overall grade (all 10 models Grade A). Avg Score = qualityMean, the unweighted mean of all eight normalized benchmark scores (Model Benchmark, HEE, PLE, PhD, SDB-100, Code, LongCtx, Tool); Code is now included. HMS = harmonic mean of min(TPS/50,1) and Avg Score.

Performance vs Token Interval — The Efficiency Frontier

The plot shows each model’s quality against its token interval (1/TPS, milliseconds per generated token — lower is faster). The dashed line is the Pareto efficiency frontier: the models that no other model beats on both speed and quality at once. 8 of the 10 catalog models sit on the frontier. The two that do not — Gemma 4 E2B and Ministral 3 14B (shown as hollow markers) — are each matched by a model that is at least as fast and at least as good on this rig. They stay in the catalog for reasons the reference rig cannot show: see the note under the chart.

Sorted by TPS descending (fastest first). 8 of the 10 models are on the strict Pareto efficiency frontier. Gemma 4 E2B is beaten by Ministral 3 3B (faster and higher quality), and Ministral 3 14B is beaten by Ministral 3 8B (same speed, higher quality) — both on this Ryzen rig. Quality is the 8-benchmark qualityMean (Code included).

Harmonic Mean Score — Single-Number Speed/Quality Ranking

The Harmonic Mean Score (HMS) folds speed and quality into one number. Higher is better. The harmonic mean punishes imbalance: a model that is fast but weak, or strong but slow, scores lower than one that is good at both. Speed is capped at 50 TPS. Past that point a model already streams faster than most people read in a chat window, so extra speed adds nothing to the experience.

Methodology

1. Speed component: speed_norm = min(TPS / 50, 1.0)
2. Quality component: quality = qualityMean — the unweighted mean of all eight normalized benchmark scores. Code Benchmark is now included.
3. Harmonic mean: HMS = 2 × speed_norm × quality / (speed_norm + quality)

HMS Ranking (all 10 catalog models)

Reading the HMS Ranking

Ministral 3 8B and Gemma 4 12B lead at HMS 0.95. Both clear the 50-TPS speed cap (71.6 and 66.8 TPS) and score 90% average quality, so they pair top speed with near-top quality. Ministral 3 8B is listed first as the faster of the two. Ministral 3 14B follows at 0.93, then Gemma 4 E4B at 0.91, then Ministral 3 3B and Phi-4 Mini tie at 0.89.

With the cap at 50 TPS, every model except the 26B reaches speed_norm = 1.0. For those models, HMS is driven almost entirely by quality. This is by design: once a model streams faster than you can read, quality is what separates them. The fastest model, Llama 3.2 1B (544.9 TPS), ranks last at 0.77 because its 63% quality is the lowest in the catalog.

The quality leader sits mid-pack on HMS. Gemma 4 26B-A4B (MoE) has the catalog’s highest quality (95%) but ranks eighth on HMS. Its 39.9 TPS is the only speed below the cap (speed_norm 0.80). For batch work, research synthesis, or high-stakes writing where speed matters less, read the quality column directly — the 26B is the top choice there.

Key Observations

8 of the 10 catalog models are on the Pareto frontier. On the Ryzen reference rig, two models are dominated. Gemma 4 E2B loses to Ministral 3 3B, which is both faster and higher quality. Ministral 3 14B loses to Ministral 3 8B, which runs at the same speed with higher quality. Both models stay in the catalog on purpose. Ministral 3 14B is there for Intel iGPU and GPU systems: it is a large, vision-capable, desktop-only model that runs on the Intel GPU, whereas the Gemma 4 models run CPU-only on Intel hardware. The reference rig is a Ryzen machine, so its numbers cannot show that gain. Gemma 4 E2B is a small, fast, vision-capable model for light hardware. The other eight models each win on some mix of speed, quality, memory footprint, or features.

Mid-size Enhanced models lead the HMS ranking. Ministral 3 8B and Gemma 4 12B (HMS 0.95) clear the 50-TPS cap and deliver 90% quality — the best speed/quality balance in the catalog. Among models fast enough to stream comfortably, the ranking rewards quality.

Gemma 4 26B-A4B (MoE) leads on quality. It takes the top spot on every knowledge and reasoning benchmark: 95% average quality, 100% HEE, 99% PLE, 100% PhD Philosophy, 80% SDB-100 (catalog leader), and 99% Code. With vision and a 256K-token context, it is the highest-quality model in the catalog. Desktop Pro only.

Vision now spans the catalog. Seven models are vision-capable: Ministral 3 3B, Gemma 4 E2B, Gemma 4 E4B, Ministral 3 8B, and Gemma 4 12B run cross-platform; Ministral 3 14B and Gemma 4 26B-A4B are desktop-only. Ministral 3 3B brings vision into the Free tier for the first time. Long Context retention is led by Ministral 3 8B (88%); Tool Calling by Gemma 4 E4B (94%, cross-platform).

Llama 3.2 1B’s knowledge scores reflect its size. At 1.2B parameters it is the fastest model in the catalog (544.9 TPS) and the Android Free-tier default. It is built for instant titling, summaries, and grammar polish on tight memory. For knowledge-heavy work, any 3B-or-larger model is clearly better.

Model Recommendations by Hardware Tier

Ultra-Light — Up to ~1.5 GB RAM Available for AI

Free-tier default: Llama 3.2 1B Instruct (HMS 0.77) — the Android Free-tier default and fastest model in the catalog at 544.9 TPS. 128K native context, text-only. Best for instant note titling, short summaries, transcription polish, and basic text cleanup under severe memory constraints. Llama 3.2 Community License.

Lightweight — ~1.5–4.5 GB RAM Available for AI

HMS co-leader (Lightweight): Ministral 3 3B (HMS 0.89) — Apache 2.0, 220.1 TPS, vision-capable, and the Electron default. The first vision model in the Free tier. 81% average quality (HEE 84%, PLE 81%, PhD 81%, Code 88%, Long Context 86%, Tool 84%), 256K native context. The best all-round default for note-taking with image support on Lightweight hardware.

MIT-licensed text default: Phi-4 Mini 3.8B (HMS 0.89) — MIT license, 220.0 TPS, text-only. PhD 87%, PLE 81%, Code 86%, Long Context 84%, Tool 82%. 128K native context. A strong MIT-licensed text default that ties Ministral 3 3B on HMS.

Llama family: Llama 3.2 3B Instruct (HMS 0.84) — 253.0 TPS, HEE 80%, PLE 71%, PhD 72%, Tool 64%. Llama 3.2 Community License, text-only, 128K native context.

Standard-tier vision option: Gemma 4 E2B (HMS 0.88) — Apache 2.0, 198.3 TPS, vision-capable, ~2.3B effective, 128K native context. A small, fast vision model for light hardware: 79% average quality (HEE 94%, PLE 87%, PhD 92%, Code 91%, Long Context 80%). On the reference rig Ministral 3 3B is both faster and higher quality, so E2B sits just off the frontier — but it stays a compact, vision-capable choice where its smaller footprint fits better.

Enhanced — ~5–9 GB RAM Available for AI

HMS leader: Ministral 3 8B (HMS 0.95) — Apache 2.0, 71.6 TPS, vision-capable. The catalog’s Long Context leader (88%) with 90% average quality (HEE 96%, PLE 92%, PhD 94%, Code 95%). 256K native context. The top speed/quality balance for AI Chat, structured actions, and image analysis on Enhanced hardware.

Tool-calling flagship: Gemma 4 E4B (HMS 0.91) — Apache 2.0, 119.0 TPS, Tool 94% (catalog co-leader), HEE 96%, PLE 92%, PhD 96%. Vision-capable, 128K native context. The strongest cross-platform choice for tool-driven AI actions.

Highest-quality cross-platform model: Gemma 4 12B (HMS 0.95) — Apache 2.0, 66.8 TPS, 90% average quality (tied for the catalog’s second-highest, behind only the desktop-only 26B MoE), Code 97%, PhD 99%, Tool 93%. Vision-capable, 12B dense, 256K native context. The top-quality model that still runs cross-platform — and it ties for the overall HMS lead.

Intel iGPU / GPU pick: Ministral 3 14B (HMS 0.93, Advanced tier, desktop only) — Apache 2.0, vision-capable, 14B dense, 256K native context, ~11 GB peak RAM. This is the large-model option for Intel systems: it runs on the Intel iGPU or GPU, where the Gemma 4 models fall back to CPU-only. On the Ryzen reference rig it measured 71.6 TPS at 87% average quality (HEE 98%, PLE 95%, PhD 97%, Code 92%, Long Context 87%), so on that machine Ministral 3 8B just edges it on the speed/quality frontier. On an Intel GPU, though, the 14B is how you run a large, high-quality vision model at usable speed. Desktop only (Windows and Linux).

Desktop Pro — ~22 GB RAM or 16+ GB GPU VRAM

Highest overall quality: Gemma 4 26B-A4B (MoE) (HMS 0.87) — Apache 2.0, desktop-only, 39.9 TPS. The catalog’s top quality model: 95% average, HEE 100%, PLE 99%, PhD 100%, SDB-100 80% (catalog leader), Code 99%, Tool 94%. 26B total / 4B active Mixture-of-Experts, vision-capable, 256K native context. The highest-quality choice for research, synthesis, and agentic workflows where speed is secondary.

Summary Recommendation Table

Complete Benchmark Data

Full benchmark results for every model evaluated in the candidate sweep — including models not selected for the catalog — are available in a dedicated reference article.

Notes on Hardware Acceleration

The benchmarks above were collected on a Ryzen 7 7800X3D system running Linux (AI Benchmark Orchestrator v2.3, Beta 5.141.38). NotesXML supports GPU acceleration on NVIDIA CUDA/Vulkan, AMD Vulkan (RADV), and Intel iGPU (Vulkan / SYCL). CPU-only inference remains fully supported on every platform.

• NVIDIA CUDA / Vulkan: Full support. RTX 4060 class and above can run all 10 catalog models. The single Desktop Pro model (Gemma 4 26B-A4B) is best on 16+ GB VRAM at Q4_K_M.
• AMD Vulkan (RADV): Full support. APU systems (Ryzen 7000 with Radeon 780M) successfully run catalog models through the Lightweight and Enhanced tiers.
• Intel iGPU: Vulkan is supported on Linux and Windows; SYCL is supported on platforms where the Intel oneAPI runtime is available. CPU-only inference is also fully supported. On Intel GPUs the Mistral-family models (including Ministral 3 14B) run on the GPU, while the Gemma 4 models fall back to CPU-only — which is why the 14B is the recommended large model for Intel iGPU/GPU systems.

About NotesXML AI

All AI runs locally on your device using llama.cpp. No subscription is needed beyond the one-time Professional license. Models download once and are stored on your machine. No internet is needed when the AI runs. Your notes, questions, and AI responses stay private.

© 2026 IWV Digital Solutions LLC. All rights reserved.