Evaluating a quantum machine learning model requires more than reporting its best accuracy. This guide presents a repeatable workflow for comparing a QML model with appropriate classical and random baselines, measuring task performance and resource costs, accounting for shot noise and parameter variance, and recording enough experiment detail to reproduce the result later.
Overview
The central evaluation question is not simply, “Did the quantum model achieve a high score?” It is, “Did the quantum component provide a useful improvement for this task under the same experimental conditions?” A defensible answer requires controlled comparisons.
Start by defining the task, the data split, the success metric, and the practical constraints. A classifier may be judged by balanced accuracy, precision, recall, F1 score, area under a suitable curve, or calibration quality. A regressor may use mean absolute error, root mean squared error, or another error measure that matches the application. For an optimization experiment, the objective value, constraint violations, feasibility rate, and time to reach a target may matter more than a generic machine learning score.
Keep a separate record of predictive quality and implementation cost. A model that improves a score slightly but requires substantially more circuit evaluations, tuning, or execution time may not be the better choice. This distinction is especially important in hybrid quantum-classical workflows, where a classical optimizer repeatedly calls a parameterized quantum circuit.
For background on comparing classical and quantum approaches, see Classical vs Quantum Machine Learning: When the Quantum Part Helps. Before running a large experiment, estimate qubits, circuit depth, shots, and runtime using the quantum circuit resource estimation worksheet.
What to track
1. Data and split integrity
Use a fixed train-validation-test design whenever possible. Fit preprocessing steps such as scaling, feature selection, and dimensionality reduction on the training data only. Apply the resulting transformation to validation and test data without refitting. Record the split method, sample counts, class balance, feature representation, and random seed.
Small QML datasets can make results sensitive to the split itself. If the dataset is limited, repeat the evaluation across several predefined seeds or use cross-validation where it is appropriate. Do not compare a quantum model trained on one favorable split with a classical model evaluated on another.
2. Task-specific performance metrics
Choose a primary metric before looking at results, then add secondary metrics that expose failure modes. For imbalanced classification, accuracy alone can conceal poor minority-class performance. For regression, report an error metric alongside a simple reference such as predicting the training-set mean. For optimization, record both the final objective and whether the solution satisfies the problem constraints.
| Task | Primary measurements | Useful companion measurements |
|---|---|---|
| Classification | F1, balanced accuracy, or a selected task metric | Confusion matrix, per-class recall, calibration, variance across seeds |
| Regression | MAE, RMSE, or another predefined error metric | Baseline-relative error, residual checks, variance across seeds |
| Optimization | Objective value and feasibility | Time to target, constraint violations, evaluation count |
3. Baselines that answer different questions
A random or majority-class baseline checks whether the model has learned more than a trivial strategy. A simple classical baseline establishes a minimum useful standard. Depending on the dataset, this could be logistic regression, a linear regressor, a small tree-based model, or another transparent method. A stronger classical baseline helps test whether any improvement survives a more competitive comparison.
Keep preprocessing, training data, evaluation splits, and tuning budgets as comparable as practical. A classical baseline should not be deliberately under-tuned while the QML model receives extensive optimization. If the quantum model uses a classical feature map or post-processing layer, describe the full hybrid pipeline rather than attributing the complete result to the circuit.
4. Variance and shot noise
Run each important configuration with more than one random seed. Track the mean, spread, and best result rather than reporting only the best run. The sources of variation may include parameter initialization, minibatch order, optimizer behavior, circuit sampling, and hardware noise.
When measurements use finite shots, record the shot count and whether the reported score comes from a single execution or repeated executions. Increasing shots can make an estimate more stable, but it also changes the resource cost. Treat shot count as an experimental variable, not as an invisible implementation detail.
5. Resource and complexity measures
Record the number of qubits, circuit depth, gate counts where available, number of circuit evaluations, shots per evaluation, optimizer iterations, wall-clock time, and whether the run used a simulator or hardware. Also note compilation or transpilation settings when they materially affect the executed circuit.
These measurements help separate an algorithmic result from an execution result. A model may have similar predictive quality across two configurations, while one uses a shallower circuit or fewer evaluations. That information can determine which implementation is more maintainable.
6. Reproducible experiment metadata
Use a structured log for every run. A practical minimum includes:
- Run ID, date, code version, and environment or dependency lockfile
- Dataset version, preprocessing configuration, split seed, and sample counts
- Framework and backend, simulator or hardware mode, and noise model if used
- Ansatz or circuit description, qubit count, depth, encoding method, and measurement design
- Optimizer, learning rate or other hyperparameters, initialization seed, epochs, and stopping rule
- Shot count, primary and secondary metrics, runtime, evaluation count, and failure notes
This record makes a QML benchmark useful beyond one notebook. It also helps identify whether an apparent improvement came from a model change, a data change, a new backend, or a different evaluation procedure.
Cadence and checkpoints
Use checkpoints that match the stage of the project rather than waiting until the end.
- Before training: Freeze the evaluation protocol, select the primary metric, define the baselines, and create the experiment-log fields.
- After a pilot run: Check that the circuit executes as intended, labels are aligned, preprocessing is leakage-free, and the loss behaves sensibly.
- After a model comparison: Review mean performance, variance, resource use, and failure cases. Do not promote a configuration based on one unusually favorable seed.
- Before a hardware run: Compare ideal simulation, noisy simulation if available, and hardware results under clearly labeled conditions. Record queue or execution conditions only when they affect the interpretation.
- Monthly or quarterly: Re-run a small reference suite when framework versions, backends, circuit construction, or preprocessing code changes. Keep an unchanged reference configuration so that trends remain interpretable.
A compact comparison table can make recurring reviews faster:
| Configuration | Primary score | Spread | Runtime | Evaluations | Qubits/depth | Decision |
|---|---|---|---|---|---|---|
| Random baseline | — | — | — | — | — | Reference only |
| Classical baseline | Record | Record | Record | Record | Not applicable | Keep or replace |
| QML candidate | Record | Record | Record | Record | Record | Advance, revise, or stop |
How to interpret changes
Look for consistent improvement, not isolated peaks. If the QML model beats a simple baseline on one seed but not on the average across seeds, describe the result as unstable. If it improves predictive quality while using much more runtime or circuit depth, report the trade-off explicitly.
When a score changes, diagnose the likely cause in a fixed order: data split, preprocessing, random seed, optimizer settings, shot count, simulator noise, hardware backend, circuit compilation, and code or dependency changes. Changing several of these at once makes the result difficult to attribute.
Separate statistical uncertainty from engineering uncertainty. Repeated seeds help estimate variation in training. Repeated executions at a fixed configuration help reveal measurement or backend variation. Neither guarantees that a result will generalize to a new dataset. A benchmark is evidence about a defined setup, not proof that the quantum approach will outperform classical methods in every use case.
For algorithm-specific context, consult the quantum algorithms reference. Hardware-oriented measurements such as fidelity or execution throughput should complement, not replace, application-level performance metrics; the quantum benchmarking methods guide provides useful context for that distinction.
When to revisit
Revisit the evaluation whenever a recurring input changes: the dataset, feature pipeline, circuit architecture, optimizer, framework version, simulator noise model, hardware backend, shot budget, or classical baseline. A quarterly review is a reasonable default for an active project, while a monthly review may be appropriate during rapid development. Record the reason for each re-run so that a new score is not mistaken for a direct continuation of an older experiment.
At each review, retain one immutable reference run and compare new candidates against it. Update the reference only when the team agrees that the protocol, data, or production constraint has genuinely changed. Archive unsuccessful configurations rather than deleting them; they provide evidence about sensitivity and prevent repeated experiments.
To put this workflow into practice, create a tracking sheet or experiment dashboard with one row per run, add the metadata fields listed above, and define a promotion rule such as: “Advance only if the candidate improves the primary metric across the selected seeds without exceeding the resource budget.” Then run the same small suite on your chosen simulator or backend, compare it with the random and classical baselines, and schedule the next checkpoint. This turns quantum machine learning evaluation from a one-time score hunt into a repeatable engineering process.
