Commercial routing APIs publish no benchmark numbers. This sheet is the counter-position: every claim below is a measurement with a unit, a date, and a named harness (including the gaps we have not closed yet). Two registers: what never varies (guarantees: structural properties of the model), and what varies, measured (benchmarks).
These are properties of the model, not results of tuning. A daily optimizer can approximate them with penalties; here they are the shape of the decision space. The verification column is where we went looking for counterexamples.
| guarantee | why it cannot drift | verified |
|---|---|---|
| Every customer keeps a fixed weekday pattern for the whole plan. | The weekday is one decision per customer, not per visit: a single visit cannot pick its own day. | 592/592 families, two production quarters · 2026-07-18 |
| 2×/week means spread pairs: Mo+Th, Tu+Fr. Never Mo+Tu. | The pattern catalog contains only spread pairs; there is no penalty weight to mistune. | 100% of pairs, neutral evaluator · 2026-07-18 |
| A sub-weekly rhythm keeps its week parity across rolling planning windows: every-2-weeks stays on its even or odd weeks. | The phase is a locked decision carried by the continuity file, anchored to a fixed calendar Monday. | 135/135 sub-weekly families across a quarter replan · 2026-07-18 |
| Two visits for one customer never share a day, unless that pair is explicitly whitelisted. | Enforced inside move application: a violating plan cannot be produced by the search, so it is never scored, never chosen. | structural; re-checked by result validation on every run |
| Replanning with the continuity file keeps every unchanged visit on its exact date. | The continuity file carries each customer's weekday set and phase; a replan pins every unchanged customer whose days are still available. | 100.0% of ~1,200 visits, every arm and seed, under 5% customer churn, at +0.2% travel · 2026-07-19 |
† Verification sources: production runs 260629/260706 (measure_stability.py), the neutral head-to-head evaluator (head_to_head.py), and the controlled churn experiment (bench_stability_churn.py, meting-02).
Public academic instances with published best-known solutions, the one comparison no vendor can grade for itself. We publish the gap either way.
| set | instances | avg gap vs best known | conditions |
|---|---|---|---|
| CVRPLIB, day routing A/B/E/M classics + Uchoa X |
11 | +0.45% | the production day path (two engines, best result kept) at the production time budget (150 ms/stop → 5–53 s/instance); 4 of 11 at the best known exactly, incl. the proven optimum A-n32-k5; single-engine was +5.7% |
| Cordeau PVRP, recurring periodic set, verified BKS ‡ |
4 † | +15.0% | full recurring pipeline: day assignment + routing; was +29.2% at first measurement, +20.3% a tuning week later, +15.0% with the geo-attraction move and the two-engine day pass |
† The 4
capacity-expressible instances of 32; the rest need per-customer
visit-day combinations, which our catalog deliberately restricts to spread patterns (on the roadmap).
‡ Best-known values transcribed from Cacchiani, Hemmelmayr
& Tricoire, Discrete Applied Mathematics 163
(2014), cross-checked against a second source.
The PVRP gap is an honest price, and we know exactly where it lives: re-solving the day routes of a finished plan at 60 s/day moves totals by less than 2%: the entire gap is day assignment. Academic PVRP objectives pack days as full as possible; ours balances weeks, because a crew that works 11 hours on Monday and 4 on Friday is not a solved problem. Where the objectives agree (pure day routing), the gap is +0.45%.
The day engine became a measured best-of pair on 2026-08-21: every day is solved by both OR-Tools and PyVRP at the same wall-clock budget and the better result is kept, so a day is never worse than either engine alone. Re-solving the full reference production quarter this way (312 days, 9,282 stops) removed 2.7% travel and 27.5% overtime without moving a single visit to another day.
scripts/cvrp_bench.py --polish · scripts/pyvrp_day_resolve.py · scripts/pvrp_convert.py · scripts/pvrp_report.py · measured 2026-07-18, day engine re-measured 2026-08-21
Same recurring problem (the public sandbox), same day optimizer, same matrix, same time budgets. The only difference: who decides which day each visit happens. The baseline is not a strawman: it fixes weekdays, uses spread pairs, balances minutes greedily, alternates sub-weekly parities. A careful human planner, automated. One neutral evaluator scores both.
† The point: day assignment is where recurring operations are won. Routing was identical on both sides; choosing the days together instead of one day at a time removed all 4,941 overtime minutes.
api/scripts/head_to_head.py · public sandbox (300 customers, 4 weeks) · measured 2026-07-18
The controlled experiment behind the continuity guarantee: change only the input (5% customer churn), replan, and count what survives. Day-level stability is something the model guarantees, not something you hope the optimizer reproduces.
| replan | unchanged visits on the same date | travel cost of stability |
|---|---|---|
| identical input, re-solved | 100.0% | ±1 min |
| 5% churn, with continuity file | 100.0% (every arm, every seed) | +0.2% |
| 5% new customers added, with continuity file | 100.0% | +4.9% |
| 5% churn, from scratch (no continuity) | ~34% | – |
† The from-scratch number is the honest contrast, measured on a harder instance than production: 85% of sandbox customers have all five workdays open, so nothing anchors them. On real production data (narrower day options), 67–77% survives even a from-scratch replan. Full write-ups: meting-01 (production baseline), meting-02 (controlled).
scripts/bench_stability_churn.py · public sandbox + pinned OSRM matrix · measured 2026-07-19
| stops | horizon | wall time | overtime | instance |
|---|---|---|---|---|
| 150 | 1 day | 7.5 s | 0 min | real data, dense day |
| 1,000 | 1 day | 10.0 s | 0 min | synthetic, clustered |
| 3,000 | 1 day | 60.0 s | 0 min | synthetic, clustered |
| 1,600 | 4 weeks | 52 s | 0 min | synthetic, periodic |
| 20,800 | 260 days, a full year | 617 s | 0 min | synthetic, periodic |
| 8,290 | 13 weeks | 65 min | 0 min | production quarter, 4 regions, 1,150 routes |
† Day-route solving is budget-driven, so wall time scales with the number of days, not the number of visits; the travel matrix is bounded by unique addresses, not task instances: a year of recurring work is matrix-cheap.
Quality at scale, measured: the standard setting (effort: fast) against a 4×-multistart reference gives up +5.7% at 100 tasks, only +1.5% at 1,000, and +0.0% at 10,000, where the reference, given 17.1 h against the standard setting's 5.2 h, returned the identical plan. Deeper search buys the most on small instances; at scale the standard setting is already at the reference. That 10k instance is not fleet-binding (zero overtime), and basin coverage pays mainly in the binding regime, so the figure describes this regime, not every one.
scripts/make_benchmarks.py + run_benchmarks.sh · snapshot 2026-05-18 · production run 2026-07 · scale-quality curve scripts/bench_scale_quality.py 2026-08-22