Chapter 70 Oracles: What Does “Correct” Mean?

tools/xna-oracle contains 39 declarative scenes and the same number of matching 256-by-256 RGBA8 PNGs. The references were captured once by hand from the real XNA 4.0 runtime under Wine. The C# runner source is checked in; its compiled executable, XNA runtime, and private 32-bit Wine prefix are not. Those external pieces are needed to regenerate references, not to compare a current CNA render against the checked-in corpus.

One renderer-agnostic C++ program implements the CNA side and is built for D3D9, EasyGL, OpenGL ES 1, FNA3D, and Skia. The comparator uses absolute per-channel differences. Its default tolerance is zero; it has no PSNR, SSIM, or perceptual metric. Optional RGB and alpha tolerances, a raw-differing-pixel budget, and an allowed rectangle make exceptions explicit rather than globally weakening the corpus.

The governing rule is stricter than “looks similar”: an increased tolerance needs a documented scene-specific reason, because silent tolerance growth turns an authenticity project into a parity project. D3D9’s registered hard gate runs all 39 scenes at tolerance zero. FNA3D’s registered corpus test is presently a render-survival gate, not a pixel-equality gate. EasyGL and OpenGL ES 1 runners are manual measurements rather than CTests.

Skia runs the nine SpriteBatch-mode scenes under a checked-in per-scene TSV policy. Seven are pixel exact. The two linear-filter scenes permit RGB delta 1, no alpha delta, a bounded number of raw differences, and a specific 80-by-80 rectangle. The runner verifies policy-to-scene and scene-to-policy set equality before comparing images.

This is a strong pattern for necessary relaxation. The tolerance is local, spatially bounded, versioned beside the references, and meta-validated against the corpus. Adding a scene without a policy or leaving a dead policy row both fail. The exception remains reviewable instead of becoming a magic global number.

PixelTestGame provides single-region samples and full golden comparison for standalone renderer tests. Its comparisons intentionally inspect RGB and ignore alpha. Seventeen 8-by-8 golden PNGs live under examples/golden; their names are EasyGL-derived, and OpenGL 2 reuses the same files.

Tolerances range from zero to 60 on an 8-bit channel. At the high end, the test catches gross shape or state regressions rather than subtle shading drift. That does not make it useless; it makes its contract different from the real-XNA corpus. A self-generated golden is a regression oracle for CNA’s chosen behavior, not independent evidence that the behavior matches XNA.

Setting CNA_UPDATE_GOLDEN to any non-empty value rewrites the reference from the live render. Updating is therefore a review operation, never a routine way to make a failure green. The change should carry the image diff, renderer, driver, and reason the new output is correct.

Single-pixel coverage is broader than golden coverage: the audited tree contains 340 ExpectPixel calls in 70 files, and 169 renderer example sources include the pixel-test harness. These cases are cheap and precise for clear colors, blend equations, scissor order, or depth outcomes.

Before running, the harness probes SDL video. Failure returns code 77, which CTest interprets as SKIPPED. A green headless run can therefore mean that none of these comparisons executed. A pixel-oracle report must state display and skip evidence, not just the final exit status.

A separate chain runs C# against a live FNA.dll and compares its JSON output with CNA’s C++ reference dumper. It covers enums, state presets, packed-vector types, and viewport cases. Numeric comparison defaults to absolute tolerance ${10}^{-4}$; non-numeric values must match exactly. It is one-directional: every FNA key must exist in CNA, while CNA-specific extension keys may remain unmatched.

This manual tool found a real IndexElementSize disagreement: semantic 16/32 values on one side versus ordinal 0/1 values on the other. The example shows why a second implementation is useful even when neither side produces pixels. It challenges assumptions embedded in the public value surface.

Not every expected value comes from an external runtime. CNA uses three recurring patterns:

Golden tables

Packed-vector values are derived separately from FNA arithmetic and checked with narrow numeric tolerances.

Recomputation ladders

glTF tests rebuild node composition through independent L2, L3, and L4 calculations rather than asking the production routine for its own answer.

Shared oracle headers

A single pixel or geometry oracle is consumed by multiple renderer tests so copies cannot drift independently.

LZX decompression adds a byte-exact differential against FNA’s decoder and checked-in reference bytes; it previously exposed a heap-buffer overflow. Direct2D and EasyGL also share a small 64-by-64 diagnostic corpus with a bounded renderer-to-renderer tolerance and an explicit feature whitelist. Excluding unstable filtering cases is preferable to widening the tolerance until everything passes.

The oracle hierarchy can be summarized as follows: real XNA defines authenticity; a live FNA implementation supplies an independent compatibility reference; independent mathematics checks algorithms; cross-renderer controls localize implementation defects; CNA-owned goldens prevent unintended visual change. CNA does not reference the sibling xna4-decomp or xna4-spec repositories in this chain.

Tolerance belongs to the authority and phenomenon being measured. Zero is appropriate for a byte-exact decode or stable reference image; one-channel rounding may justify a local pixel policy; a tolerance of 60 means only a coarse visual guard. Naming that strength is part of the verdict. “Passed the oracle” is incomplete until the reader knows which oracle, which policy, and whether it actually executed.