Chapter 73 CI Execution and Coverage

Nineteen workflows react to a push or pull request, often under narrow path filters. Direct3D and GDI remain workflow_dispatch-only. The expanded automatic set adds Apple, glTF stride and sanitizer, platform, TinyGL, NanoVG, native and Emscripten multi-renderer, and five C API workflows to the older general, Input, Devices, HTML-DOM, Metal and 32-bit jobs. A workflow file’s presence therefore proves neither that its trigger matched this tag nor that an individual job ran successfully.

The general Linux job configures Debug with tests and examples, builds the default target, then runs unfiltered CTest with a known-failure classifier. The Input matrix builds only CnaTests across five renderer/sanitizer combinations and invokes the single input-label entry, which repeats 541 selected cases five times in shuffled order: 2,705 case executions when all iterations complete. The workflow’s nearby comment still says “x3,” but the registered CTest command and current Input documentation both say five; executable configuration wins that disagreement. The Devices workflow instead runs filtered GoogleTests directly, following the preset’s warning that per-case CTest discovery can race shared temporary paths.

HTML-DOM is the most complete Web route: pinned emsdk, Playwright Chromium, six HTTP-served pages, browser verdicts, and compositor samples. Metal selects the native renderer on macOS 14, enables Metal validation variables, and runs Metal-named plus renderer-definition tests. The 32-bit job builds three small arithmetic programs with -m32; it is a narrow integer-overflow gate, not a 32-bit CNA build.

The native multi-renderer workflow configures HEADLESS;SOFTWARE;STUB, builds the generated descriptor/registry gate, selects each compiled identity through the public demo, tests the environment-variable route, and retains a singular-build control. A separate Emscripten workflow composes WEBGL2;CANVAS;HTML_DOM;SVG_DOM. These are compile and runtime-choice evidence for two compatible sets, not proof that arbitrary families may be combined.

The platform workflow names the graphics, platform and audio values independently. Its matrix exercises SDL3, SDL2 and headless tuples, cross-implementation event oracles, SDL2-native helper executables and a 2D integration demo. Terminal remains covered by source/tests outside a Windows build; the matrix does not turn its POSIX-only contract into a cross-platform claim.

Five C API workflows check generated surface coverage, pure-C header/toolchain compatibility, the ABI baseline, limitation records and the release decision record. They are substantial structural gates, but their repository snapshots do not prove that the alpha.1 tag’s workflow runs succeeded. Running the tag’s source-only release-record checker reports all ten criteria met and “READY to publish the experimental release.” An exact-tag library build nevertheless fails because the C renderer map has 49 identities while the canonical enumeration has 50. The checker proves that gate definitions exist and records agree; it does not execute the installed consumer or final library build and therefore produces a false artifact-readiness verdict. Its heading also mislabels ABI 0.7.0 as 0.1.0.

At the audited SHA, general-tests-ci.yml and two Input legs pass the retired internal token EASYGL as the current renderer option’s value. It is an implementation family, not one of the 50 public values accepted by renderer selection. The public native profile is OPENGLES3 or another named GL identity. Those jobs therefore stop at configure time.

This defect removes more than nominal redundancy. The general job is the only automatic unfiltered suite, and one dead Input leg is the only AddressSanitizer-plus-UndefinedBehaviorSanitizer combination in CI. A workflow diagram read without its configure arguments would report both as coverage even though neither reaches compilation.

The correct automation audit starts at the trigger and follows every step: branch/path filter, configure command, named build target, test selector, environment, artifact upload, and failure policy. “Workflow exists” is merely the first row of that trace.

The manual Direct3D matrix configures MSVC/Ninja for DirectX 11, DirectX 12, and Direct2D, disables examples and networking, then runs renderer-labelled CTests. Direct2D adds debug-layer log verification and a WARP lifetime pass; DirectX 11 invokes the native HLSL verifier. GDI builds 17 named executables and runs its label.

The GDI workflow’s explicit target list excludes CnaTests. The D3D workflow is subtler: its unrestricted cmake --build builds the default all target and therefore includes CnaTests, contradicting the workflow header’s claim that it does not. The DirectX 11 and DirectX 12 label runs still omit the unlabelled discovered GoogleTests. Direct2D, however, labels and runs Direct2D_Unit, whose command invokes CnaTests --gtest_filter=Direct2D*. The same header’s POSIX setenv rationale is stale because platform-incompatible process sources are filtered on Windows. Renderer-specific Windows tests remain useful, but the actual build and selector commands—not that comment—define their corpus. The native HLSL verifier also points to a pre-modularization source path and fails before it can import the shader manifest.

Manual dispatch has a further consequence: five of the seven scripts referenced anywhere in CI live only in this D3D workflow. On a routine push, the script layer exercised directly by workflows reduces to the two HTML-DOM runner layers. CTest can invoke many other scripts, but only inside a configure that registers and executes their renderer.

CNA has a root CNA_SANITIZE option and separate Devices presets that place raw sanitizer flags in CMAKE_CXX_FLAGS. CI contains two sanitizer legs in source: the dead Input ASan+UBSan leg and a Devices UBSan preset. No workflow runs ThreadSanitizer, and no general non-Devices sanitizer matrix exists. Skia tests disable vptr sanitization because the pinned no-RTTI Skia archives cannot supply the required type information.

GPU validation is renderer-specific. Vulkan enables Khronos validation and synchronization validation; three tests assert that the layer actually loaded. D3D11 and Direct2D request the debug device with a missing-SDK fallback. Metal’s workflow exports shader and debug-layer variables. WebGPU pushes validation error scopes. These mechanisms are not interchangeable, and a test must prove engagement rather than infer it from an environment variable.

Four GoogleTest source files contain eight fuzz-named cases for XACT, XNB/LZX, and SDL input. They are deterministic mutation loops inside CnaTests; there is no libFuzzer, AFL, or OSS-Fuzz target. One SoundEffect reader file implements a hand-rolled property test. These are valuable adversarial tests, but they do not provide coverage-guided exploration or a persistent crash corpus.

Mutation testing exists for Direct2D only. Its seven mutations can rebuild and invert the expected verdict, but CI runs just --dry-run, proving anchors still exist rather than proving the renderer tests kill each mutation. The distinction between mutation machinery and a completed mutation campaign must remain explicit.

No build file, preset, script, or workflow configures gcov, lcov, gcovr, llvm-cov, or compiler coverage flags. Documents named coverage.md describe API presence, feature matrices, or source-to-test mappings. Their percentages are not line, branch, or function coverage.

A credible CI summary should therefore report which configured paths executed, their skip and artifact records, and the gaps. At this revision, automatic browser evidence is real, targeted native and sanitizer evidence exists, and much of the renderer apparatus remains manual or configuration-exclusive. The honest result is not “CI complete” or “CI absent,” but a precise map of what ran and what merely exists.