WWD conformance: 73 test vectors and how a second implementation is tested

Definition

The WWD conformance suite is a set of real .wwd packages published as data, with a manifest stating what a conforming validator must report for each one. Appendix C of the WWD 0.1 draft is normative for the diagnostic codes: a validator that detects one of the listed conditions MUST report it using the code given there, and MUST NOT reuse a listed code for a different condition. Message wording is not normative. Each manifest entry names a package file, a sha256 digest, the codes that must be reported, the codes that must not be reported, whether the package is valid, and the conformance classes the package must and must not reach. A second implementation runs the suite against its own validator and compares the result with the manifest.

A vector is a real package plus one manifest entry. The package is a ZIP archive with a stored mimetype entry first, a wwd/manifest.json, and the documents the case needs. The manifest entry names the file, describes what the case asserts, cites the specification sections it is claimed from, pins a sha256 and a byte length, and carries an expect block.

The expectation is a claim about the specification, not a record of what the reference validator currently does. A suite that recorded current behaviour would prove nothing about a second implementation.

What a vector is

A package on disk and an entry in index.json.

the packageunzip -l

Archive:  test-suite/vectors/valid-as-built.wwd
  Length      Name
---------  ----------------------
       23  mimetype
      748  wwd/manifest.json
     2845  wwd/design.json
     1303  wwd/results.json

A stock ZIP tool opens it. The mimetype entry is first and stored, not deflated.

the manifest entryindex.json

{
  "name": "invalid-not-zip",
  "file": "vectors/invalid-not-zip.wwd",
  "description": "Not a ZIP archive at all: the first thing any reader must survive is being handed the wrong kind of file.",
  "spec_section": "3.1",
  "expect": {
    "valid": false,
    "must_report": [
      "P01_NOT_ZIP"
    ],
    "must_not_report": [],
    "conformance_classes": [],
    "not_conformance_classes": []
  },
  "sha256": "e3b0c442...",
  "bytes": 22
}

The digest is abbreviated here. In the manifest it is the full sha256 of the file.

Running the suite

One command, and the rules the runner applies.

run_suite.py is the reference runner. It is about 120 lines, uses the standard library only, and does not import the reference implementation. It shells out to whatever command is named and applies the rules stated in index.json.

the commandtest-suite/

python3 run_suite.py --validator "path/to/your-validator --json"

The --validator string is required. The runner splits it, appends the package path as the final argument, and reads stdout as JSON, so the string must include whatever flag makes the validator print JSON. Both reference implementations spell it --json.

What run_suite.py does per vector
Step	Behaviour
Read the manifest	run_suite.py loads index.json from the suite directory and iterates its vectors array.
Verify the digest	The sha256 of the package file is recomputed and compared with the pinned value. A mismatch fails the vector with "the vector has been modified" and the validator is not run.
Shell out	shlex.split of the --validator string, with the package path appended as the final argument. The runner does not import the reference implementation.
Parse findings	json.loads of stdout. A dict is searched for findings, diagnostics, results or messages; a list is taken as the findings. Each finding needs a code and a severity, and everything else is ignored.
Compare	Codes are collected into a set, and the errors are the subset whose severity is the string error. The manifest rules are then applied to those two sets.
Report	A failing vector prints its name, each problem, and the first 96 characters of its description. Exit status is 0 when every vector passes and 1 otherwise.

the output shape the runner reads--json

{
  "package": "invalid-bare-number.wwd",
  "valid": false,
  "refused": false,
  "as_built": false,
  "disposition": null,
  "findings": [
    {
      "code": "C02_BARE_NUMBER",
      "severity": "error",
      "message": "component/neurons/wwd:culture_temperature: bare number 37 has no unit",
      "where": "component/neurons/wwd:culture_temperature",
      "hint": "write \"37 kPa\" (a quantity string), or a field object with an explicit backing and unit"
    }
  ]
}

A finding needs a code and a severity. Everything else is ignored. A validator whose output differs changes parse_findings, which is the only WWD-specific glue in the file.

The manifest rules, from the how_to_run block of index.json
Key	Rule
must_report	Every code listed MUST appear in your findings. This is a SUBSET test, not equality: your implementation MAY report additional codes and still conform. Eleven of the shipped vectors legitimately emit codes beyond those listed, because one defect often trips several independent rules.
must_not_report	No code listed may appear. These are the false positives a plausible misreading of the specification produces.
valid	true means your validator reports no finding of severity error. Warnings are permitted in a valid package and some valid vectors carry them: warnings plus no errors is conforming.
conformance_classes	Each class listed MUST be reachable. Also a subset test; reaching additional classes is not a failure.
not_conformance_classes	Each class listed MUST NOT be reachable. This is where an overclaiming implementation is caught.
severity	A consumer MUST NOT downgrade an error to a warning. Reporting a listed code at a lower severity does not conform.
unknown_codes	Reporting a code this suite does not mention is permitted. It may be an extension diagnostic or a stricter reading, and is not a conformance failure.
message_wording	NOT normative (spec Appendix C). Only codes are. Do not compare message text.

Coverage by code family

Vectors counted against the Appendix C section their expected codes belong to.

Code families across the 65 invalid vectors
Prefix	Appendix C section	Vectors
P	C.1 Package and container	13
C, F	C.2 Coordinates, units, and fields	3
V	C.2b Property registry	4
K	C.2c Unknown keys	1
U	C.2d Physical possibility of criteria	5
M	C.2e Verdict recomputation	4
S	C.2f Sampling and decision rules	4
X	C.2g Frames and transforms	5
I	C.2h Identity and references	6
N	C.2i Interfaces	7
L	C.2k Lineage and provenance graph	3
Q	C.2s Simulation envelope	1
A	C.3 Causal axis	4
R	C.4 Requirements and results	6

A vector is counted once per family it expects a code from, so a vector expecting two codes from two families appears in both rows. The 8 valid vectors expect no code and are not counted here.

Every vector

Name, what it asserts, and the codes the manifest requires.

test-suite/index.json, 73 vectors, WWD 0.1
Vector	Valid	Expected codes	What it asserts	§
valid-minimal	yes	none	The smallest useful package: a vial of identified cells.	22.2, 22.3
valid-design-only	yes	none	An aligned muscle strip, as designed, with one acceptance criterion.	7, 12.3, 15
valid-as-built	yes	none	The same design with measured results; as-built mode is DERIVED.	16
valid-expression-field	yes	none	A property that varies over the component, given as an expression.	10.2, 11
invalid-not-zip	no	P01_NOT_ZIP	Not a ZIP archive at all: the first thing any reader must survive is being handed the wrong kind of file.	3.1
invalid-mimetype-not-first	no	P02_MIMETYPE_NOT_FIRST	The mimetype entry is not the first entry in the archive.	3.1
invalid-mimetype-compressed	no	P02_MIMETYPE_COMPRESSED	The mimetype entry is deflated rather than stored.	3.1
invalid-mimetype-content	no	P02_MIMETYPE_CONTENT	The mimetype entry holds the wrong media type.	3.1
invalid-no-manifest	no	P03_NO_MANIFEST	No wwd/manifest.json, so nothing declares what the package contains or which document is the root.	3.2
invalid-integrity-mismatch	no	P05_INTEGRITY_MISMATCH	The design document does not match its declared digest.	20
invalid-unmanifested-file	no	P03_UNMANIFESTED	A payload file present in the archive but absent from the manifest.	3.2
invalid-required-extension-unsupported	no	P08_REQUIRED_EXT_UNSUPPORTED	Declares a required extension no Core consumer implements; a conforming consumer MUST refuse the package rather than ignore it.	19
valid-optional-extension	yes	none	Carries a real third-party extension marked used-but-not-required. A Core consumer may ignore it, must report that it did, and must read the Core content identically either way.	19
invalid-required-extension-optogenetics	no	P08_REQUIRED_EXT_UNSUPPORTED	The same extension marked REQUIRED. A consumer that does not implement it must refuse before manufacture, simulation or acceptance, and must name the extension it could not process.	19
invalid-required-not-declared-used	no	P08_REQUIRED_NOT_IN_USED	An extension listed as required but absent from extensionsUsed: the two arrays have disagreed, so the declaration is defective.	19
invalid-extension-mints-core-term	no	V01_UNREGISTERED_PROPERTY	An extension inventing a term in the wwd: namespace. Extensions must namespace their own vocabulary; the Core registry owns wwd:.	7.9, 19
invalid-bare-number	no	C02_BARE_NUMBER	A quantitative slot holding a bare number with no unit.	5.3
invalid-field-no-backing	no	F03_NO_BACKING	A field object with no backing discriminator.	9
invalid-response-imposed	no	V03_CAUSAL_CLASS_MISMATCH	Declares fiber alignment as something the producer sets. Alignment is produced by the tissue; it cannot be pipetted.	7.8, 7.9
invalid-control-without-means	no	A8_CONTROL_NO_MEANS	A control that names no protocol step or apparatus imposing it. A quantity nothing can set is not a control.	7.8
invalid-no-causal-role	no	A8_NO_CAUSAL_ROLE	A scientific assertion with no causal role declared.	7.8
invalid-criterion-no-axis	no	A8_CRITERION_NO_AXIS	A criterion that does not say whether it tests the protocol or the biology. The two have different remedies when they fail.	7.8
invalid-unit-not-canonical	no	V02_UNIT_MISMATCH	Right property, wrong unit. Refused even though a conversion exists: a silently converted value is one the producer did not write.	7.9
invalid-unregistered-property	no	V01_UNREGISTERED_PROPERTY	Uses the wwd: namespace for a term the Core registry does not define. Seeding density is unregistered: it is ambiguous between volumetric and areal.	7.9
invalid-2d-alignment-no-plane	no	A03_NO_MEASUREMENT_PLANE	A planar order parameter with no measurement plane: not comparable between laboratories.	12.3
invalid-criterion-no-method	no	R01_NO_METHOD	A functional criterion with no assay method. Two suppliers measuring differently produce incomparable numbers.	15.1
invalid-dangling-result	no	R04_DANGLING_CRITERION	A measured result referencing a criterion that does not exist.	16
invalid-simulated-as-measured	no	F09_SIMULATED_AS_MEASURED	A simulated value presented as a measured acceptance result. A prediction that can pass acceptance is the failure the five-layer physics separation exists to prevent.	18.1
invalid-gtin-as-unit-id	no	R05_GTIN_NOT_A_UNIT	A GS1 GTIN identifies a product type, not an individual unit.	13.3
invalid-no-id-scheme	no	R05_NO_ID_SCHEME	A realization with an opaque identifier and no scheme.	13.3
invalid-authored-mode-flag	no	R06_AUTHORED_MODE_FLAG	As-built mode stored in the file rather than derived from the presence of measured results.	16
invalid-convenience-sample-with-test	no	R03_CONVENIENCE_SAMPLE	A statistical test applied to a convenience sample, which does not satisfy the probability-sample assumption the rule depends on.	13.2
invalid-misspelled-safety-key	no	K01_UNKNOWN_KEY	A criterion whose sampling plan is misspelled as 'smapling'. Because the key is optional, the typo is indistinguishable from omission: the criterion now states no sample size and every consumer acts on a requirement the author did not write.	7.10
valid-namespaced-extension-key	yes	none	The same object carrying a namespaced key. Extensions are required to namespace their content, so an unknown-key rule that rejected this would forbid the extension mechanism. The prefix is bound to a versioned URI in the manifest (19.6): a CURIE on its own names no extension, and a consumer comparing content against extensionsUsed cannot tell whether an unbound one is covered.	7.10, 19
invalid-criterion-wrong-unit	no	U01_CRITERION_UNIT_MISMATCH	A twitch force criterion stated in hertz. It parses, it looks like a requirement, and it is satisfied by every construct ever made including one that generates no force.	15.5
invalid-proportion-as-percentage	no	U03_IMPOSSIBLE_LIMIT	A proportion criterion with a limit of 80: a percentage written where a fraction belongs. No lot can ever satisfy it.	15.5
invalid-inverted-range	no	U03_IMPOSSIBLE_LIMIT	A range whose low bound exceeds its high bound; no value satisfies it.	15.5
invalid-failing-result-marked-pass	no	M03_VERDICT_CONTRADICTED	A measurement of 120 uN recorded as 'pass' against a critical 'median >= 500 uN' criterion. Spec 16.1 requires a consumer to recompute the verdict rather than trust it; a format whose acceptance record can disagree with its own numbers certifies whatever it is told to.	16.1
invalid-result-unit-mismatch	no	M01_RESULT_UNIT_MISMATCH	A result reported in millinewtons against a criterion stated in micronewtons: a thousandfold error that would pass trivially if the numbers were compared directly.	16.1
invalid-plan-not-executed	no	S04_NOT_EXECUTED	A criterion planning n = 12 answered by a single specimen. The sampling plan is part of the requirement; an under-sampled criterion is not a passed criterion.	15.6
invalid-unattainable-significance	no	S02_UNATTAINABLE_POWER	A one-sided test at alpha = 0.05 with n = 3. The most extreme possible outcome has probability 0.125 under the null, so no data whatsoever can reject it: the criterion is unsatisfiable in principle rather than merely demanding.	15.6
invalid-dangling-criterion-target	no	I05_DANGLING_REFERENCE	A criterion whose target names no object in the package. It occupies a line in the acceptance table, a supplier is quoted against it, and it can never fail because there is nothing for it to be about. An absent criterion is visibly absent; a dangling one looks like diligence.	4
invalid-identifier-collision	no	I04_ID_COLLISION	A region carrying the same identifier as a criterion. Identity is package-wide, not per collection: a reference to a shared id resolves to two objects.	4
invalid-frame-bad-handedness	no	X05_BAD_FRAME	A frame whose handedness is neither right nor left. Handedness cannot be inferred from axis letters, and a mirrored construct is dimensionally identical to its original.	5
invalid-frame-reordered-axes	no	X05_BAD_FRAME	Axes listed out of Core order. Reordering silently permutes every coordinate in the file.	5
invalid-transform-mirrors-the-frame	no	X03_HANDEDNESS_FLIP	A transform with negative determinant. Lengths, angles and volumes all survive a mirror; only chirality flips, so the geometry still looks correct while every director points the wrong way.	5
invalid-transform-not-rigid	no	X04_NOT_RIGID	A transform declared rigid whose linear part scales. Every length in the child frame is then wrong by a factor the file never states.	5
invalid-transform-singular	no	X02_SINGULAR_TRANSFORM	A singular transform collapses the child frame onto a plane and cannot be inverted, so no coordinate can be mapped back.	5
invalid-relative-package-id	no	I02_NOT_ABSOLUTE	A package identified as 'my-package' is identified only within whoever's filesystem it currently sits in. Once two laboratories exchange files that is not an identity.	4
invalid-malformed-uuid	no	I03_MALFORMED_UUID	A UUID URN whose variant nibble is invalid under RFC 9562. A string that merely looks like a UUID is a different kind of object that happens to contain hyphens.	4
invalid-criterion-targets-an-interface	no	I06_WRONG_REFERENT	A reference must resolve to exactly one object OF THE REQUIRED KIND. An interface is not something a lot acceptance criterion can be about.	4
invalid-load-limit-without-direction	no	N05_NO_DIRECTION	A mechanical interface stating an allowable force with no direction or sign convention. 'Allowable force 5 mN' does not say whether that is tension or compression, and those are different failures.	14
invalid-compatibility-by-label-only	no	N03_LABEL_ONLY	Compatibility asserted only by a standard name. Two vendors can both claim it while differing in pitch, electrode count, well depth and datum, so a quote given against a label is a quote given against a guess.	14
invalid-interface-targets-nothing	no	N02_NO_TARGET	An interface attached to a region that does not exist. It cannot be loaded, imaged or connected, and no criterion can be written about it.	14
invalid-statistic-mismatch	no	M02_STATISTIC_MISMATCH	A result reporting a mean against a criterion written on the median. The two answer different questions about the same data, and a skewed batch passes one while failing the other.	16
invalid-impossible-yield	no	M04_IMPOSSIBLE_YIELD	More units meeting a criterion than were tested. The yield is not merely wrong, it is unachievable, and it inflates every downstream estimate built on this lot.	16
invalid-impossible-sample	no	S01_IMPOSSIBLE_SAMPLE	A sampling plan demanding zero specimens. A criterion no specimen can be drawn for cannot be evaluated, yet it still reads as a requirement a supplier is held to.	15
invalid-decision-rule-without-a-plan	no	S03_RULE_WITHOUT_PLAN	A statistical decision rule with no sampling plan. The rule cannot be executed or reproduced, so the criterion asserts a rigour it does not have.	15
invalid-dimension-mismatch	no	U01_CRITERION_UNIT_MISMATCH, U02_DIMENSION_MISMATCH	A force criterion stated in hertz. This is the copy-paste error between criteria that unit mistakes take, and every construct ever made satisfies it.	15.5
invalid-degenerate-range	no	U04_DEGENERATE_RANGE	A range demanding exact equality on a measured quantity. No assay delivers a point value, so the criterion is unsatisfiable in principle rather than merely demanding.	15.5
invalid-lineage-cycle	no	L03_DERIVATION_CYCLE	A derivation cycle: something is its own ancestor. A recall query over this graph cannot be trusted to report everything affected.	17
invalid-lineage-dangling-edge	no	L02_DANGLING_EDGE	A derivation edge naming an entity the graph does not contain. The ancestry it claims cannot be walked, so a recall silently stops early.	17
invalid-lineage-node-type	no	L01_BAD_NODE_TYPE	A provenance node whose type is not entity, activity or agent. PROV has exactly three, and a fourth cannot be interpreted by any conforming consumer.	17
valid-assembly-netlist	yes	none	A two-part assembly: a tissue loop pinned to a lever tip, and the lever's fulcrum grounded. Every port resolves, the kinds are compatible, and something is held. This is the whole 14.7 rule set passing, and it is checkable with no solver and no geometry.	14.7
invalid-connection-dangling	no	N10_CONNECTION_DANGLING	A connection to a port that does not exist. The assembly draws a load path that nothing in the package carries.	14.7
invalid-port-kind-mismatch	no	N12_PORT_KIND_MISMATCH	An optical port connected to a mechanical one. Nothing is transmitted across it, and a consumer that inferred a load path here would simulate a machine that cannot exist.	14.7
invalid-port-kind-unknown	no	N11_PORT_KIND_UNKNOWN	A connected port whose modality 14.7 does not map to a port kind. Reported rather than defaulted: treating an unrecognised modality as mechanical is how an optical port comes to be bolted to a lever with nothing objecting.	14.7
invalid-assembly-ungrounded	no	N14_ASSEMBLY_UNGROUNDED	An assembly nothing holds. A solver given one returns the rigid-body motion of the whole assembly as though it were the answer, so the omission is caught before any solver sees it.	14.7
invalid-extension-prefix-unbound	no	P08_EXTENSION_NOT_DECLARED	A CURIE-keyed extension whose prefix the manifest never binds. Two packages may both write `acme:` and mean unrelated extensions, so the content is undeclared however the used list reads.	19.6
invalid-extension-uri-unversioned	no	P08_EXTENSION_URI_UNVERSIONED	An extension identifier with no version. It names a lineage of extensions rather than one of them, so a consumer comparing it against its supported set compares against something that changes underneath it.	19
valid-design-only-study	yes	none	A study run against a design nobody has built: models and runs in the design, a results document carrying predictions and no realization. The package stays as-designed, because as-built is keyed on the realization and a prediction is not a lot. Requiring one here would force a producer to discard the study or invent a lot that was never manufactured.	18.5
invalid-envelope-duplicated	no	Q09_ENVELOPE_DUPLICATED	The same model and run carried both in the design and beside the results. A reference that resolves twice resolves to whichever copy the consumer read first, and the two are free to disagree.	18.5
invalid-path-traversal	no	I01_NO_PACKAGE_ID, P07_UNSAFE_PATH	An archive entry that escapes the package root. It also declares no package_id: 4 makes identity a manifest fact, so a package whose design is missing is still asked for its identifier -- and that is the package a reader most needs named, because it is the one they will be trying to match against another copy.	21, 4

8 of the 73 packages are valid and expect no code. The remaining 65 expect at least one, and 62 distinct codes appear across the manifest.

The conformance classes

Section 22 of the WWD 0.1 draft. Six classes, each answerable to a stated section list.

WWD 0.1 conformance classes
Class	Sections	What it covers
WWD-Package-Core	3, 4, 19, 20, 21	A package MUST satisfy those sections and MUST contain a conforming manifest and root design document. It MAY omit geometry, fields, results and simulations when the component type does not require them. A consumer MUST verify package structure, identifiers, extension sets and required integrity.
WWD-Design-Core	WWD-Package-Core + 5-14	At least one component and one valid biological identity branch for living content. A spatial component MUST contain or reference authoritative geometry and a canonical frame. Every quantitative property MUST satisfy scalar-or-field semantics and every declared field MUST satisfy the common contract.
WWD-Field-Evaluator	8-12	An implementation MUST satisfy those sections for every backing it claims, and MUST support constant and expression for full Core evaluator conformance. Full five-backing conformance additionally requires mesh_attribute, image_volume and table. A partial evaluator MUST list its supported backings and MUST refuse evaluation of unsupported required fields. It MUST preserve provenance, value shape, units, frame, interpolation and out-of-domain semantics.
WWD-As-Built	WWD-Design-Core + 15-17	Every measured result MUST resolve to a criterion and a realization, and every measured assertion MUST map to an observation activity. The package MUST preserve designed and measured values separately, and a consumer MUST derive, not trust, as-built mode and disposition.
WWD-Simulation	WWD-Design-Core + 18	Every model, run and result reference MUST resolve. Every simulation result MUST be marked simulated and linked to exactly one run. A simulation consumer MUST NOT present simulated results as acceptance measurements. Solver-specific execution capability is not required to consume the envelope.
WWD-Manufacturing-Ready	WWD-Design-Core + retrievability	All assets, fields, interfaces and extensions required to manufacture the declared article MUST be retrievable and integrity-verified. All required dimensions, units, frames, biological identities, interfaces and process-relevant requirements MUST be unambiguous. Every manufacturing-critical field MUST be evaluable by the intended workflow. An implementation MUST refuse manufacturing readiness when a required extension, field, asset or provenance assertion is unsupported or unverifiable. The class does not assert that a particular supplier can or will manufacture the design.

11 vectors carry class claims. valid-as-built.wwd must reach WWD-Package-Core, WWD-Design-Core and WWD-As-Built. invalid-authored-mode-flag.wwd must not reach WWD-As-Built, and invalid-required-extension-optogenetics.wwd must not reach WWD-Package-Core or WWD-Manufacturing-Ready. That second array is where an overclaiming implementation is caught.

How a validator declares its classes

Section 22.1, and the shape the reference validator prints.

Obligations on a conformance claim
Obligation	Rule
Name the version and every class	A claim MUST name WWD 0.1 and every claimed class.
Name the gaps	A claim MUST identify unsupported optional features.
Do not rename structural validation	Passing structural validation alone MUST NOT be described as scientific or manufacturing conformance.
More than one class is permitted	A package MAY conform to more than one class.
State the role	An implementation MAY conform as a producer, consumer, rewriter or evaluator. A producer claim requires emitted packages to satisfy the claimed package classes. A consumer claim requires correct acceptance, refusal and interpretation behavior.
Draft packages reach nothing	A draft package MUST NOT be reported as reaching any conformance class, including WWD-Package-Core. The condition is reported as W01_DRAFT_NOT_RELEASABLE at WARNING.

the claim--conformance

valid-as-built.wwd
  PASS  (as-built)  errors=0 warnings=0  disposition=pass

  conformance (WWD 0.1)
    PASS  WWD-Package-Core
    PASS  WWD-Design-Core
    PASS  WWD-Field-Evaluator
    PASS  WWD-As-Built
    n/a   WWD-Simulation  (no simulation content)
    PASS  WWD-Manufacturing-Ready

    claim: WWD 0.1 - WWD-Package-Core, WWD-Design-Core, WWD-Field-Evaluator,
           WWD-As-Built, WWD-Manufacturing-Ready

One line per class, with n/a where the class does not apply to the content, and a claim line naming the version and every class reached. Unsupported optional features are listed beneath it. A field evaluator lists the backings it supports under the same rule.

When a vector fails

Two candidates, and the suite exists to tell them apart.

A failure means the implementation and the manifest disagree about what the specification requires. One of them is wrong, and finding out which is the purpose of publishing the vectors as data instead of keeping them as Python tests. The runner prints that sentence itself when any vector fails.

Both outcomes have occurred. While the suite was being built, one vector claimed F09_BAD_RESULT_PROVENANCE for a simulated value presented as a measurement, and the validator emitted the more specific F09_SIMULATED_AS_MEASURED. The manifest was wrong and was corrected. An implementer who disagrees with a vector and believes the specification supports them is asked to open an issue.

The two reference implementations
Implementation	Source	Suite result	Command
Python	validator/wwd, standard library only	73/73 vectors	python -m wwd package.wwd --json
TypeScript	validator-ts/src, one runtime dependency for raw DEFLATE	73/73 vectors	wwd-validate package.wwd --json

The TypeScript implementation was written against the published specification and the conformance suite, not against the Python reference. Where the two disagree, the disagreement is recorded as a finding about the specification rather than reconciled in silence.

What the suite does not test

The claims the manifest does not make.

Out of scope for the conformance suite
Not tested	Note
That a validator accepts anything	A suite of only-invalid cases cannot catch a validator that rejects everything. 8 valid packages are included for that reason.
Message wording	Appendix C is normative for the codes and not for the message text. Two implementations report the same code and may write different sentences.
Whether extra codes are correct	Reporting a code the suite does not mention is permitted. It may be an extension diagnostic or a stricter reading of the specification.
Whether the design will work	The DFM design rules are advisory and are not conformance codes. No vector expects one.
Conformance classes, in the shipped runner	The manifest carries conformance_classes and not_conformance_classes on 11 vectors, and index.json states the rule. run_suite.py compares codes and errors only; a runner that checks the class claims reads those two arrays itself.
Performance, memory or API shape	The manifest states codes, validity and classes. Nothing else is asserted about an implementation.

Adding a vector

The generator writes both the package and the manifest entry.

validator/build_test_suite.py
Step	What it does
Write the package	validator/build_test_suite.py builds packages with pack(), where every structural rule is individually defeatable: mimetype order, mimetype compression, manifest omission, extra files, manifest patches and raw bytes.
Record the expectation	vector(name, description, data, must_report=, must_not_report=, valid=, classes=, not_classes=, section=) appends one manifest entry. valid defaults to false when must_report is non-empty.
Regenerate	python build_test_suite.py writes vectors/<name>.wwd and rewrites index.json, including the sha256 and byte length of every package.
Check the runner	python3 run_suite.py --validator "your-validator --json", and python -m pytest tests/test_suite_runner.py to check the runner itself against the reference implementation.
Cite the section	Each entry carries spec_section, the sections of the WWD 0.1 draft the expectation is claimed from.

The digest and byte length are computed by the generator, so a hand-edited package fails its own vector on the sha256 check before the validator is ever invoked.

Questions

What is the WWD conformance suite?

The WWD conformance suite is a set of real .wwd packages published as data, with a manifest stating what a conforming validator must report for each one. Appendix C of the WWD 0.1 draft is normative for the diagnostic codes: a validator that detects one of the listed conditions MUST report it using the code given there, and MUST NOT reuse a listed code for a different condition. Message wording is not normative. Each manifest entry names a package file, a sha256 digest, the codes that must be reported, the codes that must not be reported, whether the package is valid, and the conformance classes the package must and must not reach. A second implementation runs the suite against its own validator and compares the result with the manifest.

How many vectors are there?

73. 8 are valid packages and 65 are invalid. Between them they expect 62 distinct diagnostic codes, and 11 carry conformance-class claims. The manifest pins a sha256 for every package, so a vector cannot drift without the manifest showing it.

What command does a second implementer run?

python3 run_suite.py --validator "path/to/your-validator --json". The --validator string is the command that validates one package and prints JSON, and the runner appends the package path as the final argument. The validator must be invoked with whatever flag makes it print JSON, because the runner reads stdout as JSON and reads nothing else. The reference implementations use --json.

Is must_report an equality test?

Every code listed MUST appear in your findings. This is a SUBSET test, not equality: your implementation MAY report additional codes and still conform. Eleven of the shipped vectors legitimately emit codes beyond those listed, because one defect often trips several independent rules.

What does valid mean in the manifest?

true means your validator reports no finding of severity error. Warnings are permitted in a valid package and some valid vectors carry them: warnings plus no errors is conforming.

What does a failing vector mean?

That the implementation and the manifest disagree about what the specification requires. One of them is wrong, and finding out which is the purpose of publishing the suite. Building the suite, a vector claimed F09_BAD_RESULT_PROVENANCE for a simulated value presented as a measurement, and the validator emitted the more specific F09_SIMULATED_AS_MEASURED. The manifest was wrong and was corrected. An implementer who disagrees with a vector and believes the specification supports them is asked to open an issue.

How does a validator declare which classes it implements?

By class and version. A claim MUST name WWD 0.1 and every claimed class, MUST identify unsupported optional features, and MUST NOT describe structural validation alone as scientific or manufacturing conformance. A package MAY conform to more than one class, and an implementation MAY conform as a producer, consumer, rewriter or evaluator. The reference validator prints the claim line under --conformance, with one PASS, FAIL or n/a per class and any unsupported feature listed beneath.

Which conformance classes exist?

Six. WWD-Package-Core, WWD-Design-Core, WWD-Field-Evaluator, WWD-As-Built, WWD-Simulation and WWD-Manufacturing-Ready, defined in section 22 of the WWD 0.1 draft.

What is the state of the reference implementations?

Two. The Python reference in validator/wwd and the TypeScript implementation in validator-ts/src. Both pass 73 of 73 vectors. The TypeScript implementation was written against the published specification and the conformance suite rather than against the Python reference, and disagreements between the two are recorded rather than reconciled in silence.

What is in a vector file?

A real ZIP package. valid-as-built.wwd carries a stored 23-byte mimetype entry, wwd/manifest.json, wwd/design.json and wwd/results.json. The manifest entry beside it records the file path, a description, the specification sections, the expectation block, the sha256 and the byte length.

What licence covers the vectors?

The vectors and the manifest are released under CC0.

Continue

Related pages.

The reference validator. Every diagnostic code and how to run it.
Design rules. The advisory DFM checks, each with its threshold, remedy and source.
The .wwd file format. The container, the manifest and the documents inside.
As-built. Designed, simulated and measured in one package.
Acceptance criteria. Statistical acceptance, recomputed verdicts, disposition and yield.
The WWD 0.1 draft specification. All normative sections with anchors.

Vectors	73
Valid packages	8
Invalid packages	65
Distinct codes expected	62
Conformance classes	6, section 22
Manifest	test-suite/index.json
Runner	test-suite/run_suite.py
Licence	CC0