inliver233
1dc1d9b707
A reviewer reported seven drifting commands in the evidence record. Reproduced, the report split
three ways, and the middle category was the useful one: the gate had been scoring them clean.
Not reproducible, with the measurement:
tests/run_tests.py -q runs. The wrapper forwards its arguments to pytest
(`args = list(argv[1:]) or ["-q"]`), and documents that in its own usage.
dcc.py already written as `cd tools/_work/bench/repos/dcc` followed by
`python dcc.py ...` -- an external checkout, not a shipped script.
Real, and fixed here: seven commands in two evidence records named a **workbench** script with no
path at all (`analyze_pair.py`, `make_skeleton.py`, `ttd_attach.py`, `b2_measure.py`). The files
exist under `tools/`, which .gitignore excludes, so a reader copying one from the repository root
gets "No such file" -- and two of those documents wrote the correct path elsewhere, so they
disagreed with themselves.
**The actual defect was in `check_commands.py`, not in the seven lines.** `resolve_script()` tried
the document's own directory first, so a bare `analyze_pair.py` inside `docs/tool-verification/`
resolved to `tools/_work/bench/unpack/b3/analyze_pair.py` -- a real file -- and passed. Resolution
now follows the order a reader would try (repository root, then the skill's scripts, then the
document's own directory, which is what `../x.py` in a nested README legitimately means); a bare
name that only resolves under `tools/` is `unqualified-workbench`; and that class counts as **drift,
not as a skip**, because it is a one-token fix that a reader cannot run. The first attempt classified
it as a low-confidence warning, which would have left all seven in place.
One exception is honoured and pinned: a code block opening with `cd tools/_work/bench/repos/dcc` is
telling the reader where to stand, so a bare `python dcc.py` after it is correct.
`documented_commands()` now reports whether a `cd` preceded the command in the same block, and that
case is skipped as `workbench-after-cd`.
Two defects of this pass's own, found while fixing it:
--json emitted the RESULT= token after the JSON document, so `--json | jq` failed with "Extra
data". The token is gone from that path; the status is a field in the document, and the exit code
carries the same meaning.
The fix's own substitution left four literal backslashes before `$` in the two files it edited --
a character a reader would copy by mistake. Removed, and verified absent.
`tests/integration/test_check_commands_gate.py` pins the resolution order, the drift classification,
the `cd` exception, and a clean pass over the committed tree, so this cannot come back silently.
Gates: check_commands 153 checked / 0 drift, check_repo 0 problem, check_refs 0 dangling / 0 warning,
check_routing coherent, check_budget 0 failure / 0 warning, pytest 613 passed / 158 skipped / 25
xfailed.
tests/ - the executable half of this repository's evidence
tests/benchmark.md is a record: what happened when a route was run once, by hand,
against a real sample and a real device. This directory is the other half: checks that run
on every commit with nothing but the standard library and pytest.
The two are not substitutes. A benchmark row can establish that a route works on a hardened target; it cannot tell you that yesterday's refactor broke the dex width table. A test in here can tell you that every time, and it can never tell you whether the route achieves the goal on a real app.
python -m pytest -q # whole suite
python tests/run_tests.py # same thing, with a clear message if pytest is missing
python -m pytest -q -m unit # one marker
python -m pytest -q -m cli # exit codes, flags, --json
python -m pytest -q -m integration # no-device flows
python -m pytest -q -ra # list every skip and xfail reason
What the markers mean
unit- pure functions and parsers of real shipped scripts: the dex width table and branch arithmetic, the scheme-less protobuf decoder, the blob rotation search, the leak rules, and the establisheddex_patch_bytesregression hash.cli- one subprocess run per script per invocation shape:--help, an unknown flag, no arguments, and every input pointed at a file that does not exist.integration- flows that were made of several parts: adb argv construction under a fake runner, logcat signal counting, tombstone parsing, the two-sample launch-health verdict.
Nothing here needs a device
No test in this suite opens a frida session or touches the network. The device-dependent behaviour is tested by intercepting the command that would have been run and asserting on the argv list.
The exception is deliberate and enforced: contract.DEVICE_OR_NETWORK_BOUND names the 15
scripts that reach a device once their arguments parse, and those are never executed with
complete arguments — only --help and an unknown flag, which argparse answers before any
device call. That blocklist exists because a development host may well have a phone attached
(the machine this harness was written on did), and an earlier version of the CLI contract test
drove it for real. General safety net: every child runs in a private temporary working
directory, so a script that defaults an output path relative to cwd cannot write into the
shipped tree.
Anything that genuinely cannot be reached this way is not faked into a pass — it is listed as a
coverage gap in docs/tool-verification/EXTENSION-test-harness.md.
Skipped is not passed
Two kinds of non-passing outcome are kept separate on purpose:
skip- the check does not apply here.tools/_work/bench/**is git-ignored, so thedex_patch_bytesregression hash self-skips on a fresh checkout and in CI, printing the reason. A --json script that needs a sample to reach its JSON path skips too.xfail- the script is present and does not meet the repository contract. The reason carries the measured value, for examplecontract says exit 2, measured 1. These are work items, not flaky tests.
pytest -q -ra prints both lists. The xfail list for the CLI layer is the exit-code
unification work order; it is reproduced in EXTENSION-test-harness.md with owners.
Ground rules
- This suite never edits anything under
skills/. A defect found here is registered in the harness report. Fixing a script's exit code or quoting strategy is its owner's change, and doing it from a test would hide the drift the test exists to expose. - The bench tree is evidence and is read-only here. The regression test copies its input
into
tmp_pathbefore patching, sotools/_work/bench/l1l3/dex/classes.patched.dexcannot be overwritten by a test run. - No real target identity is committed. The leak-scanner decoys are synthesised in
tmp_pathat run time, and token-shaped decoys are assembled from fragments so that this directory cannot itself become a finding. - Fixtures in
tests/fixtures/are small, synthetic sample files.tests/fixtures/apk/**is owned by a different work item and is not touched from here.