Testing¶

p2w's tests answer one question: does the compiled program print what CPython prints? Most of them compile a short program, run it under Node.js, run the same source under CPython, and compare the two outputs.

make test                          # the whole suite (uv run pytest tests)
uv run pytest tests/b_integration/test_generators.py
uv run pytest -m unit              # or -m integration, -m e2e
make test-cov                      # with a coverage report in htmlcov/
make lint                          # ruff, ruff format --check, ty, mypy

The integration and end-to-end tests need wasm-tools and Node.js 22 or later; the tests that need them are skipped when they are missing.

Layout¶

The suite is a pyramid, and tests/conftest.py marks each test by its directory:

tests/a_unit/ (marker unit): No I/O and no Node.js: type inference, analysis, the inliner, the tree shaker, and checks on the emitted WAT, such as every called function being defined.
tests/b_integration/ (marker integration): Compile and run. One file per area of the language (test_closures.py, test_pattern_matching.py, test_large_integers.py, …), plus test_programs.py, which runs the programs in programs/internal/, and test_unsupported.py, which checks that the constructs p2w refuses raise NotImplementedError at compile time.
tests/c_e2e/ (marker e2e): The p2w command line.

The golden programs¶

programs/internal/ holds 104 programs, each exercising one area: classes, generators, exceptions, comprehensions, string methods, pattern matching, and so on. tests/b_integration/test_programs.py runs each one under p2w and CPython. A program in WORKING_PROGRAMS must match exactly; one in OUT_OF_SCOPE is skipped. Today 102 programs are working, and the two out of scope are async.py and generator_protocol.py.

When a fix makes another program match, add it to WORKING_PROGRAMS, so the next regression fails the build.

Writing a test¶

The helpers in p2w.testing do the comparison:

from p2w.testing import compare_outputs


def test_floor_division_of_negatives():
    py, p2w, same = compare_outputs("print(-7 // 2, -7 % 2)")
    assert same, (py, p2w)

Assert on the output CPython gives, not on the output you expect: CPython is the specification.

Property-based tests¶

tests/a_unit/test_properties.py and tests/b_integration/test_properties.py use Hypothesis to generate expressions and check that p2w and CPython agree on them. They found the floor-division and modulo sign bugs, which no hand-written test had caught (see Lessons learned).

Idiom sweeps¶

Feature tests check what each feature's author thought of. A sweep checks what programs do: a few dozen three-line programs of everyday Python, each run under p2w and CPython, with the differences sorted into wrong output, crashes, and refusals. Wrong output and crashes come first. The known bugs page is kept up to date from these sweeps.