Benchmarking¶

p2w has two benchmark suites. The first measures p2w against CPython on the programs the compiler is tuned on; the second measures it against other runtimes and native compilers on the Benchmarks Game programs. Both check that every runtime prints the same output before reporting a time.

p2w against CPython¶

programs/benchmarks/ holds eleven programs: binarytrees, fannkuch, fasta, fibonacci, mandelbrot, matmul, nbody, primes, pystone, sieve and spectralnorm. Six of them also have an annotated *_native.py variant, which the runner uses. The latest figures are on the Performance page.

uv run python programs/benchmarks/run_benchmarks.py                 # run, print a table
uv run python programs/benchmarks/run_benchmarks.py --save -m "why"  # and record it
uv run python programs/benchmarks/run_benchmarks.py --list           # recorded sessions
uv run python programs/benchmarks/run_benchmarks.py --compare 44     # #44 against the latest
uv run python programs/benchmarks/run_benchmarks.py --warmup 2 --runs 5

Each benchmark is compiled once, then run --warmup times untimed and --runs times timed (3 and 10 by default), under both CPython and Node.js. The table gives medians, the speed-up of each program, and the geometric mean of the speed-ups. --save stores the session in programs/benchmarks/benchmark_results.db, a SQLite file.

The Benchmarks Game suite¶

programs/benchmarks-alioth/ holds five programs from the Computer Language Benchmarks Game: binarytrees, fannkuchredux, mandelbrot, nbody and spectralnorm, in Python and in C and Zig for the native baselines. suite.yaml lists them with their arguments. p2w-benchmark runs them:

uv run p2w-benchmark runtimes                      # which runtimes are installed
uv run p2w-benchmark run                           # p2w and CPython
uv run p2w-benchmark run --python cpython,pypy --native gcc,zig
uv run p2w-benchmark run --benchmark nbody --save -d "nbody after the change"
uv run p2w-benchmark list                          # saved sessions
uv run p2w-benchmark compare 3 5                   # session 3 against session 5
uv run p2w-benchmark check --threshold 0.05        # fail on a regression over 5%

run repeats each measurement until the coefficient of variation falls under --cv-target (1% by default), between --min-runs and --max-runs runs. check compares the two latest saved sessions and exits with an error when a benchmark got slower by more than the threshold. Results go to programs/benchmarks-alioth/benchmark_results.db unless --db says otherwise.