A cross-browser, cross-system web benchmark runner
Project description
Crossbench
Crossbench is a cross-browser/cross-benchmark runner to extract performance numbers.
Mailing list: crossbench@chromium.org
Issues/Bugs: Tests > CrossBench
Supported Browsers: Chrome/Chromium, Firefox, Safari and Edge.
Supported OS: macOS, Android, linux and windows.
Basic usage:
Chromium Devs (with a full chromium checkout)
Use the ./cb.py
script directly to run benchmarks (requires chrome's
vpython3)
Standalone installation
- Use
pip install crossbench
, - or use the "poetry" package manager, see the development section.
Run the latest speedometer benchmark 20 times with the system default browser (chrome-stable):
# Run chrome-stable by default:
./cb.py speedometer --repeat=20
# Compare chrome browser versions and a local chrome build on jetstream:
./cb.py jetstream --browser=chrome-stable --browser=chrome-m90 --browser=$PATH
Profile individual line items (with pprof on linux):
./cb.py speedometer --probe='profiling' --separate
Use a custom chrome build and only run a subset of the stories:
./cb.py speedometer --browser=$PATH --probe='profiling' --story='jQuery.*'
Profile a website for 17 seconds on Chrome M100 (auto-downloading on macOS and linux):
./cb.py loading --browser=chrome-m100 --probe='profiling' --url=www.cnn.com,17s
Main Components
Browsers
Crossbench supports running benchmarks on one or multiple browser configurations. The main implementation uses selenium for maximum system independence.
You can specify a browser with --browser=<name>
. You can repeat the
--browser
argument to run multiple browser. If you need custom flags for
multiple browsers use --browser-config
(or pass simple flags after --
to
the browser).
./cb.py speedometer --browser=$BROWSER -- --enable-field-trial-config
Browser Config File
For more complex scenarios you can use a browser.config.hjson file. It allows you to specify multiple browser and multiple flag configurations in a single file and produce performance numbers with a single invocation.
./cb.py speedometer --browser-config=config.hjson
The example file lists and explains all configuration details.
Probes
Probes define a way to extract arbitrary (performance) numbers from a host or running browser. This can reach from running simple JS-snippets to extract page-specific numbers to system-wide profiling.
Multiple probes can be added with repeated --probe='XXX'
options.
You can use the describe probes
subcommand to list all probes:
# List all probes:
./cb.py describe probes
# List help for an individual probe:
./cb.py describe probe v8.log
Inline Probe Config
Some probes can be configured, either with inline json when using --probe
or
in a separate --probe-config
hjson file. Use the describe
command to list
all options.
# Get probe config details:
./cb.py describe probe v8.log
# Use inline hjson to configure a probe:
./cb.py speedometer --probe='v8.log:{prof:true}'
Probe Config File
For complex probe setups you can use --probe-config=<file>
.
The example file lists and explains all
configuration details. For the specific probe configuration properties consult
the describe
command.
Benchmarks
Use the describe
command to list all benchmark details:
# List all benchmark info:
./cb.py describe benchmarks
# List an individual benchmark info:
./cb.py describe benchmark speedometer_3.0
# List a benchmark's command line options:
./cb.py speedometer_3.0 --help
Stories
Stories define sequences of browser interactions. This can be simply loading a URL and waiting for a given period of time, or in more complex scenarios, actively interact with a page and navigate multiple times.
Use --help
or describe to list all stories for a benchmark:
./cb.py speedometer --help
Use --stories
to list individual story names, or use regular expression
as filter.
./cb.py speedometer --browser=$BROWSER --stories='.*Angular.*'
Development
Setup
This project uses poetry deps and package scripts to setup the correct environment for testing and debugging.
# a) On debian:
sudo apt-get install python3.10 python3-poetry
# b) With python 3.8 to 3.10 installed already:
pip3 install poetry
Check that you have poetry on your path and make sure you have the right
$PATH
settings.
poetry --help || echo "Please update your \$PATH to include poetry bin location";
# Depending on your setup, add one of the following to your $PATH:
echo "`python3 -m site --user-base`/bin";
python3 -c "import sysconfig; print(sysconfig.get_path('scripts'))";
Install the necessary dependencies from the lock file using poetry:
# Select the python version you want to use (3.8 to 3.10):
poetry env use 3.10
poetry install
# For python 3.11 you have to skip pytype support:
poetry env use 3.11
poetry install --without=dev-pytype
Crossbench
For local development / non-chromium installation you should
use poetry run cb ...
instead of ./cb.py ...
.
Side-note, beware that poetry eats up an empty --
:
# With cb.py:
./cb.py speedometer ... -- --custom-chrome-flag ...
# With poetry:
poetry run cb speedometer ... -- -- --custom-chrome-flag ...
Tests
poetry run pytest
Run detailed test coverage:
poetry run pytest --cov=crossbench --cov-report=html
Run pytype type checker:
poetry run pytype -j auto .
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for crossbench-1.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93fce6147b58f1456cfe3a856be1184cba3df313238b05075c3a48ee958ff705 |
|
MD5 | 43e15b51476c0a9da3547f89f49267db |
|
BLAKE2b-256 | 5ca76f2f357d52feba65933cbea0cf30bd1d40a3fbfe32ddd077a1e5f73073f2 |