On this page
1Before you start
- Lemonade installed and running on the machine you want to measure. LemonMatrix only talks to Lemonade's HTTP API; it never reads the hardware itself.
- The model you want to benchmark, pulled in Lemonade. The most compared model on this board is
Qwen3-1.7B-GGUF:lemonade pull Qwen3-1.7B-GGUF - Python 3.10 or newer, and
git. - To submit with one command: the GitHub CLI,
logged in with
gh auth login.
2Install LemonMatrix
Use a virtual environment so nothing touches your system Python.
Linux and macOS
python3 -m venv lemonmatrix-env
source lemonmatrix-env/bin/activate
pip install lemonmatrix
Windows (PowerShell)
py -m venv lemonmatrix-env
lemonmatrix-env\Scripts\Activate.ps1
pip install lemonmatrix
If PowerShell refuses to run Activate.ps1, allow scripts for this window only with
Set-ExecutionPolicy -Scope Process Bypass and run it again.
Check it works with lemonmatrix --version. LemonMatrix is on
PyPI; to upgrade later, run
pip install -U lemonmatrix.
3Create a profile
A profile is one Lemonade instance. LemonMatrix asks Lemonade for the machine's CPU, GPU, NPU, OS and Lemonade version once and attaches that to every result from this profile.
lemonmatrix profile detect --save Radeon-RX-7900-XTX
detect scans localhost on Lemonade's usual ports (13305 first, then the older 8000). If Lemonade runs
on another machine, or needs an API key, add it directly:
lemonmatrix profile add Radeon-RX-7900-XTX --host 192.168.1.42 --port 13305 --token $LEMONADE_API_KEY
Check what was discovered:
lemonmatrix profile show Radeon-RX-7900-XTX
The profile name becomes the folder under results/ and the Profile column on the
leaderboard. Name it after the hardware, not after yourself or your computer's hostname.
4Run an LLM benchmark
This measures prefill speed, decode speed and time to first token over 5 trials, after 2 warm-up requests:
lemonmatrix run --profile Radeon-RX-7900-XTX \
--model Qwen3-1.7B-GGUF --model-class dense --quant Q4_0 \
--context-length 4096 --engine dgpu --backend llamacpp-vulkan \
--power-state plugged
| Flag | What to put |
|---|---|
--model | The model name exactly as Lemonade lists it (lemonade list). |
--model-class | dense or moe. |
--quant | The quantization in the model's file, e.g. Q4_0 or Q4_K_M. |
--engine | Where it runs: cpu, igpu, dgpu, npu or hybrid. |
--backend | The backend Lemonade uses, e.g. llamacpp-vulkan, llamacpp-rocm, llamacpp-cpu. |
--power-state | plugged, battery or power_capped (with --power-cap-w). |
--trials, --warmup | Optional. Defaults are 5 and 2. |
The result is written to results/<profile>/<run_id>.json, and the command prints the speeds
and whether the run is valid. Run the same model on another backend or engine to get a comparison row.
5Make the run valid
Only valid runs are ranked. LemonMatrix marks a run invalid, and says why in validity.notes, when:
- another model or app was using the GPU or NPU during the measured trials;
- the device Lemonade reports doesn't match
--engine, or the backend doesn't fit the engine; - Lemonade couldn't say which backend served the model;
- the model was reloaded in the middle of the run;
- the quantization doesn't match
--quant; - the hardware was thermally throttled.
Close other GPU-heavy apps, keep a laptop plugged in, and let Lemonade load only the model being
measured. --force runs past a mismatch, but the result is then always marked invalid.
6Other modalities
Speech, image, embedding and other models have their own commands, because their speed is not measured in tokens. Each writes to its own folder and shows up under Other modalities:
| Command | Measures |
|---|---|
lemonmatrix tts | Text to speech, seconds of audio per second |
lemonmatrix stt | Speech to text, seconds of audio per second |
lemonmatrix audiogen | Music and audio generation, seconds of audio per second |
lemonmatrix imagegen | Images per second |
lemonmatrix embeddings | Embeddings per second |
lemonmatrix rerank | Documents per second |
lemonmatrix classify | Classifications per second |
lemonmatrix meshgen | 3D meshes per second |
Each command's --help shows an example, for instance:
lemonmatrix tts --profile Radeon-RX-7900-XTX --model kokoro-v1 \
--engine cpu --backend kokoro-cpu --power-state plugged \
--input-text "Lemonade can speak"
7Router cost runs
A Lemonade collection.router can send easy prompts to a local model and hard ones to a cloud model.
A router run records where each request went and what that cost, compared with sending everything to the most
expensive candidate:
lemonmatrix run --profile Radeon-RX-7900-XTX --run-type router \
--model user.My-Router --context-length 4096 --power-state plugged \
--prompt-set router-mix --trials 20
- Cloud prices come from Lemonade's cost metadata.
--cost MODEL=IN,OUTsets a price in USD per million input and output tokens; local models count as free. - After the trials, each prompt the router served locally is sent once to the baseline model, so the
comparison uses the tokens that model really writes.
--estimate-baselineskips these extra requests. - Cloud API keys stay in Lemonade's environment (
LEMONADE_<PROVIDER>_API_KEY); they are never written into the result.
8Check and submit
Check the file against the same schema the leaderboard's CI uses:
lemonmatrix validate-submission results/Radeon-RX-7900-XTX/<run_id>.json
Preview the pull request, then open it:
lemonmatrix submit results/Radeon-RX-7900-XTX/<run_id>.json --dry-run
lemonmatrix submit results/Radeon-RX-7900-XTX/<run_id>.json
submit forks
LemonMatrix/lemonmatrix.github.io
(or reuses your fork), adds the file under the same results/<profile>/ path, and opens a PR. CI
validates it again; once it is merged, this site rebuilds and your run appears.
Without the GitHub CLI
- Fork the repository on GitHub.
- Add your file at
results/<profile>/<run_id>.json(modality results go inresults/<profile>/<modality>/, e.g.tts/). - Open a pull request to
main.
9What gets published
The result file is public once merged. It holds:
- the hardware names Lemonade reports (CPU, GPU, NPU), memory size, OS version, driver, Lemonade and backend versions;
- the model, quantization, engine, backend and run settings;
- the measured numbers and the validity checks.
It does not hold your hostname, IP address, user name, the model's answers or API keys. Open the JSON and read it before you submit; it is plain text.
10Troubleshooting
| Problem | Fix |
|---|---|
profile detect finds nothing | Check Lemonade is running and note its port, then use
profile add --port. |
| 401 or 403 from Lemonade | The instance has LEMONADE_API_KEY set; pass it with
--token. |
| The run is marked invalid | Read validity.notes in the JSON, fix the cause (see step 5) and
run again. |
submit fails with "Permission denied (publickey)" | The GitHub CLI clones over SSH without
a key it can use. Run gh config set git_protocol https and try again. |
Still stuck? Open an issue on Bekhouche/LemonMatrix.