๐Ÿ‹ How to submit a result

Benchmark your own Lemonade instance with LemonMatrix and add the result to the public leaderboard. Each step below is a command you run on the machine you want to measure.

On this page

  1. Before you start
  2. Install LemonMatrix
  3. Create a profile
  4. Run an LLM benchmark
  5. Make the run valid
  6. Other modalities
  7. Router cost runs
  8. Check and submit
  9. What gets published
  10. Troubleshooting

1Before you start

2Install LemonMatrix

Use a virtual environment so nothing touches your system Python.

Linux and macOS

python3 -m venv lemonmatrix-env
source lemonmatrix-env/bin/activate
pip install lemonmatrix

Windows (PowerShell)

py -m venv lemonmatrix-env
lemonmatrix-env\Scripts\Activate.ps1
pip install lemonmatrix

If PowerShell refuses to run Activate.ps1, allow scripts for this window only with Set-ExecutionPolicy -Scope Process Bypass and run it again.

Check it works with lemonmatrix --version. LemonMatrix is on PyPI; to upgrade later, run pip install -U lemonmatrix.

3Create a profile

A profile is one Lemonade instance. LemonMatrix asks Lemonade for the machine's CPU, GPU, NPU, OS and Lemonade version once and attaches that to every result from this profile.

lemonmatrix profile detect --save Radeon-RX-7900-XTX

detect scans localhost on Lemonade's usual ports (13305 first, then the older 8000). If Lemonade runs on another machine, or needs an API key, add it directly:

lemonmatrix profile add Radeon-RX-7900-XTX --host 192.168.1.42 --port 13305 --token $LEMONADE_API_KEY

Check what was discovered:

lemonmatrix profile show Radeon-RX-7900-XTX

The profile name becomes the folder under results/ and the Profile column on the leaderboard. Name it after the hardware, not after yourself or your computer's hostname.

4Run an LLM benchmark

This measures prefill speed, decode speed and time to first token over 5 trials, after 2 warm-up requests:

lemonmatrix run --profile Radeon-RX-7900-XTX \
  --model Qwen3-1.7B-GGUF --model-class dense --quant Q4_0 \
  --context-length 4096 --engine dgpu --backend llamacpp-vulkan \
  --power-state plugged
FlagWhat to put
--modelThe model name exactly as Lemonade lists it (lemonade list).
--model-classdense or moe.
--quantThe quantization in the model's file, e.g. Q4_0 or Q4_K_M.
--engineWhere it runs: cpu, igpu, dgpu, npu or hybrid.
--backendThe backend Lemonade uses, e.g. llamacpp-vulkan, llamacpp-rocm, llamacpp-cpu.
--power-stateplugged, battery or power_capped (with --power-cap-w).
--trials, --warmupOptional. Defaults are 5 and 2.

The result is written to results/<profile>/<run_id>.json, and the command prints the speeds and whether the run is valid. Run the same model on another backend or engine to get a comparison row.

5Make the run valid

Only valid runs are ranked. LemonMatrix marks a run invalid, and says why in validity.notes, when:

Close other GPU-heavy apps, keep a laptop plugged in, and let Lemonade load only the model being measured. --force runs past a mismatch, but the result is then always marked invalid.

6Other modalities

Speech, image, embedding and other models have their own commands, because their speed is not measured in tokens. Each writes to its own folder and shows up under Other modalities:

CommandMeasures
lemonmatrix ttsText to speech, seconds of audio per second
lemonmatrix sttSpeech to text, seconds of audio per second
lemonmatrix audiogenMusic and audio generation, seconds of audio per second
lemonmatrix imagegenImages per second
lemonmatrix embeddingsEmbeddings per second
lemonmatrix rerankDocuments per second
lemonmatrix classifyClassifications per second
lemonmatrix meshgen3D meshes per second

Each command's --help shows an example, for instance:

lemonmatrix tts --profile Radeon-RX-7900-XTX --model kokoro-v1 \
  --engine cpu --backend kokoro-cpu --power-state plugged \
  --input-text "Lemonade can speak"

7Router cost runs

A Lemonade collection.router can send easy prompts to a local model and hard ones to a cloud model. A router run records where each request went and what that cost, compared with sending everything to the most expensive candidate:

lemonmatrix run --profile Radeon-RX-7900-XTX --run-type router \
  --model user.My-Router --context-length 4096 --power-state plugged \
  --prompt-set router-mix --trials 20

8Check and submit

Check the file against the same schema the leaderboard's CI uses:

lemonmatrix validate-submission results/Radeon-RX-7900-XTX/<run_id>.json

Preview the pull request, then open it:

lemonmatrix submit results/Radeon-RX-7900-XTX/<run_id>.json --dry-run
lemonmatrix submit results/Radeon-RX-7900-XTX/<run_id>.json

submit forks LemonMatrix/lemonmatrix.github.io (or reuses your fork), adds the file under the same results/<profile>/ path, and opens a PR. CI validates it again; once it is merged, this site rebuilds and your run appears.

Without the GitHub CLI

  1. Fork the repository on GitHub.
  2. Add your file at results/<profile>/<run_id>.json (modality results go in results/<profile>/<modality>/, e.g. tts/).
  3. Open a pull request to main.

9What gets published

The result file is public once merged. It holds:

It does not hold your hostname, IP address, user name, the model's answers or API keys. Open the JSON and read it before you submit; it is plain text.

10Troubleshooting

ProblemFix
profile detect finds nothingCheck Lemonade is running and note its port, then use profile add --port.
401 or 403 from LemonadeThe instance has LEMONADE_API_KEY set; pass it with --token.
The run is marked invalidRead validity.notes in the JSON, fix the cause (see step 5) and run again.
submit fails with "Permission denied (publickey)"The GitHub CLI clones over SSH without a key it can use. Run gh config set git_protocol https and try again.

Still stuck? Open an issue on Bekhouche/LemonMatrix.