Installing ScarfBench
ScarfBench uses the scarf command line tool to pull benchmark assets, run evaluations, and work with submissions.
Install Options
Section titled “Install Options”Choose one installation method.
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/scarfbench/scarf/releases/latest/download/scarfbench-cli-installer.sh | shbrew tap scarfbench/tapbrew install scarfbench-clicargo install scarfbench-clinpm install @scarfbench/scarfbench-cligit clone https://github.com/scarfbench/scarf.gitcd scarfcargo install --path . --root "$HOME/.local"Add $HOME/.local/bin to your PATH if needed.
Verify Installation
Section titled “Verify Installation”scarf --helpYou should see CLI help output. For most users, the core commands needed post install are:
| Group | Command | Purpose | Link |
|---|---|---|---|
| Benchmark setup | scarf bench pull | Download benchmark files for local evaluation | Command help |
| Agent evaluation | scarf eval | Evaluate agents on benchmark | Go to section |
scarf eval run | Run agent evaluation jobs | Command help |
Getting the Benchmark
Section titled “Getting the Benchmark”Use scarf bench pull to download the benchmark to a local directory.
Basic usage
Section titled “Basic usage”scarf bench pull --dest /path/to/scarfbench-dataThis creates or updates a local benchmark checkout in /path/to/scarfbench-data.
Pull a specific benchmark version
Section titled “Pull a specific benchmark version”scarf bench pull --dest ./scarfbench-data --version <VERSION>Replace <VERSION> with the release or version identifier you want to evaluate against.
Parameters
Section titled “Parameters”--dest <DIR>: required destination directory where benchmark files will be pulled--version <VERSION>: optional version selector; if omitted, the latest available version is pulled
Next Step
Section titled “Next Step”After pulling, use scarf eval ... commands to run your agent evaluations. Continue with the Quickstart Guide to run your first evaluation.