Skip to content
ScarfBench

Scarf (short for Self-Contained Application Refactoring) benchmark is a suite of Java applications across frameworks: Jakarta EE, Quarkus, and Spring for evaluating agentic transformation between the frameworks. This suite enables systematic assessment of AI agents’ ability to migrate enterprise Java applications while preserving functionality, idiomatic patterns, and architectural integrity across different runtime environments.

The benchmark includes comprehensive examples ranging from focused layer-specific demonstrations to complete production-grade applications, each with verified implementations across all supported frameworks.

Manual Conversions with Developer Verification All applications in this benchmark have been manually converted and verified by experienced developers. Each implementation has undergone rigorous testing to ensure functional correctness, adherence to framework-specific idioms, and preservation of architectural integrity across Jakarta EE, Quarkus, and Spring frameworks.

Getting Started with the Benchmark

Section titled “Getting Started with the Benchmark”

Quickstart A quick walkthrough to get you started with ScarfBench, including how to set up the environment and run your first benchmark evaluation.

Installing ScarfBench Step-by-step instructions for installing ScarfBench on your local machine, including system requirements and troubleshooting tips.

Leaderboard Explore the current leaderboard to see how different solutions are performing on the ScarfBench benchmark, and compare results across various metrics and frameworks.

Submit Solutions Submit your agentic transformation solutions to the ScarfBench benchmark and contribute to the community by sharing your results and insights with other researchers and practitioners.

Citation

Section titled “Citation”

If you use scarfbench, please consider using the following citation.

@misc{scarfbench,
  title        = {ScarfBench},
  author       = {{IBM}},
  year         = {2026},
  howpublished = {\url{https://github.com/scarfbench}},
  note         = {GitHub repository}
}

Contact us!

Section titled “Contact us!”

For any questions, feedback, or suggestions, please contact the authors:

NameEmail
Rahul Krishnai.m.ralk@gmail.com
Bridget McGinnbridget.mcginn@ibm.com
Raju Pavuluripavuluri@us.ibm.com