Expert-refined questions
525 literature-based cold spray questions.
Evaluating how language models reason about cold spray materials and manufacturing.
SprayBench evaluates language models on open-ended cold spray questions involving process conditions, materials, and mechanical performance. An LLM judge scores answers against references, with a blinded expert audit to assess reliability. Retrieval tests compare settings where the question’s source document is excluded or eligible for retrieval.
525 literature-based cold spray questions.
Compare answers with and without access to the source document.
Reference-guided judging with a blinded expert audit.
Expanded results from Stephen Price, Bryer C. Sousa, Ronan Flynn, Kiran G. Judd, Elke A. Rundensteiner, and Danielle L. Cote, SprayBench: Benchmarking LLMs on Open-Ended Materials Reasoning in Additive Manufacturing, NeurIPS 2026 AI4Mat Workshop. Research conducted while at Worcester Polytechnic Institute.
Questions, references, and outputs on GitHubClosed-book model comparison
* Indicates a high refusal rate. See the leaderboard for details.
Pass rate is the share of evaluated responses judged to pass; known errors and refusals count as failures. Score is the average overall-quality rating (0–5). Each row shows its evaluated question count. The base results snapshot is dated August 18, 2026; the expanded results differ from the paper in configurations and review coverage. Latency and speed depend on hardware, provider, and settings.
Scroll within the table to see all models.
| Rank |
|---|
Retrieval
Source-eligible does not guarantee source retrieval.
| Rank by excluded gain |
|---|
Atomix Research Systems