SprayBench

Evaluating how language models reason about cold spray materials and manufacturing.

Benchmark
525 questions
Model configurations tested
-- configs
Retrieval comparison
-- settings

About the paper

SprayBench evaluates language models on open-ended cold spray questions involving process conditions, materials, and mechanical performance. An LLM judge scores answers against references, with a blinded expert audit to assess reliability. Retrieval tests compare settings where the question’s source document is excluded or eligible for retrieval.

Expert-refined questions

525 literature-based cold spray questions.

Source-controlled retrieval

Compare answers with and without access to the source document.

Calibrated evaluation

Reference-guided judging with a blinded expert audit.

Expanded results from Stephen Price, Bryer C. Sousa, Ronan Flynn, Kiran G. Judd, Elke A. Rundensteiner, and Danielle L. Cote, SprayBench: Benchmarking LLMs on Open-Ended Materials Reasoning in Additive Manufacturing, NeurIPS 2026 AI4Mat Workshop. Research conducted while at Worcester Polytechnic Institute.

Questions, references, and outputs on GitHub

Overall pass rate

Closed-book model comparison

Variants

* Indicates a high refusal rate. See the leaderboard for details.

Model leaderboard

Pass rate is the share of evaluated responses judged to pass; known errors and refusals count as failures. Score is the average overall-quality rating (0–5). Each row shows its evaluated question count. The base results snapshot is dated August 18, 2026; the expanded results differ from the paper in configurations and review coverage. Latency and speed depend on hardware, provider, and settings.

Scroll within the table to see all models.

Primary SprayBench model leaderboard
Rank

Open Weight Models

API Models

MoE

Verbosity

Token Generation Speed

RAG Gains

Retrieval

Retrieval comparison

Source-eligible does not guarantee source retrieval.

SprayBench RAG model comparison
Rank by excluded gain

Comparing architecture performance and efficiency

Atomix Research Systems

Discuss a benchmark or evaluation project.

Discuss a project