LMSpeed scoring methodology
Category Score V3.0
A benchmark-only, dimension-balanced capability estimate that calibrates different benchmark cohorts through models evaluated on both, while keeping uncertainty and coverage separate.
Version 3.0 · Index range 0–100 · Uncertainty interval 80%
What the score means
Each category score estimates relative observed capability within the eligible benchmark population for a dated run. Uncertainty, coverage, and rating maturity are reported separately rather than deducted from the ability score.
It is not a success rate, IQ score, or raw average of unrelated benchmark scales.
Fixed blueprint
Every category defines four capability dimensions. Eligible benchmark families are combined inside their dimension first, then observed dimensions contribute equally to the category score. A family-rich dimension cannot outweigh the other capabilities.
| Category | Four explanatory dimensions |
|---|---|
| Agents | Planning and decomposition · Tool use · Environment and long-horizon execution · Recovery and completion reliability |
| Coding | Code generation · Repository engineering · Debugging and testing · Tool-assisted development and quality |
| Reasoning | Abstract logic · Scientific and causal reasoning · Multi-step constraints · Evidence integration and verification |
| Knowledge | Broad knowledge · Professional knowledge · Factuality · Retrieval and open-book use |
| Math | Foundational math · Competition math · Advanced proof · Applied and tool-assisted math |
| Multilingual | Cross-language understanding · Multilingual generation · Reasoning transfer · Low-resource robustness |
| Multimodal | Perception and OCR · Document and spatial understanding · Visual reasoning · Video and grounded action |
| Instruction following | Constraint adherence · Structured output · Novel instruction generalization · Multi-turn and long instructions |
Eligible evidence
V3 scores only evidence that has a declared construct and a comparable evaluation protocol. A synchronization adapter may propose a metric, but it cannot make that metric eligible by itself.
- The versioned catalog declares every eligible metric's category, dimension, canonical benchmark family, protocol, direction, transform, and scoring status. Uncatalogued metrics remain display-only.
- Aggregate, estimated, inferred, direction-unknown, and protocol-unknown values are excluded. A genuine raw score of zero remains valid evidence.
- Aliases and sources are deduplicated within a canonical family and protocol. Selection then prefers verified, official, reported, and self-reported data, followed by confidence, freshness, and stable ID.
- At least three models are required for a benchmark to score. Multiple versions and protocols from the same benchmark family share a precision cap of one inside their capability dimension.
Normalization and evidence quality
Percentage scale is read from the stored unit, never guessed from magnitude: score is divided by 100 and score_fraction is used directly. Unsupported percentage units are excluded. Fractions receive a clamped logit; ratings keep their declared identity scale, and lower-is-better metrics are reversed. Values are centered on the median and robustly scaled within each run; final z-scores are clamped to −3 through 3.
Evidence quality q combines model coverage and developer diversity. Measurement variance uses the larger of normalized reported error and a source floor, divided by q. Source standard-deviation floors are 0.25 verified, 0.35 official, 0.50 reported, and 0.75 self-reported.
fraction = raw / 100 if unit = score
fraction = raw if unit = score_fraction
q = min(1, n / 8) × min(1, developerCount / 3)
z = clamp((x − median) / robustScale, −3, 3)
variance = max(normalizedStderr², sourceFloor²) / q
How it is calculated
- Only catalogued metrics with a known protocol, direction, and construct are eligible; aggregate, inferred, estimated, and display-only rows are excluded.
- Each benchmark is robustly standardized across at least three models. Source quality, model coverage, developer diversity, and reported error determine its variance.
- Repeated versions and protocols from one benchmark family share one precision cap inside their dimension, preventing duplicates from manufacturing score or confidence.
- Within each dimension, a score-based Gaussian model jointly estimates model ability and benchmark-cohort offsets from overlapping models. It uses no parameter-count, vendor, release-date, or model-family prior. The category mean remains the equal-weight mean of observed dimensions.
- A numeric category score requires at least two observed dimensions and two independent families. Missing dimensions widen the 80% interval instead of being counted as measured failure.
precisionf = min(1, Σi∈f(1 / variancei))
zmb = θmd + βb + εmb
θmd ~ N(0, 1)
βb ~ N(0, 10)
εmb ~ N(0, 1 / precisionmb)
μd = E[θmd | eligible benchmark evidence]
σd² = Var[θmd | eligible benchmark evidence]
k = observedDimensionCount, k ≥ 2
μc = (1 / k) × Σd∈observed μd
σc² = Σd∈observed(σd² / k²) + (4 − k) / 16
score = clamp(50 + 15 × μc, 0, 100)
central80 = [μc − 1.281552 × σc, μc + 1.281552 × σc]
Rated: observedDimensions ≥ 3 and familyCount ≥ 4 and overlapCalibratedDimensions ≥ 2
How to read a score
- Rated requires at least three observed dimensions, four independent families, and overlap calibration in at least two dimensions. Estimated requires at least two dimensions and two families. Narrower evidence is Unrated and shown as a dash, while its benchmark details remain available.
- Measured dimensions shows how many of the four category capabilities have direct evidence. Observed dimensions contribute equally; additional families improve a dimension estimate but do not give it more category weight.
- The 80% interval communicates uncertainty. Family count and effective coverage indicate whether the estimate rests on several independent signals or a narrow evidence base.
- Only Rated scores receive a formal global rank, ordered by unrounded category mean, effective coverage, lower-bound diagnostic, and model ID. Estimated scores remain sortable but unranked. No overall score is created by averaging the eight categories.
Limits and interpretation
V3 uses existing third-party benchmarks and inherits their task selection, contamination, evaluator, and protocol limitations. Cohort offsets can only be learned where benchmark populations overlap; disconnected evidence retains wider uncertainty and cannot qualify a score for formal rank. Estimated scores must be read with their interval, measured dimensions, families, and evidence details.
Scores are relative to the eligible model population in a specific run. Cross-time comparisons must cite both the run date and methodology version.