> ## Documentation Index
> Fetch the complete documentation index at: https://gladia-95-jl-ml-93-amd-docs-clean.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarking

> A clear 5-step method to compare speech-to-text accuracy fairly

Use one dataset with human ground truth, and the same normalization before comparing providers. Otherwise scores are not comparable.

## Methodology

<Steps>
  <Step title="Collect representative data" icon="list">
    Use production audio that matches your traffic: noise, overlap, accents, and domains. Keep it under a DPA for the eval window, then flush it.
  </Step>

  <Step title="Human annotation" icon="user">
    Create independent ground truth. Do not score providers against each other's transcripts.
  </Step>

  <Step title="Run all STT providers" icon="diagram-project">
    Transcribe the same audio with every provider so the comparison is fair.
  </Step>

  <Step title="Normalize" icon="file-import">
    Normalize reference and predictions with the same pipeline before WER, so `Mr.` / `Mister` and `$50` / `fifty dollars` mismatches does not count as errors.

    <Card title="gladia-normalization" icon="github" href="https://github.com/gladiaio/normalization">
      Open-source text normalization for fair WER scoring
    </Card>

    ```python theme={"system"}
    from normalization import load_pipeline

    pipeline = load_pipeline("gladia-3", language="en")
    normalized_reference = pipeline.normalize(reference)
    normalized_prediction = pipeline.normalize(prediction)
    ```
  </Step>

  <Step title="Generate metrics and analyze" icon="chart-simple">
    Compute WER (and NER where it matters), then inspect where critical details fail: names, numbers, acronyms, noisy slices.

    ```text theme={"system"}
    WER = (S + D + I) / N
    ```

    Lower is better. Do not stop at one average score.
  </Step>
</Steps>

## Before you start

Decide what "good" means for your product (noise, diarization, entities, domain vocab, timestamps). If the dataset does not look like your traffic, the score will not predict production quality.

## Common pitfalls

* Different datasets per provider
* Weak ground truth
* Counting formatting as recognition errors
* Too few samples, or a wrong distribution of your real audio traffic diversity
* Reference text that is not in the audio