Testing method

How a release test works.

One call type, an agreed outcome and a set of calls the agent has never heard. Each failure can be traced to a moment in the call or a change in the record.

Protocol

From call type to repeat test

  1. Choose one call type.

    Pick a task with the team responsible for release. For an address change, agree which fields may change, what the agent must read back and when a human must take over.

  2. Commission and review the calls.

    Commission original scenarios from consenting speakers. Agree ownership and reuse rights in writing. Gulf Arabic speakers and contact-centre experts review the dialogue and expected action, and record any disagreement.

  3. Keep cases unseen.

    Hold back whole scenarios and speakers from development. Keep the expected answers private so a familiar voice or script cannot give the agent a head start.

  4. Run the agent blind.

    Run the release candidate in a test system. Give it the call and the tools it would normally use. Keep the answer key hidden and withhold the version name from the reviewers.

  5. Count failures by type.

    Compare the read-back and saved record with the agreed outcome. Report how many cases failed each check, how many were tested and where reviewers disagreed.

  6. Repeat on the next release.

    Rerun the same private cases to see which failures persist. Add fresh unseen cases to check whether a fix holds on unfamiliar calls.

The report

What your team receives

A private report identifies the tested version, call type and scoring rules. For each failed check, it gives the relevant turn, the expected action and the observed result. Your team can use the run details to reproduce the failure.

Reading the result

What the counts can tell you

A failure on a held-out call shows a specific behaviour to investigate. A small set cannot estimate a rare failure rate precisely. Passing every case still leaves open how the agent will handle other calls.

Design partners

Bring one call type.

We are looking for our first design partners. Bring a bilingual call your agent needs to handle and someone who can say what the right outcome should be. Together we will define the first release test.

Pilot enquiries open soon.