Task-Critical Word Error Rate: Extending Entity-Focused Evaluation to Voice Commands in African-Accented Speech

Voice assistants increasingly mediate everyday tasks, making accurate speech recognition important not only for transcription but also for preserving user intent. Yet conventional Word Error Rate (WER) treats all transcription errors equally, although errors in command-critical words can change or invalidate an intended action. This raises an important question for equitable voice interfaces: does aggregate WER capture the errors that matter for command execution? Recent work has moved beyond aggregate WER. AfriSpeech-MultiBench [1] evaluates African-accented English across application-specific domains, including named-entity and command utterances, while AfriVox-v2 [2] introduces Entity Error Rate (EWER) and Numeric Error Rate (NWER) for entity- and number-containing utterances. These approaches provide finer-grained evaluation but do not explicitly capture the broader vocabulary required for command execution, including action verbs (e.g., “call,” “play,” “set”) and state or temporal attributes (e.g., “on/off,” “today/tomorrow”). We propose Task-Critical Word Error Rate (TC-WER), extending entity- and numeric-focused evaluation to tokens required to preserve a command’s intended action. We situate TC-WER within a three-level hierarchy: (1) transcription accuracy (WER), (2) task-critical recognition error (TC-WER), and (3) functional command success. We further propose simulated voice commands as a controlled setting for testing whether such errors cause command-level failure. Because commercial assistants do not expose their ASR pipelines, we use Whisper [3] as an accessible testbed, not a proxy for any specific assistant. We construct country-stratified evaluation subsets from AfriSpeech-200 [4], focusing on Nigeria, Kenya, South Africa, Ghana, and Uganda, with matched utterance counts where possible. We evaluate the same speech using WER and TC-WER, with entity- and number-dense utterances enabling comparison with EWER/NWER. This work asks whether comparable WER can mask task-critical disparities across African English varieties, and whether extending entity- and numeric-focused evaluation to command vocabulary reveals them. We hypothesize that TC-WER offers a more actionable equity measure than existing metrics, and contribute TC-WER, a country-stratified evaluation protocol, and a framework for ASR equity auditing when downstream systems are inaccessible.