Arabic AI Voice Evaluation Methodology

Fixed test prompts

Use these exact prompts for every provider. Dialect prompts are drafts and require native-speaker review before they are used to publish results.

1. Modern Standard Arabic: names, numbers, and dates

   أعلنت مكتبة الرباط الوطنية اليوم إطلاق خدمة جديدة للاستعارة الرقمية. يبدأ التسجيل في 15 أكتوبر، ويمكن لكل عضو استعارة 3 كتب لمدة 21 يومًا.

2. Moroccan Darija with French code-switching

   سلام، اليوم غادي نورّيوكم كيفاش تستعملو التطبيق. دخلو للإعدادات، اختارو «votre langue»، ومن بعد كليكيوا على «télécharger» باش تحفظو الملف.

3. Egyptian Arabic

   أهلاً بيكم، النهارده هنجرب التسجيل خطوة بخطوة. افتحوا التطبيق، اختاروا الصوت، وبعدها دوسوا على زر التحميل.

4. Gulf Arabic

   هلا والله، اليوم بنشرح طريقة حفظ الملف الصوتي. افتحوا التطبيق، اختاروا اللهجة، وبعدها اضغطوا على تنزيل.

5. Arabic-English code-switching

   افتح صفحة Settings، اختر Audio format، ثم اضغط على Export MP3. إذا ظهر Error message، انسخ رقم الخطأ وأرسله إلى فريق الدعم.

6. Long-form and accessibility passage

   تبدأ الرحلة إلى المكتبة بخطوات بسيطة، سواء كنت تختار كتابًا ورقيًا أو تسجيلًا صوتيًا. افتح صفحة الفهرس، واستخدم البحث للعثور على العنوان، ثم اقرأ وصفه قبل أن تضيفه إلى قائمتك. إذا كنت تستخدم قارئ الشاشة، فستحتاج إلى سماع أسماء الأزرار بالترتيب الصحيح، مع فواصل واضحة بين العناوين والنصوص. عندما يبدأ المقطع الصوتي، ينبغي أن تبقى نبرة القراءة مستقرة، وأن تُنطق الأرقام والتواريخ من دون حذف. يمكنك إيقاف الاستماع، ثم متابعة القراءة من الموضع نفسه.

Evaluation rubric

Document pronunciation, dialect authenticity, naturalness, repeat-generation consistency, and workflow separately. Publish raw notes, settings, and the related clips.

Repeatable test protocol

This is Vocalaba's initial protocol, informed by listening-test recommendations and Arabic TTS research. It does not claim official ITU compliance or endorsement.

1. For each voice that supports a test case, use the exact prompt below for 3 independent generations with neutral available settings. Do not edit the text or post-process the audio. Record provider, model, voice, settings, date, and any available seed.
2. Keep every output, including failures, and record why any sample is missing. Never silently replace a failed clip.
3. Present clips in randomized order with anonymous IDs to at least 5 listeners fluent in the target variety. Have a native speaker review dialect prompts before publishing any comparison.
4. Each listener independently rates pronunciation, dialect fit, and naturalness. Rate consistency by comparing the 3 outputs for the same prompt and retain individual ratings.
5. Publish prompts, settings, voice and listener counts, raw ratings, and clips. Do not collapse the dimensions into one ranking; report workflow measures separately.

Listening scale

Use a descriptive 1–5 scale for each listening dimension. This is a proposed internal scale, not a standardized MOS score.

1. 1 — Very poor: large portions are hard to understand or persistent problems occur.
2. 2 — Poor: clear, repeated errors interfere with listening.
3. 3 — Acceptable: the meaning is understandable, with noticeable errors or unnaturalness.
4. 4 — Good: mostly clear and natural, with minor flaws.
5. 5 — Excellent: clear, natural, and stable with no noticeable problems.

Measure workflow separately

Record seconds from submitting a request to the first playable audio across 3 attempts, and list available controls and export formats. Do not combine these measures with listening ratings.

Responsible publication

Use the same text and comparable settings, date every test, disclose who listened to the samples, and do not claim accuracy percentages or commercial rights without evidence.

References

These references inform test design; citing them does not imply Vocalaba or this protocol is certified or endorsed by those organizations.

• ITU-T P.808 — Subjective evaluation of speech quality with a crowdsourcing approach

  Guidance on test material, experimental design, and crowdsourced speech-quality listening tests.

• ITU-T P.85 — A method for subjective performance assessment of speech output devices

  A method for subjective performance assessment of speech output devices.

• Lodagala et al. — SawtArabi: A Benchmark Corpus for Arabic TTS (Interspeech 2025)

  Research on Arabic TTS data covering MSA, Egyptian Arabic, and Arabic-English code-switching.

Limits of this version

No provider tests have been run. The proposed minimum of 5 listeners per variety supports descriptive reporting only and is not enough by itself to establish statistical superiority. Verify dialect prompts and provider settings before publishing a comparison.