AI call center quality assurance for bilingual teams.

Traditional QA is a sampling exercise. A reviewer listens to a handful of calls per agent each month and scores them against a scorecard. It works, but it’s slow, and a sample that small can miss the pattern that matters: the agent who skips the recording notice only on transfers, or the policy that confuses customers every evening.

AI changes the job, not just the speed. Every interaction gets scored, so reviewers stop hunting for problems and start judging the ones the system surfaces. The reviewer’s week moves from listening at random to checking flagged calls, settling disagreements and coaching.

What it doesn’t change is who decides. AI-assisted means a person still owns every score that affects an agent’s pay, warnings or promotion. The teams that skip this step usually lose their agents’ trust in the whole program within a quarter.

Most AI QA setups follow the same pipeline. The quality of each step limits everything after it, so check them in order.

Bilingual customers switch languages. A caller might start in English, give their address in Spanish and complain in a mix of both. Many transcription and scoring setups handle one language per interaction, so the switched part is where errors pile up. Test your setup on real code-switched calls before you trust it.

Spanish isn’t one accent. A model tuned on one variety of Spanish can struggle with another, and with regional words for everyday things. If your customers are mostly Mexican or Caribbean, test with recordings from those customers, not a generic sample.

Score in the language that was spoken. Translating a Spanish call into English and then scoring it adds a second layer of errors, and it’s worst exactly where tone matters. Write the scorecard in both languages, with examples in each, so a criterion like “acknowledged the customer’s frustration” means the same thing in both.

Watch the numbers and names. Account numbers, dates, addresses and surnames are where transcription fails most often, and they’re what accuracy criteria depend on. Check them by hand in your first weeks.

AI is consistent, and it never gets tired at four in the afternoon. It’s strongest on criteria that can be checked against the transcript: was the disclosure said, was identity verified, was the customer put on hold, did the agent promise a callback with a time. It’s weaker where judgment and context carry the meaning.

You can usually keep your existing scorecard, but most need rewriting before AI scores them well. Criteria like “showed professionalism” mean different things to different reviewers, and a model inherits that ambiguity. The fix is the same one that helps human reviewers agree.

Turn each criterion into a question that the transcript can answer. Say what counts as a yes, give an example in each language, and say when the question doesn’t apply. A retention offer criterion makes no sense on a call that was never a cancellation, so mark it not applicable instead of failing the agent.

Then decide, criterion by criterion, who scores it. Some go to AI alone, some to AI with a human check, and some stay human-only. Start conservative and move criteria to AI alone only once agreement holds up.

Treat the AI as a new reviewer who needs calibrating, not as a measuring instrument. Every week, have people score a sample of interactions the AI also scored, without seeing its scores. Track how often they agree on each criterion, separately for English and Spanish.

Where agreement is low, read the disagreements. Usually the cause is a vague criterion, not a weak model. Rewrite the criterion with clearer examples, re-run the sample and see if agreement improves. Version the scorecard and the scoring instructions, so you know which version produced which score.

Keep calibrating after launch. Products, policies and scripts change, and a model scoring against last quarter’s policy will flag agents for following this quarter’s. A monthly calibration session that includes the people who maintain the AI scoring keeps both sides honest.

This isn’t legal advice, and your counsel should decide which rules apply to each program. These are the questions that come up in almost every bilingual operation we see.

Recording consent. Some US states require every party to a call to consent to recording, so the recording notice belongs at the start of every call, in the language of the call. QA should check it on every interaction, which is one place AI coverage clearly helps.

Payment data. Card data has strict rules under PCI DSS. The safest setups never record it at all, by pausing the recording or taking payment through a separate secure channel. If card numbers do reach a recording, redaction must happen before transcripts are stored or sent to any model.

AI voices. In 2024 the FCC ruled that AI-generated voices in robocalls count as artificial voices under the Telephone Consumer Protection Act, so the consent rules for prerecorded calls apply. If you use AI voice agents for outbound calls, check that with counsel first.

Mexican customers. If you serve customers in Mexico, Mexico’s federal law on personal data held by private parties requires a privacy notice that says what you collect and why. Make sure it covers recordings and automated analysis.

Sources: FCC Declaratory Ruling FCC 24-17 on AI-generated voices (2024) · PCI Security Standards Council: document library

If AI chat or voice agents handle part of your volume, score them on the same scorecard as your people. Customers don’t care who failed them, and leadership will want to know whether the AI is as good as the team.

Sample AI conversations on purpose. Random samples of bot traffic are mostly easy questions handled well. Pull the conversations where the customer asked for a person, where the bot handed off, and where the conversation ran long. That’s where invented answers and broken handoffs show up.

A score nobody acts on is just surveillance. The point of covering every interaction is better coaching, so build the coaching loop before you switch on full coverage.

Coach one thing at a time. Each agent gets one behavior to work on each week, with two or three real examples from their own interactions. Let agents see their own scores and the evidence behind them, and give them a way to dispute a score. Disputes are free calibration data.

Celebrate the good calls too. Coverage of every interaction also finds the agent who calmed down an angry customer in two languages at once. Playing that call in a team huddle teaches more than another policy reminder.

Watch the QA program and the customer outcome together. If QA scores rise while repeat contacts and complaints stay flat, agents may be learning to satisfy the scorecard rather than the customer.

Roll AI QA out in parallel with your existing process before it replaces anything. Two months is enough to know whether it’s trustworthy in both languages.

Whether you buy software or hand the whole operation to a partner, these questions separate a working program from a demo.