Google contractor policy forces non-expert reviews of Gemini outputs
Google allegedly instructed contractor GlobalLogic to change evaluation guidelines for Gemini, requiring reviewers to assess AI responses outside their areas of expertise rather than skip them. The policy change potentially compromises the accuracy of human quality assurance, particularly for specialized domains.
What happened
Google reportedly instructed GlobalLogic, a contractor firm evaluating Gemini outputs, to change its evaluation guidelines. Previously, contractors could skip prompts outside their domain expertise with instructions stating "If you do not have critical expertise (e.g. coding, math) to rate this prompt, please skip this task." The new guidance instructs contractors not to skip prompts requiring specialized knowledge and to instead "rate the parts of the prompt you understand" while noting gaps in their expertise.
Reported impact
The change may reduce accuracy of Gemini's human evaluation process. One contractor responded to the new policy asking, "I thought the point of skipping was to increase accuracy by giving it to someone better?" The report raises questions about the rigor of testing standards Google claims to apply, particularly for responses that are inaccurate rather than harmful.
Index analysis
Non-expert reviewers may be unable to detect technical inaccuracies in specialized domains such as coding, mathematics, or law. Google stated the ratings "do not directly impact our algorithms" but serve as aggregate data points and the raters can still provide feedback on formatting and style. Google also noted it released the FACTS Grounding benchmark to check that LLM responses are factually accurate and sufficiently detailed.
Governance controls
Google stated it is "working with trusted testers and external experts and performing extensive risk assessments and safety and assurance evaluations" for Gemini 2.0. The company includes a disclaimer that "Gemini can make mistakes, so double-check it." Google emphasized that raters are instructed to note areas outside their expertise in their analysis.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.