Study reveals that LLM-based dialogue evaluation rubrics don't uniformly predict business outcomes in conversational commerce, with only 2 of 7 dimensions actually driving conversions.

Researchers tested a 7-dimension rubric using LLM-as-Judge against real conversion data from a major Chinese matchmaking platform to validate whether quality scores actually predict business results

Only Need Elicitation (rho=0.368) and Pacing Strategy (rho=0.354) showed statistically significant associations with conversion after correction, while Contextual Memory showed zero correlation

The core problem lies in rubric design and weighting rather than LLM accuracy—any evaluation judge using the same rubric would face the same structural limitations

Phase 2 analysis of 60 human conversations with verified labels revealed significant dimension-level heterogeneity in predictive validity for business outcomes