A current evaluation method for large language models produces generic explanations that do not provide actionable feedback. CriticGen proposes a fine-grained, generation-aware evaluation framework. This framework generates sample-specific evaluation dimensions and scoring criteria under categories such as subjective, objective, and self-derived constraints. These criteria are then used to jointly produce a score, a reason, an executable refinement suggestion, and a refined answer. The rubric-conditioned refinement process enables models to diagnose flaws and perform targeted answer improvement.
Experimental results show that fine-grained evaluation should be both instance-specific and actionable. CriticGen induces higher-quality rubrics, improving relevance/coverage from 3.33/4.03 to 3.97/4.24. The framework also achieves the best score correlations, with 0.9556 Pearson and 0.9560 Spearman. Criterion-grounded reasons and executable suggestions show an F1 improvement from 0.6369/0.5994 to 0.7554/0.7900.
Crucially, the feedback translates into reliable answer improvement. 73.17% of answers were improved, with a 93.28% non-degradation rate. This indicates a significant potential for using evaluation frameworks to directly enhance model performance.
Source: https://arxiv.org/abs/2609.05439