Consensus evaluation reduces individual judge bias by requiring agreement across multiple evaluators before accepting a result.
Variants
- Majority vote: Simple majority determines outcome
- Unanimous: All judges must agree
- Weighted: Some judges carry more weight
Trade-offs
- More robust than single-judge evaluation
- Higher cost (multiple evaluations per item)
- Potential for systematic shared biases