Problem Rating by Rating Contributors

Rating Scheme

• Inadequate SOTA contributors

• In the extreme, problems are easy or unsolved

• Use some non-SOTA contributors

           Number of failing rating contributors
  Rating = -------------------------------------
               Number of rating contributors                

• Makes some problems difficult rather than easy

Example Results: THM_RFO_NEQ_CNF_NHN

ALG002-1  ED:15 S:14 F: 1 SC: 3 SCS: 3 SCF: 0 NSC: 0 Rating: 0.00
ANA001-1  ED:15 S: 0 F:15 SC: 3 SCS: 0 SCF: 3 NSC: 0 Rating: 1.00
ANA002-1  ED:15 S: 0 F:15 SC: 3 SCS: 0 SCF: 3 NSC: 0 Rating: 1.00
ANA002-2  ED:15 S: 4 F:11 SC: 3 SCS: 2 SCF: 1 NSC: 0 Rating: 0.33
ANA002-3  ED:15 S: 1 F:14 SC: 3 SCS: 1 SCF: 2 NSC: 0 Rating: 0.67
ANA002-4  ED:15 S: 4 F:11 SC: 3 SCS: 2 SCF: 1 NSC: 0 Rating: 0.33
ANA003-4  ED:15 S: 4 F:11 SC: 3 SCS: 3 SCF: 0 NSC: 0 Rating: 0.00
ANA004-4  ED:15 S: 3 F:12 SC: 3 SCS: 2 SCF: 1 NSC: 0 Rating: 0.33
ANA004-5  ED:15 S: 0 F:15 SC: 3 SCS: 0 SCF: 3 NSC: 0 Rating: 1.00
ANA005-4  ED:15 S: 0 F:15 SC: 3 SCS: 0 SCF: 3 NSC: 0 Rating: 1.00
ANA005-5  ED:15 S: 0 F:15 SC: 3 SCS: 0 SCF: 3 NSC: 0 Rating: 1.00
  ...  

Problem Ratings using Current Data