Problem Rating by SOTA

Approach

• Complete analysis is impossible; empirical measurement is required
  • Problem rating is by and for ATP systems
  • Problem rating within SPCs

• Only know ranking of systems within SPCs
  • Problem ratings based on only solutions found

• Avoid influence by large number of weak systems
  • Use only SOTA contributors

Rating Scheme

• The fraction of SOTA contributors that fail to solve

           Number of failing SOTA contributors
  Rating = -----------------------------------
               Number of SOTA contributors               

• Discrete ratings: Easy - Difficult - Unsolved

• Problem Rating by Rating Contributors