Problem and System Ratings

Problems and Data

• Use TPTP problems
• Exclude biased problems
• Divide into Specialist Problem Classes
• Use TSTP data within each class
  (resource limits beyond the systems' PPPs)

System Ranking by Subsumption

• Provides a partial ranking (sample data)
• Superset systems subsume subset systems
• SOTA contributors form a SOTA system

Problem Rating by SOTA

• Use SOTA contributors to avoid weak systems
• The fraction of SOTA contributors that fail (sample data)
• Discrete ratings: Easy - Difficult - Unsolved

System Rating by SOTA

• Use difficult problems to differentiate between systems
• The fraction of difficult problems solved (sample data)

Evidence of Progress

• Declining problem ratings are evidence of progress

Reports available online at tptp.org/cgi-bin/SystemOnTPTP

---