ATP System Evaluation

Aims

• To make unbiased measurements of the relative abilities of ATP systems
• To provide a intuitively acceptable and practically realistic rating of ATP problems

Outcomes

• Framework
  • Empirical evaluation
    • Problems solved is largely independent of CPU time limit
  • Appropriate problems
    • Specialist Problem Classes
• Ranking and Rating
  • System ranking by subsumption
  • Problem rating by SOTA
  • System rating by SOTA
  • Measures for resource usage
• Observation
  • System quality and problem difficulty are intertwined