System Evaluation

Evaluation in Divisions

• Number of problems solved with acceptable proofs/models/answers output
• Average runtimes (CPU or WC) over solutions found
• Specialist measures
  • State-of-the-art contribution = average fraction of non-solvers
  • Core usage = average CPU/WC
  • Efficiency = average solution rate * fraction solved

Clear and Meaningful

• Precise recognition of system status using SZS ontologies
• Ranking scheme with clear semantics
• Commonly acceptable