System Evaluation
Evaluation in Divisions
- Number of problems solved with acceptable
proofs/models/answers output
- Average runtimes (CPU or WC) over solutions found
- Specialist measures
- State-of-the-art contribution = average fraction of non-solvers
- Core usage = average CPU/WC
- Efficiency = average solution rate * fraction solved
Clear and Meaningful
- Precise recognition of system status using
SZS ontologies
- Ranking scheme with clear semantics
- Commonly acceptable