The CASC Process

For presentation at the Deep Knowledge Representation Challenge Workshop

Abstract CASC DKRC
Language
A common language that all systems can parse, or for which there are translators to the systems' native languages. The competition problems need to be presented equally to all systems in a competition. TPTP language. TPTP2X/TPTP4X to translate. English? ACE/CELT?
Theories
Background theories upon which specific queries/conjectures can be based. TPTP axiom files. SUMO?
Problems
A collection of problems that the community agrees are representative of possible "real-world" applications of the systems. These can be used for a competition. The TPTP problem library, including the identification of problems that are "biased" - not representative of possible "real-world" applications. The biology book.
Specialist Problem Classes (SPCs)
A division of the problems so that each division is homogeneous wrt ATP systems. These form the basis for competition divisions, so that entrants in a division are, in principle, able to attempt all problems in the division. TPTP SPCs, which are based on logical, language, and syntactic characteristics. ????
Solutions
A collection of solutions for the problems, which provide the basis for deciding which problems should be eligible for a competition. The TSTP solution library. ????
Ratings
A rating of the problems that indicates the difficulty of the problems wrt potential competition entrants. Problems that all systems solve, and problems that no systems solve, are of little interest in a competition. Therefore it is necessary to know inadvance what's realistically possible. TPTP problem ratings. ????
Competition Organization
There are many aspects to this, but there are some general ideas:

A dedicated organizing team or not more than 3 people.
A panel of respected researchers, who are independent of the organizers, who adjudicate and decide on any disagreements regarding conformance and interpretation of the rules.
Pre-determined rules, which are generally agreed upon as "reasonable" by the community and the entrants. These must be announced far enough in advance for entrants to adpat their systems for the competition.

Geoff Sutcliffe, and previously Christian Suttner
Panel of 3 appointed each year, with some continuity.
Provided online about 5 months in advance.

Stimulating Environment
Do not let "evaluation" dominate. A competition must be equally about stimulating research, and providing an environment in which researchers can productively interact. CASC dinner. Real time competition in one day, with online results displayed in a public area at a conference. ????
Resources
Enough compute power is necessary to run a competition in one day, to maximize the stimulating impact. Various, but most recently the MPII cluster and the University of Manchester clusters. Hopefully soon the StarExec cluster. The StarExec cluster ????
Recognition
The system developers (entrants) are the people who do the hardest work, and their efforts must be seriously recognized. Trophies. Recognition in the published competition report. ????

Abstract	CASC	DKRC
Language A common language that all systems can parse, or for which there are translators to the systems' native languages. The competition problems need to be presented equally to all systems in a competition.	TPTP language. TPTP2X/TPTP4X to translate.	English? ACE/CELT?
Theories Background theories upon which specific queries/conjectures can be based.	TPTP axiom files.	SUMO?
Problems A collection of problems that the community agrees are representative of possible "real-world" applications of the systems. These can be used for a competition.	The TPTP problem library, including the identification of problems that are "biased" - not representative of possible "real-world" applications.	The biology book.
Specialist Problem Classes (SPCs) A division of the problems so that each division is homogeneous wrt ATP systems. These form the basis for competition divisions, so that entrants in a division are, in principle, able to attempt all problems in the division.	TPTP SPCs, which are based on logical, language, and syntactic characteristics.	????
Solutions A collection of solutions for the problems, which provide the basis for deciding which problems should be eligible for a competition.	The TSTP solution library.	????
Ratings A rating of the problems that indicates the difficulty of the problems wrt potential competition entrants. Problems that all systems solve, and problems that no systems solve, are of little interest in a competition. Therefore it is necessary to know inadvance what's realistically possible.	TPTP problem ratings.	????
Competition Organization There are many aspects to this, but there are some general ideas: A dedicated organizing team or not more than 3 people. A panel of respected researchers, who are independent of the organizers, who adjudicate and decide on any disagreements regarding conformance and interpretation of the rules. Pre-determined rules, which are generally agreed upon as "reasonable" by the community and the entrants. These must be announced far enough in advance for entrants to adpat their systems for the competition.	Geoff Sutcliffe, and previously Christian Suttner Panel of 3 appointed each year, with some continuity. Provided online about 5 months in advance.
Stimulating Environment Do not let "evaluation" dominate. A competition must be equally about stimulating research, and providing an environment in which researchers can productively interact.	CASC dinner. Real time competition in one day, with online results displayed in a public area at a conference.	????
Resources Enough compute power is necessary to run a competition in one day, to maximize the stimulating impact.	Various, but most recently the MPII cluster and the University of Manchester clusters. Hopefully soon the StarExec cluster.	The StarExec cluster ????
Recognition The system developers (entrants) are the people who do the hardest work, and their efforts must be seriously recognized.	Trophies. Recognition in the published competition report.	????