Call for Participation

The CADE-14 ATP System Competition will evaluate the performance of sound, fully automatic, 1st order ATP systems, in terms of:

• the number of problems solved, and
• the average runtime for successful solutions;

• a bounded number of eligible problems, chosen from the TPTP Problem Library, and
• a specified CPU time limit for each solution attempt,

The competition machines are SUN Ultra 140s, each with 64MB memory and running Solaris 2.5.1.

The competition is being organized by Christian Suttner and Geoff Sutcliffe. If you have any questions about the competition, please email the organizers. The competition will be overseen by a panel of knowledgeable researchers who are not participating in the event. It is planned to publish the competition results in a form that includes contributions written by entrants.

This document contains information about the:

• Competition Divisions
• Problem Selection and Preparation
• Time Limits and Timing
• System Execution
• Performance Evaluation
• FOF Demonstration Division
• Special Hardware Demonstration Divisions
• Conditions for Participation
• Competition Dinner
• System Registration Form

• ATP systems may be entered only at the division level.
• ATP systems may be entered into more than one division (a system that is not entered into a particular division is assumed to be useless for that type of problem).
• All problems will be presented in CNF. ATP systems that accept FOF problems may enter the FOF Demonstration division.
• All systems must run on a single locally provided standard UNIX workstation. ATP systems that cannot run on the locally provided standard UNIX workstations may enter the Special Hardware Demonstration divisions.

The TPTP difficulty rating scheme identifies problems as:

• Solvable by all state-of-the-art ATP systems (easy)
• Solvable by some state-of-the-art ATP systems (difficult)
• Solvable by no state-of-the-art ATP systems (unsolved)
• Theorem-hood unknown (open)

The TPTP distinguishes versions of problems as one of standard, non-standard, incomplete, or special. The competition will use standard TPTP problems.

The eligible problems for each competition division and category can be extracted from the TPTP using the tptp1T script that is distributed with the TPTP, as follows:

• MIX Division, Horn with No Equality
  tptp1T CNF NonProp Unsatisfiable Rating 0.01 0.99 Standard Horn NoEq
• MIX Division, Non-Horn with No Equality
  tptp1T CNF NonProp Unsatisfiable Rating 0.01 0.99 Standard NonHorn NoEq
• MIX Division, Horn with Equality
  tptp1T CNF NonProp Unsatisfiable Rating 0.01 0.99 Standard Horn SomeEq
  and
  tptp1T CNF NonProp Unsatisfiable Rating 0.01 0.99 Standard Horn NonUnitEq
• MIX Division, Non-Horn with Equality
  tptp1T CNF NonProp Unsatisfiable Rating 0.01 0.99 Standard NonHorn SomeEq
• UEQ Division
  tptp1T CNF NonProp Unsatisfiable Rating 0.01 0.99 Standard UnitEq
• SAT Division
  tptp1T CNF NonProp Satisfiable Rating 0.00 0.00 Standard
  and
  tptp1T CNF NonProp Satisfiable Rating 0.00 0.00 Special
  Note that all the problems are "easy". We have had to accept this to get some eligible problems. Further, Special problems have been accepted, as there are too few Standard problems. Finally, there have been a few changes to problem version status since the release of TPTP v2.0.0, so check the explicit eligible problems list.
• FOF Divisions
  tptp1T FOF Standard NonProp Theorem
  There is no Rating information available for FOF problems.

The number of problems to be used will be chosen between a minimum that ensures sufficient confidence in the competition results (the competition organizers will ensure that there are sufficient resources available), and a maximum determined from the number of workstations available, the time allocated to the competition, the number of ATP systems, and the minimal time limit:

          Number of workstations * Time for competition
Maximum = ---------------------------------------------
             Number of systems * Minimal time limit

The problems to be used will be randomly selected on the day of the competition, from the most recent TPTP release. The tptp2X utility, distributed with the TPTP, will be used to:

• replace predicate and function symbols by short, meaningless symbols,
• randomly reorder the clauses and literals,
• randomly reverse the unit equalities in the UEQ division,
• remove equality axioms that are not needed by an ATP system,
• output the problems in the format required. (If the tptp2X utility does not support the format required, then it is the entrant's responsibility to provide a reformating program. The most acceptable form for this program is a tptp2X format module. If the ATP system is publically available, then the competition organizers may be able to produce such a module.)

          Number of workstations * Time for competition
Maximum = ----------------------------------------------
          Number of systems * Minimal number of problems

The timing will be done in units of 1 second, and the minimal time to find a solution is 1 second. If an ATP system cannot solve a problem, the runtime will be set to the time limit.

• A person must be nominated to handle all issues (including execution difficulties) arising before and during the competition.
• It is not necessary for entrants to physically attend the competition.
• It is assumed that each entrant has read all the documents related to the competition (they will be made available to entrants) and will comply with the competition rules. Non-compliance with the rules could lead to disqualification.
• Entering many similar versions of the same system is deprecated. Entrants may be required to limit the number of system versions that they enter.
• The precomputation and storage of any information for individual TPTP problems for usage during the competition is contrary to the spirit of the competition. The panel may disqualify unfairly doped systems.
• Winners are expected to provide public access to their system's source code. If you want to participate without releasing your source code, please email the organizers.
• Catch-all rule: no cheating is allowed; the panel may disqualify entrants due to unfairness and may adjust competition rules in case of misuse.

Disclaimer

Note: You have to be directly associated with an entered ATP system to attend this dinner. It's an exclusive event. (Spouses are welcome to come along; please indicate this in your registration.)