This document contains information about the:
The design and procedures of CASC-15 evolved from those of CASC-13 and CASC-14. The rules, deadlines, and specifications given here, are absolute. Only the competition panel has the right to make exceptions.
Entry into the competition divisions is subject to the following rules:
The eligible problems for each division and category can be extracted from the TPTP using the following links.
Number of Problems
The minimal numbers of problems that have to be used in each division and
category, to ensure sufficient confidence in the competition results,
will be determined from the numbers of eligible problems in each division
and category
(the competition organizers have to ensure that there is sufficient CPU time
available to run the ATP systems on this minimal number of problems).
This minimal numbers of problems will then be used in determining the
time limit imposed on each solution
attempt.
A lower bound on the total number of problems that will be used will be determined from the number of workstations available, the time allocated to the competition, the number of ATP systems to be run on the general hardware over all the divisions, and the time limit, according to the following relationship:
Number of workstations * Time allocated
Number of problems = ---------------------------------------
Number of ATP systems * Time limit
It is a lower bound on the total number of problems because it assumes that
every system will use all the time limit for each problem.
Since some solution attempts will obviously succeed before the time
limit is reached, more problems can actually be used.
The actual numbers used in each division and category will be determined
according to the judgement of the competition organizers.
Problem Preparation
It is necessary to ensure that no system receives an advantage or
disadvantage due to the specific presentation of the problems in the TPTP.
To this end the tptp2X utility, distributed with the TPTP, will be used to:
CCCNNN-1.p for the symbolic links, with NNN
running from 001 to the number of problems in the respective
division or category.
The problems will be specified to the ATP systems using the symbolic link names.
In the Special Hardware Division the same problems will be used as for the general hardware, with the same tptp2X transformations applied. However, the original file names will be retained.
A minimal time limit of 180 seconds will be used. The maximal time limit will be determined using the relationship used for determining the number of problems, with the minimal number of problems as the Number of problems. The time limit will be chosen as a reasonable value within the range allowed.
The timing will be done by the UNIX /usr/bin/time command,
which returns times in units of 0.1 second.
If an ATP system cannot solve a problem, the runtime will be set to the
time limit.
On the general hardware a wall clock time limit will be imposed in addition to the CPU time limit, in order to prevent very high memory usage that causes swapping. The wall clock time limit will be double the CPU time limit.
Entering many similar versions of the same system was deprecated. Entrants could have been required to limit the number of system versions that they entered. Winners are expected to provide public access to their system's source code.
It is assumed that each entrant has read all the WWW pages related to the competition, and has complied with the competition rules. Non-compliance with the rules could lead to disqualification. A "catch-all" rule is used to deal with any unforseen circumstances: No cheating is allowed. The panel is allowed to disqualify entrants due to unfairness and to adjust the competition rules in case of misuse.
Please email the system description to the competition organizers.
The ATP systems that run on the general hardware have to be interruptable by a SIGXCPU signal, so that the CPU time limit can be imposed on each solution attempt. The default action on receiving this signal is to exit (thus complying with the time limit, as required), but systems may catch the signal and exit of their own accord. Both approaches are acceptable for the competition. If any system runs past the time limit this will be noticed in the timing data and the system will be considered to have not solved that problem. In the Special Hardware Division the wall clock time limit will be imposed using SIGALRM.
When terminating, the ATP systems have to output a distinguished string
(specified by the entrant) to stdout indicating the result:
The ATP systems are not required to output solutions (proofs or models).
However, systems that do output solutions to stdout will be
highlighted in the presentation of results.
For practical reasons excessive output from the ATP systems is not allowed.
A limit, dependent on the disk space available, will be imposed on the amount
of stdout and stderr output that can be produced.
The limit will be at least 10KB per problem (averaged over all problems so
that it is possible to produce some long proofs).
The precomputation and storage of any information for individual TPTP problems for usage during the competition is contrary to the spirit of the competition. The panel is allowed to disqualify unfairly doped systems. For every problem solved, the system's solution process has to be reproducible by running the system again.
Entrants have to ensure that their systems execute in the competition environment, according to the checks listed below. Entrants are advised to firstly perform these checks on their own machines, well in advance of the installation period. This gives the competition organizers time to help resolve any difficulties that are encountered. Secondly, entrants must perform these checks on the general hardware. Systems that do not pass these checks on the general hardware cannot compete.
prompt>which MyATPSystem
/home/tptp/bin/MyATPSystem
prompt>/home/tptp/bin/MyATPSystem /home/tptp/TPTP/Problems/GRP/GRP001-1.p
Proof found in 147 seconds.
prompt>
prompt> cd /home/tptp/tmp
prompt> ln -s /home/tptp/TPTP/Problems/GRP001-1.p CCC001-1.p
prompt> /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p
Proof found in 147 seconds.
prompt>
CPULimitedRun.c program.
For example:
prompt>which CPULimitedRun
/home/tptp/bin/CPULimitedRun
prompt>/home/tptp/bin/CPULimitedRun 10 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p
CPULimitedRun: ----------------------------------------------------------
CPULimitedRun: /home/tptp/bin/MyATPSystem
CPULimitedRun: Time limit is 10s
CPULimitedRun: PID is 5827
CPULimitedRun: ----------------------------------------------------------
CPU time limit exceeded (core dumped)
prompt>
/bin/time. For example:
prompt> /bin/time /home/tptp/bin/CPULimitedRun 200 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p
CPULimitedRun: ----------------------------------------------------------
CPULimitedRun: /home/tptp/bin/MyATPSystem
CPULimitedRun: Time limit is 200s
CPULimitedRun: PID is 13526
CPULimitedRun: ----------------------------------------------------------
Proof found in 147 seconds.
real 3:04.2
user 2:19.3
sys 7.6
prompt>
rsh. The timed command
needs to be quoted, and don't forget to make an appropriate
rhosts entry. For example:
prompt> rsh atoll.cs.jcu.edu.au '/bin/time /home/tptp/bin/CPULimitedRun 200 /home/tptp/bin/MyATPSystem /home/tptp/tmp/CCC001-1.p'
CPULimitedRun: ----------------------------------------------------------
CPULimitedRun: /home/tptp/bin/MyATPSystem
CPULimitedRun: Time limit is 200s
CPULimitedRun: PID is 25461
CPULimitedRun: ----------------------------------------------------------
Proof found in 147 seconds.
real 3:04.2
user 2:19.3
sys 7.6
After the installation deadline access to the general hardware will be denied, and no further changes to the ATP systems will be permitted. The organizers will then test the ATP systems, first to check that the systems execute correctly (according to the above checks), and secondly to check for soundness.
For the soundness testing, non-theorems (selected from the TPTP) will be submitted to the systems participating in the MIX, UEQ, and FOF Divisions, and theorems will be submitted to the systems participating in the SAT Division. Finding a proof of a non-theorem or a model for a theorem indicates that the system is unsound. The soundness testing has a secondary aim of eliminating the possibility of an ATP system simply delaying for some amount of time and then claiming to have found a solution. If an ATP system fails the soundness testing then the ATP system will be disqualified.
In the Special Hardware Division the systems will be installed on the respective hardware by the entrants, and no soundness testing needs to be performed.
perl script, provided by the competition organizers.
The jobs will be queued onto the workstations so that each
workstation is running one job at a time.
All attempts at the Nth problems in all the divisions and
categories will be started before any attempts at the (N+1)th problems.
During the competition a perl script will parse the systems'
output.
If an ATP system's success string is found then the timing information
from the time command will be extracted.
The CPU time taken, or the time limit if no solution was found, will be
recorded.
This data will be used to generate an HTML-file, and a WWW-browser will be
used to display the results.
The execution of the Special Hardware Division systems will be supervised by their entrants.
If only one ATP system registers for a particular competition division, no winner can be announced for that division, but the results for that system will still be presented.