The CADE ATP System CompetitionThe World Championship for Automated Theorem Proving |
The CADE ATP System Competition (CASC) is the annual evaluation of fully automatic, classical logic, ATP systems – the world championship for such systems. One purpose of CASC is to provide a public evaluation of the relative capabilities of ATP systems. Additionally, CASC aims to stimulate ATP research, motivate development and implementation of robust ATP systems that can be easily and usefully deployed in applications, provide an inspiring environment for personal interaction between ATP researchers, and expose ATP systems within and beyond the ATP community. CASC evaluates the performance of ATP systems in terms of the number of problems solved with an acceptable solution output, and the average time taken for problems solved, in the context of a bounded number of eligible problems and specified time limits.
CASC is held at each CADE and IJCAR conference - the major forums for the presentation of new research in all aspects of automated deduction. The competition organizer is Geoff Sutcliffe. The competition is overseen by a panel of knowledgeable researchers who are not participating in the event. If you have any questions about the competition, please email the organizer.
THF | THN | TFA | TFN | FOF | FNT | SLH | EPR | UEQ | LTB | T-Shirt Design | ||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CASC‑29 (2023) | Vampire 4.8 | | Vampire 4.8 | Vampire 4.8 | Vampire 4.8 | Vampire 4.8 | E 3.1 | | Twee 2.4.2 | | Peace | |||
CASC‑J11 (2022) | Zipperp'n 2.1.999 | | cvc5 1.0 | | Vampire 4.7 | Vampire 4.7 | E 3.0 | | Twee 2.4.1 | Vampire 4.7 | Firing Squad | |||
CASC‑28 (2021) | Zipperp'n 2.1 | | | | Vampire 4.6 | Vampire 4.6 | Ehoh 2.7 | | Twee 2.4 | Vampire 4.6 | Mobius Strip | |||
CASC‑J10 (2020) | Zipperp'n 2.0 | | Vampire 4.5 | | Vampire 4.5 | Vampire 4.5 | | | E 2.5 | E 2.5 | COVID-19 Paper | |||
CASC‑27 (2019) | Satallax 3.4 | | Vampire 4.4 | | Vampire 4.4 | Vampire 4.4 | | Vampire 4.4 | E 2.4 | Leo-III 1.4 | Sunshine Drink | |||
CASC‑J9 (2018) | Satallax 3.3 | | Vampire 4.3 | | Vampire 4.3 | Vampire 4.3 | | iProver 2.8 | | MaLARea 0.6 | Dagger in Heart | |||
CASC‑26 (2017) | Satallax 3.2 | | Vampire 4.1 | | Vampire 4.2 | Vampire 4.1 | Vampire 4.2 | iProver 2.6 | | Vampire 4.2 | Snowman | |||
CASC‑J8 (2016) | Satallax 3.0 | | Vampire 4.1 | Beagle 0.9.47 | Vampire 4.0 | Vampire 4.1 | iProver 2.5 | | Vampire 4.0 | ATP? DFM! | ||||
CASC‑25 (2015) | Satallax 2.8 | Nitpick 2015 | VampireZ3 1.0 | CVC4 1.5 | Vampire 4.0 | Vampire 4.0 | Vampire 4.0 | | Vampire 4.0 | Sailing | ||||
CASC‑J7 (2014) | Satallax‑MaLeS 1.3 | | CVC4 1.4 | | Vampire 2.6 | iProver 1.0‑SAT | iProver 0.9 | Waldmeister 710 | | Vienna Sausage | ||||
CASC‑24 (2013) | Satallax‑MaLeS 1.2 | | SPASS+T 2.2.19 | | Vampire 2.6 | iProver 1.0‑SAT | iProver 0.9 | | MaLARea 0.5 | Brain | ||||
CASC‑J6 (2012) | Isabelle 2012 | | Princess 120604 | | Vampire 2.6 | Paradox 3.0 | CNF | SAT | iProver 0.9 | | Vampire 2.6 | Troll | ||
CASC‑23 (2011) | Satallax 2.1 | | SPASS+T 2.2.14 | | Vampire 0.6 | Paradox 3.0 | E 1.4pre | | iProver 0.9 | Waldmeister 710 | Vampire‑LTB 1.8 | Flower Power | ||
CASC‑J5 (2010) | LEO‑II 1.2 | | | | Vampire 0.6 (A&P) | Paradox 3.0 (A&M) | Vampire 0.6 | | iProver 0.8 | Waldmeister 710 | Vampire‑LTB 0.6 | Bagpipes | ||
CASC‑22 (2009) | TPS 3.080227G1d | | | | Vampire 11.0 (A&P) | Paradox 3.0(A&M) | Vampire 10.0 | Paradox 3.0 | iProver 0.7 | Waldmeister C09a | Vampire‑LTB 11.0 | Guillotine | ||
CASC‑J4 (2008) | | | | | Vampire 10.0 (A&P) | MetaProver 1.0(A) Paradox 3.0(M) | Vampire 10.0 | MetaProver 1.0 | iProver 0.5 | Waldmeister 806 | SInE 0.3 | 'Roo and Koalas | ||
CASC‑21 (2007) | | | | | Vampire 9.0 (A&P) | Paradox 2.2(A&M) | Vampire 8.1 (A&P) | Paradox 1.3 (A&M) | Darwin 1.3 | Waldmeister 806 | | Fishing | ||
CASC‑J3 (2006) | | | | | Vampire 8.1 (A&P) | | Vampire 8.1 (A&P) | Paradox 1.3 (A&M) | Darwin 1.3 | Waldmeister 806 | | CASC Beer | ||
CASC‑20 (2005) | | | | | Vampire 8.0 (A&P) | | Vampire 8.0 (A&P) | Paradox 1.3 (A&M) | DCTP 10.21p | Waldmeister 704 | | ATP seems Impossible | ||
CASC‑J2 (2004) | | | | | Vampire 7.0 (A&P) | | Vampire 7.0 (A&P) | Gandalf c‑2.6‑SAT (A) Paradox 1.0‑casc (M) | DCTP 10.21p | Waldmeister 704 | | Love and Hate | ||
CASC‑19 (2003) | | | | | Vampire 6.0 | | Vampire 6.0 (A&P) | Gandalf c‑2.6‑SAT (A) Paradox 1.0‑casc (M) | DCTP 1.3-SAT | Waldmeister 702 | | State Space Explosion | ||
CASC‑18 (2002) | | | | | Vampire 5.0 | | Vampire 5.0 (A&P) | Gandalf c-2.5-SAT | E‑SETHEO csp02 | Waldmeister 702 | | Rainbow | ||
CASC‑JC (2001) | | | | | E‑SETHEO csp01 | | E‑SETHEO csp01 (A) VampireJC 2.0 (A&P) | GandalfSat 1.0 | E‑SETHEO csp01 | Waldmeister 601 | SEM | Wizard Logos | ||
CASC‑17 (2000) | | | | | VampireFOF 1.0 | | E 0.6 | GandalfSat 1.0 | | Waldmeister 600 | E‑SETHEO csp00 | Flying Saucer | ||
CASC‑16 (1999) | | | | | SPASS 1.00T | | Vampire 0.0 | OtterMACE 437 | | Waldmeister 799 | | Spray Can | ||
CASC‑15 (1998) | | | | | SPASS 1.0.0a | | Gandalf c‑1.1 | SPASS 1.0.0a | | Waldmeister 798 | | Worm and Butterfly | ||
CASC‑14 (1997) | | | | | SPASS 0.77 | | Gandalf | SPASS 0.77 | | Waldmeister | | Meat Mincer | ||
CASC‑13 (1996) | | | | | | | E-SETHEO | | | Otter 3.0.4z | |
If you would like to cite CASC (in general), please use:
@Article{Sut16, Author = "Sutcliffe, G.", Year = "2016", Title = "{The CADE ATP System Competition - CASC}", Journal = "AI Magazine", Volume = "37", Number = "2", Pages = "99-101", Comment = "TPTPCite" } @Article{SS06-SoCASC, Author = "Sutcliffe, G. and Suttner, C.", Year = "2006", Title = "{The State of CASC}", Journal = "AI Communications", Volume = "19", Number = "1", Pages = "35-48" } @Article{PSS02, Author = "Pelletier, F.J. and Sutcliffe, G. and Suttner, C.B.", Year = "2002", Title = "{The Development of CASC}", Journal = "AI Communications", Volume = "15", Number = "2-3", Pages = "79-90" }Individual CASC reports are:
@Article{SD23-CASC, Author = "Sutcliffe, G. and Desharnais, M.", Year = "2023", Title = "{The 11th IJCAR Automated Theorem Proving System Competition - CASC-J11}", Journal = "AI Communications", Volume = "36", Number = "2", Pages = "73-91," } @Article{SD22-CASC, Author = "Sutcliffe, G. and Desharnais, M.", Year = "2022", Title = "{The CADE-28 Automated Theorem Proving System Competition - CASC-28}", Journal = "AI Communications", Volume = "34", Number = "4", Pages = "259-276" } @Article{Sut21-CASC, Author = "Sutcliffe, G.", Year = "2021", Title = "{The 10th IJCAR Automated Theorem Proving System Competition - CASC-J10}", Journal = "AI Communications", Volume = "34", Number = "2", Pages = "163-177" } @Article{Sut20-CASC, Author = "Sutcliffe, G.", Year = "2020", Title = "{The CADE-27 Automated Theorem Proving System Competition - CASC-27}", Journal = "AI Communications", Volume = "32", Number = "5-6", Pages = "373-389" } @Article{Sut18-CASC, Author = "Sutcliffe, G.", Year = "2018", Title = "{The 9th IJCAR Automated Theorem Proving System Competition - CASC-29}", Journal = "AI Communications", Volume = "31", Number = "6", Pages = "495-507" } @Article{Sut17-CASC, Author = "Sutcliffe, G.", Year = "2017", Title = "{The CADE-26 Automated Theorem Proving System Competition - CASC-26}", Journal = "AI Communications", Volume = "30", Number = "6", Pages = "419-432" } @Article{Sut16-CASC, Author = "Sutcliffe, G.", Year = "2016", Title = "{The 8th IJCAR Automated Theorem Proving System Competition - CASC-J8}", Journal = "AI Communications", Volume = "29", Number = "5", Pages = "607-619" } @Article{SU16-CASC, Author = "Sutcliffe, G. and Urban, J.", Year = "2016", Title = "{The CADE-25 Automated Theorem Proving System Competition - CASC-25}", Journal = "AI Communications", Volume = "29", Number = "3", Pages = "423-433" } @Article{Sut15-CASC, Author = "Sutcliffe, G.", Year = "2015", Title = "{The 7th IJCAR Automated Theorem Proving System Competition - CASC-J7}", Journal = "AI Communications", Volume = "28", Number = "4", Pages = "683-692" } @Article{Sut14-CASC, Author = "Sutcliffe, G.", Year = "2014", Title = "{The CADE-24 Automated Theorem Proving System Competition - CASC-24}", Journal = "AI Communications", Volume = "27", Number = "4", Pages = "405-416" } @Article{Sut13, Author = "Sutcliffe, G.", Year = "2013", Title = "{The 6th IJCAR Automated Theorem Proving System Competition - CASC-J6}", Journal = "AI Communications", Volume = "26", Number = "2", Pages = "211-223" } @Article{Sut12, Author = "Sutcliffe, G.", Year = "2012", Title = "{The CADE-23 Automated Theorem Proving System Competition - CASC-23}", Journal = "AI Communications", Volume = "25", Number = "1", Pages = "49-63" } @Article{Sut11, Author = "Sutcliffe, G.", Year = "2011", Title = "{The 5th IJCAR Automated Theorem Proving System Competition - CASC-J5}", Journal = "AI Communications", Volume = "24", Number = "1", Pages = "75-89" } @Article{Sut10, Author = "Sutcliffe, G.", Year = "2010", Title = "{The CADE-22 Automated Theorem Proving System Competition - CASC-22}", Journal = "AI Communications", Volume = "23", Number = "1", Pages = "47-60" } @Article{Sut09, Author = "Sutcliffe, G.", Year = "2009", Title = "{The 4th IJCAR Automated Theorem Proving Competition}", Journal = "AI Communications", Volume = "22", Number = "1", Pages = "59-72" } @Article{Sut08, Author = "Sutcliffe, G.", Year = "2008", Title = "{The CADE-21 Automated Theorem Proving System Competition}", Journal = "AI Communications", Volume = "21", Number = "1", Pages = "71-82" } @Article{Sut07, Author = "Sutcliffe, G.", Year = "2007", Title = "{The 3rd IJCAR Automated Theorem Proving Competition}", Journal = "AI Communications", Volume = "20", Number = "2", Pages = "117-126" } @Article{Sut06, Author = "Sutcliffe, G.", Year = "2006", Title = "{The CADE-20 Automated Theorem Proving Competition}", Journal = "AI Communications", Volume = "19", Number = "2", Pages = "173-181" } @Article{Sut05, Author = "Sutcliffe, G.", Year = "2005", Title = "{The IJCAR-2004 Automated Theorem Proving Competition}", Journal = "AI Communications", Volume = "18", Number = "1", Pages = "33-40" } @Article{SS04, Author = "Sutcliffe, G. and Suttner, C.", Year = "2004", Title = "{The CADE-19 ATP System Competition}", Journal = "AI Communications", Volume = "17", Number = "3", Pages = "103-182" } @Article{SS03, Author = "Sutcliffe, G. and Suttner, C.", Year = "2003", Title = "{The CADE-18 ATP System Competition}", Journal = "Journal of Automated Reasoning", Volume = "31", Number = "1", Pages = "23-32" } @Article{SSP02, Author = "Sutcliffe, G. and Suttner, C. and Pelletier, F.J.", Year = "2002", Title = "{The IJCAR ATP System Competition}", Journal = "Journal of Automated Reasoning", Volume = "28", Number = "3", Pages = "307-320" } @Article{Sut01, Author = "Sutcliffe, G.", Year = "2001", Title = "{The CADE-17 ATP System Competition}", Journal = "Journal of Automated Reasoning", Volume = "27", Number = "3", Pages = "227-250" } @Article{Sut00, Author = "Sutcliffe, G.", Year = "2000", Title = "{The CADE-16 ATP System Competition}", Journal = "Journal of Automated Reasoning", Volume = "24", Number = "3", Pages = "371-396" } @Article{SS99, Author = "Sutcliffe, G. and Suttner, C.B.", Year = "1999", Title = "{The CADE-15 ATP System Competition}", Journal = "Journal of Automated Reasoning", Volume = "23", Number = "1", Pages = "1-23" } @Article{SS98, Author = "Suttner, C.B. and Sutcliffe, G.", Year = "1998", Title = "{The CADE-14 ATP System Competition}", Journal = "Journal of Automated Reasoning", Volume = "21", Number = "1", Pages = "99-134" } @Article{SS97, Author = "Sutcliffe, G. and Suttner, C.B.", Year = "1997", Title = "{Special Issue: The CADE-13 ATP System Competition}", Journal = "Journal of Automated Reasoning", Volume = "18", Number = "2" }
Other people are doing similar things for other types of ATP problems: | |
The Competition on Software Verification | |
The Answer Set Programming System Competition | |
The Satisfiability Modulo Theories Competition. | |
A competition for systems for proving termination. | |
The purpose of the competition is to identify new challenging benchmarks and to promote new solvers for the propositional satisfiability problem (SAT) as well as to compare them with state-of-the-art solvers. | |
The objective of MaxSAT is assessing the state of the art in the field of Max-SAT solvers, as well as creating a collection of publicly available Max-SAT benchmark instances. | |
The goal of the pseudo-Boolean competition is to assess the state of the art in the field of pseudo-Boolean solvers. | |
| An informal competition between inductive theorem proving systems. |
QBF Evaluations are a series of yearly events with the purpose of assessing the state of the art in the field of QBF solvers and QBF-based applications. The final of the evaluation are presented during the yearly SAT conferences. | |
| |
Other people are doing similar things for other types of AI problems: | |
The Robot World Cup Initiative is an international research and education initiative. It is an attempt to foster AI and intelligent robotics research by providing a standard problem where wide range of technologies can be integrated and examined, as well as being used for intergrated project-oriented education. For this purpose, RoboCup chose to use soccer game as a primary domain, and organizes RoboCup: The Robot World Cup Soccer Games and Conferences. | |
The aim of this event is to compare the expressiveness, the usability and the performance of graph and model transformation tools. A deeper understanding of the relative merits of different tool features will help to further improve graph and model transformation tools and to indicate open problems. | |
The Loebner Prize is the first formal instantiation of a Turing Test. In 1990 Hugh Loebner agreed with The Cambridge Center for Behavioral Studies to underwrite a contest designed to implement the Turing Test. Dr. Loebner pledged a Grand Prize of $100,000 and a Gold Medal for the first computer whose responses were indistinguishable from a human's. Each year an annual prize of $2000 and a bronze medal is awarded to the most human computer. The winner of the annual contest is the best entry relative to other entries that year, irrespective of how good it is in an absolute sense. |