| Higher-order Theorems | Satallax‑MaLeS 1.2 | Satallax 2.7 | Isabelle 2013 | Isabelle 2012 | LEO‑II 1.6.0 | TPS 3.120601S1b | cocATP 0.1.8 |
|---|---|---|---|---|---|---|---|
| Solved/150 | 119/150 | 116/150 | 108/150 | 98/150 | 76/150 | 47/150 | 14/150 |
| Av. CPU Time | 10.42 | 11.39 | 54.65 | 60.54 | 5.05 | 25.23 | 52.05 |
| Solutions | 0/150 | 0/150 | 0/150 | 0/150 | 76/150 | 0/150 | 14/150 |
| μEfficiency | 505 | 393 | 36 | 21 | 397 | 48 | 20 |
| SOTAC | 0.25 | 0.24 | 0.25 | 0.24 | 0.24 | 0.32 | 0.23 |
| New Solved | 25/27 | 25/27 | 20/27 | 20/27 | 14/27 | 3/27 | 1/27 |
| Typed First-order Theorems +*-/ | SPASS+T 2.2.19 | Princess 120604 | Beagle 0.4 |
|---|---|---|---|
| Solved/100 | 95/100 | 91/100 | 90/100 |
| Av. CPU Time | 5.29 | 6.63 | 4.17 |
| Solutions | 95/100 | 0/100 | 0/100 |
| μEfficiency | 434 | 412 | 731 |
| SOTAC | 0.36 | 0.35 | 0.35 |
| New Solved | 10/10 | 10/10 | 9/10 |
| First-order Theorems | Vampire 2.6 | Vampire 3.0 | E 1.8 | E‑MaLeS 1.2 | iProver 1.0 | E‑KRHyper 1.4 | Prover9 1109a | Zipperposition 0.2 | Muscadet 4.3 | CVC4 1.2 | iProver‑Eq 0.85 | iProverModulo 0.7‑0.2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Solved/300 | 281/300 | 274/300 | 249/300 | 237/300 | 167/300 | 122/300 | 119/300 | 91/300 | 60/300 | 147/300 | 112/300 | 106/300 |
| Av. CPU Time | 12.24 | 10.91 | 29.02 | 14.52 | 12.23 | 8.60 | 11.87 | 15.95 | 4.11 | 14.78 | 7.12 | 18.26 |
| Solutions | 281/300 | 274/300 | 249/300 | 237/300 | 167/300 | 121/300 | 119/300 | 91/300 | 60/300 | 0/300 | 0/300 | 0/300 |
| μEfficiency | 632 | 626 | 473 | 448 | 345 | 308 | 138 | 192 | 156 | 368 | 285 | 267 |
| SOTAC | 0.20 | 0.18 | 0.18 | 0.16 | 0.13 | 0.11 | 0.12 | 0.10 | 0.14 | 0.13 | 0.11 | 0.10 |
| New Solved | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 0/2 | 1/2 | 2/2 | 2/2 |
| First-order Non-theorems | iProver 1.0‑SAT | Paradox 3.0 | CVC4 1.2‑SAT | E 1.8 | Nitrox 2013 | Vampire 3.0‑SAT | E‑KRHyper 1.4 | iProver‑Eq 0.85 |
|---|---|---|---|---|---|---|---|---|
| Solved/150 | 122/150 | 99/150 | 96/150 | 79/150 | 79/150 | 78/150 | 67/150 | 37/150 |
| Av. CPU Time | 52.47 | 2.28 | 25.94 | 20.94 | 29.70 | 15.89 | 7.57 | 30.77 |
| Solutions | 122/150 | 99/150 | 96/150 | 79/150 | 79/150 | 78/150 | 67/150 | 0/150 |
| μEfficiency | 165 | 549 | 204 | 396 | 36 | 395 | 292 | 92 |
| SOTAC | 0.28 | 0.23 | 0.19 | 0.22 | 0.24 | 0.20 | 0.19 | 0.15 |
| New Solved | 1/4 | 1/4 | 1/4 | 1/4 | 1/4 | 1/4 | 0/4 | 0/4 |
| Effectively Propositional CNF | iProver 0.9 | iProver 1.0 | iProver‑Eq 0.85 | Vampire 3.0‑EPR | PEPR 0.0ps | E 1.8 | E‑KRHyper 1.4 | 5alarm 0.1 |
|---|---|---|---|---|---|---|---|---|
| Solved/100 | 81/100 | 77/100 | 51/100 | 47/100 | 43/100 | 23/100 | 8/100 | 0/100 |
| Av. CPU Time | 26.76 | 33.86 | 40.58 | 14.72 | 26.91 | 49.93 | 23.75 | - |
| Solutions | 37/100 | 77/100 | 0/100 | 31/100 | 0/100 | 23/100 | 8/100 | 0/100 |
| μEfficiency | 269 | 240 | 122 | 199 | 181 | 62 | 51 | - |
| SOTAC | 0.31 | 0.30 | 0.24 | 0.21 | 0.20 | 0.17 | 0.21 | - |
| New Solved | 11/26 | 8/26 | 9/26 | 7/26 | 8/26 | 2/26 | 2/26 | 0/26 |
| Large Theory Batch Problems | MaLARea 0.5 | E 1.8‑LTB | Vampire 3.0‑LTB | iProver 1.0‑LTB | TEMPLAR::leanCoP 0.8 | E‑KRHyper 1.4‑LTB |
|---|---|---|---|---|---|---|
| Solved/750 | 239/750 | 135/750 | 126/750 | 89/750 | 28/750 | 3/750 |
| Av. CPU Time | 54.80 | 28.44 | 30.81 | 61.45 | 28.23 | 7.38 |
| Av. WC Time | 15.14 | 7.48 | 10.85 | 18.83 | 12.02 | 7.42 |
| Solutions | 239/750 | 135/750 | 126/750 | 89/750 | 28/750 | 0/750 |
| μEfficiency | 27 | 62 | 45 | 13 | 6 | 1 |
| SOTAC | 0.54 | 0.37 | 0.36 | 0.33 | 0.27 | 0.18 |
| Core Usage | 3.84 | 3.91 | 2.94 | 3.76 | 2.85 | 1.00 |
| New Solved | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| THF without Equality | Satallax‑MaLeS 1.2 | Satallax 2.7 | Isabelle 2013 | Isabelle 2012 | LEO‑II 1.6.0 | TPS 3.120601S1b | cocATP 0.1.8 |
|---|---|---|---|---|---|---|---|
| Solved/50 | 40/50 | 38/50 | 31/50 | 23/50 | 20/50 | 19/50 | 8/50 |
| Av. CPU Time | 5.37 | 8.64 | 39.01 | 47.02 | 4.98 | 28.89 | 59.50 |
| Solutions | 0/50 | 0/50 | 0/50 | 0/50 | 20/50 | 0/50 | 8/50 |
| μEfficiency | 473 | 389 | 49 | 19 | 308 | 51 | 31 |
| SOTAC | 0.26 | 0.25 | 0.25 | 0.23 | 0.20 | 0.49 | 0.23 |
| New Solved | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| THF with Equality | Satallax‑MaLeS 1.2 | Satallax 2.7 | Isabelle 2013 | Isabelle 2012 | LEO‑II 1.6.0 | TPS 3.120601S1b | cocATP 0.1.8 |
|---|---|---|---|---|---|---|---|
| Solved/100 | 79/100 | 78/100 | 77/100 | 75/100 | 56/100 | 28/100 | 6/100 |
| Av. CPU Time | 12.98 | 12.73 | 60.95 | 64.69 | 5.07 | 22.75 | 42.12 |
| Solutions | 0/100 | 0/100 | 0/100 | 0/100 | 56/100 | 0/100 | 6/100 |
| μEfficiency | 521 | 395 | 30 | 22 | 442 | 46 | 14 |
| SOTAC | 0.24 | 0.24 | 0.25 | 0.24 | 0.25 | 0.21 | 0.22 |
| New Solved | 25/27 | 25/27 | 20/27 | 20/27 | 14/27 | 3/27 | 1/27 |
| TFA using Integers | Princess 120604 | SPASS+T 2.2.19 | Beagle 0.4 |
|---|---|---|---|
| Solved/50 | 50/50 | 47/50 | 42/50 |
| Av. CPU Time | 10.01 | 9.13 | 8.79 |
| Solutions | 0/50 | 47/50 | 0/50 |
| μEfficiency | 500 | 389 | 538 |
| SOTAC | 0.37 | 0.36 | 0.35 |
| New Solved | 10/10 | 10/10 | 9/10 |
| TFA using Rationals xor Reals | Beagle 0.4 | SPASS+T 2.2.19 | Princess 120604 |
|---|---|---|---|
| Solved/50 | 48/50 | 48/50 | 41/50 |
| Av. CPU Time | 0.12 | 1.52 | 2.51 |
| Solutions | 0/50 | 48/50 | 0/50 |
| μEfficiency | 923 | 480 | 325 |
| SOTAC | 0.36 | 0.36 | 0.33 |
| New Solved | 0/0 | 0/0 | 0/0 |
| FOF Theorems without Equality | Vampire 2.6 | Vampire 3.0 | E 1.8 | iProver 1.0 | E‑MaLeS 1.2 | E‑KRHyper 1.4 | Zipperposition 0.2 | Prover9 1109a | Muscadet 4.3 | iProver‑Eq 0.85 | iProverModulo 0.7‑0.2 | CVC4 1.2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Solved/100 | 99/100 | 98/100 | 95/100 | 92/100 | 91/100 | 76/100 | 62/100 | 60/100 | 31/100 | 88/100 | 84/100 | 75/100 |
| Av. CPU Time | 1.65 | 3.49 | 12.01 | 2.52 | 2.98 | 3.38 | 9.91 | 2.60 | 4.17 | 2.17 | 9.38 | 12.30 |
| Solutions | 99/100 | 98/100 | 95/100 | 92/100 | 91/100 | 75/100 | 62/100 | 60/100 | 31/100 | 0/100 | 0/100 | 0/100 |
| μEfficiency | 896 | 852 | 787 | 778 | 776 | 644 | 459 | 264 | 250 | 753 | 696 | 613 |
| SOTAC | 0.12 | 0.12 | 0.12 | 0.10 | 0.10 | 0.10 | 0.09 | 0.09 | 0.09 | 0.10 | 0.10 | 0.10 |
| New Solved | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 0/2 | 2/2 | 2/2 | 1/2 |
| FOF Theorems with Equality | Vampire 2.6 | Vampire 3.0 | E 1.8 | E‑MaLeS 1.2 | iProver 1.0 | Prover9 1109a | E‑KRHyper 1.4 | Muscadet 4.3 | Zipperposition 0.2 | CVC4 1.2 | iProver‑Eq 0.85 | iProverModulo 0.7‑0.2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Solved/200 | 182/200 | 176/200 | 154/200 | 146/200 | 75/200 | 59/200 | 46/200 | 29/200 | 29/200 | 72/200 | 24/200 | 22/200 |
| Av. CPU Time | 17.99 | 15.04 | 39.51 | 21.71 | 24.13 | 21.29 | 17.22 | 4.06 | 28.86 | 17.36 | 25.24 | 52.19 |
| Solutions | 182/200 | 176/200 | 154/200 | 146/200 | 75/200 | 59/200 | 46/200 | 29/200 | 29/200 | 0/200 | 0/200 | 0/200 |
| μEfficiency | 500 | 513 | 316 | 284 | 128 | 75 | 140 | 109 | 58 | 245 | 51 | 53 |
| SOTAC | 0.24 | 0.21 | 0.21 | 0.20 | 0.16 | 0.15 | 0.14 | 0.19 | 0.12 | 0.16 | 0.12 | 0.12 |
| New Solved | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| FOF Non-theorems without Equality | iProver 1.0‑SAT | Paradox 3.0 | Vampire 3.0‑SAT | Nitrox 2013 | E 1.8 | CVC4 1.2‑SAT | E‑KRHyper 1.4 | iProver‑Eq 0.85 |
|---|---|---|---|---|---|---|---|---|
| Solved/50 | 39/50 | 35/50 | 28/50 | 28/50 | 27/50 | 22/50 | 17/50 | 21/50 |
| Av. CPU Time | 46.22 | 5.90 | 9.11 | 28.10 | 59.90 | 36.23 | 9.42 | 17.34 |
| Solutions | 39/50 | 35/50 | 28/50 | 28/50 | 27/50 | 22/50 | 17/50 | 0/50 |
| μEfficiency | 242 | 423 | 406 | 31 | 213 | 150 | 274 | 214 |
| SOTAC | 0.37 | 0.22 | 0.21 | 0.21 | 0.23 | 0.16 | 0.14 | 0.15 |
| New Solved | 0/3 | 0/3 | 1/3 | 0/3 | 1/3 | 0/3 | 0/3 | 0/3 |
| FOF Non-theorems with Equality | iProver 1.0‑SAT | CVC4 1.2‑SAT | Paradox 3.0 | E 1.8 | Nitrox 2013 | E‑KRHyper 1.4 | Vampire 3.0‑SAT | iProver‑Eq 0.85 |
|---|---|---|---|---|---|---|---|---|
| Solved/100 | 83/100 | 74/100 | 64/100 | 52/100 | 51/100 | 50/100 | 50/100 | 16/100 |
| Av. CPU Time | 55.41 | 22.89 | 0.31 | 0.71 | 30.58 | 6.94 | 19.68 | 48.41 |
| Solutions | 83/100 | 74/100 | 64/100 | 52/100 | 51/100 | 50/100 | 50/100 | 0/100 |
| μEfficiency | 127 | 230 | 612 | 487 | 38 | 301 | 390 | 31 |
| SOTAC | 0.24 | 0.20 | 0.24 | 0.21 | 0.26 | 0.21 | 0.20 | 0.14 |
| New Solved | 1/1 | 1/1 | 1/1 | 0/1 | 1/1 | 0/1 | 0/1 | 0/1 |
| EPR Unsatisfiable CNF | iProver 0.9 | iProver 1.0 | iProver‑Eq 0.85 | PEPR 0.0ps | Vampire 3.0‑EPR | E 1.8 | E‑KRHyper 1.4 | 5alarm 0.1 |
|---|---|---|---|---|---|---|---|---|
| Solved/50 | 41/50 | 39/50 | 23/50 | 15/50 | 14/50 | 5/50 | 2/50 | 0/50 |
| Av. CPU Time | 38.93 | 46.14 | 75.60 | 59.66 | 9.55 | 97.74 | 21.04 | - |
| Solutions | 0/50 | 39/50 | 0/50 | 0/50 | 14/50 | 5/50 | 2/50 | 0/50 |
| μEfficiency | 108 | 93 | 50 | 33 | 90 | 23 | 20 | - |
| SOTAC | 0.37 | 0.37 | 0.31 | 0.26 | 0.23 | 0.17 | 0.14 | - |
| New Solved | 8/13 | 7/13 | 9/13 | 8/13 | 7/13 | 2/13 | 1/13 | 0/13 |
| EPR Satisfiable CNF | iProver 0.9 | iProver 1.0 | Vampire 3.0‑EPR | PEPR 0.0ps | iProver‑Eq 0.85 | E 1.8 | E‑KRHyper 1.4 | 5alarm 0.1 |
|---|---|---|---|---|---|---|---|---|
| Solved/50 | 40/50 | 38/50 | 33/50 | 28/50 | 28/50 | 18/50 | 6/50 | 0/50 |
| Av. CPU Time | 14.28 | 21.26 | 16.92 | 9.37 | 11.81 | 36.64 | 24.65 | - |
| Solutions | 37/50 | 38/50 | 17/50 | 0/50 | 0/50 | 18/50 | 6/50 | 0/50 |
| μEfficiency | 430 | 387 | 307 | 329 | 193 | 101 | 82 | - |
| SOTAC | 0.26 | 0.23 | 0.19 | 0.18 | 0.18 | 0.17 | 0.23 | - |
| New Solved | 3/13 | 1/13 | 0/13 | 0/13 | 0/13 | 0/13 | 1/13 | 0/13 |
| LTB HOL Light Theorems | MaLARea 0.5 | Vampire 3.0‑LTB | E 1.8‑LTB | iProver 1.0‑LTB | TEMPLAR::leanCoP 0.8 | E‑KRHyper 1.4‑LTB |
|---|---|---|---|---|---|---|
| Solved/250 | 66/250 | 29/250 | 26/250 | 16/250 | 12/250 | 0/250 |
| Av. CPU Time | 62.17 | 41.88 | 34.22 | 48.42 | 26.45 | - |
| Av. WC Time | 17.02 | 14.62 | 9.07 | 16.66 | 11.80 | - |
| Solutions | 66/250 | 29/250 | 26/250 | 16/250 | 12/250 | 0/250 |
| μEfficiency | 17 | 21 | 30 | 5 | 7 | - |
| SOTAC | 0.63 | 0.32 | 0.31 | 0.27 | 0.23 | - |
| Core Usage | 3.84 | 2.94 | 3.94 | 3.76 | 2.61 | - |
| New Solved | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| LTB Isabelle Theorems | E 1.8‑LTB | MaLARea 0.5 | Vampire 3.0‑LTB | TEMPLAR::leanCoP 0.8 | iProver 1.0‑LTB | E‑KRHyper 1.4‑LTB |
|---|---|---|---|---|---|---|
| Solved/250 | 42/250 | 38/250 | 21/250 | 12/250 | 11/250 | 0/250 |
| Av. CPU Time | 33.93 | 71.21 | 44.09 | 34.01 | 68.80 | - |
| Av. WC Time | 9.04 | 19.72 | 15.64 | 14.13 | 20.76 | - |
| Solutions | 42/250 | 38/250 | 21/250 | 12/250 | 11/250 | 0/250 |
| μEfficiency | 33 | 8 | 8 | 5 | 3 | - |
| SOTAC | 0.46 | 0.45 | 0.40 | 0.34 | 0.27 | - |
| Core Usage | 3.91 | 3.63 | 2.92 | 3.19 | 3.73 | - |
| New Solved | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| LTB Mizar Theorems | MaLARea 0.5 | Vampire 3.0‑LTB | E 1.8‑LTB | iProver 1.0‑LTB | TEMPLAR::leanCoP 0.8 | E‑KRHyper 1.4‑LTB |
|---|---|---|---|---|---|---|
| Solved/250 | 135/250 | 76/250 | 67/250 | 62/250 | 4/250 | 3/250 |
| Av. CPU Time | 46.57 | 22.92 | 22.76 | 63.51 | 16.25 | 7.38 |
| Av. WC Time | 12.93 | 8.08 | 5.88 | 19.06 | 6.38 | 7.42 |
| Solutions | 135/250 | 76/250 | 67/250 | 62/250 | 4/250 | 0/250 |
| μEfficiency | 58 | 106 | 123 | 30 | 4 | 4 |
| SOTAC | 0.53 | 0.36 | 0.34 | 0.36 | 0.22 | 0.18 |
| Core Usage | 3.71 | 2.94 | 3.98 | 3.76 | 2.92 | 1.00 |
| New Solved | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |