Higher-order Theorems Satallax
3.4
Satallax
3.3
Leo‑III
1.4
Zipperpin
1.5
Vampire
THF‑4.4
CVC4
1.7
LEO‑II
1.7.0
Solved/500 418/500 399/500 359/500 356/500 304/500 268/500 179/500
Av. CPU Time 26.30 25.51 21.45 36.34 23.52 7.86 6.73
Av. WC Time 26.25 25.49 16.00 36.59 23.24 7.84 6.78
Solutions 418 83% 399 79% 359 71% 356 71% 304 60% 268 53% 176 35%
μEfficiency 289 283 450 356 264 420 291
μWCEfficiency 291 283 417 356 264 419 290
SOTAC 0.23 0.22 0.21 0.20 0.19 0.20 0.17
Core Usage 0.97 0.97 1.44 0.92 0.95 0.86 0.89
New Solved 0/0 0/0 0/0 0/0 0/0 0/0 0/0
Typed First-order Theorems +*-/ Vampire
4.3
Vampire
4.4
CVC4
1.7
Solved/200 169/200 168/200 159/200
Av. CPU Time 16.79 22.07 18.05
Av. WC Time 16.60 21.76 18.12
Solutions 169 84% 168 84% 159 79%
μEfficiency 505 449 471
μWCEfficiency 508 449 471
SOTAC 0.38 0.37 0.43
Core Usage 0.90 0.93 0.83
New Solved 0/0 0/0 0/0
First-order Theorems Vampire
4.3
Vampire
4.4
E
2.4
CSE_E
1.1
Enigma
0.4
CVC4
1.7
iProver
3.0
GKC
0.4
Zipperpin
1.5
leanCoP
2.2
CSE
1.2
nanoCoP
1.1
Prover9
1109a
Twee
2.2
PyRes
1.0
Etableau
0.1
Solved/500 437/500 434/500 375/500 358/500 323/500 259/500 256/500 248/500 126/500 120/500 116/500 110/500 100/500 52/500 7/500 287/500
Av. CPU Time 14.67 13.90 19.28 25.12 11.40 32.49 23.00 15.35 34.74 31.75 74.00 31.35 28.09 36.87 11.53 5.55
Av. WC Time 14.51 13.76 19.30 25.00 11.35 32.54 22.96 15.36 34.82 31.09 73.82 30.69 28.20 36.88 11.55 4.64
Solutions 437 87% 434 86% 375 75% 358 71% 323 64% 259 51% 256 51% 248 49% 126 25% 120 24% 116 23% 110 22% 100 20% 52 10% 7  1% 0  0%
μEfficiency 439 455 387 304 224 211 173 228 67 43 45 40 72 30 3 304
μWCEfficiency 439 464 387 297 219 210 172 228 67 70 43 65 78 30 3 280
SOTAC 0.17 0.16 0.14 0.13 0.12 0.14 0.11 0.11 0.09 0.09 0.09 0.09 0.12 0.13 0.07 0.11
Core Usage 0.95 0.94 0.95 1.04 1.01 0.94 0.99 0.95 0.98 0.87 1.01 0.86 0.94 0.96 0.93 1.14
New Solved 35/54 32/54 26/54 32/54 23/54 15/54 21/54 24/54 10/54 6/54 8/54 6/54 10/54 5/54 1/54 20/54
FEQ with Wall clock limit Vampire
FEW‑4.4
Enigma
FEW‑0.4
iProver
FEW‑3.0
Etableau
0.1
Solved/400 371/400 313/400 191/400 215/400
Av. CPU Time 61.18 90.13 144.99 8.52
Av. WC Time 8.87 12.84 21.05 5.49
Solutions 370 92% 313 78% 191 47% 0  0%
μEfficiency 472 100 62 262
μWCEfficiency 613 370 172 267
SOTAC 0.42 0.34 0.28 0.28
Core Usage 3.90 5.64 6.81 1.12
New Solved 34/44 23/44 11/44 11/44
First-order Non-theorems Vampire
SAT‑4.3
Vampire
SAT‑4.4
CVC4
SAT‑1.7
iProver
SAT‑3.0
E
FNT‑2.4
PyRes
1.0
Solved/200 189/200 189/200 99/200 111/200 42/200 14/200
Av. CPU Time 13.31 14.75 24.75 25.40 10.21 7.98
Av. WC Time 13.16 14.56 24.78 7.29 10.21 7.98
Solutions 189 94% 189 94% 99 49% 98 49% 42 21% 14  7%
μEfficiency 402 302 199 302 121 9
μWCEfficiency 402 289 203 207 121 9
SOTAC 0.33 0.33 0.24 0.25 0.25 0.17
Core Usage 0.96 0.99 0.96 2.89 0.96 1.00
New Solved 27/28 27/28 10/28 0/28 0/28 0/28
Effectively Propositional CNF Vampire
4.4
iProver
2.8
iProver
3.0
E
FNT‑2.4
PyRes
1.0
GKC
0.4
Solved/75 29/75 26/75 22/75 8/75 2/75 1/75
Av. CPU Time 40.33 43.47 55.01 11.02 3.23 66.47
Av. WC Time 39.79 43.42 54.96 11.01 3.19 66.54
Solutions 29/75 25/75 21/75 8/75 2/75 1/75
μEfficiency 51 51 46 74 7 0
μWCEfficiency 51 51 46 74 7 0
SOTAC 0.59 0.43 0.35 0.52 0.20 0.25
Core Usage 1.00 0.99 0.97 1.00 1.01 1.00
New Solved 1/1 1/1 1/1 1/1 1/1 0/1
Unit Equality CNF Waldmeister
710
E
2.4
Twee
2.2
Vampire
4.4
MaedMax
1.3
GKC
0.4
iProver
3.0
Solved/200 164/200 161/200 151/200 142/200 102/200 88/200 78/200
Av. CPU Time 4.75 15.58 10.70 13.48 22.13 14.01 26.06
Av. WC Time 4.98 15.64 10.70 13.29 22.14 14.07 26.05
Solutions 164 82% 161 80% 150 75% 142 71% 102 51% 88 44% 78 39%
μEfficiency 564 591 466 395 212 252 159
μWCEfficiency 595 595 466 391 212 257 159
SOTAC 0.27 0.23 0.23 0.21 0.20 0.16 0.16
Core Usage 0.81 0.92 0.95 0.95 0.98 0.92 0.95
New Solved 20/48 31/48 29/48 28/48 1/48 13/48 14/48
Large Theory Batch Problems Leo‑III
LTB‑1.4
E
LTB‑2.4
MaLARea
0.8
Vampire
LTB‑4.4
MaLARea
0.6
iProver
LTB‑3.0
GKC
LTB‑0.4
Solved/10000 5440/10000 4644/10000 4640/10000 4499/10000 4380/10000 3793/10000 2755/10000
Av. CPU Time 30.79 9.80 14.24 199.44 10.36 10.41 1.86
Av. WC Time 4.32 1.85 2.99 25.00 2.37 1.69 1.86
Solutions 5440 54% 4644 46% 4640 46% 4499 44% 4380 43% 3789 37% 2755 27%
μEfficiency 22 343 62 336 65 184 204
μWCEfficiency 137 391 207 337 206 320 205
SOTAC 0.27 0.18 0.18 0.18 0.17 0.16 0.15
Core Usage 7.06 2.49 3.92 2.77 3.71 3.83 1.00
New Solved 0/0 0/0 0/0 0/0 0/0 0/0 0/0
THF without Equality Satallax
3.4
Satallax
3.3
Leo‑III
1.4
Zipperpin
1.5
Vampire
THF‑4.4
CVC4
1.7
LEO‑II
1.7.0
Solved/100 77/100 76/100 71/100 68/100 62/100 52/100 47/100
Av. CPU Time 14.09 14.93 10.83 22.23 11.21 5.26 2.18
Av. WC Time 14.13 14.97 4.04 22.78 11.10 5.33 2.23
Solutions 77 77% 76 76% 71 71% 68 68% 62 62% 52 52% 47 47%
μEfficiency 316 322 566 440 446 419 420
μWCEfficiency 327 323 541 440 446 419 415
SOTAC 0.23 0.21 0.21 0.19 0.19 0.18 0.16
Core Usage 0.95 0.94 1.34 0.88 0.91 0.78 0.84
New Solved 0/0 0/0 0/0 0/0 0/0 0/0 0/0
THF with Equality Satallax
3.4
Satallax
3.3
Leo‑III
1.4
Zipperpin
1.5
Vampire
THF‑4.4
CVC4
1.7
LEO‑II
1.7.0
Solved/400 341/400 323/400 288/400 288/400 242/400 216/400 132/400
Av. CPU Time 29.06 28.00 24.07 39.67 26.68 8.49 8.36
Av. WC Time 28.98 27.97 18.95 39.85 26.35 8.45 8.39
Solutions 341 85% 323 80% 288 72% 288 72% 242 60% 216 54% 129 32%
μEfficiency 282 273 421 335 218 420 259
μWCEfficiency 282 273 387 335 218 419 259
SOTAC 0.23 0.23 0.21 0.20 0.19 0.20 0.17
Core Usage 0.97 0.98 1.46 0.93 0.96 0.87 0.91
New Solved 0/0 0/0 0/0 0/0 0/0 0/0 0/0
TFA using Integers Vampire
4.3
Vampire
4.4
CVC4
1.7
Solved/125 100/125 100/125 87/125
Av. CPU Time 28.13 36.44 30.71
Av. WC Time 27.78 35.91 30.77
Solutions 100 80% 100 80% 87 69%
μEfficiency 285 253 259
μWCEfficiency 286 253 259
SOTAC 0.40 0.39 0.46
Core Usage 0.95 0.97 0.91
New Solved 0/0 0/0 0/0
TFA using Reals CVC4
1.7
Vampire
4.3
Vampire
4.4
Solved/75 72/75 69/75 68/75
Av. CPU Time 2.76 0.34 0.93
Av. WC Time 2.83 0.40 0.95
Solutions 72 96% 69 92% 68 90%
μEfficiency 824 871 776
μWCEfficiency 824 878 776
SOTAC 0.39 0.35 0.34
Core Usage 0.73 0.82 0.88
New Solved 0/0 0/0 0/0
FOF Theorems without Equality Vampire
4.4
Vampire
4.3
E
2.4
CSE_E
1.1
iProver
3.0
GKC
0.4
Enigma
0.4
CVC4
1.7
leanCoP
2.2
nanoCoP
1.1
CSE
1.2
Zipperpin
1.5
Twee
2.2
Prover9
1109a
PyRes
1.0
Etableau
0.1
Solved/100 87/100 87/100 80/100 79/100 78/100 70/100 68/100 56/100 49/100 45/100 43/100 33/100 12/100 11/100 2/100 74/100
Av. CPU Time 6.19 11.83 6.80 16.03 14.55 4.15 13.01 50.17 29.52 29.85 84.07 42.17 61.57 8.58 15.05 3.97
Av. WC Time 6.14 11.71 6.82 15.94 14.51 4.18 12.97 50.21 29.01 29.20 83.93 42.27 61.56 8.72 15.02 3.53
Solutions 87 87% 87 87% 80 80% 79 79% 78 78% 70 70% 68 68% 56 56% 49 49% 45 45% 43 43% 33 33% 12 12% 11 11% 2  2% 0  0%
μEfficiency 632 489 559 404 244 431 227 105 88 74 59 61 4 41 1 459
μWCEfficiency 647 494 559 377 244 431 227 105 133 115 59 61 4 41 1 364
SOTAC 0.12 0.12 0.13 0.11 0.11 0.10 0.10 0.10 0.09 0.09 0.09 0.08 0.09 0.08 0.08 0.10
Core Usage 0.97 0.95 0.98 0.99 1.00 0.91 1.01 0.96 0.87 0.91 1.01 0.98 1.00 0.92 1.00 1.30
New Solved 8/10 8/10 8/10 10/10 9/10 9/10 7/10 1/10 6/10 6/10 6/10 3/10 0/10 3/10 1/10 9/10
FOF Theorems with Equality Vampire
4.3
Vampire
4.4
E
2.4
CSE_E
1.1
Enigma
0.4
CVC4
1.7
GKC
0.4
iProver
3.0
Zipperpin
1.5
Prover9
1109a
CSE
1.2
leanCoP
2.2
nanoCoP
1.1
Twee
2.2
PyRes
1.0
Etableau
0.1
Solved/400 350/400 347/400 295/400 279/400 255/400 203/400 178/400 178/400 93/400 89/400 73/400 71/400 65/400 40/400 5/400 213/400
Av. CPU Time 15.38 15.83 22.66 27.69 10.98 27.62 19.76 26.70 32.10 30.50 68.07 33.29 32.39 29.46 10.12 6.10
Av. WC Time 15.21 15.67 22.69 27.56 10.92 27.66 19.76 26.66 32.18 30.61 67.86 32.53 31.72 29.47 10.15 5.02
Solutions 350 87% 347 86% 295 73% 279 69% 255 63% 203 50% 178 44% 178 44% 93 23% 89 22% 73 18% 71 17% 65 16% 40 10% 5  1% 0  0%
μEfficiency 426 411 345 279 223 237 178 156 69 79 41 32 31 37 4 266
μWCEfficiency 425 418 345 277 217 236 178 154 69 87 40 54 53 37 4 258
SOTAC 0.19 0.17 0.14 0.13 0.12 0.15 0.11 0.12 0.10 0.12 0.09 0.10 0.09 0.15 0.07 0.12
Core Usage 0.95 0.94 0.95 1.05 1.01 0.94 0.97 0.99 0.98 0.94 1.01 0.87 0.82 0.95 0.90 1.08
New Solved 27/44 24/44 18/44 22/44 16/44 14/44 15/44 12/44 7/44 7/44 2/44 0/44 0/44 5/44 0/44 11/44
FOF Non-theorems without Equality Vampire
SAT‑4.3
Vampire
SAT‑4.4
iProver
SAT‑3.0
CVC4
SAT‑1.7
E
FNT‑2.4
PyRes
1.0
Solved/100 95/100 95/100 81/100 65/100 19/100 1/100
Av. CPU Time 17.53 18.34 31.35 27.14 6.11 3.23
Av. WC Time 17.29 18.10 9.12 27.17 6.10 3.25
Solutions 95 95% 95 95% 81 81% 65 65% 19 19% 1  1%
μEfficiency 511 318 428 227 61 2
μWCEfficiency 511 292 292 234 61 2
SOTAC 0.27 0.27 0.26 0.25 0.29 0.17
Core Usage 0.94 0.99 2.88 0.96 0.99 0.99
New Solved 0/0 0/0 0/0 0/0 0/0 0/0
FOF Non-theorems with Equality Vampire
SAT‑4.3
Vampire
SAT‑4.4
CVC4
SAT‑1.7
E
FNT‑2.4
iProver
SAT‑3.0
PyRes
1.0
Solved/100 94/100 94/100 34/100 23/100 30/100 13/100
Av. CPU Time 9.05 11.12 20.17 13.59 9.31 8.35
Av. WC Time 8.99 10.99 20.20 13.61 2.33 8.34
Solutions 94 94% 94 94% 34 34% 23 23% 17 17% 13 13%
μEfficiency 293 286 171 181 175 15
μWCEfficiency 293 285 171 181 121 15
SOTAC 0.39 0.39 0.24 0.21 0.21 0.17
Core Usage 0.98 0.99 0.95 0.94 2.92 1.00
New Solved 27/28 27/28 10/28 0/28 0/28 0/28
EPR Unsatisfiable CNF Vampire
4.4
iProver
2.8
iProver
3.0
E
FNT‑2.4
GKC
0.4
PyRes
1.0
Solved/50 18/50 13/50 9/50 5/50 1/50 0/50
Av. CPU Time 54.29 72.52 86.11 17.54 66.47 -
Av. WC Time 53.58 72.44 86.01 17.53 66.54 -
Solutions 18/50 12/50 8/50 5/50 1/50 0/50
μEfficiency 16 5 3 51 0 -
μWCEfficiency 16 5 3 51 0 -
SOTAC 0.74 0.52 0.36 0.70 0.25 -
Core Usage 1.01 1.00 1.00 1.01 1.00 -
New Solved 0/0 0/0 0/0 0/0 0/0 0/0
EPR Satisfiable CNF iProver
2.8
iProver
3.0
Vampire
4.4
E
FNT‑2.4
PyRes
1.0
GKC
0.4
Solved/25 13/25 13/25 11/25 3/25 2/25 0/25
Av. CPU Time 14.41 33.47 17.48 0.15 3.23 -
Av. WC Time 14.40 33.46 17.24 0.15 3.19 -
Solutions 13/25 13/25 11/25 3/25 2/25 0/25
μEfficiency 143 133 121 120 21 -
μWCEfficiency 143 133 121 120 21 -
SOTAC 0.34 0.34 0.36 0.22 0.20 -
Core Usage 0.97 0.95 0.98 0.98 1.01 -
New Solved 1/1 1/1 1/1 1/1 1/1 0/1
Unit Equality CNF Waldmeister
710
E
2.4
Twee
2.2
Vampire
4.4
MaedMax
1.3
GKC
0.4
iProver
3.0
Solved/200 164/200 161/200 151/200 142/200 102/200 88/200 78/200
Av. CPU Time 4.75 15.58 10.70 13.48 22.13 14.01 26.06
Av. WC Time 4.98 15.64 10.70 13.29 22.14 14.07 26.05
Solutions 164 82% 161 80% 150 75% 142 71% 102 51% 88 44% 78 39%
μEfficiency 564 591 466 395 212 252 159
μWCEfficiency 595 595 466 391 212 257 159
SOTAC 0.27 0.23 0.23 0.21 0.20 0.16 0.16
Core Usage 0.81 0.92 0.95 0.95 0.98 0.92 0.95
New Solved 20/48 31/48 29/48 28/48 1/48 13/48 14/48
LTB HOL4 Theorems Leo‑III
LTB‑1.4
E
LTB‑2.4
MaLARea
0.8
Vampire
LTB‑4.4
MaLARea
0.6
iProver
LTB‑3.0
GKC
LTB‑0.4
Solved/10000 5440/10000 4644/10000 4640/10000 4499/10000 4380/10000 3793/10000 2755/10000
Av. CPU Time 30.79 9.80 14.24 199.44 10.36 10.41 1.86
Av. WC Time 4.32 1.85 2.99 25.00 2.37 1.69 1.86
Solutions 5440 54% 4644 46% 4640 46% 4499 44% 4380 43% 3789 37% 2755 27%
μEfficiency 22 343 62 336 65 184 204
μWCEfficiency 137 391 207 337 206 320 205
SOTAC 0.27 0.18 0.18 0.18 0.17 0.16 0.15
Core Usage 7.06 2.49 3.92 2.77 3.71 3.83 1.00
New Solved 0/0 0/0 0/0 0/0 0/0 0/0 0/0