Five Data-based Results

Overall

• 25325 problems analysed
• 19762 (78%) solved in v6.3.0, 20227 (80%) in v8.2.0
• 5563 (22%) unsolved when added, 1009 (4%) solved by v8.2.0
• 8984 (35%) rated 0.00 in v8.2.0, 2965 (12%) rated higher before
• Contributions vary across the coherent SPC sets

Coherent SPC Sets

• CNF unit equality (UEQ)
  • 1140 problems - N:120-86 S:1020-1049 A:38-233
  • Progress in v6.4.0 due to Twee 2.0, Waldmeister 710 (5 newly solved)
  • Progress in v7.4.0 due to E 2.5
  • Progress in v8.1.0 due to Twee 2.4, CSE_E 1.3
  • Waldmeister 710 had the highest Shapley values
  • Least problems solved in v7.5.0 and v8.0.0

• CNF unsatisfiable (CNF UNS)
  • 4441 problems - N:569-391 S:3873-3966 A:1004-1780
  • Small consistent decline in ratings
  • Progress in v7.0.0 due to Vampire 4.2 (33 newly solved)
  • Progress in v8.2.0 due to SnakeForV 1.0 (26 newly solved)
  • Progress in CNF → progress elsewhere

• CNF satisfiable (CNF SAT)
  • 1042 problems - N:155-147 S:887-889 A:476-598
  • Progress in v6.4.0 due to Vampire 4.0.5 (4 newly solved)
  • Less problems solved in v7.0.0 due to missing Prover9 1105 data
    • Reason is lost in the mists of time
    • Interesting that Prover9 could uniquely solve some problems

• Effectively propositional (EPR)
  • 1425 problems - N:78-43 S:1347-1360 A:1027-1311
  • Progress in v7.0.0 due to iProver 2.6
  • Progress in v7.3.0 due to iProver 3.0
  • Regress in v8.2.0 due to iProver 3.7, SnakeForV 1.0, Vampire 4.7
    • Maybe the common FOF to CNF translator?

• FOF theorems (FOF THM)
  • 7202 problems - N:1116-818 S:6086-6235 A:696-971
  • The best known FOF problems, most systems
  • Ratings quite flat
  • Number of problems solved increases
  • Progress in v7.0.0 due to Vampire 4.2 (72 newly solved), ET 0.2
  • Progress in v7.4.0 due to Enigma 0.4, Vampire 4.5

• FOF non-theorems (FOF CSA/SAT)
  • 1028 problems - N:282-256 S:746-753 A:481-709
  • Progress in v6.4.0 due to Vampire 4.0.5 (10 newly solved), iProver 2.5

• TF0 theorems without arithmetic (TF0 NAR)
  • 397 problems - N:120-103 S:277-268 A:117-123
  • Progress in v7.0.0 due to Vampire 4.2, CVC4 1.5.2
  • Progress in v8.2.0 due to SnakeForV 1.0
  • Less problems solved between v7.4.0 and v8.1.0

• TF0 theorems with arithmetic (TF0 ARI)
  • 1087 problems - N:172-58 S:915-1022 A:763-785
  • Simplest TPTP language that includes arithmetic
  • Progress in v6.4.0 due to Vampire 4.0.5, CVC4 1.5, Princess 150706

• TH0 theorems (TH0)
  • 3183 problems - N:461-305 S:2722-2814 A:617-1244
  • Ratings decline moderately
  • Progress in v7.0.0 due to Satallax 3.2
  • Progress in v7.5.0 due to Zipperposition 2.0 (18 newly solved)