What is Required?

Rigorous, Scientific Method

• Researchers need to know ...
  where the challenges lie
• Developers need to understand ...
  which techniques work well for what types of problems
• Users need to know ...
  which systems are most likely to solve given problems

Evaluation of Systems is a Key

• Complete analysis is impossible (or P = NP)
• Developer reported emperical analysis can be inadequate
• A community accepted framework is required

Two Intuitively Acceptable Criteria

• What Problems can they Solve?

• What Resources do they Need?