AAAI presidential panel – AI evaluation

Marcin Wilkowski / More info please / Licenced by CC-BY 4.0

The Future of AI Research report, published in March 2025, aims to clearly identify the trajectory of AI research in a structured way. The report was led by outgoing AAAI President Francesca Rossi and covers 17 different AI topics. Members of the report team, and other selected AI practitioners, are taking part in a series of video panel discussions covering selected chapters from the report.

In the next discussion in the collection, the panellists discuss AI evaluation. Specifically, they cover the following topics:

• Why standard software testing methods fail for AI systems
• The limits of benchmark-driven testing (and how Goodhart’s Law creeps in)
• The four dimensions of evaluation the report says we’re missing: capability, usability, human-system performance, and legal/ethical compliance
• Open research challenges — from monitoring systems that evolve after deployment to evaluating agentic AI safety

Panel Members

• Dawn Song, University of California, Berkeley
• Sara Hooker, Co-Founder, adaption
• Karen Myers, Carnegie Mellon University

Moderator

• Francesca Rossi, AAAI past president, IBM Fellow and AI Ethics Global Leader

Lucy Smith is Senior Managing Editor for AIhub.

Lucy Smith is Senior Managing Editor for AIhub.