About The Intelligence Game
Welcome to The Intelligence Game. This research project looks at new methods of eliciting and collecting judgments of uncertain future events. Our goal is to improve geopolitical intelligence forecasting. This is a web-based project, with monthly questions about current world events.
We are basing our research on a modified version of the Delphi method, basically a process of estimate-feedback-estimate. Delphi is based on the principle that forecasts (or judgments) from a group of individuals are more accurate than an expert on any particular single judgment. This is commonly known as "collective intelligence".
Research Grant:
This project is funded by IARPA (Intelligence Advanced Research Projects Activity) — an initiative of the US Office of the Director of National Intelligence. You can find out more about IARPA here.
The project is part of the Aggregative Contingent Estimation (ACE) program. A synopsis of the ACE program is provided on the IARPA website.
University of Melbourne Australian Centre of Excellence for Risk Analysis (CEBRA) has been engaged as a research partner. You can find out more about CEBRA here. Professor Mark Burgman is the Director of CEBRA, and you can find out more here.
Background to the study:
In 2011, Intelligence Advanced Research Projects Activity (IARPA) engaged five U.S. collaborative research groups to look at how best to make judgements. The University of Melbourne Australian Centre of Excellence for Risk Analysis (CEBRA) is the only research partner participating from outside the U.S.A. CEBRA has joined one of the five U.S. teams, headed by George Mason University, called DAGGRE. After one year in the project, our Delphi teams beat the competition benchmark by 48%. We look forward to another exciting year, and even better performance!
Delphi Study Groups:
Melbourne Centre of Excellence for Biosecurity Risk Analysis (CEBRA) will recruit volunteers to participate in groups from both Australia and the U.S.. Participants will be selected from the volunteers to form groups that are as diverse and balanced as possible.
CEBRA have developed a website where individuals in groups can share information and have discussions about relevant topics within their group.
Data:
All of the data collected from participants in the research will be de-identified and stored securely in CEBRA, at the University of Melbourne. We will report group results and illustrative examples only, that is, nothing that could identify an individual. Only researchers or students working with us (and listed below) who have agreed to respect the confidentiality and security of data will be given access to it, for the purposes of our research program. All data will be destroyed 5 years after final publication.
Researchers and students working on this project:
University of Melbourne: Mark Burgman (CEBRA), Fiona Fidler (CEBRA), Louisa Flander (School of Population Health & CEBRA), Neil Thomason (Department of Philosophy), Raquel Ashton (Student, Botany), Marissa McBride (Student, Botany), Geoff Saw (Student, Psychology & CEBRA), Bonnie Wintle (Student, Botany).
External: Aidan Lyon (University of Maryland), Donald Gantz (George Mason University), Charles Twardy (Project Leader, George Mason University), Kenneth Olson (George Mason University), Steven Mascaro (Bayesian Intelligence).
Brief Biographical Sketches of Key Researchers:
Mark Burgman, Ph.D. is Managing Director of the Centre of Excellence for Biosecurity Risk Analysis (CEBRA) and the Adrienne Clarke Chair of Botany in the School of Botany at the University of Melbourne. He works on risk assessment and expert elicitation, with a focus on biosecurity. He received a BSc from the University of New South Wales (1974), an MSc from Macquarie University, Sydney (1981), and a PhD from the State University of New York (1987). He worked as a consultant ecologist and research scientist in Australia, the United States and Switzerland during the 1980s before joining the University of Melbourne in 1990. He has published five authored books, two edited books, over 150 research papers, and more than 50 reviewed reports and commentaries. He was elected to the Australian Academy of Science in 2006.
Fiona Fidler, Ph.D. is a Senior Research Fellow at the Australian Centre of Excellence for Risk Analysis at the University of Melbourne. She received her PhD in History and Philosophy of Science from the University of Melbourne, and was an Australian Research Council Postdoctoral Fellow in the School of Psychological Sciences at La Trobe University. She publishes on statistical cognition — researchers and students' understanding of statistical concepts — and on institutional inertia and resistance to change in statistical practice in science. Her substantive cognitive psychology interest is judgment and decision making under conditions of uncertainty, including evidence-based practice in expert elicitation.
Louisa Flander, Ph.D. is a Senior Research Fellow at the Centre for Molecular, Environmental, Genetic & Analytic Epidemiology at the University of Melbourne, and a member of CEBRA, the Australian Centre for Excellence in Risk Analysis at the University of Melbourne. She received her PhD from the University of Colorado (Anthropology), and did post-doctoral training in epidemiology at the University of California, San Francisco. In collaboration with CEBRA scientists and others, she looks at environmental health risks with unknown probabilities and uncertain, potentially severe outcomes.
Marissa McBride, B.Sc. is a doctoral student in the School of Botany at the University of Melbourne. She earned her bachelor's degree in mathematics from the University of Queensland before joining the doctoral program at the University of Melbourne. Her dissertation research focuses on methods for evaluating and improving the use of expert knowledge in environmental decision-making.
Steven Mascaro, Ph.D., is a consultant at Bayesian Intelligence. In this role, he provides advice on Bayesian networks, causal modelling and knowledge engineering and develops BN and related software solutions. He also currently sits on the board of the Australasian Bayesian Network Modelling Society. He received his Ph.D. from Monash University in 2008, which proposed the use of evolutionary Artificial Life simulations to investigate ethical scenarios. He is co-author of the book Evolving Ethics: The New Science of Good and Evil and has interests in Bayesian networks, knowledge engineering, decision support, software development, artificial life, evolutionary psychology, evolutionary ethics and the epistemology of simulation.
Neil Thomason, Ph.D. has long been interested in improving the quality of the IC's analytic products; he was a NIC Associate for several years. He received a Ph.D. in Philosophy from Berkeley. His first refereed publication "No-First-Use Unknowables" (Lieberman and Thomason 1986) proposed a way to improve the logic of political scientists' arguments. Rieber and Thomason's "Better Intelligence Analysis Requires a National Institute for Analytic Methods" (2005 "Studies in Intelligence") argues the necessity of careful scientific testing of proposed analytic techniques, a concern that drove his 2009 National Research Council report on the Analysis of Competing Hypotheses.
Bonnie Wintle, B.Sc. is a doctoral student in the School of Botany at the University of Melbourne. She completed her undergraduate in environmental science and geography at the University of Melbourne, and her Honours in plant ecology at the University of Tasmania. She has since turned to cognitive psychology to examine expert judgement in environmental decision making. Her PhD develops strategies to improve the calibration of scientific judgements under uncertainty.