A new test bed for automatic incident detection (AID) systems uses real-time traffic video and data feeds from the Ontario, Canada, Ministry of Transportation COMPASS advanced traffic management system. This new test bed, called the AID comparison and analysis tool (AID CAAT), consists largely of a data warehouse storing a significant amount of traffic video, the corresponding traffic data, and an accurate log of incident start and end times. Also presented is a proof-of-concept field evaluation whereby the AID CAAT is used to calibrate and then analyze the performance of three AID algorithms: California Algorithm 8, the McMaster algorithm, and the genetic adaptive incident detection algorithm. In the calibration and testing process, nuisance rate and false normal rate are introduced as two new performance measures to supplement the three traditional measures (detection rate, false alarm rate, and mean time to detection). Further, the pilot evaluation shows the considerable advantages of AID CAAT in its ability to investigate the impact of freeway geometry, traffic flow rate, and traffic sensor spacing on the performance of the three AID algorithms. This work represents the first stage in a series of further tests to develop a set of AID algorithm deployment guidelines.