Cycle-level vs. Second-by-Second Adaptive Traffic Signal Control using Deep Reinforcement Learning

With unrelenting growth in population and urbanization, cities face escalating challenges in providing fast and reliable mobility. A significant source of delay for urban commuters is related to signalized intersections. optimizing traffic signals to maximize the capacity of intersections is of critical importance for cities. Although this topic has been around for decades, recently, Deep Reinforcement Learning (DRL) approaches have begun to be used for intelligent traffic signal control. The result is a promising new generation of traffic signal controllers (TSCs). They are model-free, adaptive, and capable of exploiting newer pervasive sensory technologies (e.g., connected vehicles). While most of the RL-based TSCs focus on second-by-second level decision making, the industry favors controllers that manipulate signals less frequently. This changes the RL control problem from being in a discrete to a continuous action space. With the lack of RL-based TSCs that can handle a continuous action space, we propose a novel RL-based cycle-level TSC that determines the phase timings once every cycle. Our controller uses Proximal Policy optimization (PPO) as one of the most promising continuous action RL algorithms to produce signal timings in a cycle. We test our proposed controller against one RL-based second-level controller as well as an optimized fixed-time traffic signal controller and compare the results.