Gravitational instability of planetary gaps and its effect on orbital migration

Abstract Gap formation by giant planets in self-gravitating disks may lead to a gravitational edge instability (GEI). We demonstrate this GEI with global 3D and 2D self-gravitating disk-planet simulations using the ZEUS, PLUTO and FARGO hydrodynamic codes. High resolution 2D simulations show that an unstable outer gap edge can lead to outward migration. Our results have important implications for theories of giant planet formation in massive disks.