Bifurcation Analysis of a Predator-Prey System with Nonmonotonic Functional Response

We consider a predator-prey system with nonmonotonic functional response: $p(x)=\frac{mx}{ax^2+bx+1}$. By allowing b to be negative ($b > -2\sqrt a$), p(x) is concave up for small values of x > 0 as it is for the sigmoidal functional response. We show that in this case there exists a Bogdanov--Takens bifurcation point of codimension 3, which acts as an organizing center for the system. We study the Hopf and homoclinic bifurcations and saddle-node bifurcation of limit cycles of the system. We also describe the bifurcation sequences in each subregion of parameter space as the death rate of the predator is varied. In contrast with the case $b\geq 0$, we prove that when $-2\sqrt a < b < 0$, a limit cycle can coexist with a homoclinic loop. The bifurcation sequences involving Hopf bifurcations, homoclinic bifurcations, as well as the saddle-node bifurcations of limit cycles are determined using information from the complete study of the Bogdanov--Takens bifurcation point of codimension 3 and the geometry of the system. Examples of the predicted bifurcation curves are also generated numerically using XPPAUT. Our work extends the results in [F. Rothe and D. S. Shafer, Proc. Roy. Soc. Edinburgh Sect. A, 120 (1992), pp. 313--347] and [S.Ruan and D. Xiao, SIAM J. Appl. Math., 61 (2001), pp. 1445--1472].