The influence of geographical environmental factors on regional economic disparity has always been the focus and hot issue of scholars at home and abroad. Detecting the influence of each factor on regional economic disparity and scientifically revealing the mechanism of each factor will provide important guidance for the formulation of regional economic development strategy. Although there are many studies on that question, the previous studies were lacked of contrastive analysis of impact factors between poor counties and affluent counties in Jiangxi Province and also ignored the interactive and combined influence. To solve these problems, using multiple regression and the geographic detector, we synthetically detected the influence of terrain, landform, land use, traffic location and other factors on economic differences among typical poor counties and affluent counties in Jiangxi Province and analyzed the influence, interaction and instruction of various factors. The results show that: (1) The 9 selected geographical environment factors all have an impact on the economy of poor counties and affluent counties, and different factors have different influence and the dominant factors are also different between the two types of countries. (2) Poor counties are mainly affected by natural conditions and geographical location and affluent counties are mainly affected by resource abundance and traffic location. (3) Comparing the dominant factors between the two types of counties, we find that from the needy areas to the affluent areas, dominant factors gradually change from natural endowments and other uncontrollable factors to traffic, technology and other controllable factors. (4) According to the dominant factors of regional economic disparity, economic development should adapt to local conditions and implement different strategies on different county types. At the same time, paying attention to the improvement of enhanced interaction-factors will play obvious effect on the development of economy.