Agricultural pollution poses a considerable challenge to grain security and human health, especially in economically developed areas. Mineral exploitation, chemical enterprise operation, pesticide and fertilizer application, sewage discharge, and vehicle emissions are the pollution sources of agricultural land. Identifying and assessing potential agricultural pollution (PAP) is, therefore, the most urgent task to achieve grain security and the human health. Large-scale (e.g., regional or national) PAP assessment can be very expensive, which could also generate a certain amount of information that usually discourages evaluation by decision-makers. To identify areas for regional priority investigation, here we proposed an assessment framework for PAP in economically developed areas. The framework consisted of PAP assessment, vulnerability assessment, hazard assessment, and socio-economic assessment. Then, we conducted a case study by using the proposed framework in one of China’s economically developed areas, Zhejiang Province. The results showed that PAP, especially soil heavy metal pollution, soil acidification, and surface water pollution involved almost the entire study area. High-vulnerability high-hazard areas were mainly associated with high socio-economic development or high grain yield. These areas had negatively affected grain security and increased carcinogenic risk, potentially contributing to the formation of cancer villages. Based on the results, we proposed measures for environmental risk managers to alleviate the impact of PAP on grain security and human health in economically developed areas.