It is now well established that for non-associated materials such as geomaterials, a broad domain exists, strictly within the plastic limit, where different failure modes can coexist. In particular, material instability as defined by Hill [Hill, R., 1958. A general theory of uniqueness and stability in elasticplastic solids. J. Mech. Phys. Solids, 6, 236249], related to the vanishing of the second-order work, can potentially occur. In this paper, we focus on granular materials and establish from theoretical considerations that the macroscopic second-order work on the assembly scale is related to the microscopic second-order work computed on each contact. This question is examined from a multi-scale approach, and the relation between macroscopic and microscopic second-order works is brought to light using a micro-mechanically based model. Finally, these theoretical results are confirmed from numerical simulations based on a discrete element method.