Accurate models for the viscous densification of snow (understood here as a density below 550 kg m−3) and firn (a density above 550 kg m−3) under mechanical stress are of primary importance for various applications, including avalanche prediction and the interpretation of ice cores. Formulations of snow and firn compaction in models are still largely empirical instead of using microstructures from micro-computed tomography to numerically compute the mechanical behavior directly from the physics at the microscale. The main difficulty of the latter approach is the choice of the correct rheology/constitutive law governing the deformation of the ice matrix, which is still controversially discussed. Being aware of these uncertainties, we conducted a first systematic attempt of microstructure-based modeling of snow and firn compaction. We employed the finite element suite Elmer FEM using snow and firn microstructures from different sites in the Alps and Antarctica to explore which ice rheologies are able to reproduce observations. We thereby extended the ParStokes solver in Elmer FEM to facilitate parallel computing of transverse isotropic material laws for monocrystalline ice. We found that firn densification can be reasonably well simulated across different sites assuming a polycrystalline rheology (Glen’s law) that is traditionally used in glacier or ice sheet modeling. In contrast, for snow, the observations are in contradiction with this rheology. To further comprehend this finding, we conducted a sensitivity study on different ice rheologies. None of the material models is able to explain the observed high compactive viscosity of depth hoar compared to rounded grains having the same density. While, on one hand, our results re-emphasize the limitations of our current mechanical understanding of the ice in snow, they constitute, on the other hand, a confirmation of the common picture of firn as a foam of polycrystalline ice through microstructure-based simulations.