Climate change threatens the Earth’s biggest terrestrial organic carbon reservoir: permafrost soils. With climate warming, frozen soil organic matter may thaw and become available for microbial decomposition and subsequent greenhouse gas emissions. Permafrost soils are extremely heterogenous within the soil profile and between landforms. This heterogeneity in environmental conditions, carbon content and soil organic matter composition, potentially leads to different microbial communities with different responses to warming. The aim of the present study is to (1) elucidate these differences in microbial community compositions and (2) investigate how these communities react to warming. We performed short-term warming experiments with permafrost soil organic matter from northwestern Canada. We compared two sites characterized by different glacial histories (Laurentide Ice Sheet cover during LGM and without glaciation), three landscape types (low-center, flat-center, high-center polygons) and four different soil horizons (organic topsoil layer, mineral topsoil layer, cryoturbated soil layer, and the upper permanently frozen soil layer). We incubated aliquots of all soil samples at 4 °C and at 14 °C for 8 weeks and analyzed microbial community compositions (amplicon sequencing of 16S rRNA gene and ITS1 region) before and after the incubation, comparing them to microbial growth, microbial respiration, microbial biomass and soil organic matter composition. We found distinct bacterial, archaeal and fungal communities for soils of different glaciation history, polygon types and for different soil layers. Communities of low-center polygons differ from high-center and flat-center polygons in bacterial, archaeal and fungal community compositions, while communities of organic soil layers are significantly different from all other horizons. Interestingly, permanently frozen soil layers differ from all other horizons in bacterial and archaeal, but not fungal community composition. The 8-week incubations led to minor shifts in bacterial and archaeal community composition between initial soils and those subjected to 14 °C warming. We also found a strong warming effect on the community compositions in some of the extreme habitats: microbial community compositions of (i) the upper permanently frozen layer and of (ii) low-center polygons differ significantly for incubations at 4 °C and 14 °C. Yet, the lack of a community change in horizons of the active layer suggests that microbes are adapted to fluctuating temperatures due to seasonal thaw events. Our results suggest that warming responses of permafrost soil organic matter, if not frozen or water-saturated, may be predictable by current models. Process changes induced by short-term warming can be rather attributed to changes in microbial physiology than community composition. This work is part of the EU H2020 project “Nunataryuk”.