Climate change is opening the Arctic Ocean to increasing human impact and ecosystem changes. Arctic fjords, the region’s most productive ecosystems, are sustained by a diverse microbial community at the base of the food web. Here we show that Arctic fjords become more prokaryotic in the picoplankton (0.2–3 µm) with increasing water temperatures. Across 21 fjords, we found that Arctic fjords had proportionally more trophically diverse (autotrophic, mixotrophic, and heterotrophic) picoeukaryotes, while subarctic and temperate fjords had relatively more diverse prokaryotic trophic groups. Modeled oceanographic connectivity between fjords suggested that transport alone would create a smooth gradient in beta diversity largely following the North Atlantic Current and East Greenland Current. Deviations from this suggested that picoeukaryotes had some strong regional patterns in beta diversity that reduced the effect of oceanographic connectivity, while prokaryotes were mainly stopped in their dispersal if strong temperature differences between sites were present. Fjords located in high Arctic regions also generally had very low prokaryotic alpha diversity. Ultimately, warming of Arctic fjords could induce a fundamental shift from more trophic diverse eukaryotic- to prokaryotic-dominated communities, with profound implications for Arctic ecosystem dynamics including their productivity patterns.