The first interaction of water column-derived organic matter with benthic microbial communities takes place in surface sediments which are acting as biological filters catalyzing central steps of elemental cycling. Here we analyzed the bacterial diversity and community structure of sediment top layers at seven sites in the North Sea where sediment properties ranged from coarse-grained and highly permeable to fine-grained and impermeable. Bacterial communities in surface sediments were richer, more even and significantly different from communities in bottom waters as revealed by Illumina tag sequencing of 16S rRNA genes. Sediment permeability had a clear influence on community composition which was confirmed by CARD-FISH. Sulfate-reducing Desulfobacteraceae (2–5% of total cells), Flavobacteriaceae (3–5%) were more abundant in impermeable than in highly permeable sediments where acidobacterial Sva0725 dominated (11–15%). Myxobacterial Sandaracinaceae were most abundant in medium permeable sediments (3–7%). Woeseiaceae/JTB255 and Planctomycetes were major groups in all sediments (4–6%, 8–22%). Planctomycetes were highly diverse and branched throughout the phylum. We propose Planctomycetes as key bacteria for degradation of high molecular weight compounds and recalcitrant material entering surface sediments from the water column. Benthic Flavobacteriaceae likely have restricted capabilities for macromolecule degradation and might profit with Sandaracinaceae and Acidobacteria from low molecular weight compounds.