Bacteria of the Vibrio genus are the most predominant infectious agents threatening marine wildlife and aquaculture. Due to the large genetic diversity of these pathogens, the molecular determinants of Vibrio virulence are only poorly understood. Furthermore, studies tend to ignore co-evolutionary interactions between different host populations and their locally encountered Vibrio communities. Here, we explore the molecular targets of such co-evolutionary interactions by analyzing the genomes of nine Vibrio strains from the Splendidus-clade showing opposite virulence patterns towards two populations of Pacific oysters introduced into European Wadden Sea. By contrasting Vibrio phylogeny to their host specific virulence patterns, we could identify two core genome genes (OG1907 and OG 3159) that determine the genotype by genotype (G × G) interactions between oyster larvae and their sympatric Vibrio communities. Both genes show positive selection between locations targeting only few amino acid positions. Deletion of each gene led to a loss of the host specific virulence patterns while complementation with OG3159 alleles from both locations could recreate the wild type phenotypes matching the origin of the allele. This indicates that both genes can act as a genetic switch for Vibrio-oyster coevolution demonstrating that local adaptation in distinct Vibrio lineages can rely on only few genes independent of larger pathogenicity islands or plasmids.