From a large number of case studies on terrestrial deposits we know that Late Holocene climate fluctuations have an important impact on the terrestrial environments. However, it is hitherto not clear how the marine sediments can be used to shed light on the environment and climate change of the catchment. To provide such insight, we used the major element, trace element and rare earth element (REE) compositions in the southern Yellow Sea (SYS) sediments to reconstruct the changes in weathering and erosion of their source regions. The sediments originate predominantly from the upper crust of East China and are transported into the basin especially by the Yellow River (Huanghe) and to a lesser extent by the Yangtze River (Changjiang). The chemical index of alteration (CIA; 53.7–59.7) suggests low chemical weathering of the source rocks since the Middle Holocene. This is consistent with the relatively cool and arid climate in North China after the Holocene Megathermal. Comparison of element ratios, including Al/Ca, K/Ca, Al/Na, K/Na, Rb/Sr, Li/Ba and the CIA shows that we can use the latter as a robust proxy for climate change. The CIA-based mean annual precipitation and mean annual temperature show a series of climate fluctuations in the catchment. A relatively warm and humid Period I (5.3–2.9 cal. ka BP), a relatively cool and dry Period II (2.9–0.9 cal. ka BP) and an increasingly cool and dry Period III (0.9–0.3 cal. ka BP). These periods can be linked to climate intervals recognized elsewhere.