Climate variability during the Holocene has become a hot spot in the frame of the paleoclimate research. The relatively stable background climate condition in this period enables us to gain more knowledge for the present climate, by a better comprehension of the internal variability of the climate system. Contemporarily, understanding of the long-term climate change during the Holocene, attributed to the external forcing, can assist us to better constrain the prediction of future climate change. Climate modeling studies, with the advantage of high resolution in both spatial and temporal, can supply essential information for the physical mechanisms involved in each process and thus contribute to such an effort, which cannot be achieved only by proxy data. Several long-term simulations for the different Holocene periods have been performed with the coupled general circulation model COSMOS. Comparison among the simulations reveals that change in the orbital forcing is the primary factor that drives the climate evolution during the Holocene. The most intriguing feature is that the seasonality is increased significantly over the North Hemisphere, in particular the mid-high latitude. Even over the low latitude, e.g. the Caribbean region, such a change can result in approximately 1°C stronger seasonality in the sea surface temperature of the Caribbean Sea, supported by coral based reconstruction. Existing of the Laurentide ice sheet (LIS) and its melting have strong cooling effect on the regional scale. This influence can be extended globally by ocean-atmosphere circulation. Large amount of water trapped by the LIS, together with the orbital induced movement of the Intertropical Convergence Zone, cause a shift in fractionation of the water isotope from early to mid-Holocene, observed in different proxy records. The ocean circulation is greatly modulated by these forcing, with decreased by more than 2 Sv during the mid-Holocene due to the orbital forcing and further weakened by freshwater injection from the LIS melting during the early Holocene. Although the external induced climate mean states exhibit distinct global and regional features, the internal variability of the climate system shows no remarkable change during the Holocene. The Atlantic Multidecadal Oscillation, associated with the long-term variation in the Atlantic meridional overturning circulation, has a similar quasi periodicity of 50-80 years in each simulation as indicated by observation. Another long-term variation in the global meridional overturning circulation, associated with the Southern Hemisphere westerly winds, demonstrate a similar pattern in both pre-industrial and mid-Holocene conditions. On shorter time scales, the El Niño-Southern Oscillation and North Atlantic Oscillation, indentified as the major forcing mechanisms controlling the seasonal and interannual variability of the Caribbean climate, show a common feature during the Holocene as found in the present-day studies. However, the climate influence of these phenomena could vary in the magnitude and regional scale under different background climate conditions. The results suggest that a larger fluctuation in the mean climate induced by the external forcing might amplify the influence of the internal variability, and eventually add it to the mean climate as a positive feedback.