The energy budgets over land and oceans are key determinants of terrestrial and maritime climates. Traditionally, however, large uncertainties have been inherent in the estimates of these budgets, which is still reflected in largely differing energy budgets in the latest generation of global climate models (CMIP5). We combine a comprehensive set of radiation observations with 43 state-of-the-art global climate models from CMIP5 to infer best estimates for downward solar and thermal radiation averaged over land and ocean surfaces. Over land, where most direct observations are available to constrain the surface fluxes, we obtain 185 and 305 Wm-2 for the solar and thermal downward radiation, respectively. Over oceans, with weaker observational constraints, our best estimates are around 186 and 356 Wm-2 for the solar and thermal downward radiation. These values closely agree, mostly within 3 Wm-2, with the respective quantities independently derived by recent state-of-the-art reanalyses (ERA-Interim) and satellite-derived products (surface CERES EBAF). This remarkable consistency enhances confidence in the determined flux magnitudes, which so far caused large uncertainties in the energy budgets and often hampered an accurate simulation of surface climates in models. Considering additionally surface albedo and emission, we infer an absorbed solar and net thermal radiation over land of 138 and -67 Wm-2, and over ocean of 170 and -53 Wm-2, respectively. Best estimates for the surface net radiation thus amount to 71 Wm-2 over land and 117 Wm-2 over oceans, which may provide better constraints for the respective sensible and latent heat fluxes. Combining these surface budgets with satellite-determined TOA budgets (CERES-EBAF) results in an atmospheric solar absorption of 75 and 82 Wm-2 and a net atmospheric thermal emission of -165 and -190 Wm-2 over land and oceans, respectively.