Various societal systems around the globe have endorsed a scientific-technocratic approach to decision-making. This is not a novel observation, however, within this approach, issues such as “knowledge transfer”, “stakeholder engagement”, “participation” rank high. Thus, improving exchange and communication at the nexus of the science/policy interface has seen considerable attention over the course of the last decade. The creation of multiple web-based platforms, such as the Earth System Knowledge Platform (ESKP) of the Helmholtz Foundation, are a case in point. Even if beyond reproach in terms of political and somewhat equitable participation of society at large, these well intended initiatives may not lead necessary to well-informed and sustainable justified decision-making. Indeed, only a limited number of publications are available on the issue of what kind of “metrics of success” can be employed and how to chart the impact these efforts of stakeholder inclusion and forms of dialogue. We present here a first evaluation effort on natural science projects, which were conducted under the umbrella of the ESKP. We accompanied six projects over the course of the entire project cycle, all of which had explicit knowledge transfer components and stakeholder orientation. We conducted an ex-ante and ex-post evaluation of these research projects as well as various interviews, to assess progress, identify new success criteria for research as well as to understand how and in which way natural science scientists approach and engage with stakeholders outside of their direct academic sphere. First results of this ongoing assessment indicate that new types of output and corresponding success criteria classified to better describe and evaluate success must be developed on a case specific basis. This pertains especially to the normative question of what is success and for whom. It can be concluded from these first lesson that navigating between wishful thinking, empirical evidence and practical relevance is the challenge every knowledge platform faces.