National Information Centre for European Research
One of the indicators that is monitored regularly and on a long-term basis in the context of FPs is the involvement of research teams from given countries in the process of preparation of FP project proposals and the subsequent success rate of these project proposals with their participation.
In this blog, we will look at the success rate of project proposals with participation of EU countries submitted to the Horizon Europe calls evaluated so far (data 09/2022).
Let us note at the beginning of the analysis that the calculated success rate applies only to eligible full proposals (i.e. proposals submitted to a single-stage calls or 2d stage of 2-stage calls). The success rate calculation does not include stage 1 of two-stage calls due to the lack of structured data on applicants in the first phase of the evaluation process.
There are a number of ways how to measure success in the HE programme. It is possible to determine the success rate of calls for project proposals, types of institutions, countries, etc. Success rate can be related to the different phases of project proposal evaluation. When calculating success rates, "counts" or "amounts" and possibly other dimensions can be used.
In this article, we will determine the success rate of project proposals of the given countries according to the formula mentioned below.
Proposals success rate of the given country = (number of funded proposals in single-stage calls with participation of the given country + number of funded proposals in the 2d stage of 2-stage calls with participation of the given country)/ (number of evaluated eligible proposals in single-stage calls with participation of the given country + number of evaluated eligible proposals in the 2d stage of 2-stage calls with participation of the given country) *100
(Author's note: the European Commission uses the same success rate calculation very often)
Figure 1 is the result of that simple formula applied to EU countries.
Figure 1 - Horizon Europe: Success rate of eligible full proposals – EU countries
The most successful project proposals have been those involving Belgium and the Netherlands. There are 6 EU-13 countries and Denmark at the top of the list of EU countries. The technologically advanced countries follow behind - starting with Austria and ending with Sweden. The positions of Latvia and Slovakia may be somewhat surprising, as project proposals involving research teams from these countries have an above-average success rate. This leads to the question of what this unexpected position of these states is due to. Are Latvia and Slovakia the champions of this framework programme?
For this reason, it is useful to analyse the success rate of project proposals with country participation in Horizon Europe in a little more detail. One possibility is to divide the submitted project proposals into two groups according to the types of actions, namely:
If we analyse the success rates for these two categories of project proposals, we get a very different result than in the previous case. See Figures 2 and 3 for details.
Figure 2 – Horizon Europe: Success rate of „research and/ or Innovation“ oriented eligible full proposals – EU countries
Figure 3 – Horizon Europe: Success rate of “non research (non-innovative)” eligible full proposals – EU countries
In the case where the success rate calculation concerns research and/or innovation-oriented project proposals, the old Member States OMS, EU-14) with a higher summary innovation index (SII according to the EIS 2022) are at the forefront, while those with a lower value of this index are at the back (see Figure 2).
If the success rate for project proposals with country participation is set for project proposals mapping specific scientific research areas or for proposals aimed at organising information, training, and networking events ("non-research (non-innovative)" proposals), the EU-13 countries are at the top of the list of EU countries (see Figure 3).
Figure 4 presents the reaction of EU countries to calls for project proposals in terms of the proportion of project proposals focusing on research and innovative activities compared to non-research project proposals. This ratio shows that research and/or innovation-oriented project proposals are predominant in most of the OMS (EU-14). This prevalence is not so high for the EU-13 countries. Project proposals without research and/or innovation intent are more frequent in Malta and Slovakia.
Figure 4 – Ratio of the number of “research and/or innovation” oriented eligible full proposals to the number of “non – research (non – innovative)” eligible full proposals
From Figure 5 it is possible to derive the strength of the influence of non-research (non-innovative) project proposals on the overall success rate.This figure presents the shares of research and innovation oriented funded projects to all funded projects and similarly the shares of funded projects without research and innovation potential to all funded projects.It is clear that successful, i.e. funded, projects are predominantly devoted to pure science and innovation in most of the EU-14 (OMS) countries, whereas in the EU-13 (NMS) countries we find among the successful projects to a much greater extent those that we describe as non-scientific and non-innovative.
Figure 5 – Horizon Europe: Ration of the number of funded „research and/ or innovation oriented projects to funded „non – research (non – innovative) projects – EU countries
The values and ratios of success rates are consistent with this finding (Figure 6). For all EU-13 countries, the success rate of project proposals is significantly higher for that do not include research and (or) innovation activities - at least 1.5 times higher. For the OMS (EU-14), the success rate of “non-research” project proposals is also higher than the success rate of “research and/or innovation”-oriented project proposals, but not significantly (10 EU-14 countries have a ratio below 1.5).
Figure 6 – Horizon Europe: Ratio of success rate of „non – research (non – innovative) eligible full proposals to success rate of „research and/ or innovation“ oriented eligible full proposals – EU countries
The different nature of national participation and activity in Horizon Europe can be clearly seen in the relationship between the overall success rate of project proposals with national participation and the proportion of funded non-research (non-innovative) project proposals with national participation. For NMS (EU-13) or countries with lower SII, there is a moderately strong correlation between these two indicators.
The higher the proportion of funded non-scientific (non-innovative) project proposals (i.e. non-scientific - non-innovative funded projects) involving a given country, the higher the overall success rate of that country in Horizon Europe programme. In other words, the size of the proportion of funded projects whose content is not science and innovation but other activities has a significant impact on the overall success rate in Horizon Europe. This is valid mainly for the EU-13 countries (NMS).
This relationship does not apply to innovation-strength countries with high SII, and funded coordination and support actions and funded projects for outreach, advisory and networking activities involving these countries do not have such an impact on their overall success in Horizon Europe.
Figure 7 - Horizon Europe: Relationship between success rate of eligible full proposals and percentage of funded "non - research (non - innovative: proposals - EU countries
The fact that the success rate of project proposals in Horizon Europe with participation of OMS (EU-14) and partly of countries with strong innovation potential (strong innovators, Innovation leaders) can to a large extent be related to the Summary Innovation Index (SII) can be clearly seen in Figure 8. The higher the SII the higher the overall success rate of project proposals. For NMS (EU-13) and for countries with lower SII, the relationship between innovation potential and success in Horizon Europe is much weaker. As we have shown in the previous analysis, the overall success rate of EU-13 countries is rather influenced by the high share of successful projects focused on coordination, support, information and networking activities.
Figure 8 - Relationship between SII of EU countrie and success rate of eligible full proposals with participation of given EU countries
Brief conslusion
The factors mentioned in this analysis affecting the success of project proposals resp. success of EU countries in Horizon Europe programme are not the only ones. The values of the correlation and determination coefficients confirm this.
The degree of success rate is determined by a number other of factors. These are not only concerned with the methodological and scientific level of the research teams but also with their ability to become engaged in international consortia of importance capable of creating the aggregate research capacity that is indispensable for addressing the key issues upon which the EC has allocated its funding. However, these factors are very difficult to analyse without a detailed knowledge of the specific and real project proposals and the members of the research consortia.
It can be said that the overall success rate of project proposals in Horizon Europe, calculated only by the ratio of positive cases to all cases, should be treated with caution. In particular, the position of individual countries should be interpreted with caution when comparing the overall success rates of project proposals of individual countries. It is clear that for many EU-13 countries the success rate of project proposals is influenced by the high proportion of successful project proposals that are not purely research and/or innovation in nature. This explains the position of Latvia or Slovakia among the other countries when considering the overall success rate of project proposals in Horizon Europe. For old Member States or those with a higher innovation potential, where the successful, i.e. funded projects tend to focus on science and innovation, their overall success rates in H2020 may be related to their higher innovation index.
The resulting success rate of project proposals of EU countries in Horizon Europe is to a large extent a reflection of their innovative performance or, conversely, their capacity and level of engagement in funded projects focusing on coordination, support, training, information and networking activities.
Finally, it should be added that the criterion of success alone does not necessarily constitute a univocal reflection of the importance of engaging the teams in the programme. It depends on the types of projects, the structure of participants, and the budgets allocated to individual projects. At the same time, the fact should be reckoned with that the ”overall success rate” summarises the success of all projects or participants, regardless of whether their contribution to the project consisted in extensive research activities of fundamental importance (and probably also required the mobilisation of a large budget) or in activities less significant for research, e.g. participation in a research training network (where the costs were more or less constituted by travel expenses relating to attendance in working process meetings or to visits of institutions).
Author: Daniel Frank, TC CAS, frank@tc.cz, 14/10/2022
References:
ÚČAST ČR V PROGRAMU H2020 A V PROGRAMU EURATOM 2014–2020 7. zpráva k červnu 2021, PŘÍLOHA 3– 4/2021, Vyšlo Echo 3-4 2021 | Portál Horizont Evropa. [online]. Copyright © 2022 Technologické centrum AV ČR [cit. 10. 10. 2022]. Dostupné z: https://www.horizontevropa.cz/cs/mohlo-by-vas-zajimat/hodnoceni-ramcovych-programu/narodni-studie-monitoring/informace/yiifnews/591/vyslo-echo-3-4-2021
Publication and dissemination of the content of this blog or its parts and attachments in any way in Czech or any other language is only possible with mention and citation of the source.
18/04/2024
18/03/2024
04/03/2024
