Measuring achievements: Can cohesion policy programmes effectively monitor their performance?

1. Introduction

Cohesion policy is the main investment policy of the European Union (EU) and the world’s largest regional development programme (McCann et al., 2021) with a budget of 392 billion euro for the period 2021-2027. Its main goal is to tackle regional disparities and promote regional competitiveness in Europe. Both the significant financial size and political importance of cohesion policy have attracted considerable academic interest resulting in a vast literature debating its performance (Fratesi and Wishlade, 2017). Research on the subject has been conducted mainly from two perspectives. The first one investigates the broader socio-economic impact of the funds (see for instance: Boldrin and Canova, 2001; Becker et al., 2010; Di Caro and Fratesi, 2021; Fiaschi et al., 2018; Crescenzi and Giuia, 2020). The second one, appearing more recently, explores the capacity to spend (and absorb) allocated resources (e.g. Incaltarau et al., 2020; Surubaru, 2017; Tosun, 2014; Santos et al., 2024).

So far, academic work on the administrative performance of cohesion policy has focused mostly on the absorption rate of funds (Cunico et al., 2021). However, there exists another less-explored dimension which relates to the ability by the bodies managing cohesion policy programmes, traditionally national or regional governments, to define clear policy outputs and successfully achieve them (Mendez and Bachtler, 2022). More specifically, programmes are legally required to establish specific indicators and associated targets reflecting their main objectives. For monitoring purposes they must report progress towards the achievement of these targets throughout the programming cycle^[1]. This exercise is particularly important for assessing the effectiveness of programmes, as it provides valuable information on the quality of their management and implementation. The collected data are also an important basis for (ex-post) evaluation analyses of cohesion policy funding. Setting out accurate indicators and corresponding targets is therefore a key pre-condition for a reliable assessment of programmes’ performances. Nevertheless, there has been so far very limited analysis of this process and its potential challenges. This paper represents a first contribution to bridge this research gap. It explores how European Regional Development Fund (ERDF) programmes have set out and modified their indicators and targets (policy outputs) throughout the programming period 2014-2020^[2]. The analysis focuses in particular on the frequency and intensity with which targets have been revised. It investigates these variations by year, Member States, thematic objectives, categories of regions, etc., to identify specific patterns. The primary interest of this analysis is that it provides insights regarding both the capacity of authorities managing the programmes to design reliable monitoring and evaluation frameworks and the inherent limitations of the monitoring process. From a policy perspective, the analysis is also relevant in light of the debate as to whether the application of performance-based approaches or performance conditionalities to cohesion policy can contribute to addressing its long-standing efficiency and complexity predicaments (on this topic see for instance: Wostner, 2008; Crescenzi et al., 2022). Cohesion policy already harbours various performance-based elements, but it remains largely based on a real cost-based model, just like most EU instruments: that is, most of its funding is disbursed by the European Commission based on actual expenditures incurred and reported by programmes or projects. By contrast, the recently adopted Recovery and Resilience Facility (RFF) instrument – the main pillar of the EU post-Covid recovery programme named NextGenerationEU (NGEU) – has introduced a novel approach in the realm of EU funds (barring the external aid family) in that it is the first instrument that is entirely performance-based: that is, the disbursement of all its resources by the European Commission is contingent upon the achievement of pre-agreed milestones (reforms) and investment targets set by Member States in so-called National Recovery and Resilience Plans (NRRPs).

The performance-based philosophy underpinning the RRF is likely to represent an important reference for the design of post-27 EU funding instruments. It will certainly have a positive influence on the discussion about how reinforcing the performance orientation of future cohesion policy in order to achieve more simplification and result-orientation. The paper can therefore contribute to this debate by identifying potential bottlenecks and challenges that need to be taken into account in designing a more performance-oriented cohesion policy in the vein of the RRF. Despite major differences, the monitoring system of cohesion policy follows a similar logic to the RRF, as it also entails setting performance targets but does not link the payments to their achievement (except for the Performance Reserve).

The paper is structured as follows. Section 2 provides a short literature review, while section 3 details the data and the methodology employed. Section 4 presents the results of our analysis followed by a discussion of their potential determinants (section 5). Finally, section 6 outlines the policy implications of our work, in particular how the analysis feeds into the current debate on the future of cohesion policy funds.

2. Literature review

Performance budgeting has been in use for many decades with its uptake having increased sharply from the 90s onwards (Sapala, 2019). There is therefore a substantial body of academic works on the topic, notably in the field of public management studies (see for instance: Robinson, 2005; Robinson, 2007). Multiple definitions of performance budgeting are provided in literature reflecting different applications. Nevertheless, we can generally characterise it as the systematic use of performance information in budget planning or allocation (OECD, 2019: p. 13). The extent to which this concept is operationalised into the management of public finances can vary significantly. The OECD identifies three broad categories: presentational, performance informed, and direct performance budgeting (OECD, 2017). The first two types entail respectively the simple presentation of performance information in parallel with the annual budget (e.g., for transparency purposes) and the use of performance information to make budgetary decisions. In contrast, direct performance budgeting links the allocation of resources to the achievement of outputs and results. It is this form of performance budgeting that this paper focuses on, as aspects of it are already built into cohesion policy and represent the underlying funding logic of the RRF.

The EU budget has included some performance elements for many years. Increasing interest in performance budgeting at EU level culminated in the presentation of the “EU Budget Focused on Results” initiative in 2015 (European Commission, 2017), with the stated goal of reinforcing the result-orientation of European funding. The revision of the EU financial regulation in 2018 required spending programmes to establish performance indicators based on specific objectives to enable accurate performance monitoring^[3].

Prior to the RRF, however, the overall EU approach to performance budgeting was largely presentational (Downes et al., 2017). Only European Structural and Investment funds (ESIF) included aspects of direct performance budgeting. This is the main reason why there is still very little literature on the application of performance-based models in the area of EU funds. Related studies in the field of cohesion policy are essentially theoretical and approach the topic in terms of the relationship between policy conditionalities and performance (Bachtler and Ferry, 2013; Bachtler and Mendez, 2020). Different forms of conditionality designed to enhance the accountability and performance of the funds have been applied to cohesion policy since its 1998 inception (Bachtler and Ferry, 2013). One example is the N+2 rule (now N+3), introduced in 1999^[4], which mandates that committed funds be spent within two (or three) years to prevent programme forfeiture. This provision aims to ensure the timely utilization of allocated resources (Davies and Polverari, 2011).

The Barca report (Barca, 2009) was pivotal in two ways. First, it provided a compelling rationale for using conditional grants to address performance issues arising from administrative capacity limitations, elite capture, and market failures (Berkowitz, 2017). Second, it advocated for a stronger emphasis on outcomes and results rather than solely on implementation (Barca and McCann, 2011). These ideas were partially incorporated into specific performance-based features for the 2014-2020 period. These included ex-ante conditionalities^[5], a performance framework (performance reserve)^[6] and performance-based funding models (from 2019)^[7].

The performance framework, which links the disbursement of a small percentage of funding (the so-called performance reserve equivalent to 6% of programmes) to the achievement of (mainly) financial and outcome targets, is of particular interest for our analysis^[8]. While it foreshadowed the performance-based approach of the RRF, there are distinct differences between the two. However, its implementation has faced challenges. The European Court of Auditors (ECA) reported that approximately 55% of targets or indicators selected by programmes for the performance framework were modified between 2014 and 2018 (ECA, 2021). Although the ESI funds general regulation for 2014-2020 (Regulation EU 1303/2013) permits amendments only in duly justified cases, such as when a target was based on incorrect assumptions, programmes’ authorities made extensive use of this possibility (ECA, 2021). This highlights the difficulty in setting realistic targets and indicators over time. Notably, several authorities surveyed on the matter reported that a high degree of flexibility was needed to make this approach work (McCaster and Kah, 2017).

Existing empirical analyses of performance-budgeting mechanisms in cohesion policy are limited, primarily consisting of reports by the European Court of Auditors (ECA, 2019; ECA, 2021). However, two additional quantitative papers are worth mentioning. Mendez and Bachtler (2022) explored the correlation between regional quality of government and three variables capturing the administrative performance of programmes, of which one is the percentage of targets achieved by programmes at the end of the period 2007-2013. Dicharry (2023) investigated the influence of the N+2 rule on cohesion policy funding’s impact on GDP per capita growth in EU regions. This study found that faster absorption to meet this rule translates into a lower impact on growth in lagging regions.

More comprehensive and systematic analyses on the application of performance budgeting aspects in cohesion policy are thus needed, building on the recent work of the European Court of Auditors. This paper offers several novel insights. Firstly, it provides the first comprehensive analysis of all ERDF output indicators and their targets. Secondly, it represents one of the first attempts to assess the monitoring capacity of programmes authorities. Thirdly, it presents new evidence regarding the application of performance budgeting to EU funding instruments. The establishment of the RRF is expected to stimulate additional academic interest in performance budgeting. Existing RRF studies have largely focused on the legal aspects of the performance budgeting model and its operationalization in the planning phase (preparation of National Recovery and Resilience Plans). Some scholars have identified room for improvement, for instance by pointing to the spatially-blind nature of milestones/targets (Corti and Ruiz de la Ossa, 2023) or their disproportionate focus on outputs rather than results (Darvas et al., 2023). However, an in-depth assessment of the RRF performance budgeting model remains challenging due to its early stage of implementation. This lack of evidence hinders the debate on whether to extend the approach to other instruments in the future.

3. Data and methodological approach

Cohesion policy programmes are required to set out indicators and corresponding quantitative targets for each investing priority in order to assess implementation and progress towards achieving their specific objectives^[9]. The identification of both indicators and targets, as well as their modification during the programming period, is subject to the assessment and approval by the European Commission as part of the standard process of adopting or amending programmes. The programmes’ authorities report annually on the implementation of indicators. Three types of indicators are set out by the regulation: financial indicators (measuring the financial execution), output indicators (direct outputs produced/generated by projects) and result indicators (effects of funding interventions) (ECA, 2021). Additionally, indicators can be common or programme specific.

This paper focuses on output indicators, and their targets, selected by European Regional Development Fund (ERDF) programmes. The main reason for circumscribing the analysis to only one of the three cohesion policy funds (i.e. the ERDF) is that the European Social Fund (ESF) programmes do not have a legal obligation to express target values for all indicators^[10] whereas the Cohesion Fund (CF) has a limited geographical coverage. The choice to restrict the research scope only to output indicators is due to the fact that direct performance budgeting settings in cohesion policy (through the Performance Reserve) and the RRF feature very few result indicators. Moreover, setting accurate result targets is a much more difficult exercise as, to a certain extent, they might be influenced by external factors outside the control of the authorities.

The main source of data used for this study is the European Commission’s Cohesion Open Data Platform, more specifically two different datasets on common indicators^[11] and programme-specific indicators^[12]. The datasets comprise, amongst others, the indicators for each programme, the corresponding final target values (target to be achieved by 2023), and implemented values throughout time (i.e. yearly progress reported by programmes’ authorities towards achieving the final target). All the data are categorised by year allowing to discern if and how the indicators or their targets have been modified from a year to another in the context of one of the frequent revisions of programmes.

Common indicators are predefined ones defined by the European Commission (Annex I; Regulation EU 1301/2013) and selected by the responsible public body overseeing the programs. Some examples are the number of firms receiving grants, the amount of private investment for matching public support to enterprises and the direct employment increase in supported enterprises. Specific indicators refer to bespoke indicators defined by the managing authorities on the basis of programmes’ specific objectives. For example, the extent or coverage of a newly constructed broadband network, energy gain in the residential sector and the total amount of underlying new debt finance originated by the financial intermediary.

After merging the two-abovementioned datasets, we removed all the indicators related to Covid-19 pandemic, REACT-EU^[13] and Thematic Objective 13 (Fostering crisis repair and resilience) measures, since they are related to new actions created from 2020 due to unexpected events. Including these indicators in our analysis would have biased the overall results and their policy implications. Inter-regional collaboration programmes (like INTERREG), as well as Thematic Objective 12 (Outermost & Sparsely Populated) and Technical Assistance indicators, are also not included in the analysis due to their specific geographical scope and/or the nature of the targets. Programmes that are discontinued over time because are merged with others are also left out from the analysis.^[14]

To identify Covid-19-related indicators, we use text-mining analysis applied to the indicator name^[15] together with the indicator code. Indicators with the letters “CV” in the indicator code are related to Covid-19 measures. REACT-EU indicators are tagged as such in the datasets. Inter-regional programmes are identified by the code “TC” in the Member States in charge of the programme. We also removed all indicators in the dataset for which all years do not have values for targets or equal zero.

As we can have the same name of the indicator appearing several times but under different categories, we define our unit of analysis (ID) using seven dimensions: programme code, thematic objective, category of region, type of indicator, priority code, investment priority and measurement unit. The final dataset covers 202 ERDF programmes for the period 2014-2020 from 27 EU Member States (UK programmes are not taken into consideration). The time series spans the period 2015-2022: 2015 is the first year where a substantial number of programmes started implementation owing to delays in adoption of programmes; 2022 is the last year for which data are available.

To provide a comprehensive overview of the indicators’ characteristics, we employed various descriptive statistics, like the mean to summarize and present the key data points, as well as the t-test statistical technique to determine whether there is a significant difference between the means of two groups. The null hypothesis (H0) assumes no difference between the group means, whereas the alternative hypothesis assumes a difference exists. A significance level of 0.10 determines the threshold for rejecting the null hypothesis.

4. Results

4.1. Change in the number of output indicators

Before delving into the analysis of the targets, it is necessary as a preliminary step to look at the indicators on which they are based. Table 1 shows that the total number of output indicators selected by programmes varies marginally in the period 2016 through 2022 (2015 being less relevant)^[16]. However, we observe a certain turnover of indicators (Table 2) throughout the period analysed, with the highest share observed in 2018. The reasons might be diverse, including monitoring failures, the establishment of new investment priorities, the effects of Covid-19, etc. It is possible that the substantial replacement of indicators observed in 2018 is partially driven by programmes’ efforts to secure the first tranche of the performance reserve by introducing new indicators whose (intermediate) targets were more attainable by end 2018.

Overall, 1,268 indicators were removed from 2017 to 2022 (Table ), whereas 1,129 were added. These results suggest a certain stability in the use of indicators. This means that, in spite of substantial re-programming, the overall objectives of the programmes, which are indirectly reflected in the indicators used to monitor the implementation, have not changed markedly. Yet, the share of indicators substituted over the course of the programming period is non-negligible.

Table 1.

Number of output indicators observed in the dataset per year, 2015-2022

NB: The table shows the total number of indicators by year and the number of years the same indicator has been used by the programme (from 1 to 8 years). This number varies because programmes remove or add indicators over time, which means some indicators can be observed only for less years than the overall observed period. For instance of the 5,825 indicators in 2015, 12 are only kept for 1 year, 39 for 2 years, and so on. Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Table 2.

Number of new indicators added or removed compared to the previous year, 2015-2022

Note: (1) It refers to the cumulative number of indicators in each year, using only the information in the first row on the number of new indicators added each year. (2) It refers to the net number of cumulative indicators that appear each year, after removing the indicators that were removed and reported in the second row. Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

4.2. Changes in target values

As explained above, programmes are allowed to adjust their final targets, i.e. targets to be achieved by 2023, throughout the whole programming period. Such changes must be negotiated and approved by the European Commission, which implies a revision of programmes. Our analysis shows that very extensive changes to the target values took place during the period which we have analysed. On average, 65.8% of the programmes’ targets were revised at least once (Figure 1). In other words, programmes revised their initial estimates in relation to the final outputs of their interventions for roughly 2/3 of the selected indicators. It is noteworthy that more than a quarter of all targets (26.6%) was changed at least two times over the observed period (39.2%) whereas 39.2% were amended only once. Breaking down the data per category of regions (according to the 2014-2020 Cohesion criteria classification), we see only minor differences between less developed, transition and more developed regions.

Figure 1.

Number of indicators with at least one change in target value (%), by category of region, ERDF 2014-2020, EU27

Figure 2.

Number of times the target value of the indicator has changed(%), category of region , ERDF 2014-2020, EU27

Note: Category of regions refers to the cohesion policy eligibility criteria established in the period 2014-2020 (art. 90, Regulation EU 1303/2013). Total number of indicators: 9,669. This number is slightly lower than total number of indicators in table 2 as only indicators observed 2 or more consecutive years are included in this analysis. Indicators not assigned to a specific category of region are also included. Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Results mean-comparison t-tests: Number of indicators with at least one change in target value (%), by region category, ERDF 2014-2020, EU27

Note: Category of regions refers to the cohesion policy eligibility criteria established in the period 2014-2020 (art. 90, Regulation EU 1303/2013). Only indicators observed 2 or more consecutive years are included in the analysis. Indicators without region classification are also included. Total number of indicators: 9,669. Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Target values were revised downwards for 29% of the indicators and upwards for 24% (figure 3). 12% of the targets were modified both upwards and downwards over the same period. The intensity of the changes is also an important aspect to look at. Our analysis finds that targets were increased by a median of 58.4% and decreased by a median of 39.2% (figures 4 and 5). This suggests that the intensity of changes is in general quite significant. Overall, the results illustrated in figures 1 to 5 highlight wide-spread challenges in setting accurate targets. This in turn led programmes’ authorities to modify not only a relatively high number of targets over time (including more than one time) but also to revise their values substantially.

Figure 3.

Number of indicators with at least one change in target value (%), by category of region and by type of change, ERDF 2014-2020, EU27

Note: Category of regions refers to the cohesion policy eligibility criteria established in the period 2014-2020 (art. 90, Regulation EU 1303/2013). Only indicators observed 2 or more consecutive years are included in the analysis. Indicators without region classification are also included. Total number of indicators: 9,669.Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Figure 4.

Intensity of target change (median) when negative change is observed between two periods, ERDF 2014-2020, EU27

Figure 5.

Intensity of target change (median) when positive change is observed between two periods, ERDF 2014-2020, EU27

Note: Regions category refers to Cohesion criteria classification. Only indicators observed 2 or more consecutive years are included in the analysis. Indicators without region classification are also included. Total number of indicators: 9,669. Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Table 4.

Results mean comparison t-test: Number of indicators with at least one change in target value (%), by region category and by type of change

Note: Category of regions refers to the cohesion policy eligibility criteria established in the period 2014-2020 (art. 90, Regulation EU 1303/2013). Only indicators observed 2 or more consecutive years are included in the analysis. Indicators without region classification are also included. Total number of indicators: 9,669. Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Unsurprisingly, the percentage of changes is not constant over time (Figure 5). It varies widely across the period observed. Two years stand out showing a much higher degree of changes. The first one is 2020 with 50.6%: this percentage can be largely attributed to a Covid-19 effect whereby considerable re-programming of funds to face the shock has logically resulted in a revision of final targets. The other year showing a high degree of changes is 2018, barely two years into the de facto implementation of programmes. This figure could be partially explained by the fact that many programmes adjusted their intermediate targets (for 2018) for the Performance Framework: under this mechanism, as explained above, programmes receive an additional allocation (the so-called performance reserve) if they achieve certain targets (financial and output indicators) in 2018 and 2023. This process may have influenced the setting of final targets of the overall monitoring framework, as some indicators overlap with those of the Performance Framework ones. Regardless, this suggests that a considerable share of the estimates related to the targets already seemed to be incorrect by 2018, 2 or 3 years after they were set. It also appears plausible that absorption issues as well as major shocks such as the Russian military aggression towards Ukraine and Europe’s energy crisis led to more modifications in 2023 in some Member States although at the time of writing we do not have data for this year. Finally, statistics for the UK (see Tables A1 and A2 in appendix) suggests that Brexit had an impact on funding decisions, resulting in the modification of a non-negligible share of targets in the country in 2017 (following the 2016 referendum).

Figure 6 shows that decreases in target values were greater than increases in all but two years.

Figure 6.

Number of indicators with a change in target value (% total) by year, ERDF 2014-2020, EU27

Figure 7.

Number of indicators with a change in target value (% total) by type of change and by year, ERDF 2014-2020, EU27

Note: Only indicators observed 2 or more consecutive years are included in the analysis. The figured include 56.153 year-observations. Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Another important aspect is whether estimates about target values change more frequently for some categories of spending than others. Our analysis points to substantial differences in relation to thematic objectives (TO) (Figure 8), with indicators for TO01 (Research and Innovation), TO03 (Small and Medium-sized enterprises - SMEs) and TO04 (Low carbon economy) exhibiting above average targets’ modifications. This is likely to be caused by the different intrinsic nature and complexity of targets across different thematic investment areas.

Figure 8.

Number of indicators with at least one change in target value (%), by Thematic Objective, ERDF 2014-2020, EU27

Note: Only indicators observed in 2 or more consecutive years are included in the analysis. Total number of indicators: 9,896 (excluding indicators without thematic objective classification). Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Table 5.

Results mean-comparison t-tests: Number of indicators with at least one change in target value (%), by Thematic Objective, ERDF 2014-2020, EU27

Note: Only indicators observed in 2 or more consecutive years are included in the analysis. Total number of indicators: 9,896 (excluding indicators without thematic objective classification). Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

An even greater difference can be observed by country: the changes range from 91.4% in the case of Denmark to only 10% for Luxembourg (Figure 9). A bit surprisingly, some countries with relatively small allocations have a very high percentage of changes. In general, however, targets modifications appear to be proportional to the size of national allocations.).

Finally, the percentage of changes for targets associated with common indicators is higher than for programme-specific ones (Figure 10).

Figure 9.

Number of indicators with at least one change in target value (%), by country, ERDF 2014-2020, EU27

Note: Only indicators observed in 2 or more consecutive years are included in the analysis. Total number of indicators: 9,926. Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Figure 10.

Number of indicators with at least one change in target value (%), by type of indicator and type of change, ERDF 2014-2020, EU27

Note: Only indicators observed in 2 or more consecutive years are included in the analysis. Total number of indicators: 9,926 (excluding indicators with classification of typology). Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Table 6.

Results mean-comparison t-tests: Number of indicators with at least one change in target value (%), by type of indicator and type of change, ERDF 2014-2020, EU27

Note: Only indicators observed in 2 or more consecutive years are included in the analysis. Total number of indicators: 9,926 (excluding indicators with classification of typology). Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Figure 11.

Number of times the target value of the indicator has changed (%), by type of indicator, ERDF 2014-2020, EU27

Note: Only indicators observed in 2 or more consecutive years are included in the analysis. Total number of indicators: 9,926 (excluding indicators with classification of typology). Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

Figure 12.

Intensity of target change (median) when negative change is observed between two periods, ERDF 2014-2020, EU27

Figure 13.

Intensity of target change (median) when positive change is observed between two periods, ERDF 2014-2020, EU27

Note: Only indicators observed in 2 or more consecutive years are included in the analysis. Total number of indicators: 9,926 (excluding indicators with classification of typology). Source: Own elaboration based on European Commission’s Cohesion Open Data Platform.

5. Discussion

Our analysis reveals significant challenges faced by ERDF programmes in defining accurate and realistic target values for monitoring and evaluation. This is evident in the high number of indicators with changed targets throughout the observed period, often with substantial adjustments. Endogenous factors, such as the size of allocations, spending category, or type of indicator, appear to influence the frequency and magnitude of changes. External factors, such as the Covid-19 crisis, also play a role. The multi-level governance structure of cohesion policy, involving different government tiers, may also contribute to the high number of changes. The diverse application of this model across Member States cover different levels of administrative capacity, with some regions or local authorities struggling to set realistic targets due to limited resources. Conversely, in countries with less extensive multi-level governance, national authorities may lack adequate knowledge of local contexts, necessitating frequent and significant target amendments.

Other potential causes include inherent difficulties to define final target values for a de facto ten year long funding programme. However, the high rate of changes observed in 2018 suggests, albeit indirectly, problems in defining accurate targets even in the short term. Additionally, evolving priorities over time as well as unforeseen events (such as economic crises) prompt programmes to rearrange the objectives of the programmes and resources’ distribution across them, which inevitably entails a revision of final targets. Finally, lack of administrative capacity, having a negative impact on the absorption and economic impact of the funds (Rodríguez-Pose and Garcilazo, 2015; Milio, 2007), can equally hamper programmes’ authorities ability to produce accurate estimates. Adequate evaluation expertise may still be missing in some administrations whereas lack of quality data may also represent a considerable problem (Polverari, 2015).

There is also a more “behavioural” dimension that needs to be taken into account. The existence of risk-aversion attitudes in programmes’ authorities (Mendez and Bachtler, 2011) can result in an underestimation of target values. Political factors may also influence the definition of target values (e.g. lower targets are set out to make sure they are achieved to avoid criticism from local decision-makers or because the local government is keen to get credit for a successful implementation; the opposite is also true: higher targets are set to impress local decision-makers). These general causes may not explain alone the high frequency and intensity of changes. The analysed period has been also marked by specific legislative and contextual factors that exerted influence of their own on the amendments of the targets: the first one is the novel performance framework, which partially explains the high rate of changes in 2018 as pointed out above; the second is the slower implementation since the onset of the period; the third is the prolonged period of crisis owing to the effects on the economy of the Covid-19 and war in Ukraine which has in turn elicited an unprecedented re-programming of resources across investment priorities. It is important to note that both the increasing frequency of shocks and the risk of implementation delays appear to be affecting the current programming period as well.

There are two aspects that is worth highlighting in assessing these results. Firstly, extensive target adjustments, particularly towards the end of the period, can undermine the objective evaluation of programmes’ administrative performance. In other words, an objective assessment into whether a programme has achieved its objectives is compromised by the very possibility to revise extensively and frequently the targets associated with its policy outputs. What are we really measuring if we can constantly change our targets? How can we expect to assess objectively the performance of a programme if we can adjust at any time, and by a large scale, its policy outputs? These are important questions that need to be explored. On the other hand, the long life-span of cohesion policy programmes alongside the need to re-programme on the grounds of evolving circumstances and unforeseen events beg for a flexible approach to target-setting. Thus, there are inevitable trade-offs at the heart of the monitoring and evaluation framework. One could also see merits in the process from a policy learning perspective: in some cases targets may be revised iteratively as the consequence of a sort of learning process; in which case the process implies a successful learning by doing.

Secondly, the ease of changing targets may disincentivize authorities from adopting a rigorous approach to indicators and targets setting, potentially hindering efforts to strengthen evaluation capacity, especially in those administrations that are still underperforming in this area.

6. Policy implications and avenues for future research

The future shape of cohesion policy will be a subject of intense academic and policy debate in the coming years, with a focus on whether enhancing the performance-based dimension can improve efficiency. Discussions may also extend to the potential adoption of direct performance-based models across a broader range of EU funds. Against this background, our empirical analysis offers some avenues for reflection.

Firstly, to minimize target “instability”, authorities should adopt a more rigorous approach to monitoring and evaluation, employing additional expertise and more sophisticated methodologies for more accurate estimates. The Commission could provide tailored support through guidelines, off-the-shelf mechanisms, communities of practices, peer-to-peer learning, technical assistance, etc.

Secondly, the analysis reveals that some thematic areas (e.g., climate; infrastructures) are inherently more prone to change than others. This finding underscores the need for specialized evaluation approaches in these fields and greater flexibility for authorities in defining and modifying policy outputs due to the inherent difficulty in setting accurate targets.

Thirdly, our time series shows that shocks inevitably trigger significant changes in the number and values of targets (also leading to replacing indicators). As we are living in an increasingly crisis-prone times, the predictability of targets might be further affected in the future. To mitigate this risk, streamlined mechanisms for rapid targets adjustments in response to shocks could be envisaged. Alternatively, target values could be defined in the form of ranges between a maximum and a minimum values, instead of a fixed value, potentially reducing the need for frequent changes in unforeseen circumstances.

Fourthly, the analysis indicates that targets related to common output indicators have required more changes than programme-specific ones, perhaps due to their more general and less tailored scope. Conversely, it is possible that programme-specific targets are more stables because their degree of customization ensures a better ownership and thus understanding of their use. While there is a strong rationale for adopting more common indicators in the future to further enable a comparative evaluation of different programmes, the analysis shows that programme-specific indicators should not be abandoned altogether because they can play an important role in ensuring an accurate monitoring of programmes.

Overall our findings suggest that while performance-based mechanisms can enhance the efficiency and result-orientation of EU funding instruments (Barca, 2009), their application poses considerable challenges. The implementation of such mechanisms should be therefore proportionate to the effective capacity of funding authorities or beneficiaries to navigate their complexity, suggesting a differentiated approach across EU funds. For instance, full performance-based models could be suitable for EU direct management funds such as Horizon 2020, while shared management funds might require a combination with real-cost budgeting to ensure greater accountability. Furthermore, EU fund regulations should more clearly define the conditions for amending targets and indicators. Given the long time span of programmes, it is essential to consider the influence of specific megatrends on targets. In turn, this requires complementing a strong evaluation expertise with a foresight capacity or culture which is still missing in many managing authorities and should be supported by the Commission through specific technical assistance measures.

Despite significant differences with cohesion policy, our analysis provides valuable insights for the ongoing implementation of the RRF. While evidence regarding the operationalization of direct performance-based approaches in the RRF is limited due to its recent launch, our analysis of target-setting and amendment patterns under cohesion policy suggests that National Recovery and Resilience Plans might encounter similar challenges. The scope for modifying RRF indicators and targets is restricted by regulation, allowing amendments only in cases of unachievability due to “objective circumstances” beyond Member States' control. Member States need to bring detailed evidence to the Commission as to the circumstances and how they impact the targets. Inherent predictability issues, the occurrence of shocks as well as shifting priorities have resulted in the past months in the revision of many National Recovery and Resilience Plans. Such developments hint at the existence of challenges akin to the ones identified in our paper on the side of the RRF as well.

The study has several potential limitations that future research could address. First, it performs a comprehensive descriptive analysis of changes in indicators and targets, but does not empirically explore the potential factors influencing the underlying trends. It would be important in the future to ascertain if dimensions such as the administrative capacity of programmes’ authorities, the design of programmes (thematic concentration; number of operations; etc.), or specific socio-economic factors have an impact on the capacity to define realistic targets. Another limitation of the study is that it lacks a more qualitative understanding of the underlying drivers of the high rates of changes, which would require at the very least surveying a sample of programmes’ authorities to test some of the hypotheses formulated in the paper. The analysis is also incomplete in that it focuses only on output indicators and leaves result indicators outside the scope of the research. Finally, this study refers to cohesion policy which has very specific features both in terms of governance and delivery mechanisms. In this sense, the implications discussed in the paper, especially in regards to the application of performance budgeting, are not necessarily valid for other funding streams.

The views expressed are purely those of the authors and may not in any circumstances be regarded as stating an official position of the European Commission

JEL classification: H11; H83; R58.

redalyc-journal-id: 289