1 INTRODUCTION

As of its origin, public procurement emerged as a way of supplying the State so that it could perform its typical activities. However, since the XIX century (MCCRUDDEN, 2004), such procurement gained a second role: it became a tool to implement public policy. From then on, such use has progressively attracted the interest of Governs, intergovernmental organizations, private companies and, consequently, of both national and international literatures (SOARES, 2005; FLYNN; DAVIS, 2014). Therefore, procurement has granted the State the necessary flexibility to achieve its political-regulatory goals on economic, social regional and environmental matters (JUSTEN FILHO; JORDÃO, 2011).

Among the various ways of using procurement in the public policy context, we can highlight the new models of public acquisition, such as public procurement for innovation, sustainable procurement and the establishment of competitive advantages for determined economic segments. In such situations, everyday administrative procedures end up being determinant for the induction of desirable standards of conduct in companies that supply both goods and services to the Public Administration, configuring themselves as important tools for the public policy formulation cycle.

In 2016 alone, the Brazilian Federal Administration spent R$ 50,978,191,734.13 on public procurement, in a total of 105,534 public processes (MINISTÉRIO DA ECONOMIA, 2017). Due to its economic and political significance, procurement has been the subject of research on a global scale, assessing its impact on the configuration of both market and society. In Brazil, we can highlight the studies conducted by the Brazilian Institute of Applied Economic Research (Ipea), which aimed to assess the potential of the State’s procurement power to stimulate technological innovation and improvement in the quality of products of industrial companies hired by the Federal Government between 2001 and 2003 and from 2001 to 2010. (SOARES, 2005; SQUEFF, 2014).

However, after a few decades of using public procurement for purposes secondary to its initial function of providing the Public Administration with goods and services, there is no concrete knowledge on whether the performance of public procurement has been subject to evaluation and, consequently, whether the Brazilian Government has been successful in such use. In order to contribute with this debate, this study elected the following research question: how has recent literature assessed the performance of government procurement in Brazil?

In order to answer the question above, this study develops a comprehensive literature review of the production from 2007-2017 in order to map and present the literature’s state of the art on public procurement performance in the Brazilian Public Administration field. The choice of inserting a ten-year time frame is a common practice in social sciences and, above all, in publications in the Administration area that propose to develop such states of the art. Thus, by choosing to compose a state of the art, it is reiterated that this is "one of the possible histories" for the field, but it is not exclusive as to other possible clippings, due to its temporal design (FERREIRA, 2002, p. 269). At the end, the research gaps to be explored by future studies are mapped out, on which occasion a research agenda is composed.

2 PUBLIC PROCUREMENT: FROM SUPPLYING ACTIVITY TO PUBLIC POLICY INSTRUMENT

The procurement or supplying activity is related to the ability to manage resources in order to achieve organizational objectives, occurring in all organizations, from the purchase of raw materials to the acquisition of manufactured products (ENSSLIN et al., 2014). In the public sector, according to a study by Ipea, public procurement can be defined as "the process by which the government seeks to obtain services, materials and equipment necessary for its operation, in accordance with the laws and regulations in force" (SQUEFF, 2014, p. 7).

Beside their ordinary objectives (supplying the resources necessary for the operation of the State’s administrative apparatus), the literature has recently recognized new purposes to procurement, associated with the formulation and implementation of public policies in various thematic areas (FLYNN; DAVIS, 2014). Soares (2005) points out that the State’s procurement power is an instrument used by many countries, with the main purpose of strengthening their national industry and service sectors and leveraging the technological development of contracted companies. For the author, the relevance of this instrument is amplified due to the significant volume of public money spent on procurement.

Brazilian law evidences the practical applicability of the use of this State’s procurement power in different fields of public policy. Within the innovation scenario, it is estimated that it would be possible to establish a preference margin in public biddings in order to hire products and services that meet Brazilian technical standards, taking into consideration the development of technological innovations carried out in the country. There is also the specific hypothesis of direct hiring, through a bidding waiver, for products focused on research and development (R&D), hereby defined as those "goods, resources, services and construction works required for scientific and technological research activities, technology development or technological innovation, listed in a research project approved by the contracting institution" (BRASIL, 1993, p. 10).

Regarding the so-called “public procurement for innovation”, Georghiou et al. (2014) point out that, in face of the expressive proportion represented by public contracts on the global demand for goods and services, such contracts are increasingly perceived as a viable instrument for implementing innovation policies. The promotion of R&D is still emphasized as a key element to the 2000 Lisbon Strategy, which included as one of its goals raising the EU investment on research and development to 3% of their GDP until 2010, aiming for Europe to become the most competitive and dynamic knowledge-based economy in the world (ASCHHOFF; SOFKA, 2009).

Both national and international literature on “green procurement” has been gaining strength as countries acknowledge, on a global scale, the potential carried by public procurement to stimulate a more rational use of environmental resources (BRAMMER; WALKER, 2011). Amann et al. (2014) point out that sustainable public procurement – SPP, which combine both environmental and social responsibilities, are gaining ground in all EU member States, what has bee promoting a series of policies and work initiatives towards boosting sustainable consumption patterns.

Albano et al. (2015) report that the European Union also developed policies to stimulate the economic and productive growth of MSEs, highlighting that Europe's 2020 adopted the so-called Small Business Act (SBA), aiming to "improve overall policies for entrepreneurship, irreversibly anchoring the 'Think Small First' principle in policymaking for the regulation of public services, and promoting the growth of MSEs by helping them solve the problems that still hinder their development" (ALBANO et al., 2015, p. 230).

3 PERFORMANCE ON THE PUBLIC SECTOR: THE CASE OF PUBLIC PROCUREMENT

Brignall e Modell (2000) note that during the 1990s, within the context of the New Public Management (NPM), some concerns and pressures emerged in order to make many public services more efficient and effective, with the potential to reduce taxes while maintaining the volume and quality of services provided. Thus, reform movements sought to introduce various management techniques of private companies into the public sector, under the premise that the so-called dysfunctions of the bureaucratic model could be mitigated by an entrepreneurial and innovative government (OSBORNE; GAEBLER, 1994).

Modell (2004) adds that a notable feature of the administrative reforms that started with the New Public Management (NPM), without overlooking its variants, is the introduction of performance measurement to control public organizations, in response to concerns about fiscal probity and accountability, as well as to provide some kind of legitimization to governmental actions.

Osborne (2006) points to the emergence of New Public Governance as a new model or regime (which does not necessarily replace the previous one, but coexists and interacts with it), which, based on organizational sociology and network theory, differs from economic theories and begins to understand the state as a plural organism. According to Osborne (2010), New Public Governance (NPG) has become the predominant regime of public policy implementation and public service delivery, with emphasis on the development of sustainable policies and services and the governance of interorganizational relationships. However, for the author, "[...] However, that does not take away either the importance of the public policy process or the necessity to manage individual organizational resources and performance in an efficient and effective manner" (OSBORNE, 2010, p. 414).

Therefore, in face of the heterogeneity and complexity of public services, literature has highlighted difficulties inherent to the direct transposition of evaluation tools from the private sector, considering that these tools would not perfectly fit the dynamic political context of government organizations and would not be configured as an effective possibility of management in view of the natural alternation of the management of public agencies, among other idiosyncrasies of the sector (GOMES; LEAL; ASSIS, 2013). According to Ittner and Larcker (1998), in this course, perhaps the most fundamental question is to assess whether the private sector notions of performance measurement and accountability apply to the field of public organizations.

For Van Dooren, Bouckaert e Halligan (2015), in association to the Performance term, discussions on management, public sector reform and public policy should be developed, as well as a distinguishment between the concepts of Performance ("defined as the attainment of public values such as efficiency, effectiveness, fairness, robustness, openness, and transparency"), Performance Measurement (“the process of acquiring information on performance”) and Performance Management (“incorporating and using the information on performance in the decision making process (VAN DOOREN et al., 2015, p. 39).

In Brazil, there is a methodological reference developed by the Ministry of Planning to define and measure performance in the public sector. The Document regards that an ideal model of "Governance for Performance" needs to be multidimensional, in order to recognize the various dimensions of efforts required to achieve the expected results (MINISTRY OF ECONOMY, 2010). Brignall and Modell (2000) point to the same direction, mentioning that multidimensional performance measurement systems can be applied to the public sector, which would require, from the perspective of institutional theory, changing the focus of attention to power and to the pressures exerted by different interest groups (stakeholders), as well as to investigate how this would affect the use of information on the performance of organizations.

Atkinson, Waterhouse and Wells (1997) further highlight that traditional performance measurement systems (based on accounting) do not fit current organizations whereby relationships between employees, clients, suppliers and other stakeholders demand the evaluation of intangible and flexible organizational aspects. For these authors, a performance measurement approach based on the stakeholders' theory would also be able to assimilate issues related to strategic planning, highlighting that the choices of a company in this area influence the organizational performance measurement system itself.

When it comes to specific dimensions to measure public procurement performance, Schooner, Gordon and Clark (2008) emphasize that the implementation of objectives and goals in this context is unique. In such procurement processes, the goals may be abstractly defined by law or by regulation, limiting the flexibility of specific purchases. Also, there are more players in the public procurement arena when compared to private contracts (SCHOONER et al., 2008). Thus, the authors divide procurement objectives into two clusters, which would correspond to two possible groups of public procurement performance dimensions.

As pointed out by Rezende (2018), Schooner et al. (2008) gathers in the first group those objectives focused on the achievement of the procurement primary objective, in other words, the satisfaction of the final users of the acquired products. Thus, the very satisfaction of final users arises as a first dimension of performance, considering that the buyer in public procurement is rarely its final user, not being able to capture the real needs of the demander (SCHOONER et al., 2008). The ‘economy’ dimension requires the acknowledgement of budget restrictions in public procurement management and best value is seen as the result of the final-user satisfaction limited by the need for cost reduction, which requires the combination of the ‘price’ criterion with issues related to product quality (SCHOONER et al., 2008). In the authors' perspective, the efficiency is related to how the public procurement system works, including elements such as how fast procurement processes happen and waste reduction in terms of both human and financial resources.

Rezende (2008) also sums up that the second group of dimensions described by Schooner et al. (2008) focuses on the conformity of public procurement processes and gathers those objectives that favour the decisions made by public managers, even when at the expense of the very satisfaction of the final user addressed in Group 1 (SCHOONER et al., 2008). In this scope, ‘integrity’, which seeks to fight corruption, requires the verification of elements such as competition, transparency and accountability. ‘Uniformity’ translates into the standardization of rules and procedures for administrative processes, while ‘transparency’ relates to the access to official information on government contracts. Finally, the dimensions of ‘accountability’ and ‘competition’ are not considered as ends in themselves in public procurement, but as means to avoid corruption and maximize the procurement results. ‘Accountability’ includes issues related to responsibility, accountability and the existence of formal internal control mechanisms. On the other hand, ‘Competition’ is focused on the search for better costs and higher quality of acquired goods/services, also emphasizing the supression of corruption in State contracts (SCHOONER et al., 2008).

4 METHODOLOGY

As for its design, this is a descriptive study, since it aims to describe the characteristics of a population or a specific phenomenon and to establish relationships among variables (GIL, 2002). As for its approach, this is a qualitative-quantitative research, with a longitudinal time frame and non-probabilistic sampling. The data collected are primary and refer to variables observed in the set of selected articles.

Regarding the selection of the articles that compose the state of the art, we collected all published studies which contents concerned the discussion of the "government procurement performance" construct in articles applied to Public Administration in Brazil, considering the academic production published between 2007 and 2017. The consultations were undertaken in the Scientific Periodicals Electronic Library (Spell) database and in the Capes Periodicals Portal, remembering that the latter brings together information from 126 other reference bases.

In both cases, we applied the following keyword combinations, both in Portuguese and in English, entered to all search fields: “compras governamentais” or “compras públicas” (“public procurement”), (+) “desempenho” (“performance”) and (+) “Brazil” (qhen searching for articles in English). Since the research targeted the production applied to Brazil, this study its search with keywords in Portuguese. Later, the search criteria were replicated using the keywords in English. However, no additional papers were identified in English.

Initially, the queries pointed to a total of 74 articles, three of which from the Spell base and the remaining coming from the Capes Periodicals Portal. In order to refine the selection, we excluded all duplicate works and those whose abstracts indicated a lack of thematic adherence to the research object. In the end, 25 articles remained for bibliometric analysis and for the elaboration of the state of the art, which represent the entire national production on the theme published during the period under consideration. This is the base from which this study develops its analyses.

The following information were collected from each article added to the database: journal of origin, journal classification according to Qualis Capes parameters for the area of Administration in force in 2018, year of publication, authors, nature of the research (theoretical or empirical), design (descriptive or correlational), empirical approach (qualitative or quantitative) and research methods. We developed our own categorization, which is presented in this study, based on the research proposal developed in each article. For the variables’ analysis, we used descriptive statistics, and for the classification of studies into categories, we applied a content analysis to the research objectives extracted from each article (FLICK, 2013). From the constructed categories, we prepared a time series with the performance of each category over the years inserted in the research cut-out.

5 PUBLIC PROCUREMENT PERFORMANCE: BIBLIOMETRIC ANALYSIS AND STATE OF THE ART IN THEMATIC CATEGORIES

Zupic and Carter (2015) consider bibliometry as an alternative way of mapping science, using quantitative methods to verify the existing relation between two disciplines, fields and specialties, hence reducing, according to the authors, the subjective bias carried by the traditional literature review method. Within this scope, representative studies on the production applied to the Brazilian reality were sorted in a support spreadsheet according to demographic, methodological and theoretical variables (namely, i. does it discuss topics related to public procurement performance?; ii. does it adopt a multicriteria model to assess procurement performance?; iii. does it address the use of the State's procurement power?; and iv. in which area does it address the use of procurement power?).

The gathered studies are predominantly empirical (24), which demonstrates that the issues connected to “public procurement performance” and to the “use of the State’s procurement power” are not restricted to the theoretical field, pertaining to the practical reality of those organizations involved with public procurement. In such cases, mostly case studies were carried out (18) to elaborate descriptive researches (21). Regarding the nature of the data, there is an alternation between qualitative (11), quantitative (9) and mixed approaches (5).

Based on the formulated theoretical variables and considering the thematic affinity between the articles, three categories of analysis were constituted resulting from content analysis of the articles’ abstracts. Such categories are used as the basis for the literature review and for the elaboration of the state of the art of the national production. Table 1 depicts these thematic categories, which content will be presented in the following subsections.

Table 1- Thematic categories used in the state of the art

Source: research data.

By analysing the trajectory of publications in each of the categories and how they are dispersed throughout a time series over the period 2007 to 2017, it is possible to clarify some of the field’s characteristics. Although search searched started with articles from 2007, it was only in 2010 that the first articles addressing the performance theme associated with the use of public procurement were registered. Thus, in effect, it is recorded that the production in the field is concentrated in only eight years.

Category 1 has been a frequent choice for researchers since the first year of the time series (2010), registering at least one publication in six of the eight years of the series. Category 2 registers only one article, published in the last year of the time series (2017). In turn, category 3 gathers the largest number of articles among the three, and its first publications only occur in the third year of the series (2012). Although its appearance is delayed in relation to the first category, its frequency of publications stands out. Since its emergence in 2012, this category has registered, on average, two publications per year in an uninterrupted manner. These trajectories are presented in Graph 1 and detailed in the following sections.

Graph 1- Time series for the state of the art categories

Source: research data.

The collected data also allows to characterize which journals have published the most on public procurement performance applied to Brazil. In all, 12 journals published all of the 25 selected articles. However, their participation in the field isn’t equally distributed. Revista de Administração Pública (RAP) gathers most of the published articles, with 36% of the entire literature. In second place, we can highlight Holos, Revista de Administração Mackenzie, Revista de Ciências da Administração, Revista de Gestão and Revista Metropolitana de Sustentabilidade, who published two papers on the theme each.

5.1 Results of the use of public procurement

This category gathers those articles that proposed discussions on the results generated by the use of public procurement in a generic manner. It is worth noting that such articles do not develop multicriteria analysis models in their empirical studies, rather focusing on the evaluation of one or two variables associated with the results of procurement individually. More than a generalized research option in the field, the high incidence of such articles (44%) reveals the frequent option for reductionist research designs regarding the phenomenon which, by nature, is known to be complex and requires to be evaluated under bi-dimensional parameters (such as the dimensions of user satisfaction and quality of the procurement process) and multicriteria (e.g. with the observation of the variables integrity, uniformity, transparency, accountability and competition) (SCHOONER et al., 2008). We present below the studies that constitute this category.

Faria, Ferreira, Santos and Silveira (2010) investigated the factors that are determinant to price variation of those products purchased via electronic auction. To this end, an empirical research and a correlational statistical analysis were carried out. Results showed that the variables "number of suppliers", "assets specificity", "quantity" and "frequency of transactions" have a 67.4% joint explanation power for price variations.

Medeiros et al. (2014) assessed the quality of products and services purchased in procurement processes of the ‘lowest price’ type, based on a case study in a City Council in Rio Grande do Sul. The authors performed document analysis, in loco observation and applied questionnaires, to investigate the perception of the organization’s employees in relation to the quality of contracted products and services. They found that the level of quality of products is considered satisfactory, despite being recognized as inferior in relation to those acquired through processes that were not guided by the ‘lowest price’ criterion.

Jorge et al. (2010) measured the relative relevance of the use of various procurement models in the expenses of a public institute, in order to identify indicators to quantify the effect of improving these intermediate activities. An empirical research was conducted, based on a case study at the Evandro Chagas Clinical Research Institute (IPEC) of the Oswaldo Cruz Foundation (Fiocruz), from 2004 to 2007. It was pointed out that the promotion of cost effectiveness in procurement activities generated positive practical results in relation to the expansion of competition in procurement processes.

Tridapalli, Fernandes and Machado (2011) develop a theoretical operational model for supply chain management for public authorities as from the procurement processes and based on information technology techniques, process management and electronic commerce. An empirical and descriptive study was carried out, based on structured interviews with actors involved in the procurement processes of the 26 Brazilian states and in Federal Government units, in order to verify the stages of evolution of their procurement and supply management systems, in the years 2006 and 2007. It was pointed out that 96% of the surveyed units have the capacity for improvement, since they are below the expected transformation stage.

Borges, Walter and Santos (2016) verified the possible improvements in public procurement processes of goods, based on the use of on-site auctions with price registration. The authors carried out a bibliographic research, a document research and a case study at the Regional Electoral Court in the State of Paraíba. It was pointed out that the studied process is characterized by bureaucratic obstacles, which delay its flow due to the excessive concentration of activities in high-level units of the organization.

Santos, Fank and Varela (2012) assessed the levels, the determinants and the evolution of the public supply stocks of the Southern Region States’ capital cities. It was carried out a descriptive research with a quantitative approach. The results showed the incidence of issues related to the lack of supplies in the city halls' warehouses, possibly generated by lack of planning and strategy in the management of public stocks.

Freitas and Maldonado (2013) investigated ways to obtain cost reduction when hiring continuous services, as well as the potential effects of such reduction in contracted organizations and outsourced service providers. An exploratory study was carried out, based on a bibliographical survey of related scientific, normative and document productions. The study concluded that the use of the electronic auctions has resulted in greater agility, transparency and competitiveness in procurement processes.

Faria, Ferreira and Gonçalves (2013) checked on the factors related to potential risks arising from the use of electronic auctions, based on the transaction costs theory. The authors carried out an exploratory and descriptive research, with a quantitative approach, in order to classify the companies supplying for the Federal Government, according to the risk degree verified in their supply contracts. It was revealed that public agencies still need to develop mechanisms to mitigate opportunistic attitudes.

Vaz and Lotta (2011) identified and organized the logistical components related to public policies management, based on the premise that logistical decisions would be critical in discussions about the improvement of the State and of public services. A theoretical and conceptual study and an empirical research were conducted based on public agencies websites, management reports and the researchers' own personal experience. It was concluded that logistical decisions are capable of significantly impacting the effectiveness of governmental actions and should be considered in public management reform processes.

Dias et al. (2013) investigated the wastefulness factors associated with irregularities in the management of municipal resources transferred by the Federal Government to be used in the healthcare area, considering, among others, the spending on public procurement. An empirical research was conducted, based on the reports of audits undertaken by the Office of the Comptroller General (CGU) in 2010, in 102 inspected municipalities. The results did not show an adequate level of significance for active waste. Regarding passive waste the following factors were found: a) administrative inadequacy; b) weak supervision; and c) low level of compliance.

Ensslin et al. (2014) conducted a survey and a bibliometric analysis of articles published in scientific journals as of 2000 on the "Government procurement from the perspective of performance evaluation" theme. It was carried out an exploratory and descriptive research, using the method known as Knowledge Development - Constructivist (ProKnow-C). Results enabled the identification of the main journals, articles, authors and keywords in line with the theme, considering the criteria established by researchers.

In general, the works gathered in this category discussed matters related either to pricing/costs, the quality of purchased goods or to the logistic decisions made by public procurement managers. This way, the performance phenomenon, whenever applied to the case of public procurement, was investigated based on these variables in an isolated manner. In other words, without proposing or applying multicriteria evaluation models, able to contemplate the different dimensions related to public procurement performance, which have been mentioned by the literature (SCHOONER et al., 2008).

Finally, it is worth mentioning that none of the aforementioned studies addressed the results of the use of public procurement within the context of the elaboration of public policies based on the State’s demand, in order to verify, in concrete cases, if the Brazilian State has succeeded in such use.

5.2 The use of multicriteria models to assess public procurement performance

The present category gathers only one study that used a multicriteria evaluation structure in order to perform an empirical assessment of public procurement performance. Although it might seem questionable to create a category within a state of the art based only in one sole article, we advocate in favour of maintaining the category for understanding that a great deal of the literature on public procurement would effectively migrate to it within the next few years.

While theory already recognizes the imperative of designing studies of a multicriteria nature in order to asses complex matters related to the performance of State Actions, such as public procurement, empiric evidences within the recent national production reveal that the compliance to such theoretical recommendation is still latent. It is necessary that future researches rely on previous experiences of testing individual factors to then evolve in the proposition of more complex and multicriteria research designs. Thus, we outline a research gap in order to compose a collective research agenda for the field: to propose and test multi-criteria evaluation models for public procurement performance, in accordance with the multifaceted nature of the phenomenon. Following the example of the article presented in this category, which evolves previous research designs, the cumulative character of generation is emphasized, learning the adoption of research designs of multicriteria nature.

For now, this category comprises the study of Nishiyama et al. (2017), which used a multicriteria model to evaluate the performance of procurement processes of the Central Animal Facility of the Federal University of Santa Catarina (UFSC), from the perception of the head of the supply department of the organization. It was carried out an exploratory research, of qualitative-quantitative approach, based on a case study. For the proposed performance evaluation, the so-called "Multicriteria Constructivist Decision Support (MCDA-C)" methodology was employed. According to researchers, this methodology presents itself as a branch of traditional MCDA to support decision makers in complex, conflicting and imprecise environments.

5.3 The use of the State’s procurement power

This category includes articles that addressed the use of government procurement as a strategic tool for demand-based public policymaking (RAUEN, 2017). In essence, these are not studies that discuss issues related to public procurement performance, but to its own use as a public policy tool. Thus, because they do not refer directly to objective performance criteria, their propositions are summarized in Chart 1.

Chart 1- Articles from the bibliographic portfolio of thematic category 3

Source: the authors.

Among the studies that verified the use of public procurement as a tool for public policy, it is possible to highlight their analytical focus: 42% are focused on environmental sustainability; 35% investigate innovation; and 15% explore such premise applied to Micro and Small Enterprises. In these cases, the studies laterally analysed the performance phenomenon in order to evaluate public policies and/or organizations (both public and private) inserted in the context of the use of the State’s procurement power.

For example, Moreira and Vargas (2012a) and Moreira and Vargas (2012b) investigate the process of inducting innovation in companies that supply for the Government, based on procurement requisites established by Governmental customers. Within the Environmental Sustainability theme, Alencastro, Silva and Lopes (2014) assessed the process to implement sustainable procurement by the Federal Public Administration, highlighting its objectives, normative regulations, potentialities and limits. Regarding fomenting both economic and productive development for micro and small businesses, Cabral, Reis and Sampaio (2015) investigate the conditioning factors for the segment's participation and success levels in public procurement, in light of the main legislative changes in this regard.

Following the same trend, the performance of public procurement on itself, considering its economic, financial and administrative impacts, was not the focus of the authors of the remaining articles in this category; their focus lies on the performance of the selected policy, in terms of stimulating desired behaviours in their target audience.

6 CONCLUSIONS

This study explored the field of study ‘Public Procurement as an instrument for public policy in Brazil’. Seeking to answer to the question “how recent literature has assessed the performance of the use of Public Procurement in Brazil?”, a literature review of the production applied to Brazilian Public Administration from 2007-2017 was developed and the State of the Art for the field was mapped, which is developed around three major thematic lines.

Under the perspective of the established time frame, recent studies from 2010 are configured as investigations on the results of the use of public procurement, with major focus on the technical criteria that relate to procurement, such as pricing or quality of acquisitions. Two years later, in 2012, we noted the appearance of a new set of studies focused on the use of procurement as strategic tools for public policy, such as their use in the promotion of innovation and sustainability. Ni 2017, there was the first effort to test multicriteria models to perform an empirical assessment of public procurement performance, in response to the recommendation made by the literature (SCHOONER et al., 2008). Thus, it is understood that two major common approaches guided the field of study until 2017, from when national literature should be enhance towards filling the first research gap highlighted by this study: to develop researches that test the performance of public procurement under a compared multicriteria perspective.

Among the significant results presented in this study, we highlight the low incidence of articles that effectively discuss the variables or dimensions related to measuring procurement performance. This information is more complex than it seems: Brazilian recent literature on public procurement performance is rather expressive (there are 25 published articles in 10 years) and with high quality (84% of the studies were published in B2 or higher journals). However, it is unspecific (only 48% of the papers objectively discuss criteria that represent procurement performance). Among those, many of them adopt technical criteria rather than theoretical ones: pricing, quality of products and used procurement models. Such criteria are focused on the technical execution of procurement processes and not to their use as an instrument for public policy – on the other hand of the theoretical recommendations, which foresee variables that would go beyond the procedural character of the evaluation of procurement procedure, such as user satisfaction (SCHOONER et al., 2008).

Thus, we present the second research gap that composes the collective research agenda for the field: proposing research designs to measure performance in public procurement that contain criteria, dimensions, and variables of a theoretical nature that are typically representative of the use of procurement as a tool for public policy, such as criteria associated with user satisfaction, transparency, or accountability.

The group of results obtained in this study allows us to therefore conclude that the literature applied to procurement in the Brazilian Public Administration investigates its performance, although in a restricted manner. Frequently, the studies examine procurement performance under the proxy of the performance of public policies that use such tool. This is a useful research design, however limited in its comprehensiveness of the cross-cutting role of public procurement. Even though procurement processes are tools for State public policies, they also transcend their original role as a way to supply (MCCRUDDEN, 2004). Thus, we can point out that procurement management is a typical organizational function to which investigative efforts should also be directed regarding its efficiency, effectiveness, and economic, financial, administrative, and institutional impacts.

In summary: if in recent years the literature has focused on investigating our ability to use public demand policies, within the next ten years authors are urged to revisit the original function of procurement to answer an emerging question: after using procurement as a public policy instrument for more than a decade, have we become good public purchasers?

Within the limitations of the present study, it is worth mentioning the fact that the scope was restricted to literature applied to the Brazilian Public Administration. We suggest the development of studies that compare the national literature to the international production. When considering international production on public procurement performance, it is expected that it would be possible to additionally summarize evidences (possibly already investigated) for further testing in evaluation models of national procurement.

REFERENCES

ALBANO, G. L.; RUSSO, F. A., CASTALDI, G.; ZAMPINO, R. Evaluating Small Businesses' Performance in Public e‐Procurement: Evidence from the Italian Government's e‐Marketplace. Journal of Small Business Management, v. 53, p. 229-250, 2015.

ALENCASTRO, M. A. C.; DA SILVA, E. V.; LOPES, A. M. D. Contratações sustentáveis na administração pública brasileira: a experiência do Poder Executivo federal. Revista de Administração Pública, v. 48, n. 1, p. 207-236, 2014.

ALVARENGA, R. M.; DE QUEIROS, D.; DO CARMO, M.; DE SOUZA, L. B. Arranjo produtivo local e desenvolvimento sustentável: uma relação sinérgica no município de marco-Ceará. Revista de Administração Mackenzie, v. 14, n. 5, 2013.

AMANN, M. K.; ROEHRICH, J.; EBIG, M.; HARLAND, C. Driving sustainable supply chain management in the public sector: The importance of public procurement in the European Union. Supply Chain Management: An International Journal, v. 19, n. 3, p. 351-366, 2014.

DE ARAÚJO, G. C.; TEIXEIRA, C. E. Operacionalização das compras públicas sustentáveis na secretaria de administração penitenciária do estado de São Paulo. Revista Metropolitana de Sustentabilidade, v. 6, n. 3, p. 63-86, 2016.

ASCHHOFF, B.; SOFKA, W. Innovation on demand - Can public procurement drive market success of innovations? Research policy, v. 38, n. 8, p. 1235-1247, 2009.

ATKINSON, A. A.; WATERHOUSE, J. H.; WELLS, R. B. A stakeholder approach to strategic performance measurement. MIT Sloan Management Review, v. 38, n. 3, p. 25, 1997.

BARBOZA, R. B.; FONSECA, S. A.; DE FREITAS, G. C. O papel das políticas públicas para potencializar a inovação em pequenas empresas de base tradicional. REGE-Revista de Gestão, v. 24, n. 1, p. 58-71, 2017.

BORGES, L. M.; WALTER, F.; SANTOS, L. C. Análise e redesenho de processos no setor público: identificação de melhorias em um processo de compra. Holos, v. 1, p. 231-252, 2016.

BRAMMER, S.; WALKER, H. Sustainable procurement in the public sector: an international comparative study. International Journal of Operations & Production Management, v. 31, n. 4, p. 452-476, 2011.

BRASIL. Lei nº 8.666, de 21 de junho de 1993. Regulamenta o art. 37, inciso XXI, da Constituição Federal, institui normas para licitações e contratos da Administração Pública e dá outras providências. Diário Oficial União, 1993.

BRIGNALL, S.; MODELL, S. An institutional perspective on performance measurement and management in the ‘new public sector’. Management accounting research, v. 11, n. 3, p. 281-306, 2000.

CABRAL, S.; REIS, P. R. D. C.; SAMPAIO, A. D. H. Determinantes da participação e sucesso das micro e pequenas empresas em compras públicas: uma análise empírica. Revista de Administração, v. 50, n. 4, p. 477-491, 2015.

CASTRO, J. K.; DE FREITAS, C. L.; DA CRUZ, F. Licitações sustentáveis: um estudo em instituições federais de ensino superior na região sul do Brasil. Revista Metropolitana de Sustentabilidade, v. 4, n. 1, p. 31-51, 2014.

DE FREITAS, M.; DE VARGE, J. S. O pregão eletrônico e as contratações de serviços contínuos. Revista de Administração Pública, v. 47, n. 5, p. 1265-1282, 2013.

DIAS, L. S.; MATIAS-FERREIRA, J.; FARIAS, M. S.; PAMPLONA, V. S. Fatores associados ao desperdício de recursos da saúde repassados pela união aos municípios auditados pela Controladoria Geral da União. Revista Contabilidade & Finanças, 2013.

DOS SANTOS, P. S. A.; FANK, O. L.; VARELA, P. S. Determinantes do nível dos estoques públicos: mensuração nas capitais dos estados da região Sul do Brasil. Revista Eletrônica de Estratégia & Negócios, v. 5, n. 2, p. 28-55, 2012.

ENSSLIN, S. R.; ENSSLIN, L.; NISHIYAMA, M. A.; CHAVES, L. C. Compras governamentais sob a ótica da avaliação de desempenho: um mapeamento do tema conforme as delimitações postas pelos pesquisadores. Gestão & Regionalidade, v. 30, n. 90, 2014.

FARIA, E. R.; FERREIRA, M. M.; DOS SANTOS, L. M.; SILVEIRA, S. R. Fatores determinantes na variação dos preços dos produtos contratados por pregão eletrônico. Revista de Administração Pública, v. 44, n. 6, p. 1405-1428, 2010.

FARIA, E. R.; FERREIRA, M. A. M.; GONÇALVES, M. A. Avaliação dos riscos do pregão eletrônico: uma abordagem pela teoria da nova economia institucional. Revista de Ciências da Administração, v. 15, n. 37, 2013.

FERREIRA, N. A. As pesquisas denominadas" estado da arte. Educação & sociedade, v. 23, p. 257-272, 2002.

FERREIRA, V. S.; NAJBERG, E.; FERREIRA, C. B.; BARBOSA, N. B.; BORGES, C. Inovação em serviços de saúde no Brasil: análise dos casos premiados no Concurso de Inovação na Administração Pública Federal. Revista de Administração Pública, v. 48, n. 5, p. 1207-1228, 2014.

FLICK, U. (Ed.). The SAGE handbook of qualitative data analysis. Sage, 2013.

FLYNN, A.; DAVIS, P. Theory in public procurement research. Journal of public procurement, v. 14, n. 2, p. 139-180, 2014.

FREITAS, M.; MALDONADO, J. M. S. V. O pregão eletrônico e as contratações de serviços contínuos. Revista de Administração Pública - RAP, v. 47, n. 5, 2013.

GEORGHIOU, L.; EDLER, J.; UYARRA, E.; YEOW, J. Policy instruments for public procurement of innovation: Choice, design and assessment. Technological Forecasting and Social Change, v. 86, p. 1-12, 2014.

GIL, A. C. Como elaborar projetos de pesquisa. São Paulo, v. 5, n. 61, p. 16-17, 2002.

GOMES, R. C.; LEAL, A. S.; ASSIS, V. A. Indicadores para avaliar o desempenho de prefeituras municipais. Tecnologias de Administração e Contabilidade-TAC, v. 3, n. 1, p. 1-15, 2013.

ITTNER, C. D.; LARCKER, D. F. Innovations in performance measurement: Trends and research implications. Journal of management accounting research, v. 10, p. 205, 1998.

JORGE, M. J.; AVELLAR, C. M.; DE MELO, L. C.; PIGATTO, J. M.; BATISTA, D. L. Indicadores de Efetividade em custo de atividades-meio no modelo de gestão para resultados: a experiência do Ipec/Fiocruz. Revista de Educação e Pesquisa em Contabilidade, v. 4, n. 2, p. 1-22, 2010.

JUSTEN FILHO, M.; JORDÃO, E. F. A contratação administrativa destinada ao fomento de atividades privadas de interesse coletivo. Revista Brasileira de Direito Públic, v. 9, 2015.

LIMA, D. H.; DE VARGAS, E. R. Estudos internacionais sobre inovação no setor público: como a teoria da inovação em serviços pode contribuir?. Revista de Administração Pública, v. 46, n. 2, p. 385-401, 2012.

LOPES, R. G.; DE MOURA, L. R. Responsabilidade socioambiental: uma análise do projeto “campus verde–gestão ambiental do IFRN”. HOLOS, v. 3, p. 135-147, 2015.

MCCRUDDEN, C. Using public procurement to achieve social outcomes. In: Natural resources forum. Oxford, UK: Blackwell Publishing Ltd., 2004. p. 257-267.

MEDEIROS, F. B.; DOS SANTOS, S. X.; DENARDIM, É. S.; ABBADE, E. B. A qualidade dos produtos e serviços em licitações do tipo menor preço: um estudo em uma câmara de vereadores do Rio Grande do Sul. REGE-Revista de Gestão, v. 21, n. 4, p. 491-508, 2014.

MINISTÉRIO DA ECONOMIA. Compras governamentais. Painel de Compras do Governo, 2017. Disponível em: https://paineldecompras.planejamento.gov.br/. Acesso em: 11 maio 2018.

MINISTÉRIO DA ECONOMIA. Guia referencial para medição de desempenho e Manual para construção de indicadores. 2010.

MODELL, S. Performance measurement myths in the public sector: a research note. Financial Accountability & Management, v. 20, n. 1, p. 39-55, 2004.

MOREIRA, M. F.; DE VARGAS, E. R. Compras para a inovação: casos de inovações induzidas por clientes públicos. Revista de Administração Mackenzie, v. 13, n. 5, 2012a.

MOREIRA, M. F.; DE VARGAS, E. R. Quando o governo é o mercado: compras governamentais e inovação em serviços de software. Revista de Administração e Inovação, .(2), 175-197, 2012b.

NISHIYAMA, M. A.; DE LIMA, M. A.; ENSSLIN, L.; CHAVES, L. C. Modelo Multicritério para Avaliação de Desempenho: um estudo de caso para gestão de compras no setor público. Revista de Ciências da Administração, v. 1, n. 1, p. 09-28, 2017.

OLIVEIRA, B. S.; LUZIO, L. M. Compras públicas como política para o desenvolvimento sustentável. Revista de Administração Pública, v. 49, n. 1, p. 189-206, 2015.

OLIVEIRA, M. F.; DA SILVA, J. F.; DA MOTTA, L. F. J. A vantagem competitiva das nações e a vantagem competitiva das empresas: o que importa na localização?. Revista de Administração Pública, v. 46, n. 3, p. 701-720, 2012.

OSBORNE, D; GAEBLER, T. Reinventando o governo: como o espírito empreendedor está transformando o setor público. Mh Comunicação, 1994.

OSBORNE, S. P. The new public governance?. Public Management Review, p. 377-387, 2006.

OSBORNE, S. P. Public governance and public services delivery: a research agenda for the future. The new public governance, p. 413-428, 2010.

RAUEN, A. O. Políticas de inovação pelo lado da demanda no Brasil. Brasília. Instituto de Pesquisa Econômica Aplicada, 2017.

REZENDE, E. C. M. Da política pública à performance: mapeamento do desempenho de compras governamentais no Simples Nacional. Dissertação de Mestrado Profissional em Administração Pública. Programa de Pós-Graduação em Administração – PPGA. Universidade de Brasília, Brasília, 2018.

SANTOS, P. A.; FANK, O. L.; VARELA, P. S. Determinantes do nível dos estoques públicos: mensuração nas capitais dos estados da região sul do Brasil. Revista Eletrônica de Estratégia & Negócios, v. 5, n. 2, p. 28-55, 2012.

SCHOONER, S. L.; GORDON, D. I.; CLARK, J. L. Public procurement systems: unpacking stakeholder aspirations and expectations. GWU Law School Public Law Research Paper, n. 1133234, 2008.

SQUEFF, S.; DE HOLANDA. O poder de compras governamental como instrumento de desenvolvimento tecnológico: análise do caso brasileiro (No. 1922). Texto para Discussão, Instituto de Pesquisa Econômica Aplicada. 2014.

TRIDAPALLI, J. P.; FERNANDES, E.; MACHADO, W. V. Gestão da cadeia de suprimento do setor público: uma alternativa para controle de gastos correntes no Brasil. Revista de Administração Pública-RAP, v. 45, n. 2, p. 401-433, 2011.

VAN DOOREN, W.; BOUCKAERT, G.; HALLIGAN, J. Performance management in the public sector. Routledge, 2015.

VAZ, J. C.; LOTTA, G. S. A contribuição da logística integrada às decisões de gestão das políticas públicas no Brasil. Revista de Administração Pública, v. 45, n. 1, p. 107-139, 2011.

ZUPIC, I.; ČATER, T. Bibliometric methods in management and organization. Organizational Research Methods, v. 18, n. 3, p. 429-472, 2015.

IR