1 Introdution

The constant need for the evolution of public organizations requires the search for optimized standards of performance, ethics, transparency, and responsibility in management, based on the evaluation of their procedures and improvement in the way they perform their functions (ASSIS, 2015; MANHANI JUNIOR; LIMA, 2016).

In this context, government purchases gained relevance, as they are directly related to the services provided by the GovernmentGovernment and, consequently, to the impacts suffered by society (ENSSLINet al., 2014). The concern with the efficient use of public resources results in initiatives to control expenses with purchases and contracts, whose expenses represented around 36% of budgets (RAPOSO et al., 2016).

Over the past four decades, pressures for more transparent and efficient management have challenged researchers to propose new procedures to support public sector procurement management, and performance evaluation has emerged as a way to develop decision-making models for the sector (NISHIYAMAet al., 2017). This work analyzes the performance of the public procurement process through the customer's lenses s and process operators. It proposes improvements to leverage the maturity level of the purchasing process of the researched company.

Despite the importance of public procurement, limited scientific research has been undertaken to examine factors influencing the performance of the procurement function (WANYONYI; MUTURI, 2015). Since public services must be provided with increasingly smaller resources, seeking increasing performance levels, its measurement becomes important (LEGATO; MALIZIA; MAGZA, 2016).

For Dezolt and Barbosa (2016), the definition of evaluation criteria for the procurement model used by the states must consider a multidimensional view to producing a more comprehensive assessment of the current situation and opportunities for improvement.

This research uses the concepts of the purchasing function as one of the main elements of logistics and supply chain management. This research contributes to science in this context, as it uses variables used to assess the chain's performance. To analyze the performance of a public organization's procurement process.

In the public sphere, it is essential to recognize that, despite all the uniformity and rigidity of the legal framework, the plurality and heterogeneity of public institutions in the federal, state, municipal spheres (with their different objectives and contexts several) affect the performance of the purchasing process (SILVA, 2017). Given this finding, the need to delimit the research was identified, opting for the performance analysis of 2 units of a public research company located in the state of Ceará to produce reflections, advances, and improvements in the process.

2 Theoretical Reference

This section presents the themes that permeate the evaluation of performance in the public procurement process, divided into the public procurement process; and performance evaluation in supply chain processes.

2.1 The Public Procurement Process

Dlamini (2016) conceptualizes purchasing as a systematic process of purchasing goods and services within the organization. This action involves the need assessment, supplier selection, price negotiation, and follow-up to ensure delivery and provide the user with the right quality and suitable quantities of products and services at the right time and at the right price. This concept can be applied in the public and private spheres, but there is a need to follow the rite of legislation (FENILI, 2016).

Public purchasing is a procedure that enables the GovernmentGovernment to obtain services, materials, and equipment necessary for its operation (SQUEFF, 2014). It is a dynamic process that involves bidders, regulators, and purchasing agencies, including identification of needs, selection of sources, preparation and award of contracts, and all phases of contract administration until the end of a service contract or the useful life of an asset (SRIVASTAVA; AGRAHARI, 2017).

The rules for the acquisition of goods and contracting of services by the Brazilian public sector is the observance of the rite of public bidding, provided for in the Federal Constitution of 1988 and regulated by Law No. 8666 (1993) (Bidding and Contract Law), and its amendments (FENILI, 2016), conceptualizing bidding as a technical-legal means, an administrative procedure carried out in a succession of steps (OLIVEIRA, 2015).

Public procurement has already gone through several changes to promote technological improvements and innovations to reduce bureaucracy in the process (BRANDÃO, 2016), but they have barely changed their bureaucratic logic (ASSIS, 2015).

Among the main developments, we can highlight the advent of Law 10.520/2002, which instituted the auction, an essential step in the modernization of hiring procedures by the administration. It contributed to the speed of the process, reducing the term disclosure of the bid, the inversion of the qualification and judgment phases, the possibility of formulating bids, and the concentration of the appeal phase only at the end of the procedure, but applied only to the acquisition of everyday goods and services (ZYMLER, 2017).

Among the main changes introduced by the State-owned Law is the autonomy granted to public and mixed-capital companies (ZYMLER, 2017). The legislator left spaces to be filled by the bidding regulations of each state-owned company (SILVA, 2018). This explicit and implicit normative delegation can also be interpreted as a sign of legislative deference to the efficiency of state-owned companies. Less rigid legal provisions allow state-owned companies to bring their contracting regimes closer to private sector practices and reduce them. With its competitors, the intrinsic operational disadvantage of its purchasing processes (SILVA, 2018), but the bidding procedure remains the rule (ZYMLER, 2017).

Silva (2018) shows that state companies inserted in highly competitive market environments have a greater institutional need to make their bidding procedures more efficient. It is crucial to reduce the adverse effects of their condition as a government entity on them compared to their private competitors.

2.2 Performance evaluation in supply chain processes

Purchasing management is a strategic process for the GovernmentGovernment. Any improvement can represent a better public administration performance (NISHIYAMA, 2016), highlighting the opportunity to use performance assessment to manage public procurement.

There is a need to expand the scope of purchasing management to rationalize public spending, based on the integral management of the supply chain (RAPOSOet al, 2016). Purchasing management is an integrated supply chain activity in a dynamic and increasingly competitive environment, assuming a strategic condition for achieving results (SILVA, 2017).

However, knowledge on the subject is dispersed and still underexplored, lacking studies that build consolidated knowledge regarding the relationship between performance evaluation and public procurement to understand the alignment of the literature with organizational practices in the sector (ARAÚJO; MATOS; ENSSLIN, 2020).

In this context, it was searched in the literature, the performance evaluation models of the supply chain (Chart 1) with the indication of the used variables. It is worth noting that proposals for new models were not found in the last ten years, but applications of existing models.

Stewart (1995) developed nine metrics divided into four critical areas for evaluating the supply chain: delivery performance, flexibility and responsiveness, logistics costs, and asset management. Beamon (1999) identified three types of performance measures needed in any supply chain:

resource measures (R), related to resource availability

output measures (O), related to customer responsiveness, quality, and quantity of the final product

flexibility measures (F), which address the ability to accommodate volume fluctuations

Gunasekaran, Patel and Tirtiroglu (2001) developed a framework to measure the performance of a supply chain's strategic, tactical and operational level, consisting of a list of metrics, referring to suppliers, delivery performance, customer service, and inventory costs and of logistics.

The SCOR is a reference model that links business processes, performance metrics, and supply chain management best practices to support the description, evaluation, and comparison of activities and chain performance. The model provides an architecture basedon six business processes, which aim to support planning, supply, production, delivery, return, and implementation of processes and activities. It is evaluated by a set of metrics subdivided into performance attributes and performance indicators, whose performance attributes are reliability, responsiveness, agility, costs, and asset management (LIMA JUNIOR; CARPINETTI, 2019).

Christopher's (2007) approach to measuring logistical performance based on costs and benchmarking considers a set of customer service goals. They must be achieved with a focus on cost analysis and revenues and on the idea of benchmarking. The author also proposed an approach to evaluate performance in customer service and indicates the use of the BSC for evaluating logistical performance, proposing the quality of service, processing time, cost, and customer relationship.

Ambe and Badenhorst-Weiss (2011) proposed the BSC as a performance measurement system for the supply chain in the public sector, identifying as components of the BSC in the public sector, the citizen (customer), finance or resources, internal processes, and learning and innovation. Vay (2011) developed the conceptual model for evaluating logistical performance under the customer's perception, with a set of fifty-seven items, divided into the attributes availability, delivery quality, communication/order processing, and agility.

Finally, Deshpande's (2012) model was studied, which developed the global framework for the links between the dimensions of Supply Chain Management (SCM), SCM performance, and organizational performance. The dimensions of SCM are: long-term relationships, concurrent engineering, and strategic purchasing; SCM measures are: delivery flexibility, inventory cost, and customer responsiveness; and organizational performance measures are: financial and market performance and customer satisfaction.

For this research, the nine variables identified in Table 1, seven variables (cost, order processing, customer service, availability, delivery, processes, and return) were analyzed through a questionnaire detailed in Chart 2. A proposal for the variable "improvement" and the variable flexibility was not analyzed in this study. Due to the obligation to follow the rite of legislation, the public sphere makes the evaluation of this variable unfeasible.

Dezolt and Barbosa (2016) state that using processes as a performance dimension will better understand problems, bottlenecks, and possible solutions. The authors developed a referential maturity model containing five levels according to the characteristics of the purchasing model used, as shown in Chart 3, which will be used to define the level of maturity of the purchasing process of the studied company.

3 Methodological procedures

This article aims to analyze the procurement process performance from the point of view of customers and process operators and propose improvements to leverage the procurement process's maturity level in a public research company. To accomplish this goal, we opted for a mixed approach. Its nature is applied research, as it generates knowledge for practical application, aims to solve problems, and involves local interests (GERHARDT; SILVEIRA, 2009). As a research strategy, a single case study was chosen, with the questionnaire being applied in two units of the selected public company.

The data collection techniques used were document analysis; bibliographic research; questionnaire developed by the author; interview with the supervisor of the property and supplies sector of Unit II and own report on the purchasing process in Unit I, applied according to the following script: preparation and validation of the questionnaire, application of the questionnaire, collection, and analysis of quantitative data, preparation of the interview script, conducting an interview and reporting one's own experience using the same script as the interview, combined analysis of qualitative and quantitative research information, and elaboration of a framework to propose improvements.

The questionnaire was prepared from the analysis of Chart 4 to verify the performance of the public procurement process. As a component of the supply chain, it can be measured using seven attributes (variables). Chart 4 shows the variables and items of the questionnaire to which they are related.

The questionnaire was sent to all employees of the units located in Ceará, totaling a population of 349 employees. The submission was made electronically using Google Docs., resulting in 93 completed questionnaires, a result higher than the sample calculation of 78, representing a participation of 28% of the population and 38% of analysts and researchers (principal applicants of the process).

The chosen public company operates in the research area and has units throughout the country, having 9,329 employees when the questionnaire was applied. It would be impossible to apply the entire population or a sample as it is a large and dispersed population. For this reason, the two units located in the state of Ceará were chosen. The researcher's facility in accessing the population also contributed to the choice of units studied. These units are in the same region of the country, having the same peculiarities, facilities, and difficulties in the acquisition process.

The sampling method used was non-probabilistic for convenience. The criteria for choosing the units were intentional, opting to work with available individuals, and relevant information could be extracted, but the data cannot be generalized (CARMO; FERREIRA, 2008). A 4-point Likert scale was used in the questionnaire, in which (1) low performance, (2) low to medium performance, (3) medium to high performance, and (4) high performance.

An advantage of the Likert scale is the possibility of identifying the direction of the respondent's attitude towards each statement, whether it is positive or negative (TROJAN; SIPRAKI, 2015). The midpoint of the scale was omitted. According to Trojan and Sipraki (2015), the existence of the midpoint does not oblige the respondent to assume a position concerning the research object; on the contrary, the four-point scale induces a positive or negative response. For Dalmoro and Vieira (2013), the neutral point can generate ambivalence and indifference on the part of the respondent, clashing with the genuine opinion, which can be remedied with the use of an option such as "no conditions to give an opinion," which was the option of this research.

3.1 Data processing

The first analysis performed was the reliability of the questionnaire scale, using Cronbach's Alpha, which is often used as a reliable and straightforward way to validate the instrument, whose result was 0.9, which indicates good reliability of the scale. Alonso and Santacruz (2015) indicated that a Cronbach's Alpha value between 0.70 and 0.90 indicates good internal consistency.

Then, the analysis of the result of each variable was performed, using the techniques of descriptive statistics and Pearson's correlation to determine the correlation between the items in the questionnaire. After analyzing the quantitative data, a divergence was identified in the assessment between the surveyed units, generating the need to obtain more information, opting for an interview with the purchasing supervisor in Unit II and a report by the area supervisor in Unit I

The data analysis used in this research was connected to explain the results and use sequential analysis. Qualitative themes are linked to quantitative results, having as steps: quantitative data collection, quantitative data analysis, planning qualitative elements, qualitative data collection, and qualitative data analysis (CRESWELL; CLARK, 2013).

Based on the research results, a set of propositions for the variable "improvement" of the company's performance and leverage of the purchasing process maturity level was suggested as an action plan. This step aims to contemplate the only variable that did not exist been analyzed by applying the questionnaire, interviewing with the Unit II supervisor, and reporting the experience of the Unit 1 supervisor.

4 Results

Here are presented the description of the current purchasing process in the studied company, the analysis of the result of the process performance evaluation, and the improvement proposals.

4.1 Purchasing process in a public research company

The procurement process in the public research company was regulated by Law 8.666 (1993) and 10.520 (2002) until June 2018, being currently under the aegis of Law 13.303 (2016) and it is regulation.

The purchase planning is guided by the annual purchase calendar defined by the headquarters. It determines deadlines for the conclusion of the notices, operation of the bids, and the commitment of the contracts, considering the budget available for the year. In the internal phase of the process, which involves: object requisition, value estimation, expenditure authorization, designation of the bidding committee, preparation of draft notices, and legal analysis, the units work differently, as there is no corporate system that aids in the planning of purchases.

In the external phase of the process, consisting of notice publications, qualification, judgment, and classification of proposals, approval, and adjudication, the company uses Comprasnet, the leading platform used to operate the Federal Government's electronic procurement processes (CABRAL; REIS, 2018). The steps and deadlines are explicit in the law (OLIVEIRA, 2015; LOBÃO, 2015).

The public company object of this research prioritizes bidding in the electronic bidding modality using the Price Registration System (SRP) due to the non-mandatory requirement to carry out the acquisition (CABRAL; REIS, 2018), allowing the use of consumption forecasts to define the quantity to be tendered.

Once the public bidding session is closed and the winning suppliers are approved, the contracting phase is carried out internally, according to the notice and current regulations. In the public research company, the average time of a bidding process is 111 days, and processes of waiver and unenforceability of bidding are 20 days, which confirms the slowness of the procurement process in the public sector, as mentioned by Fenili (2016).

4.2 Evaluation of the performance of the procurement process of the public research company

The public procurement process performance in the company was positive, with a frequency of 56% with good evaluation, a mean of 2.57, a median of 3, and a standard deviation of 0.92. However, there was no significant difference in the frequency of low performance, which was 44%, and the mean did not depart significantly from the midpoint of the 2.5 scales.

Of the 28 items evaluated, 16 performed well, 1 had an average below the midpoint and a median above the midpoint, and 11 were assessed as performing poorly. Figure 1 summarizes the process evaluation.

The following items stand out as positive points in the process: 2) previous price surveys carried out before the bidding process, 4) self-assessment regarding the fulfillment of requests; 5) efficiency of the order processing system 6) service in the purchasing area; 8) conformity of products/services as specified in the order; 9) product/service acquisition information by sending an OCS/contract to the applicant; 10) involvement of the applicant in the evaluation of the proposals; 13) self-assessment regarding the follow-up of the request by the applicant; 14) response time from the purchasing area regarding the status of orders; 16) Quality of purchased products/services; 18) information technology resources used in the process; 20) conformity in the quantity of products with what is specified in the order; 22) the technical training of buyers; 23) number of personnel involved in the purchasing process; 27) performance evaluation of suppliers; and 28) notification and penalty for suppliers, and the variables that presented the best results were: order processing, delivery and return.

As in this research, Lobão (2015) highlights the use of information technology in the public procurement process as the closest point to the private sector. In addition, Wanyonyi and Muturi (2015) and Lobão (2015) identified in their research that information technology positively affects the performance of purchasing functions and the development of employees' competence.

Contrary to what was found in this research, Almeida and Sano (2017) stated a lack of qualified personnel in the purchasing sector, which harms the entire process's performance. This divergence may have happened because the author carried out his research with the auctioneers. The results from the customers and the purchasing team also identified the lack of qualified personnel.

Items that need to be improved to leverage the performance of the process, the following stand out:

1.observation of the lowest price criterion in the acquisition of quality products and according to the request; 2.price savings in contracts arising from the bidding process; 3.availability of order status information; 4.total order cycle time; 5.monitoring of acquisitions/contracts by the purchasing area; 6.agility, flexibility, and speed of suppliers in replacing products; 7.availability of information about in-stock products at the time of ordering; 8.compliance with the contract term by suppliers; 9.performance of the Bidding Law for the acquisition of quality products; 10.performance of the Bidding Law about the order fulfillment period; 11.performance of the Bidding Law in the selection and hiring of suitable suppliers.

Variables need improvement: cost, customer service, availability, order processing, delivery, and process.

As in this study, Fenili (2016) identified that the public procurement process needs to improve products' quality using the lowest price criterion. The lack of feedback and description of products by customers, the slowness of the process, and the cycle time due to excessive procedures, dysfunctional bureaucracy, and the lack of a support structure for the purchasing area also need improvement.

Cabral and Reis (2018) state that problems arising from proposals with shallow values , opportunistic behavior on the part of suppliers may act in other performance dimensions, such as deviations in the quality of the object or the contracted specifications, and even in terms of withdrawal from the contract.

The performance of the Bidding Law in the acquisition of materials with quality and on time, from the selection of suitable suppliers evaluated by items 24, 25, and 26, was low. It confirms that government policies and legislation impact execution time and process performance is necessary, as identified by Ferreira (2015) and Almeida and Sano (2017), reinforcing the need to strengthen the relationship between supplier and government (OLIVEIRA, 2015).

About the result of the interview and the experience report of the supervisors of the 2 units studied, they reported that: the teams have the same number of buyers, with training for this function, but they assess this quantity as insufficient to meet the demands; the company does not have an integrated supply chain management system, which makes it challenging to carry out the purchasing and supply activity in a strategic manner; the area provides information on hiring and stock balance to applicants, but they do not follow up on their requests; about 70% of deliveries are made on time; the cycle time of the purchasing process within the company is extended due to the need to comply with legal deadlines, the availability of a budget and the company's purchasing planning, which is semiannual; the planning of acquisitions is carried out based on demand and assessment of the minimum stock for everyday consumption items; and the company does not have a tool to evaluate suppliers, only the delivery period is monitored and the appropriate penalties are applied when the contract is not fulfilled.

After analyzing the results of the quantitative and qualitative research, an agreement is identified between the main results, with differences between the results of the interview and the questionnaire in the following items: a sufficient number of buyers to meet the demand, considered good in quantitative research and insufficient in qualitative research; in the monitoring of acquisitions by the purchasing area considered good in qualitative and poor in quantitative, and in the fulfillment of delivery deadlines by the suppliers evaluated as inferior in the quantitative research and sound in the qualitative one.

The result indicates that the public company surveyed is still at maturity level 1 of its purchasing process. According to the classification of Dezolt and Barbosa (2016), the lack of planning and a system integrated purchasing system, despite having some characteristics of the other levels, such as the use of an electronic purchasing system and administrative sanction.

Based on the results, Chart 5 presents a set of propositions for the variable improving company performance and leveraging the maturity level of the procurement process from level 1 to level 3 initially. Subsequently reaching level 5, reaching the characteristics listed by Dezolt and Barbosa (2016), identifying which variables and characteristics of each proposal's maturity level will work.

It is important to note that implementing these improvement proposals can leverage the company from a maturity level of 1. There is a lack of standardization of purchasing items, absence of purchase demand planning, purchases carried out in a decentralized manner, and lack of information on contract performance for level 3. A demand plan is implemented with standard centralized purchases. There is an electronic purchasing system, reaching level 5, standardization of items, and integrating the storage process with the demand planning with automation of the entire demand planning, purchasing, and contract management process (DEZOL; BARBOSA, 2016). The primary responsibility for this increase is implementing a centralized infrastructure of systems.

Some of the company's advances in this regard, such as implementing its bidding and contract regulation in June 2018 and acquiring an integrated system for managing contracts, infrastructure, finance, budget, and people, began operating in September 2020.

The company's intention with the implementation is to integrate processes and information, allowing an improvement in the planning and management of the company as a whole, as it is a company with several units spread across Brazil and some around the world. The expectation is that the system will be consolidated during 2021 and bring gains in planning demands and standardization from 2022.

5 Conclusion

Through the performance analysis of the public procurement process, it was possible to conclude that the assessment was considered good from the point of view of customers and process operators. However, it is necessary to implement some improvement proposals to leverage the performance and maturity level of the purchasing process.This research corroborates those of Lobão (2015) and Wanyonyi and Muturi (2015). They highlight the use of information technology and the competence of employees as strengths, positively affecting the performance of the purchasing function.

These research results and Fenili (2016) and Cabral and Reis (2018) are similar. The authors also identified the following: low quality of the purchased products; lack of customer feedback regarding the quality of items; the inadequate and incomplete description of the products; process length and cycle time; dysfunctional bureaucracy; the lack of a support structure for the purchasing area; delivery monitoring by the purchasing area and; selection of suppliers.

To improve the company's performance in the analyzed variables and leverage the maturity level of the purchasing process, the main change to be implemented is the centralized infrastructure of systems allowing the automation of the entire demand, purchasing, and planning process contract management (DEZOLT; BARBOSA, 2016).

This research provided contributions in the theoretical-scientific, methodological and managerial fields by analyzing the performance of public procurement and contracting processes. Variables from the performance evaluation of the supply chain were used; proposing improvements to the process, giving the manager a tool to increase their performance; and developing an instrument for evaluating the performance of the public procurement process used in other surveys with similar purposes.

As a suggestion for further research, it would be relevant to have a comparative study using the same instrument after a complete exercise using the new legislation and regulations. It would be interesting to use the developed instrument in other public administration bodies. New research in other sectors with the adaptation of the instrument; and carrying out qualitative research using the identified variables would also be compelling.

References

ALMEIDA, A. A. M.; SANO, H. Função Compras no Setor Público: Desafios para o Alcance da Celeridade dos Pregões Eletrônicos. Revista de Administração Pública, v. 52, n. 1, p. 89-106., 2018. Doi: http://dx.doi.org/10.1590/0034-7612164213.

ALONSO, J. G.; SANTACRUZ, M. P. Cálculo e interpretación del Alfa de Cronbach para el caso de validación de la consistencia interna de un cuestionario, con dos posibles escalas tipo Likert. Revista Publicando, v. 2, n. 1, p. 62-67, 2015. Disponível em: https://nbn-resolving.org/urn:nbn:de:0168-ssoar-423821. Acesso em: 16 mar. 2022.

AMBE, I. M. & BADENHORST-WEISS, J. A. Managing and Controlling Public Sector Supply Chains, Supply Chain Management, Dr. pengzhong Li (Ed.), ISBN: 978-953-307-184-8, InTech, 2011. Recuperado de: http://cdn.intechweb.org/pdfs/15532.pdf.

APOSTOLOVA, B.Z.; KROON, M.J.; RICHTER, M.; ZIMMER, I.M. Strategic Purchasing: A Global Perspective. Global Edition, Groningen, 2015.

ARAÚJO, S. S., MATOS, L. DOS S., & ENSSLIN, S. R. Compras Públicas sob a Perspectiva da Avaliação de Desempenho: uma Revisão de Literatura e Agenda de Pesquisa.Revista Gestão &Amp; Conexões, v. 9, n. 1, p. 99–127. 2020. DOI: https://doi.org/10.13071/regec.2317-5087.2020.9.1.27486.99-127.

ASSIS, S. A. F. Modernização da burocracia das compras públicas: sistema de registro de preços em uma Instituição Federal de Ensino Superior. 2015. 173 p. Dissertação (Mestrado Profissional em Administração Pública) - Universidade Federal de Lavras, Lavras, 2015. Recuperado de: http://repositorio.ufla.br/handle/1/9750

BEAMON, B. M. Measuring supply chain performance. International Journal of Operations & Production Management, v. 19, n. 3, p. 275-292, 1999. Doi: https://doi.org/10.1108/01443579910249714.

BRANDÃO, F. T. Eficiência na gestão das compras públicas: um estudo de caso da modalidade pregão eletrônico nos processos licitatórios do Instituto Federal de Educação, Ciência e Tecnologia do Sertão Pernambucano. 2016. 157f. Dissertação (Mestrado Profissional em Gestão de Processos Institucionais) - Centro de Ciências Humanas, Letras e Artes, Universidade Federal do Rio Grande do Norte, Natal, 2016. Recuperado de: https://repositorio.ufrn.br/jspui/handle/123456789/21930.

BRASIL. Lei nº 13.303. Diário Oficial da União, Brasília, Distrito Federal, Brasil, junho, 2016.

CABRAL, S.; REIS. P. R. C. Para além dos preços contratados: fatores determinantes da celeridade nas entregas de compras públicas eletrônicas. Revista De Administração Pública, v.52, n. 1, p. 107-125, Rio de Janeiro, Rio de Janeiro,Brasil, 2018. Doi: http://dx.doi.org/10.1590/0034-7612164442.

CARMO, H.; FERREIRA, M. M. Metodologia da investigação, guia para a autoaprendizagem. Universidade Aberta, Lisboa, Portugal, 2008.

CRESWELL, J. W.; CLARK, V. L. P. Pesquisa de métodos mistos. São Paulo: Penso, 2013.

CHRISTOPHER, M. Logística e gerenciamento da cadeia de suprimentos.São Paulo: Pioneira, 2007.

DEZOLT, A. L. P.; BARBOSA, G. P. IX Congresso CONSAD (Conselho Nacional de Secretários de Estado da Administração), Brasília, Distrito Federal, Brasil, junho, 2016. Recuperado de: http://consad.org.br/wp-content/uploads/2016/06/Painel-42-01.pdf.

DALMORO, M.; VIEIRA, K. M. Dilemas na construção de escalas Tipo Likert: o número de itens e a disposição influenciam nos resultados? Revista Gestão Organizacional, v. 6, n. 3, p. 161-174, 2013. Recuperado de: Documento :: SPELL – Scientific Periodicals Electronic Library

DESHPANDE, A. Supply Chain Management Dimensions, Supply Chain Performance and Organizational Performance: An Integrated Framework. International Journal of Business and Management, v. 7, n. 8, p. 2-19, Canadá, 2012. Doi: 10.5539/ijbm.v7n8p2.

DLAMINI, W.Determining procurement best practices in south african comprehensive universities. ( Dissertação de mestrado) University of South Africa, África do Sul, 2016. Recuperado de: https://core.ac.uk/download/pdf/43178003.pdf.

ENSSLIN, S. R.; ENSSLIN, L.; NISHIYAMA, M. A.; CHAVES, L. C.Compras governamentais sob a ótica da avaliação de desempenho: um mapeamento do tema conforme as delimitações postas pelos pesquisadores. Revista Gestão & Regionalidade, v. 30, n. 90, p. 32 – 49, set-dez, 2014. Doi: https://doi.org/10.13037/gr.vol30n90.2259

FENILI, R. R. Desempenho em processos de compras e contratações públicas: um estudo a partir da inovação e das práticas organizacionais. 2016. XXIII, 340 f., il. Tese (Doutorado em Administração) Universidade de Brasília, Brasília, 2016. Recuperado de: https://repositorio.unb.br/handle/10482/22451

FERREIRA, C. B. Gestão de compras públicas: desafios, dilemas e perspectivas na Universidade Federal de Viçosa. (Dissertação de mestrado) Universidade Federal de Juiz de Fora, Juiz de Fora, São Paulo, Brasil, 2015. Recuperado de: https://repositorio.ufjf.br/jspui/handle/ufjf/1626.

FERREIRA, M. A. M.; MEDINA, S. A.; REIS, A. O. Pregão Eletrônico e Eficiência nos Gastos Públicos Municipais. Administração Pública E Gestão Social, v. 6, n. 2, p. 74-81. 2014. Recuperado de https://periodicos.ufv.br/apgs/article/view/4599.

GERHARDT, T. E.; SILVEIRA, D. T. Métodos de pesquisa.Porto Alegre: Editora da UFRGS, 2009.

GUNASEKARAN, A.; PATEL, C.; TIRTIROGLU, E., Performance measures and metrics in a supply chain environment, International Journal of Operations & Production Management, v. 21, n. 1/2, p. 71-87, 2001. Doi: https://doi.org/10.1108/01443570110358468

KANYARU, G. M.; MORONGE, M. Determinants of public procurement legal framework on the performance of public institutions in Kenya. A case of judicial service commission. The Strategic Journal of Business & Change Management, v. 4, n. 3, p. 148-163, 2017. Recuperado de: http://strategicjournals.com/index.php/journal/article/view/499/526.

LACERDA, L. C. N. Uma análise da variabilidade dos preços adjudicados em pregões eletrônicos na administração pública federal (Dissertação de mestrado) Fundação Getúlio Vargas, Rio de Janeiro, Rio de Janeiro, Brasil, 2012. Recuperado de: https://bibliotecadigital.fgv.br/dspace/handle/10438/9931.

LEGATO, P.; MALIZIA, L.; MAZZA, R. N. Simulation-based performance measurement: assessing the purchasing process in a public university. 30th European Conference on Modelling and Simulation. 2016. Doi: 10.7148/2016-0033

LIMA JÚNIOR, F. R.; CARPINETTI, L. C. R. Modelos de decisão para avaliação de desempenho de cadeias de suprimentos baseados no Scor®: uma revisão da literatura. Brazilian Journal of Development, v.. 5, n. 7, p. 10301-10318, 2019. Brasil. Doi: https://doi.org/10.34117/bjdv5n7-186.

LOBÃO, J. C. S. A influência do contexto organizacional no desempenho das compras públicas na UFBA. (Dissertação de mestrado) Universidade Federal da Bahia, Salvador, Bahia, 2015. Recuperado e: https://repositorio.ufba.br/ri/handle/ri/18564.

MANHANI JUNIOR, M.; LIMA, M. R. Avanços e perspectivas nos processos de compra e contratação de serviços realizados pela secretaria de estado da administração e da previdência, por intermédio do departamento de administração de material. IX Congresso CONSAD (Conselho Nacional de Secretários de Estado da Administração), Brasília, Distrito Federal, Brasil, junho, 2016. Recuperado de: http://consad.org.br/wp-content/uploads/2016/06/Painel-41-03.pdf.

NISHIYAMA, M. A. Gestão de compras no setor público: Uma proposta multicritério para Avaliação de Desempenho. (Dissertação de mestrado) Universidade Federal de Santa Catarina,Florianópolis, Santa Catarina, Brasil, 2016. Recuperado de: https://repositorio.ufsc.br/handle/123456789/167763.

NISHIYAMA, M. A.; LIMA, M. V, A.; ENSSLIN, L.; CHAVES, L. Modelo multicritério para avaliação de desempenho: um estudo de caso para gestão de compras no setor público. Revista de ciências da administração, v. 19, n. 47, p. 9-28, Florianópolis, Santa Catarina, abril, 2017. Doi: https://doi.org/10.5007/2175-8077.2017v19n47p9

OLIVEIRA, M. C. Eficiência do calendário de compras para os processos de licitação: estudo de caso da Universidade de Brasília. 2015. 133 f., il. Dissertação (Mestrado Profissional em Economia) Universidade de Brasília, Brasília, 2015. Recuperado de: https://repositorio.unb.br/handle/10482/18974.

OLIVEIRA, R.C.R. Licitações e contratos administrativos, teoria e prática. 4. Ed. São Paulo: Método. 2015.

PEREIRA JUNIOR, J. T.; DOTTI, M. R. A Lei das Estatais contribui para simplificar e elevar a segurança jurídica de licitações e contratos? Revista TCU, v. 1, n. 141, p. 87-105, Janeiro/Abril, 2018. Recuperado de: https://revista.tcu.gov.br/ojs/index.php/RTCU/issue/view/85.

RAPOSO, M. H.; FREITAS, M. S.; SILVA FILHO, A. T.; FERNANDES, M. S. B.; SILVA, R. L. A importância do planejamento de compras para a gestão estratégica de suprimentos. IX Congresso CONSAD (Conselho Nacional de Secretários de Estado da Administração), Brasília, Distrito Federal, Brasil, junho, 2016. Recuperado de: http://consad.org.br/wp-content/uploads/2016/06/Painel-44-03.pdf.

SILVA, G. M. Modelo construtivista de avaliação de desempenho do processo de planejamento de compras no poder judiciário do estado de Santa Catarina. (Dissertação de mestrado) Universidade do Sul de Santa Catarina, Florianópolis, SC, Brasil, 2017. Recuperado de: http://www.riuni.unisul.br/handle/12345/3642.

SILVA.P. I. P., Regulamentos de licitações editados sob a lei 13.303/2016: instrumentos de inovação ou de repetição?(Dissertação de mestrado) Fundação Getúlio Vargas. Rio de Janeiro, Rio de Janeiro, Brasil, 2018. Recuperado de: http://bibliotecadigital.fgv.br/dspace/handle/10438/27262.

SQUEFF, F. H. S. O poder de compras governamental como instrumento de desenvolvimento tecnológico: análise do caso brasileiro.Texto para discussão. Instituto de Pesquisa Econômica Aplicada- IPEA, Rio de Janeiro, RJ, Brasil, 2014. Recuperado de: http://www.ipea.gov.br/portal/images/stories/PDFs/TDs/td_1922.pdf.

SRIVASTAVA, S. K.; AGRAHARI, A. Benchmarking approach to improve public procurement process, Economic and Political Weekly, v. 52, N. 20, p. 58 – 67, 2017. Recuperado de: https://ssrn.com/abstract=3451808.

STEWART, G. Supply chain performance benchmarking study reveals keys to supply chain excellence. Logistics Information Management, v. 8, n. 2, p. 38 – 44, 1995. Doi: https://doi.org/10.1108/09576059510085000.

TROJAN, R. M.; SIPRAKI, R. Perspectivas de estudos comparados a partir da aplicação da escala Likert de 4 pontos um estudo metodológico da pesquisa TALIS. Revista Ibero-Americana de Estudos em Educação, V. 10, N. 2, p. 275-300, 2015. DOI: https://doi.org/10.21723/riaee.v10i2.7761

VAY, I. H. Avaliação de desempenho logístico no serviço ao cliente baseada na Teoria da resposta ao item (Tese de Doutorado) Universidade Federal de Santa Catarina, Florianópolis, Santa Catarina, Brasil, 2011. Recuperado de: https://repositorio.ufsc.br/xmlui/handle/123456789/94947.

WANYONYI, S. C.; MUTURI, W. Factors affecting performance of procurement function among public technical training institutions in kisumu county, kenya. International Journal of Economics, Commerce and Management, v. III. N. 5, p. 325-337, 2015. Recuperado de: ijecm.co.uk/wp-content/uploads/2015/05/3520.pdf.

ZYLBERMAN, M. A gestão das compras públicas dos estados brasileiros: a experiência do Rio de Janeiro com a opção pela descentralização.(Dissertação de Mestrado) Fundação Getúlio Vargas, São Paulo, São Paulo, Brasil, 2015. Recuperado de: https://bibliotecadigital.fgv.br/dspace/handle/10438/13712.

ZYMLER, B. Considerações sobre o estatuto jurídico das empresas estatais (Lei 13.303/2016). Interesse Público – IP, v. 19, n. 102, p. 15-26, Belo Horizonte, Minas Gerais, 2017. Recuperado de: https://www.editoraforum.com.br/wp-content/uploads/2017/05/estatuto-empresas-estatais.pdf.

IR