1 Introduction

The use of GNSS technology has caused a revolution in navigation and positioning activities. Nowadays, geodetic and topographic studies are performed faster, more precisely and at lower cost. As the positioning techniques have evolved, several real-time and post-processing applications have emerged, greatly expanding the scope of RBMC. making the scope of RBMC increasingly broad.

The Global Navigation Satellite System (GNSS) encompasses the United States Global Positioning System (GPS), Russia Glonass System, the European Union Galileo System and China Beidou or Compass System (Monico 2007).

In operation since 1996, the Brazilian Network for Continuous Global Navigation Satellite System Monitoring (RBMC) was the first network of its kind implemented in South America (IBGE 2017). The implementation, maintenance and operation of the network are conducted by the Brazilian Institute of Geography and Statistics (IBGE), with the support of dozens of other public institutions in the country. Each station has a receiver and geodetic antenna, Internet connection and constant supply of electricity that allows the continuous operation of the station.

There are RBMC stations that received meteorological sensors that record temperature, pressure and relative humidity every minute, information that is published together with the observation and navigation data.

RBMC contributes more effectively to national and global climate studies, georeferencing of rural properties, energy generation and transmission, road, air and sea fleet control, among others.

In the geodetic and topographic applications of GNSS, the use of the relative method is implicit, meaning that at least one known coordinate station is also occupied simultaneously with the occupation of the desired points. The RBMC stations have the role of establishing the known coordinate points belonging to the Brazilian Geodetic System (SGB), eliminating the need for user to immobilize receivers at points that often are very hard to access. In addition, the receivers that equip the RBMC stations are high performance, providing high quality observations and reliability.

In this regard, the advantage of the RBMC is that all its stations are part of the SIRGAS Reference Network (Geocentric Reference System for the Americas), whose final coordinates are accurate to the order of ± 5 mm (IBGE 2017), forming one of the most accurate networks in the world. Another important role of the RBMC is that since 1997, its observations have contributed to the regional densification of the International GPS Service for Geodynamics (IGS) network, ensuring better precision of the IGS products - such as precise orbits - over the Brazilian territory.

The Complex Holographic Assessment of Paradoxical Problems (CHAP²) is a systemic problem-solving method, based on metacognitive maps, which helps structuring (not solve) high-complexity problems and involves interactions of human, technological, organizational and environmental components (Lins &Antoun Netto 2018). This method allows rapid investigation of the problem, since it is able to represent the associated complexity, articulating a multi-methodological process.

According Estellita Lins et al. (2021), the multimethodology CHAP2 is based on the principles of systems science (Mobus & Kalton 2015), social complexity theory (Castellani & Hafferty 2010), systemic thinking (Midgley 2000; Mingers 2006), positive psychology (Suchman, Sluyter & Williamson 2011), ToM (Malle & Hodges 2005), multiplicity (Behars 1985; Watkins & Watkins 1997), metacognition (Tarricone 2011; Briñol & DeMarre 2012), and concept mapping (Hyerle 2009), the latter is a tool applied to the management of social problems in several soft systems methodologies-SSM (Rosenhead & Mingers 2001). Pereira et al. (2014) present a literature review on SSM.

Data envelopment analysis (DEA) is applied in the context of conceptual mapping, thus shedding light on both quantitative and qualitative factors that influence RBMC performance. Our aim is to propose a methodology for performance indicators to support implementation of new RBMC stations, with regard to the equipment used, location, existing infrastructure, etc., using quantitative indicators. Quantitative results allow inefficient RBMC station manager to understand the causes of their overall efficiency in terms of particular low partial DEA efficiencies.

This work intends to contribute to increasing RBMC performance. With this purpose, it makes use of the interface of qualitative management sciences and quantitative methods.

Three golden rules for the design of a set of indicators, according to Franceschini et al. (2008) are:

• Understand the system defined by the immediate and the contextual processes.

• Identify stakeholders: clients, servers, and other owners affected by the decisions.

• Assess both efficacy, efficiency and effectiveness.

This work partially fulfill the first and second rules, as it develops maps for the system and identify factors connected to agents' activities that should be considered to compose the performance indices.

Besides providing technical support and engagement of managers, effective improvements in RBMC management require cultural changes, strongly conditioned by world view paradigms.

According to Estellita Lins et al. (2021) and Estellita Lins, Antoun Netto and de Castro Lobo (2019) the integration of quantitative indicators with qualitative descriptions of context is a noticeable demand from many different scientific disciplines since it contributes to linking theoretical and practical approaches to problem solving. This work presents a mixed quantitative and qualitative methodological approach to aid formulation and structuring of performance measurement of the Brazilian Network for Continuous Global Navigation Satellite System Monitoring (RBMC).

This article is divided into three more sections. Section 2 presents basics concepts about CHAP² Method, Concept Map, Data Envelopment Analysis (DEA) and Quantitative Analysis through the DEAFrontier Software. Section 3 outlines the case study to disclose tools for the formulation and structuring of RBMC network and the results obtained with the implementation of CHAP2, Concept Maps and Data Envelopment Analysis (DEA). Finally, Section 4 presents our concluding remarks.

2 Methodology

This section presents basics concepts about CHAP², Concept Map, Data Envelopment Analysis (DEA) and Quantitative Analysis through the DEAFrontier Software.

Reference to Quantitative Analysis through the DEAFrontier Software appears three times in the text. I suggest dropping two of them.

If it is necessary to provide here a review of the foundational literature regarding multi methodology and problem structuring problems, like those of Mingers, Midgley and Rosenhead.

2.1 Complex Holographic Assessment of Paradoxical Problems (CHAP²)

The CHAP2 was developed at the PSIGMA Lab at the UFRJ as a multimethodology that uses concept maps to increase the degree of awareness to the emergence of both coherent and conflicting issues resulting from the complexity of living systems. The greatest motivation for the development of the method is the deadlock and the lack of alternatives to deal with problems in public policies, in the sense that the use of common resources should result in guaranteed benefits for the several social agents (Estellita Lins et al. 2021)

CHAP2 emphasizes and makes explicit the role of metacognition, as self-image modulates every subjective political issue, with the identification of the different views of the analyzed problem, extending the concept of concept maps to metacognitive maps (Lins & Antoun Netto 2018).

The CHAP² method requires the articulation of several recent developments (Lins et al. 2021; Lins &Antoun Netto 2018), namely:

• Theory of Mind as a set of important knowledge elements to understand different human behaviors, such as the psychology of personalities, the expression of emotions and organizational cultures.

• Systemic thought, successor of cybernetics and the General Theory of Systems, which reveals imperceptible properties and phenomena under the dominant organizational theory.

• The theory of social complexity, which in particular allows identification of existential paradoxes and polarities, which need to be managed.

• Problem-based methods, which were born in UK operational research focused on the problem-modeling phase, and allow for contextual validation and commitment to concrete results.

• The representation of perspectives in relation to problems, which can be structured through conceptual/cognitive maps, and allows the integration of quantitative and qualitative aspects.

• Theory about multiplicity and dialogical self, as well as metacognition versus self-deception, both in line with the expression of structural paradoxes.

The method covers two instances: the perception of the problem and the intervention in the problem, taking into account the localized external and distributed internal regulation, seeking to integrate the perspectives of the analysts and the agents involved and integrate qualitative and quantitative approaches.

External regulation is essential in centralized and legal decision making, requiring the establishment of quantitative indicators. However, internal regulation is essential to the perception of complex real-world problems.

The CHAP² model approach comprises six steps, indicated in the Figure 1 below.

Figure 1
Steps of the CHAP² (adapted from Lins & Antoun Netto 2018).

The phase 1 from de CHAP² model had the objective of characterizing the real system and relevant agents, that is, describing the problem in a generalized way and identifying the possible relevant agents to be interviewed in Phase 3. A single map was developed at this stage.

Phase 2 from de CHAP² model consisted of seminars, evaluations and metacognitive dynamics considering that it is important for the facilitators (those who are working with the method) to express the principle of holographic representation in the conduct of the work with the focus group. The presentation of the method for the agents interviewed in phases 1 and 3 was done in person, as well as the validation of the maps with the agents, where it was possible to identify corrections to be applied.

In phase 3 from de CHAP² model, the perception patterns of the actors identified to deal with the diversity of agents involved was characterized. At this stage, the relevant agents for the characterization of the problem were interviewed. Two agents were used in this phase:

• An individual who works directly with the study at the Brazilian Institute of Geography and Statistics (IBGE);

• A user of RBMC stations.

The identification of different patterns of perception in phase 3 supported the creation of conceptual and paradoxical models enabling the elaboration of a workshop, the main objective of phase 4.

In the phases 5 and 6, we validated the formal modeling using interviews with qualified engineering professionals, researches and the manager of Geodesy Division of Brazilian Institute of Geography and Statistics (IBGE) to determine quantitative variables that could be used in Data Envelopment Analysis (DEA) model.

Insert the explanation for phase four, even if it was not carried out as prescribed. Also insert phase five, which consists of the data envelopment analysis methodology used. Insert phase six, focusing on the potential use by IBGE and contacts with potential users, demonstrating efforts towards effective implementation of the methodology.

2.2 Concept Mapping

Concept mapping is important to make explicit when one is seeking proposals to deal with complex problems and not just solving a simplified part of a problem from a particular perspective (Rosenhead & Mingers 2001). In this former approach, the structuring of matters, issues and situations are one of the stages of the modelling at the very beginning of the decision-making process. In line with Okada, Buckingham and Sherbone (2014), concept maps are graphical tools used for knowledge representation, so that two concepts can be connected via a link phrase, generating a proposition. Novak (1998) presents concept mapping as a useful tool for student learning at different levels, as well as to assist in troubleshooting in organizations. A graphical representation can be more effective than a text to communicate complex content because the mental processing of images can be less cognitively demanding than the verbal processing of a text (Vekiri 2002). Guidelines on the formulation of concept maps can be found in Ruiz-Moreno et al. (2007).

2.3 Data Envelopment Analysis (DEA)

Data Envelopment Analysis (DEA), which was proposed by Charnes, Cooper and Rhodes (1978), is a mathematical tool for measuring the efficiency of homogeneous production units called DMU (Decision Making Unit), with the objective is to compare a number of DMUs that perform similar tasks and differ in the quantities of the resources consumed and the outputs produced.

DEA has been used in the calculation of performance indicators and to establish benchmarks for supporting regulation of public sectors. The method lends itself to use in multidisciplinary issues and complex systems, to incorporate the opinion of specialists, in line with multi-criteria methodologies.

Scientific methods for measurement and follow-up of regulated firms, in particular DEA, have been developed and are already being used by regulatory agencies in several countries. Sherman and Zhu (2006) proposed expanding applications of DEA in the various sectors of the economy,

Homogeneous productive units are called DMUs (Decision Making Units) and must be comparable and operate subject to the same conditions, considering either contextual factors or scale of operation (Wu, Liang & Song 2010). As to the contextual qualitative factors, they can be identified through concept mapping and used in cluster analysis whenever data are available (Antoun Netto 2012). However, here we do not present an extensive exposition of the mapping and clustering techniques, but rather focus on their interfaces with DEA modelling, as far as variable selection and homogeneous clusters are concerned.

Two models are considered classic in DEA: the CCR (Charnes, Cooper and Rhodes) and the BCC (Banker, Charnes and Cooper). The first, also known as the CRS model (Constant Returns to Scale), was originally presented by Charnes, Cooper and Rhodes (1978) and works with constant returns to scale, that is, any variation in inputs produces proportional variation in outputs. The second, also known as VRS (Variable Returns to Scale), was introduced by Banker, Charnes and Cooper (1984), considers returns scale variables and replaces the axiom of proportionality between input and output by the axiom of convexity (Antoun Netto 2012).

There are 2 (two) formulations used in the DEA Models, namely: Envelopment, which defines a viable production region and works with a projection of each DMU on the border of this region; and Multipliers, which uses the weighted sum ratio of products and resources. Regarding the orientation, the efficient frontier is sought, minimizing the inputs and keeping the level of observed outputs constant; or maximizing the outputs and keeping the level of observed inputs constant convexity (Antoun Netto 2012).

2.4 Quantitative Analysis

The DEAFrontier software uses Excel Solver as a mechanism to solve Data Envelopment Analysis models. It is important to report that numeric weight restrictions were not introduced in the models. In determining the RBMC’s goals and indicators, each station is represented as a decision making unit (DMU) with autonomy. In addition to choosing variables and DMUs, DEA modeling requires the choice of the model, either in relation to orientation or return to scale. In this case study, we used the DEA VRS model with oriented output.

While classical models assume that inputs must be minimized and outputs maximized, the production process can generate undesirable outputs that must be minimized. One of the most commonly used approaches to deal with this problem is the multiplicative inverse transformation (MLT), which was proposed by Golany and Roll (1989), and consists of calculating the inverse value of each unwanted output, transforming it into a desired output. Another approach was adopted, which considers the unwanted output as input to the DEA model.

Data Envelopment Analysis optimizes each individual observation in order to calculate an efficiency frontier boundary, determined by units that are Pareto efficient. The Pareto efficiency concept seeks a set of units not comparable to each other, which are extreme-efficient and better than the others in at least one aspect or dimension. That is, they focus on characterizing the efficient faces of the problem solving space (Stewart 1996).

Once the variables have been properly identified, a report containing the ranking in terms of efficiency and the benchmarks is generated. Then all that is needed is to correct assessment of the report to obtain the quantitative analysis of the problem.

3 Results and Discussion

This section presents and comments on the results of the evaluations of Conceptual Map and Data Envelopment Analysis (DEA).

3.1 Conceptual Map

The conceptual map was the final product of the qualitative analysis, being used to identify the variables of the problem under study and possible suggestion of feasible solutions, the aim of the fifth and sixth phase of the method. The paradoxical map was not elaborated, as there was no disagreement among the respondents. The concept maps depicted relevant variables to apply for the quantitative analysis using DEA.

This work used a computational tool called CmapTools, version 4.11, developed by the Institute for Human and Machine Cognition (IHMC) of the University of West Florida, which allows users to construct, navigate, share and criticize knowledge models represented as concept maps.

The resulting formulation and structuring of Brazilian Network for Continuous Global Navigation Satellite System Monitoring (RBMC) is consolidated in the concept maps shown in Figures 2 through 4. Note that the original concept map is divided into 3 (three) parts in order to permit visualization and interpretation of the concepts, linking phrases and existing connectors.

The concepts with a gray background correspond to quantitative variables and also suggest using a management tool that can be used in hard operations research methods, such as data envelopment analysis (DEA).

Conceptual Map area called referenced with the letter “C” in Figure 2 shows that one of the variables identified in this problem was “days of operation”, which is the number of days that the station was working perfectly in the year.

Conceptual Map area referenced with the letter “B” in Figure 2 shows that another variable identified was the multipath effect, which is the error that results in the GPS signal that reaches the receiver through more than one path (such as when its location is near buildings or other elevations, metallic structures or water surfaces). This effect appears because the satellite signal does not always travel directly to the antenna. It can hit some object before and be reflected to the antenna, creating a false measurement. The effects can be estimated using a combination of frequency f1 and f2, code and carrier. It can usually be computed by L1 (MP1) and L2 (MP2).

The last variable identified was the quantity of free electrons in Conceptual Map area referenced with the letter “A” in Figure 3, caused by solar explosions, present in the

ionosphere, which cause a delay in the GNSS signal. A correction is made for the effect of the ionosphere on this captured signal and therefore this variable does not appear in

the reports of the stations and was not considered for the quantitative analysis of this work.

Figure 2
Conceptual map for RBMC (Part 1).

Figure 3
Conceptual map for RBMC (Part 2).

Figure 4
Conceptual map for RBMC (Part 3).

Table 1 containing the identified variables of the problem that were used to identify them.

Table 1

Identified variables.

3.2 Data Envelopment Analysis (DEA)

Although classic DEA models assume that inputs must be minimized and outputs maximized, the production process can also generate undesirable results (outputs) that should be minimized. This is the case for the multipath 1 (MP1) and multipath 2 (MP2). One of the most commonly used approaches to deal with this problem is the multiplicative inverse transformation (MIT), which was proposed by Golany and Roll (1989), and consists of calculating the inverse value of each unwanted output, turning it into a desired output in radial VRS (Lins et al. (2011) and Seiford and Zhu (2002)) and SBM (Slacks-Based Measure) DEA models which was introduced by Tone (2001) put aside the assumption of proportionate changes in inputs and outputs, and deal with slacks directly. Scheel (2001) discusses various transformations of undesirable output data. The first indirect approach to incorporating the undesirable outputs in the model is by transforming them by the additive inverse method, using values f(Q) = −Q. This approach was suggested by Koopmans (1951). The technology set defined by both approaches are identical.

Explanation regarding multiplicative inverse transformation is duplicated. I suggest dropping the second one.

We used the MIT approach, assuming multipath 1 (MP1) and multipath 2 (MP2), an undesired output. The database used in the DEA model was the information system of the RBMC for the year 2016. The variables used were operating days, multipath 1 (MP1) and multipath 2 (MP2). All entered as outputs in the software used. The variables MP1 and MP2 entered as undesirable outputs and therefore the values 1/MP1 and 1/MP2 were used.

The software generated a report that can be seen in the Figure 5 below. Implementation of the DEA model was performed through the DEAFrontier software, developed by Professor Joe Zhu of Worcester Polytechnic Institute. In determining targets and indicators of RBMC station, each station is represented as a DMU (decision making unit) endowed with autonomy.

Figure 5
DEAFrontier software Report.

In the generated report, some DMUs, although the calculated efficiency resulted in the value 1, are in the Pareto inefficient region. They would only be efficient if the benchmark was itself efficient (see the DMUs COAM, MTVB, SPS1 and UFPR in Figure 5). In this way, efficiencies were recalculated using the output-oriented Russell measure, where the new values replaced those "false results".

One of the first non-radial efficiency measures, Russell's measure (as proposed by Färe & Lovell 1978), minimizes the unweighted arithmetic mean of the proportional reduction of all individual inputs while maintaining the level of outputs constant. This measure was later generalized to the non-oriented case in which inputs and outputs can change the productivity frontier. The Russell efficiency graph (Färe, Grosskopf & Lovell 1985) with reference to a general technology T: (ɵio, and βro are, respectively, the input and output values).

In Equation 1, when βro is not considered, the above function reduces to Σɵio/m, and this is the input to Russell's measure of efficiency as first proposed by Färe and Lovell (1978). The Russell measure can therefore be defined by an input or output orientation or no orientation (as above). In any case, the outcome of the efficiency measure reflects all sources of inefficiency.

Afterward, the inverses of the efficiencies were calculated and the new values were placed in descending order to obtain the final ranking, as can be seen in the Table 2 below.

Table 2

Final DEA Efficiency.

A Geodesy Division of Brazilian Institute of Geography and Statistics helped to make an evaluation for the first and the last ten stations from the generated ranking to learn the possible conditions explaining the presented order.

Some stations had a worse result because they were down for a while, but that does not mean the data or the Internet connection to those stations was bad.

The RIOD station is on the list of the 10 best ones, because it is located inside the IBGE’s premises.

Most of the stations that were ranked as the best ones are at universities, like UFPR, PPTE MGRP, SMAR, GOUR, SPFR, and RJCG. This is an advantage for the universities that collaborate to operate the stations.

The 10 worst stations are located mostly in the Amazon and Midwest regions where the most serious problems with Internet, electric discharge and power outage exist. MSCO, for example, was disabled in 2016, explaining its poor evaluation.

4 Conclusion

This work illustrates the use of CHAP² to establish a qualitative context that can support the design and structuring of RBMC. Qualitative modeling is performed through concept mapping, which can support variable selection used in Data Envelopment Analysis (DEA).

This work also testifies the multidisciplinary approach of Engineering, that combines Geodesy and Operational Research to evaluate the Brazilian Network for Continuous Global Navigation Satellite System (GNSS) Monitoring (RBMC).

We intend to extend the method to include other Geomatics applications and their use in engineering. Finally, it is important to inform that the qualitative and quantitative results will be further validated by qualified engineering professionals, regarding the potential for improvements and the observation of the recommendations of Brazilian Network for Continuous Global Navigation Satellite System (GNSS) Monitoring (RBMC).

5 References

Antoun Netto, S.O. 2012, 'O uso de Multimetodologia para a determinação de Metas e Indicadores de Desenvolvimento Municipal na Área da Saúde'. Tese (Doutorado), Universidade Federal do Rio de Janeiro.

Banker, R.D., Charnes, A. & Cooper, W.W. 1984, 'Some models for estimating technical and scale inefficiencies in Data Envelopment Analysis', Management Science, vol. 30, no. 9, pp. 1031-1142, DOI:10.1287/mnsc.30.9.1078.

Behars, J.O. 1985, Unity and Multiplicity: Multilevel Consciousness of Self in Hypnosis, Psychiatric Disorder and Mental Health, Bruner Meisel U, New York.

Briñol, P. & DeMarree, K.G. (eds). 2012, Social Metacognition, Psychology Press, East Sussex.

Castellani, B. & Hafferty, F. 2010, Sociology and Complexity Science, Springer, Berlin.

Charnes, A., Cooper, W.W. & Rhodes, E. 1978, 'Measuring efficiency of decision-making units', European Journal of Operational Research, vol. 2, no. 6, pp. 429-444, DOI:10.1016/0377-2217(78)90138-8.

Estellita Lins, M.P., Antoun Netto, S.O. & de Castro Lobo, M.S. 2019, 'Multimethodology applied to the evaluation of Healthcare in Brazilian municipalities', Health Care Management Science, vol. 22, pp. 197-214, DOI;10.1007/s10729-018-9432-z.

Estellita Lins, M.P., Pamplona, L., Estellita Lins, A.B. & Lyra, K.F. 2021, 'Metacognitive attitude for decision-making at a University Hospital', International Transactions in Operational Research, vol. 1, pp. 1-21, DOI:10.1111/itor.12975.

Färe, R., Grosskopf, S. & Lovell, C.A.K. 1985, The Measurement of Efficiency of Production, Springer, Dordrecht.

Färe, R. & Lovell, C.A.K. 1978, 'Measuring the Technical Efficiency of Production', Journal of Economic Theory, vol. 19, no. 1, pp. 150-162, DOI:10.1016/0022-0531(78)90060-1.

Franceschini, F., Galetto, M., Maisano, D. & Mastrogiacomo, L. 2008, Properties of performance indicators in operations management - a reference framework. International Journal of Productivity and Performance Management, vol. 57, no. 2, pp. 137-155, DOI:10.1108/17410400810847401.

Golany, B. & Roll, Y. 1989, 'An Application Procedure for DEA, OMEGA', International Journal of Management Science, vol. 17, no. 3, pp. 237-250.

Hyerle, D. 2009, Visual Tools for transforming Information into Knowledge, Corwin Press, Thousand Oaks.

IBGE - Instituto Brasileiro de Geografia e Estatística 2017, viewed 06 January 2017, <http://www.ibge.gov.br/>.

Koopmans, T.C. 1951, 'Analysis of production as an efficient combination of activities', in T.C. Koopmans (ed.), Activity analysis of production and allocation, Cowles commission. Wiley, New York, pp. 33-97.

Lins, Marcos Estellita; Oliveira, Luciano Basto; da Silva, Angela Cristina Moreira; Rosa, Luiz Pinguelli; Pereira, Amaro Olímpio 2011. Performance assessment of Alternative Energy Resources in Brazilian power sector using Data Envelopment Analysis. Renewable & Sustainable Energy Reviews, v. 16, p. 898-903.

Lins, M.P.E. & Antoun Netto, S.O. 2018, Estruturação de Problemas Sociais Complexos: Teoria da Mente, Mapas Metacognitivos e Modelos de Apoio à Decisão, Editora Interciência Rio de Janeiro, Rio de Janeiro.

Malle, B.F. & Hodges, S.D., 2005, Other Minds, How Humans Bridge the Divide Between Self and Others, Guilford, New York.

Midgley, G. 2000, Systemic Intervention, Philosophy, Methodology and Practice, Kluwer Academic, Dordrecht.

Mingers, J. 2006, Realising Systems Thinking: Knowledge and Action in Management Science, Springer, Berlin .

Mobus, J.E. & Kalton, M.C. 2015, Principles of Systems Science, Springer, Berlin .

Monico, J.F.G. 2007, Posicionamento pelo GNSS - Descrição, Fundamentos e Aplicações, Editora Unesp, São Paulo.

Novak, J.D. 1998, Learning, creating, and using knowledge: concept maps as facilitative tools in schools and corporations, Lawrence Erbaum Associates, Mahwah.

Okada, A., Buckingham, S. & Sherborne, T. 2014, Knowledge Cartography: software tools and mapping techniques, 2nd edn, Springer-Verlag, London.

Pereira, T.F., Montevechi, J.A.B., Miranda, R.C. & Friend, J.D. 2014, 'Integrating soft systems methodology to aid simulation conceptual modeling', International Transactions in Operational Research, vol. 22, no. 2, pp. 265-285, DOI:10.1111/itor.12133.

Rosenhead, J. & Mingers, J. 2001, Rational Analysis for a Problematic World Revisited: Problem Structuring Methods for Complexity, Uncertainty and Conflict, 2nd edn, Wiley, Chichester.

Ruiz-Moreno, L., Sonzogno, M.C., Batista, S.H.S.E. & Batista, N.A. 2007, 'Mapa conceitual: ensaiando critérios de análise', Ciência Educação, vol. 13, no. 3, pp. 453-463, DOI:10.1590/S1516-73132007000300012.

Scheel, H. 2001, 'Undesirable outputs in efficiency valuations', European Journal of Operational Research, vol. 132, no. 2, pp. 400-410, DOI:10.1016/S0377-2217(00)00160-0.

Seiford, L.M. & Zhu, J. 2002 'Modeling undesirable factors in efficiency evaluation', European Journal of Operational Research, vol. 142, no. 1, pp. 16-20, DOI:10.1016/S0377-2217(01)00293-4.

Sherman, H.D. & Zhu, J. 2006, Service productivity management: improving service performance using data envelopment Analysis (DEA), Springer, Boston.

Stewart, T.J. 1996, 'Relationships Between Data Envelopment Analysis and Multicriteria Decision Analysis', Journal of the Operations Research Society, vol. 47, no. 5, pp. 654-665, DOI:10.2307/3010016.

Suchman, A.L., Sluyter, D.J. & Williamson, P.R. 2011, Leading Change in Healthcare-Transforming Organizations Using Complexity, Positive Psychology and Relationship-Centered Care, Radcliffe Publishing, Abingdon.

Tarricone, P. 2011, The Taxonomy of Metacognition, Psychology Press, East Sussex .

Tone, K. 2001, 'A slacks-based measure of efficiency in data envelopment analysis', European Journal of Operational Research, vol. 130, no. 3, pp. 498-509, DOI:10.1016/S0377-2217(99)00407-5.

Vekiri, I. 2002, 'What is the value of graphical displays in learning?' Educational Psychology Review, vol. 14, no. 3, pp. 261-312, DOI:10.1023/A:1016064429161.

Watkins, J. & Watkins, H. 1997, Ego States: Theory and Therapy, WW Norton and Company, New York.

Wu, J., Liang, L. & Song, M. 2010, 'Performance based clustering for benchmarking of container ports: an application of DEA and cluster analysis technique', International Journal of Computational Intelligence Systems, vol. 3, no. 6, pp. 709-722.

Notes

Author notes

E-mail:sergioantoun@gmail.comE-mail:bruna.quintan@hotmail.comE-mail:andregeffer@hotmail.com