Influence of the directives of the wide field of applied social sciences guidelines on Qualis scientific Management journals

Influência das diretrizes do amplo campo de ciências sociais aplicadas no Qualis dos periódicos científicos em Administração

The economic, social and political development of a country is mostly propelled by research and development by society, especially Higher Education Institutions (HEI), through their stricto sensu post-graduation programs. In the specific case of Brazil, although this does not represent all Brazilian intellectual production, these programs are responsible for most of the country’s research and studies (Marchlewski; Silva; Soriano, 2011; Hutz et al., 2010).

In Brazil, intellectual production is mainly evaluated by the Qualis system, which is used to classify the scientific production of post-graduation programs when articles are published in scientific journals (CAPES, 2016a). Lucena and Tibúrcio (2009) explain that the Qualis system is the main parameter used by funding agencies and the CAPES to evaluate the scientific production of academic researchers.

This system is necessary to ensure quality in scientific development and guarantee the reliability and relevance of what is produced and published (Costa; Yamamoto, 2008). It should be highlighted that the process was created to meet the specific needs of the evaluation system and is based on the information provided by the stricto sensu post-graduation programs through the data collection application, known as Sucupira system.

As a result, a list called Qualis is published classifying and raking journals according to their relevance in the different fields that make up the evaluation system. This list of journals is used by post-graduation programs to divulge their production (CAPES, 2016a).

The wide field of applied social sciences is made up of: (1) management, accountancy and tourism; (2) architecture, urbanism and design; (3) applied social sciences; (4) law, (5) economics; (6) urban and regional planning/demography; and (7) social services. It is also subject to the evaluation criteria for intellectual production. All of these fields have their own criteria for classifying journals according to their specific nature.

However, it would be natural to expect similar evaluation levels or criteria since these sciences belong to the same wide field of knowledge. We can draw comparisons between the fields to gauge their degree of maturity in scientific production and determine how these fields are reflected in their respective areas in the Qualis system. In this context, we introduce the research question: What is the composition of the Qualis system in the wide field of applied social sciences and its influence on the Qualis Journals on management?

To answer this question, we intend to study the fields that constitute the wide-ranging applied social sciences to identify their degrees of maturity as reflected in it ranking from the Qualis Periodical system. We also intend to determine whether the field of management, accountancy and tourism is influenced by the directives issued for the wider field. Furthermore, it will be possible to determine whether the fields are freed to set their own criteria without interference from the others or the wider field of applied social sciences.

To discuss the importance of intellectual production, we first address the evaluation of post-graduation programs (PGPs), as the research and teaching activities conducted by these programs are essential for the evolution of society. They must have a socially recognized function and be legitimized legally and bureaucratically with an evaluation by the CAPES (Mello; Crubellate; Rossoni, 2010).

The evaluation of the National Post-graduation System (SNPG) conducted by the CAPES determines the classification of Brazilian academic production. It also aids teacher training at all level to ensure human resources are adequately prepared for the work market in general and strengthens the entire scientific, technological and innovative base. The evaluation system is the main tool for judging the entry and continuation of PGPs in the Brazilian system, whether they offer professional master’s degrees, academic master’s degrees or doctorate degrees (CAPES, 2014a).

This evaluation is performed by ad hoc consultants from the academic and scientific community under the guidance of the CAPES Evaluation directorate. The aim is to: a) certify the quality of Brazilian PGPs, which is used as a reference for allocating research resources and b) provide guidance for the creation and expansion of PGPs by identifying regional differences and differences in the strategic fields of science in the SNPG (CAPES, 2014a).

The evaluation processes are conducted according to each of the 48 fields of knowledge, with their own documents, in which the elements used to record the quadrennial evaluation are registered, their current status, their characteristics and perspectives and the issues that are considered priorities in the evaluation of PGPs (CAPES, 2014a).

It should be emphasized that the evaluation process is dependent on the recognition, reliability and assured quality of peer analysis (ad hoc referees), using criteria that have been debated and updated by the academic and scientific community. Another important aspect of this process is transparency in the publication of decisions, ractions and results in accordance with the basic requirements established by the Scientific and Technical Council of Higher Education (CTC-ES) (CAPES, 2014a).

The quadrennial evaluation results in a scale of grades varying from 01 to 07 awarded to the PGPs. A grade of 01 to 02 means that the program does not meet the minimum requirements. These programs will not be recognized or authorized and their master’s degrees and/or doctorate degrees will be cancelled. These programs will not be allowed to open new classes until they have achieved a minimum grade of three.

A grade of three is awarded to programs with minimum (regular) quality in their quadrennial evaluations. Thus, 03 is the minimum grade required to maintain a PGP with CAPES approval. A grade of 04 means good performance, while a 05 is considered very good, having achieved a level of national excellence (Maccari, 2008).

It should be highlighted that 05 is the highest possible grade for programs offering only a master’s degree. Finally, a grade of 06 or 07 is awarded to programs whose performance is comparable to the finest international research centers (CAPES, 2015b).

2.1 Evaluation of applied social sciences

To facilitate evaluation, the 48 fields of knowledge are subdivided and aggregated for their affinity at two levels. The first level is called Colleges, and there are three of them: Colleges of Life Science (encompassing the broad fields of Agrarian Sciences, Biological Sciences and Health Sciences); College of Exact, Technological and Multidisciplinary Sciences (with the broad fields of Exact and Earth Sciences, Engineering and Multidisciplinary Sciences); and the College of Humanities (with the fields of Human Sciences, Applied Social Sciences, Linguistics, Languages and Art) (CAPES, 2014b).

The present study focuses on the Applied Social Sciences in the College of Humanities. The purpose is to conduct a more in-depth investigation of how evaluations are conducted, mainly by analyzing the composition of the Qualis Periodical in each of the fields in question. This directly affects the evaluation of the intellectual production involved, which is the criterion used in the decision to award the grade for the CAPES evaluation (CAPES, 2014a; Maccari; Nishimura, 2014).

The present study focuses on the Applied Social Sciences in the College of Humanities. The purpose is to conduct a more in-depth investigation of how evaluations are conducted, mainly by analyzing the composition of the Qualis Periodical in each of the fields in question. This directly affects the evaluation of the intellectual production involved, which is the criterion used in the decision to award the grade for the CAPES evaluation (CAPES, 2014a; Maccari; Nishimura, 2014).

Table 1 shows a summary of the number of PGPs in all the fields of Applied Social Sciences.

Table 1
Numbers of PGPs per field

Source: based on CAPES (2016).Key: ME – Academic Master DO – Doctorate MF – Professional Master ME/DO – Master/Doctorate

The above table was based on a consultation of the CAPES Sucupira Platform in August 2016. It shows the distribution of Academic Master (ME), Doctorate (DO) and Professional Master (MF) programs, and programs offering a Master and Doctorate (ME/DO) simultaneously. When analyzing this table, we observed the magnitude of the Management, Accounting and Tourism field in the set of programs in the wider field of Applied Social Sciences. In this sense, the field represents 32.57% (184/565) of the total number of programs, as follows: 25% of the Academic Master’s Degrees (sum of data from Column ME + ME/DO / Total ME + Total ME/DO = 107/428); 28.50% of the Doctorates (Sum of DO + ME/DO / Total DO + Total ME/DO = 62/221); and, over half the professional Master’s Degrees in the wider field, with 56.68% (MF/Total MF = 75/132).

The study is qualitative in nature, and we sought to study the evaluation directives of the CAPES in relation to all the fields in the wider field of Applied Social Sciences. For this purpose, we analyzed the Qualis classification criteria for journals in Applied Social Sciences, comparing the different categorization criteria of each field, their differences and the impact on the composition of the Qualis Periodical system for these fields.

We used secondary data retrieved from the CAPES website, including data manuals, field documents (of the seven fields that make up the wider field of Applied Social Sciences), documents, ministerial bulletins and other sources. These materials constitute the bases that determine the criteria for all fields of science and will support the collection and analysis of data and information (Cervo; Bervian; Silva, 2007).

The research efforts aimed to investigate how intellectual production is evaluated in the different fields and what criteria are given more importance. We then sought to identify the similarities and differences that result in discrepancies in the classification of journals and the quality of what is produced in each field.

When we observe the wide field of Applied Social Sciences, we expect each field to have criteria that differ somehow from the others due to their nature and specific characteristics. However, we should also expect criteria that do not differ very much as all the fields belong to the same wider field of knowledge, and we imagine that scientific evaluation should employ similar degrees of rigor for all. Another point that we highlight is that if the fields are all part of a wider field, it is because they have more similarities than differences.

Thus, each field in the wider field of Applied Social Sciences has its own criteria for classifying journals, taking into account the degree of maturity of each field and its understanding of the correct level of quality for each classification stratum for its journals.

For a scientific journal to be included in the CAPES Webqualis database (online base from which we can consult the Qualis Periodical system) or to move to a higher stratum, the scientific editors and editorial teams (editorial and scientific staff) follow international norms or editorial practices. These are known as intrinsic elements (quality) and extrinsic elements (form/presentation) for editorial management. One of the most important criteria used by the committees in each field is the impact factor.

For a journal to be included in the upper strata (A1 or A2) of the Qualis system, it must be indexed in international databases, such as the Scopus (Elsevier), Web of Science (Thomson Reuters) and SCIELO (FAPESP) and have short-term citations (Sandes-Guimarães; VERASDE; Diniz, 2013).

The selection criteria for all the databases are very similar (EBSCOhost, Redalyc, DOAJ, Gale Cengage Learning, Latindex, Proquest and Dialnet): scientific and editorial quality (evaluation of articles), editorial staff, original articles, production quality, journals published on schedule, institutional diversity of authors and standardized references.

According to Santos (2016, p. 23), the indexing sources (better known as the databases) can be classified as:

Commercial or Proprietary Indexing Sources: Commercial databases. Access is permitted by subscription for users from institutional libraries, with no indexing cost. These sources are EBSCOhost, Proquest, Web of Science (Thomson Reuters), Gale Cengage Learning and Scopus (Elsevier).

Public Indexing Sources: Open access bases for scientific journals: Redalyc, Directory of Open Access Journals (DOAJ), SciELO (Brazil), Latindex, Google Scholar, Spell (Anpad), base with focus on Management, Red Iberoamericana (REDIB), Dialnet, ErihPlus and PKP Index (the most recent).

Automatic or Autonomous Indexing Sources: Bases that use web crawlers that capture and copy information and digital content.

These bases are intended to facilitate the tracing and recovery of information with the help of search engines (such as Google), thus expanding the number of versions (Ebrahim, 2014; Ebrahim et al., 2014) of the same digital file.

The Impact Factor (IF) of the Journal Citation Reports (JCR) and the Scopus H-Index were incorporated by the fields of evaluation such as Management, Accounting and Tourism, Architecture, Applied Social Sciences I, Economics, Urban Planning and Social Services, as summarized in Table 2a and 2b.

Table 2a
Impact criteria for the fields of Applied Social Sciences

Source: Documents published by the field Committees (2016).

Table 2b
Continuation of Impact criteria for the fields of Applied Social Sciences

Source: documents published by the field Committees (2016)

With particular regard to the upper strata in Table 2a and Table 2b (A1, A2 and B1), Management, Architecture, Applied Social Sciences I, Economics and Urban Planning include scientific journals indexed in the Scopus, Web of Science (Thomson Reuters) and SCIELO (FAPESP) databases, in addition to Impact Factor (IF) indices of the Journal Citation Reports (JCR) and the Scopus (Elsevier) H-Index, to move to a higher stratum.

Concerning the field of Social Services, a periodical only needs to be indexed/listed in the SCIELO, Scopus and ISI databases. The criteria adopted for the Qualis in Economics and Architecture are far more rigorous between strata A1, A2, B1 and B2, with only the Impact Factor (IF) as the predominant criterion to move up the Qualis system.

For B1, between Applied Social Sciences I and Urban Planning, the requirements are more flexible, as the journals without an impact factor should be indexed in international databases for inclusion in this stratum. It is interesting that Urban Planning includes the CAPES Portal as a database for journals, unlike the other fields, which do not list it as a requirement for the Qualis system.

A general comparison of the fields shows that only Law uses the exogenous percentage as a requirement for the evaluation of scientific journals in the Qualis system. This has a negative impact on the work of editors and editorial staff, as without well-defined criteria, they cannot outline strategies for improving their position in the Qualis ranking system.

According to Sandes-Guimarães, Verasde and Diniz (2013), the CAPES Scientific and Technical Council (CTC-ES) restricted the percentage of journals eligible for inclusion in the upper strata. This was due to the proliferation of journals in the fields. Therefore, the adoption of the impact factor emerged as a deciding criterion for the Qualis system. By complying with this directive, the council for Management eliminated some criteria from the previous three-year period (2010-2012) and added new criteria/requirements (such as the impact factor) in this quadrennial (2013-2016) for strata A1, A2 and B1.

With the impact factor as a criterion for the upper strata, several journals were relegated in the Qualis/CAPES Management ranking. The scientific editors and the institutions that managed the journals considered other alternatives and different ways to remain in the upper strata. However, the closest stratum was the Qualis B2. A further difficulty for editors was that the only more practical way was for their journals to be indexed in the Mexican Redalyc and the Brazilian SCIELO bases. However, the Redalyc and SCIELO were overwhelmed with submissions from Brazilian editorial teams and their requests could not be addressed in time by the understaffed technical staff of these databases.

Yet another frustration for editors was that the third item listed as a requirement was that the periodical had to be published by an international publishers stipulated by the field. It should be remembered that institutions, especially public ones, do not have sufficient resources to outsource the editorial management of journals, as costs are high (Diniz, 2016).

From this context, it was clear that the Management journals were not prepared for the new criteria. Why did this happen? Unlike the other fields, Management journals were not included in international databases and were not listed in bases that calculate the Impact Factor (IF).

This scenario was made clear with the publishing of the Impact Factor list of the Web of Science (Thomson Reuters) for Management journals, as shown in Table 3.

Table 3
Impact factor of Brazilian management journals in the Web of Science

Source: JCR (2016).

Rodrigo Assunção Rosada, a doctoral student in Management at the Getúlio Vargas Foundation (FGV/EAESP), conducted a survey to identify the journals in Management, consulting the following databases: Google Scholar, SCIELO (FAPESP), CAPES Journals and the library base Spell (ANPAD). In this study, he identified at least 220 scientific journals with a focus on Management. This information was posted on the ‘SCI&ORG – Ciência e Organizações’ blog. Of these 220 journals in Management, Accounting and Tourism, only three Brazilian journals were included on the list of the Journal Citation Reports (JCR) managed by the Web of Science (Thomson Reuters). The highlight was the journal Custos e Agronegócio Online, registering two impact factors: Scimago Journal Rank (SJR), published by Scopus and Journal Citation Reports® (ISI/JCR), published by the Web of Science/Thomson Reuters. This journal is published by the Federal Rural University of Pernambuco and is classified as B2 in Management. In Urban and Regional Planning/Demography, the Qualis stratum is A2.

Another interesting point to analyze is the number of A1 journals (considered as the highest quality by Qualis) in the fields compared with the total number of journals in their respective fields, as shown in Figure 1. This analysis shows the compliance of all the fields to the checks stipulated by the CTC (maximum of 12% of all journals may be considered A1).

Figure 1
Percentage of A1 journals
Source: prepared by the authors (2016).

We also looked at the language in which the A1 journals are published. All the journals on Management, Accounting and Tourism, Architecture and Social Services are published in English, whereas all the A1 journals on Law are in Portuguese. The journals on Applied Social Sciences I, Economics and Urban Planning are also published in Spanish. These data are summarized in Figure 2.

Figure 2
Languages of A1 journals per field
Source: prepared by the authors (2016).

We also analyzed the internationalization of journals. Although the A1 journals of Applied Social Sciences are published in only three languages, we found a wide variety regarding the country of origin of their editorial staff, as shown in Table 4. This origin was confirmed when we analyzed the country of origin of the editors in chief of all the A1 journals in the wider field of Applied Social Sciences.

Table 4
Countries of origin of the editors of A1 journals

Source: prepared by the authors (2016).

Continuing to focus on Management, Accounting and Tourism in comparison with the other fields, this is the one with the most A1 journals of all the Applied Social Sciences (68 journals). They are all published in English and show little diversity in terms of country of origin, with 43 of the 68 journals (63%) from the United States and United Kingdom.

Architecture and urbanism is second place in the number of A1 journals, with 27, and is similar to Management in that all the journals are published in English and are largely concentrated in North America. Of the 27 journals, 12 (44.4%) are of this origin.

Applied Social Sciences I has more distinct characteristics in relation to Management, with 16 journals, 8 of which are published in English in different countries, 4 in Spanish and 4 in Portuguese in Brazil.

Law is the most divergent field, as all 12 of its journals are published in Portuguese and in Brazil. This represents the specific nature of the field. Publication in Brazil in the native language makes sense because these are studies of the specific norms and laws of the country.

Most of the journals on Economics are published in English. Eleven are published in the United States, two in the United Kingdom and one in Japan. There is also one journal in Spanish.

As in most of the fields, the A1 journals on Urban Planning are also published in English, 18 out of 26 (69%). These are mainly from the United States and United Kingdom, with 8 and 5 journals, respectively. Five are published in Portuguese in Brazil and three in Spanish in Chile.

Social Services is the last field under analysis and also the field with the fewest A1 journals, 4 in all. They are all Canadian and published in English.

Returning to Management, Accounting and Tourism, the main focus of analysis, we noted from the data that in this field the emphasis is on international journals published in English, with no Brazilian journals. This could mean that, in combination with the Qualis criteria for the field, the Brazilian journals have yet to achieve an impact factor or that their impact factors are insufficient to be considered first class. It could also mean that the Management criteria are not in keeping with the status of Brazilian journals, or that the Brazilian field of Management, Accounting and Tourism is somewhat detached from the field in international terms due to linguistic differences between English and Portuguese.

In comparison with the other fields of Applied Social Sciences, Management is the only one, other than Architecture and Urbanism (we can disregard Social Services due to the low number of journals) that has no A1 journals published in any language other than English. This shows that there is room for Brazilian journals in the A1 stratum in this and the other fields.

We also listed the distribution of Brazilian journals in the A2 stratum, as well as the A1 in the Qualis via the Sucupira Platform. The distribution and total number of journals for the fields in question are summarized in Table 5.

Table 5
Distribution of Brazilian journals in strata A1 and A2

Source: CAPES (2016b)

We noted that there is a high concentration of Brazilian scientific journals in the upper strata for the field of Urban Planning, with 30 journals. This is followed by 29 in Architecture and finally Law, with 26 journals concentrated in the upper strata. Concerning the A1 stratum, as shown in Figure 2, we noted an absence of Brazilian journals in the fields of Management, Architecture and Economics. There are two reasons for this: a) the Qualis system for these fields allows journals to be registered or classified in different fields in the strata, thus removing the space that could be allocated for the periodical in that field; and b) there is no policy for incentives or flexibility (alternative indexations) for low impact Brazilian journals, and in practice greater value is placed on foreign journals. Consequently, the best findings in scientific research are published in foreign journals or in ‘high-ranking Brazilian journals’.

This study sought to gauge the influence of the composition of the Qualis Periodical system for the wide field of Applied Social Sciences on the Qualis Journals in the field of Management. To this end, we analyzed the documents, manuals and CAPES guides, articles by other authors on the subject and other works to determine the proposals and classification criteria for each field.

We found that the criteria established for the Qualis in Management are considerably more demanding that in the other Applied Social Sciences. High impact factors are required for Management, whereas for the others only indexers in some databases are necessary.

This does not necessarily mean that the criteria set for Management are too demanding. However, it does mean that they are not in keeping with the reality of Brazilian scientific production in Management. There are also several possible reason for this phenomenon. We might infer that the demands for a Qualis A1 classification in Management are unnecessary. There may be a lack of maturity in the other fields in terms of their post-graduate programs and Qualis system. It could be the lack of an equal evaluation system for the wider field of Applied Social Sciences or that the specific characteristics of each field lead to differences.

Another factor is the predominance of journals published in English, especially those from the United States and United Kingdom. These publications are considered elite and their content is of higher quality internationally and with regard to Brazilian science. The lack of Brazilian journals in Management is a reflection of the Qualis being allowed to classify journals in different strata and the lack of incentive for journals with low impact factors.

Regarding the international indexers SciELO, Web of Science (WOS) and Scopus, prior to submission we recommend that journal editors check whether the journal has been cited in other journals. They should also divulge and monitor citations from articles. These databases mainly seek original quality articles, articles in English, high-impact citations, visibility (indexing helps) and an international readership. If the journal does not meet these expectations, we recommend against submission because when a periodical is not approved, it will be blocked (SCOPUS and WOS) from making further submissions. There is a logical explanation for this, as a journal’s citation index is calculated every two years.

The evaluated fields could review the Qualis criteria or enable actions to improve the quality and position of the journals in the Webqualis/CAPES ranking.

With the present study, we do not see any way of determining what must be done, nor do we affirm the existence or non-existence of problems in relation to the Qualis periodical system criteria. The present study is intended to initiate discussions on the theme.

We hope we have contributed to beginning a more detailed discussion on some gaps and diverging criteria that we identified and which deserve further research in future studies.