1 Introduction

Participation in tertiary education in Brazil varies remarkably among different population groups (PEDROSA et al., 2007). Household income, parents’ educational attainment, and race are among the most decisive factors that impact the student’s probability of attending an institution of higher education (SILVA and HASENBALG, 2002). In order to decrease inequalities, many public institutions in Brazil implemented affirmative action policies and increased the number of slots in recent decades (FRANCIS and TANNURI-PIANTO, 2012; PEDROSA et al., 2007; TELLES and PAIXÃO, 2013). Consequently, there was an increase in the proportion of Black/Pardo/Indigenous students and those from low-income families attending public universities in Brazil.

Moreover, other recent changes in the selection process may also affect the selection of students at federal public universities in Brazil. Until recently, most universities had their own exams; in recent years, the National Exam of the Secondary Level (ENEM) began to be used as part of, or as the unique exam for, the selection of students for tertiary education (LIMA and MACHADO, 2016).

In particular at the Federal University of Minas Gerais (UFMG), until 2010 two stages of a UFMG-specific exam were used to select students. From 2011 to 2013, four of the ENEM exams (Mathematics, Language, Science, and Humanities) were used as the first stage, while the second stage continued to be UFMG’s own exams combined with the composition score from the ENEM. Since 2014 the five ENEM exams and the Unified Selection System (SISU) have been used as the main methods of student selection (LIMA and MACHADO, 2016; NOGUEIRA et al., 2017).

All these policy changes may have influenced the associations between performance in the entrance exams and posterior academic performance at institutions of higher education in Brazil. This paper has three main objectives associated with these relation ships: to address the determinants of academic performance, to analyze whether temporal trends exist, and to observe whether there are differences between the results for 2009 and 2012 due to the change in the entrance exam.

Other authors discussed similar topics in different settings (BAI and CHI, 2011; BETTINGER et al., 2013; FRYER JR. et al., 2008; LEONARD and JIANG, 1999; LOURY and GARMAN, 1993; ROTHSTEIN, 2004). However, this paper analyses the three questions mentioned above within a developing country where there have been many recent changes in the process by which public universities select students. To the best of my know ledge, these investigations have not yet been addressed with Brazilian data.

The paper uses official records of UFMG developed by the Permanent Commission of the Vestibular (COPEVE) and by the Department of Academic Registry (DRCA). The databases were kindly made available for research purposes, and the anonymity of the students was preserved.

Besides this introduction, the paper is further divided into four sections. Section Two presents the literature review. Section Three describes the methodology. Section Four depicts the empirical results, and the last section concludes the paper.

2 Literature review

This section is divided into two subsections. The first briefly presents the recent expansion of tertiary education in Brazil. The second describes some of the factors associated with academic performance at institutions of higher education.

2.1 Recent expansion of higher education in Brazil

Higher education in Brazil expanded remarkably recently (BARROS, 2015; SALATA, 2018). Barros (2015) shows that the number of students participating in tertiary education in Brazil increased from a little over 3 million in 2001 to over 6.3 million in 2010. This increase was mostly in private institutions, but public institutions also showed an amazing increase in the number of students. Salata (2018) shows that this tendency continued at least until 2014.

The number of annual slots also increased in institutions of higher education in Brazil, from fewer than 1.5 million in 2001 to more than 3 million in 2010. However, the number of secondary school graduates did not increase between 2001 and 2010. Consequently, the number of empty slots increased from less than 400,000 in 2001 to more than 1.5 million in 2010, the great majority of which were in private institutions (BARROS, 2015).

According to Barros (2015) and Salata (2018), some recently implemented policies were partially responsible for these tendencies. These include the Educational Loans Program (Fies), the University for All Program (ProUni), and the Support Program for the Restructuring and Expansion of Federal Universities (Reuni). Fies was implemented in 1999 and provides low-interest loans for students in private institutions of higher education. Pro Uni was implemented in 2005 and has as its main purpose to give scholarships to students from low-income households. In 2011, more than 250,000 students had partial or full scholarships. The Reuni policy was implemented in 2007 and had as its main objective to increase the number of slots in public institutions of higher education. All three policies promoted an increase in the number of students, especially from low-income households, in private and public institutions. Moreover, the Brazilian Open University (UAB)was created in 2006 with the promotion of on-line courses.

2.2 Factors associated with academic performance

Many factors are associated with academic performance at different schooling levels. Among these determinants are individual attributes (sex, race, age, etc.), household characteristics (parents’ schooling level, income, number of siblings, etc.), and school factors (infrastructure, administration, teachers, etc.) (BARROS et al., 2001). Many authors described these determinants with Brazilian data using standardized tests for elementary and secondary students (ARAÚJO and SIQUEIRA, 2010; FERNANDES and NATENZON, 2003; MACHADO et al., 2008; SOARES, 2005; SOARES and ALVEZ, 2013).

Nonetheless, the focus here is on higher education. In particular for this level in Brazil, Francis and Tannuri-Pianto (2012) observed that male, Pardo, and Indigenous students had lower academic performances. Golgher et al. (2015) verified that students at UFMG who had studied at municipal, federal or private secondary schools, who hadn’t attended a specific course preparing for the entrance exams of institutions of higher education (prévestibular), who did not work or who worked up to twenty hours per week, who came from higher-income households, and who had computers at home had higher performances.

Besides the above-mentioned socioeconomic and demographic determinants of academic performance at the tertiary level, performance on the entrance exams of institutions of higher education may also affect academic performance at these institutions. Concerning this topic, Bai and Chi (2011) investigated whether the Chinese College Entrance Examination score predicted college academic success. They found that the total and subject test scores of this exam predicted undergraduate GPAs for all four years in college. They present other results, including the findings that females had a better GPA than males and that minorities had similar performances to non-minorities.

Loury and Garman (1993) compared SAT scores with GPAs of Whites and Blacks attending institutions of higher education at different levels of selectivity. The authors observed that higher SAT scores were positively correlated with GPAs for both groups. Similarly, Fryer Jr. et al. (2008) also analyzed the determinants of college GPA sincluding SAT scores as the explanatory variable. They observed a positive correlation between both. In addition, they verified that parental education was positively correlated with academic performance.

Bettinger et al. (2013) emphasized that improving college performance and retention can be difficult and costly. They proposed a simple and low-cost change in the way colleges use the ACT exam in their admission decisions. The ACT covers four subjects, Mathematics, English, Reading, and Science, and nearly all colleges use a composite score of the four exams of the test in their admissions process. Similarly to the authors mentioned previously, they observed a strong positive correlation between ACT composite scores and college outcomes. However, they observed that the use of only two exams, Mathematics and English, could effectively predictoutcomes in college GPA for the first and third years. They concluded that Reading and Science tests did not contribute effectivelyto the selection of students.

Leonard and Jiang (1999) observed that exams such as the ACT and SAT under-predicted women’s performance in tertiary education, as women obtain higher grades in college than men with identical scores in the entrance exam. Thus, females tend to be under representedamong selected freshmen when the selection process is based on the SAT exam. Similarly, Rothstein (2004) observed that female students had higher freshman GPAs, but lower SAT scores than males.

Lastly, authors described some of the limitations of using exams such as the SAT for student selection. Rothstein (2004) observed that many studies have found that exam scores are given disproportionate weight relative to their predictive power and tend to disadvantage lower-income and minority students; he stressed that, as a consequence, several colleges had deemphasized the SAT exam in the selection process. He points out that, if academic success is the main objective for admissions, other information besides SAT scores should be included in student selection. Scott-Clayton et al. (2014) also proposed the use of other types of information.

Similarly to the studies mentioned above, this paper analyses associations between performance on UFMG’s entrance exams in 2009 and in 2012 and subsequent academic performance at this institution. The focuses are the recent policy changes in the selection process. The applied methodology is detailed in the next section.

3 Methodology

The methodological section is divided into three subsections. The first describes the policies implemented at UFMG between 2009 and 2016, the second presents the database, and the third details the empirical strategy.

3.1 Policies implemented at UFMG between 2009 and 2016

This subsection describes the main changes that occurred at UFMG regarding the affirmative action policies, the number of slots, and the student selection exams. Table 1 presents the main changes that occurred at this university between 2009 and 2016. Aranha et al. (2012) describe some of these policies in detail.

UFMG implemented a bonus policy in the entrance exam of 2009. This policy increased by 10% the grades obtained in the entrance exam by individuals who had attended public schools in the last seven years of elementary and secondary education. In addition, an extra 5% bonus was given to those who considered themselves Black/Pardo/Indigenous. This policy, with minor changes, remained valid until the entrance exam of 2012. In 2012 the national federal law on quotas was passed; this law was implemented in all federal higher education institutions over four years (TELLES and PAIXÃO, 2013). Beginning with the entrance exam of 2013, at least 12.5% of the students in each course had to have attended public secondary schools. This number increased to 25%, 37.5%, and 50% respectively in the years of 2014, 2015, and 2016.

Although they have similar final objectives, bonus and quota polices differ in some aspects. In a quota system, spots are reserved for minorities as a proportion of the student body. In contrast, the bonus policy gives students additional points that are proportional to their performance. The bonus policy has some advantages over quota policy (PEDROSA et al., 2007); however, this type of policy does not guarantee the participation of a minimum number of individuals from minorities in each course. Hence, minorities may be underrepresented in more competitive courses, as observed at UFMG by Golgher et al. (2014, 2015).

UFMG also implemented other policies over this period. Mostly due to the Reuni policy, there was a remarkable increase in the number of slots between 2008 and 2012, from 4,600 to 6,600 annually (ARANHA et al, 2012; LIMA and MACHADO, 2016). During this period, 27 new courses were created and 23 preexisting courses increased their number of slots. Nearly all of this increase took place in the years 2009 and 2010.

Moreover, there were changes in the entrance exam. Until recently, most universities had their own exams. In recent years, the ENEM exam began to be used as the first stage or as the sole exam used by institutions of higher education to select students (LIMA and MACHADO, 2016). In UFMG, there have been two changes in the entrance exam since 2009. Until 2010, there was a UFMG-specific exam in two stages. From 2011 to 2013, the ENEM was used as the first stage, while there was a UFMG-specific exam for the second stage. Since 2014, the ENEM and SISU has been used as the main method for student selection (LIMA and MACHADO, 2016).

The government predicted three positive results of the implementation of ENEM/SISU (NOGUEIRA et al., 2017). Students’ spatial mobility was expected to increase, as the same exam could be used in all states and at most public universities in Brazil. Students of secondary schools in one state could apply to institutions of higher education in another state with little cost. The government also predicted that costs would be lowered and the selection process would become more efficient, with a decrease in the proportion of empty slots. Finally, there was to be an increase in the proportion of minorities in the university, particularly because of the quota policy, but also due to the reduction of costs of the selection process for individuals (NOGUEIRA et al., 2017). Nonetheless, Nogueira et al. (2017) and Silveira et al. (2015) concluded that these expectations were not fulfilled.

In addition, there are other differences between selecting students based on UFMG’s own exam and selecting them using ENEM as the first or unique stage. The exams differ in size and scope. The first stage of UFMG’s own exam had eight multiple-choice questions in each of the following subjects: Biology, Physics, Geography, History, Foreign Languages, Portuguese, Brazilian Literature, Mathematics, and Chemistry (UFMG, 2008). The ENEM has 180 multiple choice questions divided among four exams: Natural Sciences, Human Sciences, Mathematics, and Language. Moreover, Sobrinho and Dos Santos (2014) explain that another of the focal point of the ENEM is contextualization. Oliveira (2014) argues that anemphas is on contextualized learning has among its objectives, not only to provide required knowledge in many subjects, but also to enhance the practice of citizenship.

3.2 Databases

The paper uses as databases official records of UFMG that were developed by COPEVE and by DRCA. The databases were kindly made available for research purposes, and the anonymity of the students was preserved. They contain information for all students who applied, were selected, and registered at UFMG in 2009 and in 2012. The databases are continuously updated, but are not continuously available for research purposes, only for administrative objectives. Besides the administrative information, the databases contain affirmative action policy information and socioeconomic variables, as all students who register at UFMG answer a survey which collects non-administrative information. Details about some of the variables in the database are given in subsection 3.3. These or similar databases were used in other studies (GOLGHER et al., 2014, 2015); however, these other analyses had different approaches and objectives.

Among students who had registered in 2009, 5,809 had a GPA record in at least one semester; however, the database does not include socioeconomic data on some of these students. This number for 2012 is 6,438. Many students dropout of UFMG. Others temporarily stop attending this institution and return after a short period. To prevent this last group of students who show a null GPA in specific semesters from biasing the analysis, only students with positive GPA in a specific semester were included in the empirical analysis. The numbers of students who had available data on socioeconomic variables and a positive GPA in 2009 were 4,992, 4,644 and 4,222, respectively in the first, third and sixth semester. These same numbers for 2012, for the first and third semesters only, were 5,335 and 4,826. The number of observations decreases mostly due to students dropping out. Thus, most students had a positive GPA in the three semesters, but a sizable minority did not. Only students who did not change courses were included in the analysis.

3.3 Empirical strategy

The main questions which this paper empirically addresses are: 1) What are the associations between performance on the entrance exams and subsequent academic performance at UFMG?; 2) Have these associations altered due to changes in the first stage of the selection process?; 3) How do these associations change when freshmen are compared to sophomores and to juniors?

The dependent variable is the GPA in each specific semester. For students who entered UFMG in 2009 there was data available for eight semesters. I selected three of them: the first, the third, and the sixth semesters. By doing so, I could analyze time trends. Besides, I avoided later semesters when many students may have already graduated. For students who entered UFMG in 2012, there was data for the first four semesters. I selected the first and the third semesters in order to make comparisons with the data from 2009.

Given that GPA is continuous and approximately normally distributed, OLS models with robust and clustered standard errors could be used. Another possibility would be to transform the GPA into an ordered variable by rounding the continuous variable. A multinomial ordered model could be used in this case. However, these approaches would result in the loss of information. Hence, I choose to use the GPA as a continuous variable.

However, academic performance can only vary between 0 and 5. That is, the dependent variable is censored, which is a very common feature in microeconomic data (GREENE, 2003). Here, as GPA is continuous and approximately normally distributed, it is assumed that GPS has a truncated standard normal distribution. For values that would be negative, it is given the value zero. For values that would be above five, the value five was given. Based on these features, following this author, double-censored Tobit models were used.

Initially, the explanatory variables of main interest are performance on the first stage of UFMG’s own exam in 2009 and onthe ENEM in 2012. In the first stage of the selection process, all applicants do the same exams. In 2009, the exams were Portuguese, Foreign Language, Mathematics, Science, and Humanities. The first two exams are grouped in the ENEM in order to make comparison between 2009 and 2012 more insightful. The results of the ENEM’s composition exam were not included in the first set of analysis.

The second-stage exams differ depending on the chosen course. To analyze each course separately is not feasible due to small sample limitations. However, groups of courses with similar entrance exams in the second stage were analyzed separately, and this analysis was also incorporated into the paper.

The groups of courses are the following, with the second-stage exams in brackets: Medicine (Biochemistry); Language and Linguistics (Portuguese and History); General Engineering, Physics, Chemistry, and Geology (Physics, Chemistry, and Mathematics); Architecture, Computers Science, Statistics, and Mathematics (Physics and Mathematics); Business,

Accounting, Economics, and Information Science (History, Geography and Mathematics); Agricultural and Husbandry Sciences, Health Sciences, Biology, and Dentistry (Biology and Chemistry); and Social Science, Law, and Teaching (History and Geography). All individuals took a composition exam. The results in these exams are also explanatory variables of main interest in this paper.

The models also included several control variables: features of the individuals (sex, race, civil status, workload, whether the individual had already graduated, previous attendance at a pré-vestibular, previous knowledge in reading foreign languages, and place of residence), characteristics of the household (the father’s and the mother’s schooling levels, the household income, and the father’s and the mother’s occupations), features related to the student’s secondary education (years since high school graduation, regime of secondary school, and type of secondary school) and assets in the household (domestic servant, fridge, car, and computer). Most of these variables are commonly used in studies that address the determinants of schooling performance at different levels, as described in the theoretical section. Moreover, a dummy for each course was also included in the models.

In a similar vein to Bettinger et al. (2013), I estimated the following equation, which was estimated for each of the specific semesters mentioned above:

GPA. =α+∑β_1jSubject_1ij +∑_k β_2k Subject_2ik +δX_i +ε_i, where i represents individuals, Subject_1ij is the performance of i in the subject j in the first stage of the selection process, Subject_2ik is the performance of i in the subject k in the second stage of the selection process, X_i is a set of controls, and ε. are the stochastic errors.

4 Results

The results are presented in two subsections. The first presents descriptive statistics and the second shows the results of the econometric models.

4.1 Descriptive statistics

Table 2 shows the mean values for selected explanatory variables for entrance exam scores in the first stage of the UFMG exam and for GPAs in the first, third and sixth semester for students who registered at UFMG in 2009. I included these three different semesters in order to observe whether students from certain categories caught up with their peers in academic performance while at UFMG. Those in the third semester would still be near the beginning of the course, while those in the sixth semester would be close to graduating. The results for those who registered at UFMG in 2012 were very similar and are not shown.

The results of the entrance exam are those of the first stage, at which all students did the same exam. They were transformedin to a normal distribution with mean zero and variance one. The GPAs vary between 0 and 5, but only positive values were entered into the statistics; data on students who temporarily or permanently left UFMG is not included. As already mentioned, the numbers of observations are respectively 4,992, 4,644, and 4,222 for the first, third and sixth semesters. That is, the number of observations decreases due to students dropping out.

Dropout rates at Brazilian universities are not strongly correlated with race and income. Pedrosa et al. (2007) observed that the socioeconomic profile of those who dropped out from the university was similar to that of those who graduated. Francis and Tannuri-Pianto (2012) verified that Black students had similar college attrition to lighter-skinned siblings. In particular for UFMG, Aranha et al. (2012) analyzed dropout rates between 2009 and 2010 for those who did and did not benefit from the bonus policy and observed that those who benefitted from the bonus policy were less likely to drop out in nearly all fields of study. These results suggest that dropout rates for disadvantaged students may be similar to or lower than those of their more advantaged peers in Brazil.

The discussion of the results in table 2 has three main objectives. The first is to observe differences for each category for each selected explanatory variable for the results of the entrance exam and academic performance at UFMG. The second is to compare the results of the entrance exam with those for academic performance at the tertiary level. The third is to observe the dynamics of the academic performance of the different groups during the students’ trajectory at an institution of higher education.

The statistical significance of the differences between categories in a particular variable was accessed. For dummy variables, a two-group comparison, the means were compared by t-tests. Comparisons that showed statistically significant differences are marked with asterisks. For categorical variables, ANOVA and ad hoc Bonferroni tests were used to compare more than two groups. . stands for the larger value and . stands for the smaller value in a particular comparison if differences were statistically significant. As multiple comparisons can be done, numbers are included in the table to represent each comparison. Further explanations are given as the results are presented.

Differences between males and females are all statistically significant, as shown by the asterisks. Males performed better in the entrance exam, but showed a worse GPA in all three semesters in the university, without any signaling of catching up. This results are similar to those observed by Leonard and Jiang (1999) and by Rothstein (2004).

For race, differences were statically significant for the results of the entrance exam and for GPAs in the first and third semesters. The values for the sixth semester did not differ significantly between racial groups. The results for the entrance exam indicate that the value for those who did not declare their race was statistically larger than for those who self-declared as Black/Pardo/Indigenous. The letters M1 and m1 respectively for these two groups show that the difference between them was significant. Similarly, the value for White/Asian individuals was statistically larger than for Black/ Pardo/Indigenous individuals, as shown by the letters M2 and m2. No other comparisons were statistically significant. Concerning GPA differences in the first semester, those who did not declare their race had a value larger than that of either of the other two categories (M1 against m1). White/Asian individuals had a value larger than that of Black/Pardo/Indigenous individuals (M2 against m2). The results for the third semester indicate that the values for those who did not declare their race and for White/Asian individuals were statistically larger than for Black/Pardo/Indigenous individuals. Not all the differences were statistically significant in the sixth semester, indicating that Black/Pardo/Indigenous individuals catch up while at the university.

Those who did not work had the best performance in the three categories of the work variable (M1 against m1), and those who worked more than twenty hours per week had the worst performance in the entrance exam (M2 against m2). For GPA in the first semester, those who didn’t work or who worked up to twenty hours had the best performances. In the third semester, those who worked up to 20 hours weekly had the highest GPAs, even higher than those who did not work (M1 against m1), suggesting that they may represent a positively selected sample. The results for GPA in the sixth semester were similar to those in the first. All these results suggest that it is feasible to work and study if workloads are not large.

For father’s and mother’s schooling, the results for the entrance exam indicate that those with more educated parents had higher performances (Notice that m1-3 indicate that the categories had a smaller value than the categories marked M1, M2 and M3). However, differences for GPA are mostly non-significant for the third and sixth semester, and they are significant in the first semester only for the father’s education. Similar trends were observed for house hold income. This is a clear indication that minorities do catchup while at the university, despite their worse results on the entrance exam.

Finally, differences in entrance exam scores for students from different types of secondary school attended are as expected. Students from private and federal schools had better performances than those from state and municipal schools. For GPA in the first semester, students from state secondary schools had a lower performance, but those from municipal schools showed statistically similar results to those who had attended private and federal secondary schools. All differences in the third and sixth semesters were non-significant, indicating, once more, that minorities catch up while at the university. This catching-up could be partially caused by selective dropout rates. However, as discussed by Aranha et al. (2012), Francis and TannuriPianto (2012) and Pedrosa et al. (2007), this is not expected.

4.2 Econometric models

This section presents the results of the econometric models. The objectives are three fold: to address the determinants of academic performance in a more controlled analysis, to analyze whether temporal trends exist, and to observe whether there are differences between 2009 and 2012 due to the change in the entrance exam. Table 3 shows the results for four models that address this first objective. All models include dummies for courses as controls. Results are not shown for these coefficients.

Concerning the first model, the dependent variable is the results on the entrance exam in the first stage of the selection process in 2009. I briefly comment on the results here, as this is not the focus of the paper. This dependent variable is continuous and normally distributed. The model is an OLS with robust and clustered errors, as shown in the last line of the table. Individuals who were male, who did not declare their race, who knew how to read foreign languages besides Spanish, who lived in higherincome households, who had attended secondary regular schools, and who had attended secondary private or federal schools had higher performances. Notice that most of these results are expected; however, they are biased, as the database includes only those who were selected and registered at UFMG.

The dependent variable in the next three models is the GPA in the first semester for students who entered UFMG in 2009. Different models were estimated with the objective of observing the determinants of GPA performance with different sets of explanatory variables. Notice that this justifies the slightly different number of observations due to missing data on some variables. All of three models were estimated as double censored Tobit models, as shown in the last line of the table.

Model 1 has only one explanatory variable besides the controls for courses. GPA in the first semester was positively correlated with the general results in the first stage of the entrance exam, as expected. Model 2 divides the entrance exam results into five subjects: Portuguese, Mathematics, Humanities (History and Geography), Foreign Language (English, French or Spanish), and Science (Physics, Chemistry and Biology). Notice that only three exams showed significant and positive correlations, the first two and the last. That is, as also found by Bettinger et al. (2013), only some subject test results seems to effectively predict GPA in the first semester. Model 3 includes the socioeconomic and demographic variables as controls. The model shows that all exams had significant and positive coefficients after controlling for the socioeconomic and demographic variables, although Humanities and Foreign Language coefficients were of smaller magnitude. That is, all exams significantly explain variations in GPA in the first semester, although Portuguese, Mathematics, and Science seemed to have a larger explanatory power.

Concerning the results of the sociodemographic variables, as observed by Leonard and Jiang (1999) and Rothstein (2004), female students had higher

GPAs, although they had lower performances on the entrance exam. Black/ Pardo/Indigenous individuals had worse performances on the entrance exam and in the first semester at UFMG, even in a more controlled analysis. A lack of ease in a new environment might partially explain this result, as model 3 controls for the results on the first stage of the entrance exam. Catchingup is expected, as is shown in the next table. Notice that White/ Asian individuals had worse performances on the entrance exam than those who did not declare race. A detailed study of this last group is beyond the scope of this paper; however, it should be addressed in future analyses. Civil status showed non-significant coefficients for performance on the entrance exam and in the first semester’s GPA.

Those who worked more than 20 hours weekly, who had not graduated from another undergraduate course, who came from higher-income households, who had graduated from secondary school more than four years prior to their entrance exams, and who had not attended regular or vocational schools had worse performances. Some explanations are readily available. Those who work more than 20 hours weekly may face a shortage of time. Those who have already graduated from another undergraduate course are a positively selected sample, with more experience, and possibly with positively selectedun observable traits. Those from lower-income households may be catchingup with their richer colleagues in the university. Those who have been out of high school for more than four years may have become unaccustomed to doing academic work. Those who have not attended regular or vocational schools may have attended lower-quality secondary schools that have features harming the potential to do academic work which are not captured by the worse performance on the entrance exam.

Moreover, other variables showed significant results. Students who had attended federal secondary schools had better performance. These schools have a strict process for selecting incoming students, and those who attend these schools tend to be a positively selected sample. Students who lived in a household without a fridge had worse performances, indicating that very low SES is prejudicial for performances.

The next analysis has two main objectives: to observe time trends in selected variables, and to compare the results for 2009 and 2012 as the first stage in the entrance exam changed from the UFMG’s own exam to the ENEM. Table 4 presents five models, all with the same set of explanatory variables and estimated as a Tobit model. The variables Portuguese and Foreign Language were grouped in the 2009 data in order to make comparisons with 2012 more insightful, as the ENEM has only four exams instead of the five mentioned in table 3.

In order to apprehend time trends while students progress at the university, for 2009 data, results for the first, third, and sixth semesters are shown. For 2012, results for the first and third are shown, as data for the sixth was not available. Notice that the pseudo R. decrease with time at the university. That is, the predictive power of the explanatory variables decreases as the students advance at university.

First, notice that performance on most subject tests of the entrance exam showed positive and significant coefficients. For UFMG’s own exam in 2009, the exams for Humanities lost their significance in the third and sixth semesters. For the ENEM in 2012, all coefficients were positive and significant. These results are similar to those observed by Bai and Chi (2011) for China.

The presentation continues with the time trends of the selected socioeconomic and demographic variables. Males had worse performances in all models and did not show any tendency to catch up to females. Black/ Pardo/Indigenous individuals had a worse performance only in the first semester for those who entered UFMG in 2009. All coefficients for 2012 data were non-significant. That is, the slight difference between races disappeared during their university attendance.

Those who worked more than 20 hours weekly caught up with those who did not work for 2009 data. Differences for 2012 data were non-significant. Those who worked up to 20 hours showed better performance after a time at the university for those who entered UFMG in 2009, suggesting a positively selected group of students. Again, differences for 2012 data were nonsignificant.

Concerning SES, most income coefficients were non-significant, but those who lived in higher-income households had worse performance in the first and third semester in 2009. Conversely, those who did not had a fridge at home, mostly poor individuals, showed a worse performance in the first semesters in 2009 and 2012, but differences were non-significant for the third and sixth semesters, indicating that poor individuals catch up with other students.

Those who had finished high school four years or more before entering the university, and those who had not attended regular or vocational schools, showed lower performances in most or all models, with no clear catchingup trend. That is, apparently these two features negatively affect academic performance at institutions of higher education, and limitations caused by them are not overcome as the students advance at the university. Concerning the type of secondary school, those who had attended secondary federal schools tend to lose their initial advantage, and those from municipal or state schools tend to catch-up with other students.

All the other results were non-significant, including those for the variables included as controls, as shown in the bottom of the table.

The last two analyses include the results of the second stage of the selection process. As described in the methodology, seven groups of courses were created: medicine; language and linguistics; general engineering, physics, chemistry, and geology; architecture, computer science, statistics, and mathematics; business, accounting, economics, and information science; agricultural and husbandry sciences, health sciences, biology, and dentistry; and social sciences, law, and teaching.

Tables 5 and 6 show the results for GPAs in the first semester for these groups of courses, respectively for those who entered UFMG in 2009 and in 2012. The upper panels show the results of the Tobit models with all the controls. Results for controls are not shown. The objective is to observe which exams in the first and second stage had a significant predictive power when both stages are included in the analysis. The bottom panel shows the good ness of fit for models that includes different sets of controls. The objective is to observe the predictive power of sets of controls, similarly to what was done by Rothstein (2004).

For Medicine (group 1), none of the variables were significant for those who entered UFMG in 2009. That is, none of the exams in the first or second stage had a significant predictive power for first-semester GPAs. For the 2012 data, the ENEM’s Humanities and Science exams showed positive and significant coefficients. Moreover, the biochemistry exam in the second stage also showed a positive and significant coefficient. These results suggest that the use of the ENEM has a greater predictive power for academic performance than the use of UFMG´s own exam. Besides, the second stage of UFMG’s exam became significant with the use of ENEM as the first stage. Concerning the predictive power of sets of controls shown in the bottom panel, the UFMG’s exam in 2009 showed a very small pseudo R² of 0.008. The explanatory variables used as controls had a much larger predictive power, as pseudo R² increased to 0.136. The results for 2012 (table 6) show that the first stage had a much greater explanatory power than that observed in 2009, 0.030, but still most of the predictive power was due to the controls.

For Language and Linguistics (group 2), the coefficients in the second stage for composition and Portuguese in both years, and for History in 2012, were positive and significant. Language, Humanities, and Science in the first stage showed non-significant predictive power. Moreover, the coefficient for the first-stage mathematics exam was positive and significant in both models, suggesting that this last exam captures features that are not represented in the second stage. The predictive power of the entrance exams in 2009 and 2012 were much larger than previously observed for medicine. The magnitudes of the predictive power of both stages were reasonably close to the magnitude observed for the predictive power of the controls.

For group 3, comprising general engineering, physics, chemistry, and geology, most coefficients in the second stage were significant, including composition. Conversely, only one coefficient inthe first stage was significant, which was language in 2009. That is, the second stage of the entrance exam apparently had most of the predictive power, especially in 2009. Notice that the predictive powers of the exams net of the predictive power of the courses are similar to the predictive powers observed for the controls.

The fourth group of courses is composed of architecture, computer science, statistics, and mathematics. For the second stage, both coefficients for mathematics and one for physics were significant, but none for composition. For the first stage, only coefficients for mathematics and science were significant, while for language and humanities they were not significant. For this group, the predictive power of the controls seems to be greater, although the predictive powers of the exams are not negligible.

For business, accounting, economics, and information science (group 5), regarding the second stage, all coefficients for composition were nonsignificant, while all for mathematics, one for geography, and one for history were significant. In the first stage, all coefficients in 2009 were nonsignificant. In 2012, humanities showed a negative coefficient, which was not anticipated, and science showed a positive coefficient. Similarly to the previous group, the predictive power of the controls for this group seems to be greater, although the predictive power of the exams is not minor.

For agricultural and husbandry sciences, health sciences, biology, and dentistry (group 6), for the first stage,the coefficients for science were significant and positive in both models and for humanities this happened in 2012. In 2009, only chemistry showed a significant coefficient in the second stage. When the ENEM was used as the first stage, all the coefficients in the second stage were significant. This suggests that the use of ENEM as the first stage increased the predictive power of the use of UFMG exam in the second stage. For this group, the predictive power of exams and controls are quite similar.

Finally, for social sciences, law, and teaching (group 7), the first stage shows mostly non-significant coefficients, while the coefficient for mathematics in 2012 is negative and significant. That is, apparently the first stage has a very small predictive power, as also observed in the bottom panel. The composition and history coefficients of the second stage were significant in both models, while geography showed non-significant coefficients. However, the predictive power of the second stage is small. That is, most of the predictive power is due to controls.

Overall, one out of fourteen coefficients was significant for language. The ENEM exam for language showed only non-significant results. Correlations between language in the first stage and the other exams in the first and second are much stronger with the ENEM exam, and this might explain part of this results. For mathematics in the first stage, four coefficients were significant, but one was negative, contrary to expectations. Two of the positive and significant coefficients were for the language and linguistics group that does not have mathematics in the second stage. That is, except for this group, the predictive power of mathematics in the first stage is close to non-significant. All the coefficients for humanities in the first stage in UFMG’s own exam were non-significant, and two coefficients for the ENEM were positive and significant, while one was negative. That is, the predictive power of this exam is also small. Five of the coefficients for science were significant, although the results of this exam are highly correlated with math, physics, chemistry, and biology in the second stage. Taking the four exams together, only science seems to have a reasonable predictive power when a second stage is included as explanatory variable. Comparing the results of the bottom panel for courses, and courses and first stage, the predictivepower of the ENEM’s first stage is much larger that that observed for UFMG’s own exam.

Notice that the coefficients for composition were significant in seven models, including both models for groups 2, 3, and 7. Correlations with other exams were much smaller than those observed for other exams. Is this because other features are tested? All coefficients for mathematics, Portuguese, and chemistry in the second stage were significant. For physics, biology, history, geography and biochemistry, most coefficients were nonsignificant in 2009 and most were significant in 2012, although correlations with the exams in the first stage were much higher for the latter. Comparing the results of the bottom panel for courses and first stage with courses, and first and second stages with courses, the predictive power of the second stage is similar in both years

Comparing the results of the bottom panel for courses; courses and first and second stage; and courses, first and second stage and controls, the predictive power of UFMG’s own exam in two stages had a predictive power smaller than that of the controls, while results for ENEM’s first stage and UFMG’s second stage were similar to that observed for controls.

5 Conclusion

There were many recent changes in the selection process of institutions of higher education in Brazil, such as changes in affirmative action policies, the passage of the Reuni policy, and the implementation of the ENEM as part of or as the unique exam for student selection for tertiary education. All these policy changes may have influenced the associations between performance in the entrance exams and subsequent academic performance at the university. The main objective of this paper was to analyze these associations at UFMG in the years of 2009 and 2012. Some of the main results are summarized below.

Results clearly indicate that minority students catch up while at the university.

For students who entered UFMG in 2009 and took UFMG’s own firststage exams, Portuguese, Mathematics and Science exams had a larger predictive power, while Humanities and Foreign Language exams played smaller roles in predicting GPA outcomes. For those who were approved in the entrance exam of 2012, when the ENEM was used as a first stage of the selection process, all four exams (language, mathematics, humanities, and science) showed positive and significant results. When both stages were analyzed conjointly, the predictive power of the first stage was much smaller.

For the second stage, the coefficients for composition were significant for only a few courses, indicating the specificity of the predictive power of the exam. All coefficients for mathematics, Portuguese, and chemistry in the second stage were significant. For physics, biology, history, geography, and biochemistry, most coefficients were non-significant in 2009 and most were significant in 2012. These results suggest that the second stage of UFMG’s exam had a greater predictive power when ENEM was used as the first stage of the selection process.

Among the four subjects of the ACT exam – mathematics, English, reading, and science – Bettinger et al. (2013) proposed that the selection process should include only the first two. Based on the empirical results of the present paper, could I propose something similar? Not really. Nowadays the ENEM is basically the unique exam used to select students in federal universities in Brazil. All four exams had significant and positive coefficients in the models without the results of the second stage as explanatory for those who entered UFMG in 2012, indicating the significant predictive power of all exams.

Rothstein (2004) proposed that information other than SAT scores should be use din the student’s selection. Based on the empirical results above, could I propose something similar? Given that controls have a significant predictive power and that minorities catch up during their time at the university, affirmative action policies based on socioeconomic and demographic variables, such as those implemented by the quota system, may not harm overall performance at the university if they are well designed. Thus, periodical analysis of this topic should be performed to evaluate such policies.

Finally, the results of the second stage are mostly significant when the ENEM is used as the first stage. Thus, the implementation of an ENEM second stage, similar in some aspects to the second stage of UFMG’s own exam, would be a good choice to better select students, improving academic performance, while inducing changes at the secondary level, in particular regarding the depth and coverage of lectures on particular subjects.

References

ARANHA, Antônia; PENA, Carolina; RIBEIRO, Sérgio (2012), “Programas de inclusão na UFMG: o efeito do bônus e do reuni nos quatro primeiros anos de vigência – um estudo sobre acesso e permanência”. Educação em Revista, v. 28, n. 04, pp. 317-345.

ARAÚJO, Fernando; SIQUEIRA, Liedge (2010), “Determinantes do desempenho escolar dos alunos da 4ª série do ensino fundamental no Brasil”. Economia e Desenvolvimento v.9, n. 1, pp. 70-103.

BAI, Chong-em; CHI, Wei. (2010), “Determinants of undergraduate GPAs in China: college entrance examination scores, high school achievement, and admission route”. MPRA[S.l.] v.5, n. 1.

BARROS, Aparecida. (2015), “Expansão da educação superior no Brasil: limites e possibilidades.” Educ. Soc. v. 36, n. 131, pp. 361-390.

BARROS, Ricardo; MENDONÇA, Rosana; SANTOS, Daniel; QUINTÃES, Giovani (2001), “Determinantes do desempenho educacional do Brasil.” Pesquisa e Planejamento Econômico v. 31, n. 1, pp. 1-42.

BEHRENDT, Amy; EISENACH, Jeffrey and JOHNSON, Willian.(1986), “Selectivity bias and the determinants of SAT scores”. Economics of Education Review v. 5, n. 4, pp. 363-371.

BETTINGER, Eric; EVANS, Bent; POPE, Devin. (2013), “Improving college performance and retention the easy way: unpacking the ACT exam”. American Economic Journal: Economic Policy v. 5, n. 2, pp. 26-52.

FERNANDES, Reynaldo; NATENZON, Paulo. (2003), “A evolução recente do rendimento escolar das crianças brasileiras: uma reavaliação dos dados do Saeb”. Estudos em Avaliação Educacional, n. 28, pp. 3-22.

FRANCIS, Andrew; TANNURI-PIANTO, Maria (2012), “Using Brazil’s racial continuum to examine the short-term effects of affirmative action in higher education.” Journal of Human Resources v. 47, n. 3, pp. 754-784.

FRYER, Roland; LOURY, Glenn; YURET, Tolga (2008), “An Economic Analysis of Color-Blind Affirmative Action”. Journal of Law, Economics and Organization v. 24, n. 2, pp. 319-355.

GOLGHER, André; AMARAL, Ernesto; NEVES, Alan (2014), “Avaliação de impacto do bônus sociorracial da UFMG no desempenho acadêmico dos estudantes”. Mediações - Revista de Ciências Sociais v. 19, n. 1, pp. 214-248.

GOLGHER, André; AMARAL, Ernesto; NEVES, Alan (2015), “Desempenho acadêmico dos estudantes da UFMG: uma análise da política de bônus sociorracial”. Ciências Sociais em Perspectiva v.14, n. 26, pp. 120-145.

GREENE, Willian (2003), Econometric Analysis. 5. Ed. Upper Saddle River: Prentice Hall.

LEONARD, David; JIANG, Jiming (1999), “Gender Bias and the College Predictions of the SATs: A Cry of Despair”. Research in Higher Education v. 40, n. 4, pp. 375-407.

LIMA, Edileusa; MACHADO, Lucília (2016), “Reuni e Expansão universitária na UFMG de 2008 a 2012”. Educação e Realidade v. 41, n. 2, pp. 383-406.

LOURY, Linda; GARMAN, David (1993), “Affirmative action in higher education”. American Economic Review v. 83, n. 2, pp. 144-158.

MACHADO, Ana Flávia; MORO, Sueli; MARTINS, Ludiemyand; RIOS, Juan (2007), “Qualidade do ensino em matemática: determinantes do desempenho de alunos em escolas públicas estaduais mineiras”. Economia v. 9, n. 1, pp.23-45.

NOGUEIRA, Claudio; NONATO, Bréscia; RIBEIRO, Gustavo; FlONTINO, Sandra. (2017),“Promessas e limites: o Sisu e sua implementação na Universidade Federal de Minas Gerais”. Educação em Revista n. 33, pp. 61-90.

PEDROSA, Renato; DACHS, Norberto; MAIA, Rafael; ANDRADE, Cibele (2007), “Academic performance, students’ background and affirmative action at a Brazilian university”. Higher Education Management and Policy v. 19, n. 3, pp. 58-77.

ROTHSTEIN, Jesse. (2004), “College Performance Predictions and the SAT”. Journal of Econometrics v. 121, n. 1-2, pp.297-317.

SALATA, André (2018), “Ensino superior no Brasil das últimas décadas – Redução nas desigualdades de acesso?”. Tempo Social, Revista de Sociologia da USP v. 30, n. 2, pp. 219-253.

SCOTT-CLAYTON, Judith; CROSTA, Peter; BELFIELD, Clive (2014), “Improving the targeting of treatment: evidence from college remediation”. Educ. Eval. Policy Analysis, v. 36, n. 3, pp. 371-393.

SILVA, Nelson; HASENBALG, Carlos (2002), “Recursos Familiares e Transições Educacionais”. Cadernos de Saúde Pública, n. 18 (suplemento), pp. 670-76.

SILVEIRA, Fernando; BARBOSA, Márcia; SILVA, Roberto (2015), “Exame Nacional do Ensino Médio (ENEM): uma análise crítica”. Revista Brasileira de Ensino de Física v. 37, n. 1, pp. 1101.

SOARES, Tufi (2005), “Modelo de três níveis hierárquicos para a proficiência dos alunos de 4ª série avaliados no teste de língua portuguesa do SIMAVE/PROEB-2002”. Revista Brasileira de Educação n. 29, pp. 73-88.

SOARES, José; ALVES, Maria (2013), “Effects of schools and municipalities in the quality of basic education”. Cad. Pesqui v. 43, n. 149, pp. 493-517.

TELLES, Edward; PAIXÃO, Marcelo (2013), “Affirmative action in Brazil”. LASA FORUM v. 14, n. 2, pp.10-12.

Notes