The Instituto Nacional de Evaluación Educativa(INEEd, National Institute of Educational Evaluation) of Uruguay proposed the development of a National Assessment of Educational Achievement system called Aristas, which aims to provide information for the design of evidence-based policies, from a broad conception of educational achievement that transcends the frontier of measuring reading, mathematics or scientific skills. It includes – among other components –, the evaluation of the SES of students in the sixth year of primary education as an indicator of some of the competences that the educational system intends to promote, as well as a potentially explanatory factor for reading and mathematics performance and the climate of coexistence and participation in educational centers.

Currently, there is a great diversity of frameworks to conceptually operationalize SES. Organizations such as the Organization for Economic Cooperation and Development define them as the individual capacities that are manifested in persistent patterns of thoughts, feelings and behaviors and that can be developed from formal and informal learning experiences that significantly influence the social and economic results of the individuals throughout their life (Organization for Economic Cooperation and Development, 2015). They can also be understood as affective, behavioral, cognitive and regulatory tools, which affect one’s emotional state and the relationship with others, and the understanding of the impact of actions on events and others. These are skills that come to play and interact to achieve goals, a key element in learning (Organization for Economic Cooperation and Development, 2015).

In the framework of Aristas, SES are defined as a set of cognitive, emotional and social tools that have the function of adapting the individual to the environment, and that facilitate personal development, social relations, learning and well-being (INEEd, 2018a; Zins & Elias, 2007). They become configured dynamically from the interaction of the individual with the context, through experiences and learning, and in this sense they are considered as the result of development processes, which extend throughout the entire life cycle (Johnson, 2008).

Our broad definition encompasses the wide variety of specific skills that have been addressed in the academic literature and the field of educational evaluation (INEEd, 2018a; Morrison-Gutman & Schoon, 2013). Three selection criteria were applied to determine the skills to be evaluated: their ability to be modified from specific interventions in the school environment (malleability criteria), their incidence on learning and school life, their relevance for the development and well-being of students. The summary of the evidence that supports our selection and grouping is reported in INEEd (2018a). Table 1 shows the taxonomy developed for our evaluation of socio-emotional skills in the sixth year of primary education and the corresponding definitions.

The objective of this paper is to present the validation of the INEEd socio-emotional skills questionnaire for sixth year primary school students. The instrument is operationalized at 3 levels, in accordance with the taxonomy reported in Table 1: the first level covers the SES subscales, the second level groups three scales (Interpersonal Skills, Intrapersonal Skills and Motivation and Self-regulation), and the third level refers to the full SES scale.

Stage 1: Development of Items. Based on the conceptual decisions summarized in Table 1, a list of items was made for each of the selected SES, according to their operational definition. In the first instance, the items used in consolidated instruments were used. They were identified based on a systematic review of the literature, both in English and Spanish, and the items in English were translated into Spanish and back-translated. Seeking to address all the proposed constructs, in a second instance, the elaboration of own new, customized items was used. To obtain inputs for the linguistic and contextual adaptation of the items, the item proposal was submitted to experts, teachers and sixth-year students from primary school. The experts received a preliminary version, together with the operational definitions; they were requested to review the items according to the following criteria: understanding of the item and adequacy in the national context; relationship between the item and the dimension it intends to measure; sufficiency of the operationalization of the dimensions (Escobar-Pérez & Cuervo-Martínez, 2008). The systematization of these data led to a first adjustment of the instrument.

In a following stage, interviews and focus groups were conducted with 40 teachers from seven regions of the country to verify the linguistic and contextual adequacy of the instrument, as well as to highlight suggestions. Finally, the individual application of the instrument was carried out with 11 sixth-grade students, in public education centers in four departments of the country. Students were required to read each item, outloud voice, and to describe in their own words their understanding and indicate their responses and the rationale for their choice. Based on this experience, the final adjustments of the instrument were made.

Stage 2: Internal Structure and Validation. The application of the instrument was made online using the one computer per child public policy, in the classroom, within the curricular schedule, administered by INEEd applicators. As for ethical requirements, the requirements of the Law on Protection of Personal Data and Action of “Habeas Data” (nº 18.331) and Decree nº 379/008 on research with humans were followed. This research is framed on the National Educational Assessment, established at the Educational Law n° 18.437 and approved by the Ministry of Education and Culture. The application of the socio-emotional skills questionnaire was part of a battery of instruments applied in the same instance, as part of the pilot and final Aristas study (INEEd, 2018b).

Results Principal Component Analysis: The Principal Component Analysis retains 8 components with eigenvalues greater than 1, accounting for 46.02% of the variance in the sample (Table 2). The first factor, corresponding to the Self-Control Scale, explains 16.55% of the variance. The factorial solution reproduces the theoretical structure of the instrument with one exception: the item “If my work doesn’t turn out well, I get upset and stop doing it” has a double load, both on the factor that corresponds to the Self-Control Scale (corresponding to its theoretical structure shows a load of 0.37), as in the Academic Perseverance Scale (negative load of -0.42). The other items are grouped according to the construct for which they were developed, showing good factor loadings ranging from 0.33 to 0.79. The Cronbach alpha values of the subscales (level 1) range satisfactorily between 0.65 and 0.74. At the second level of analysis (scales), alpha values of 0.60 are observed for motivation and self-regulation, 0.77 for interpesonal skills and 0.55 for intrapersonal skills. The SES total scale, meanwhile, shows good internal consistency (α = 0.76) (Table 3).

Internal Correlations: At the level of the scales (level 2), moderate correlations of the Motivation and Self-regulation scale with the Interpersonal (r = 0.50) and Intrapersonal (r = 0.40) Skill Scales stand out. The latter show a weak correlation of 0.17. Internally, moderate to high correlations are observed between the subscales and their relevant scales, although two exceptions stand out. First, regarding the Intrapersonal Skills Scale, a very high correlation is observed with the Self-control subscale (r = 0.98) and a low correlation with the Emotional Regulation subscale (r = 0.31). Second, the Growth Mindset subscale shows a very weak correlation (r = 0.07) with the Motivation and Self-Regulation Scale, as well as with the other components that make up this scale. It only shows a correlation – albeit a weak one –, with the Academic Perseverance subscale (r = 0.20).

External Correlations: Interpersonal skills show low but considerable correlations, with the scales of bond with the teacher (r = 0.30) and bond between peers (r = 0.41). The Motivation and Self-Regulation Scale is also correlated with the bond with the teacher (r = 0.34). Regarding the correlations of the scales (level 2) with the scores obtained in the math and reading tests, its correlation – although weak –, with the Motivation and Self-Regulation Scale (r = 0.18 in both cases) stands out, being higher compared with the correlations of these variables with the Inter- and Intrapersonal Skills Scales (Table 4).

This work tried to capture the process of development and validation a large-scale evaluation instrument of SES of sixth year primary school children in Uruguay. The SES evaluation questionnaire allows a first approach to a series of skills in our educational system, which will lead to awareness and promotion of the relevance of socio-emotional development as outcome and mediators of the learning processes fostered in school (Garcia & Weiss, 2016; Thomson et al., 2018; Weissberg et al., 2015).

The instrumental development process presented in this work led to an instrument with satisfactory psychometric characteristics, in terms of its factor structure and internal consistency of the full scale (α = 0.76), and the subscales (with α values ranging from, 0.65 and 0.74). The scale that groups Interpersonal Skills (α = 0.72) is also considered good and the scale that groups skills related to Learning Motivation and Self-regulation (α = 0.60) is acceptable. The internal consistency of the Intrapersonal Skills scale is considered insufficient (α = 0.55) and is due to the very low correlation between its two subscales. Therefore, the Self-control subscale explains almost all the variance of the Intrapersonal Skills Scale. Based on these data, it is suggested to review the Emotional Regulation subscale and to expand the specificity of its evaluation, which under its current conceptual definition is limited in scope, encompassing only the cognitive strategies of emotional regulation (INEEd, 2018a), which leaves out a series of strategies focused on the background of the emotional response (Gross, 1998), and can explain the relative psychometric weakness of this subscale. In addition, certain indicators of the scales convergent validity were observed, obviously in those that were more reliable, and positive correlations between Interpersonal Skills and indicators of social well-being such as the bond with teachers and with classmates (INEEd, 2018b; Wentzel, 1998). The Motivation and Self-regulation Scale, on the other hand, shows a considerable correlation with the relationship reported by the students with their teacher, which is consistent with the adjustment to the normative imperatives of school culture and the correct interaction according to social norms (INEEd, 2018b; Viscardi & Alonso, 2013). Both scales at the same time demonstrate a weak but positive relationship (ranging from r = 0.17 to 0.18) with the scores obtained on the math and reading tests. Considering the multi-causality of academic performance (Garcia & Weiss, 2016; Richardson, Abraham, & Bond, 2012), the observation of correlations of this size is considered relevant, highlighting, however, the need to deepen further studies through multilevel models and structural equations.

The data found yield relevant data for the development of this type of measurements. First, persistent difficulties encountered for the Growth Mindset ability are highlighted. We consider that these may be due to the fact that this construct, although it has several antecedents for educational intervention, has been rarely evaluated (Sisk, Burgoyne, Sun, Butler, & Macnamara, 2018); we therefore have few inputs for the development of our instrument (Dweck, Chiu, & Hong, 1995). In turn, the item qualitative validation stage revealed the students’ difficulties in understanding the concept of intelligence, being mainly associated with academic behaviors matching the definition of “a good student”.

Second, despite the initial conceptual definition of the constructs to be measured, in their operationalization a certain overlap between skills could not be avoided, resulting, among others, in the double factor load, indicating the relevance of the item “If my work doesn’t turn out well, I get upset and I stop doing it” for the measurement of the concept of Self-control, as well as for Perseverance (even with a negative load, as it corresponds conceptually). Academic perseverance implies putting into play a series of self-control processes for performing academic activities (Gordeeva et al., 2017), for which reason it is expected that this conceptual relationship also manifests itself in the psychometric results of the instrument. However, it highlights one of the challenges presented in the evaluation of SES, which should be – according to McKown (2015) –, specific enough to capture the SES essential for socio-emotional development, while remaining comprehensive.

Finally, throughout the process of elaboration and validation of the scales, difficulties were found with the inclusion of inverse items. These difficulties may be due to different aspects. On the one hand, Suárez-Alvarez et al. (2018) state that the cognitive processes required to answer inverse items, are usually different, benefiting participants with better language skills. On the other hand, there is the principle of acquiescence (the tendency to indicate according to the series of items on the scale) or attention problems to understand the subtlety of the inverse item (van Sonderen, Sanderman, & Coyne, 2013). The scale was not elaborated from the beginning to address these difficulties, for example using a balance between direct and inverse items, as suggested by Ferrando & Lorenzo-Seva (2010). Therefore, and considering this controversy, it was decided to use scales composed of items in a single sense to improve the psychometric quality of the instrument.