Abstract: Screening tools in health and education provide quick indicators that anticipate diagnostic evaluation and treatment. Faced with the socio-emotional competences of children, the Ages & Stages Questionnaires: Social-Emotional (ASQ:SE) was used in the population of children enrolled in public pre-schools in the city of Rio de Janeiro during 2011. The objective of this work was to investigate aspects of its validity and reliability. Data from 23,334 children (50.6% boys) being an average age of 5 years old (SD: 3 months) and enrolled in 625 pre-schools were analyzed. After an intensive data analysis, the Exploratory and Confirmatory Factor Analysis, the results were favorable to the multidimensional model with a social and emotional dimension: χ2 (463) = 46363.495, p <0.001; RMSEA = 0.067; CFI = 0.918; TLI = 0.913. Reliability indicators were adequate. The results confirmed validity aspects of the ASQ:SE, thereby verifying its use for children aged 5 years old.
Keywords: Child developmentChild development,PsychometricsPsychometrics,ValidityValidity,Factor AnalysisFactor Analysis.
Resumo: Instrumentos de rastreio em saúde e educação oferecem indicadores rápidos que antecipam a avaliação diagnóstica e o tratamento. Frente às competências socioemocionais infantis, o Ages & Stages Questionnaires: Social-Emotional (ASQ:SE) foi utilizado na população de crianças matriculadas em pré-escolas públicas do município do Rio de Janeiro durante 2011. Posto isso, o objetivo desse trabalho foi investigar aspectos de validade e fidedignidade desse instrumento. Dados de 23.334 crianças (50.6% meninos), com idade média de 5 anos de idade (DP: 3 meses) e matriculadas em 625 pré-escolas foram analisados. A partir da Análise Fatorial Exploratória e Confirmatória, os resultados foram favoráveis ao modelo com um fator emocional e outro social: χ2 (463) = 46363.495, p < 0,001; RMSEA = 0,067; CFI = 0,918; TLI = 0,913. Indicadores de fidedignidade foram adequados. Os resultados evidenciaram aspectos de validade do ASQ:SE e possibilitam seu uso em crianças com 5 anos de idade.
Palavras-chave: Desenvolvimento infantil, Psicometria, Validade, Análise fatorial.
Resumen:
Instrumentos de rastreo en salud y educación ofrecen indicadores rápidos que anticipan la evaluación de diagnóstico y tratamiento. Ante las competencias socio-emocionales infantiles, el Ages & Stages Cuestionarios: Social-Emocional (ASQ:SE) fue utilizado con la población de niños preescolares de escuelas públicas del municipio de Rio de Janeiro en el año 2011. El objetivo de este trabajo fue investigar aspectos de validez y veracidad de este instrumento. Fueron analizados datos de 23.334 niños (50.6% de sexo masculino) con edad promedio de 5 años(DE: 3 meses) matriculados en 625 escuelas preescolares. A partir del Análisis Factorial Exploratorio y Confirmatorio, los resultados fueron favorables al modelo con un factor emocional y otro social: X2(463) = 46363.495, p<0,001;RMSEA = 0,067;CFI =0,918;TLI =0,913. Los indicadores de veracidad fueron adecuados y los resultados señalaron aspectos válidos del ASQ:SE que posibilitan su uso en niños con 5 años de edad.
Palabras clave: desarrollo infantil, psicometría, validez, análisis factorial.
Papers
Factor Structure of a Social-Emotional Screening Instrument for Preschool Children
Estrutura Fatorial de um Instrumento de Rastreio Socioemocional em Crianças Pré-Escolares
Estructura Factorial de un Instrumento de Investigación Socio-Emocional en niños en edad preescolar
Luis Anunciação
Chieh-Yu Chen
Danilo Assis Pereira
J. Landeira-Fernandez
Received: 20 December 2017
Revised document received: 22 August 2018
Accepted: 28 September 2018
Aspects of social and emotional development during childhood are widely recognized as crucial for the future development of cognitive function (Carter, Briggs-Gowan, & Davis, 2004; Saarni, Campos, Camras, & Witherington, 2008). However, child development does not always progress without complications and the early identification and detection of delays rely on developmental screening. The Ages and Stages Questionnaires: Social-Emotional (ASQ:SE) is a tool that was designed to assess these skills in 6-month-old to 5-year-old children, and is part of the ASQ System, which includes tools for assessing and monitoring child development (Squires, Bricker, & Twombly, 2002).
ASQ:SE is comprised of eight questionnaires that contain Likert-type items. Summing the items allows a rapid and objective indicator of child development with three classification possibilities: typical, questionable (or monitor), and at risk (Squires, Bricker, Heo, & Twombly, 2001). The classification of typical development occurs when a child’s score falls below the cut-off for referral, whereas delayed development is the result of a child’s score falling above the cut-off.
According to the theoretical model of the ASQ:SE, the social and emotional domains are highly correlated, but some differences can be mapped and represent different constructs. The social domain is related to the ability to negotiate social situations, and it is associated with a positive relationship among peers. The emotional domain refers to the ability to regulate one’s own emotions in different social situations. An important feature of this domain is the ability to recognize both one’s own and others’ emotions (Saarni et al., 2008).
Furthermore, seven behavioral areas can be found in its structure: (1) Self-regulation, (2) Compliance, (3) Communication, (4) Adaptive functioning, (5) Autonomy, (6) Affect, and (7) Interaction with people. Self-regulation is the ability to adjust to environmental or physiological conditions. Conformity is the ability to conform with and obey orders. Communication refers to the ability to respond to verbal and non-verbal cues. Adapted functioning is the ability to cope with physiological needs. Autonomy is the ability to take initiative. Affection is the ability to show feelings and empathy for others. Interaction is the ability to respond to parents, other adults, and peers (Laurent & Gorman, 2018; Squires & Bricker, 2009; Squires et al., 2002).
Because the theoretical model of ASQ:SE is linked to child development milestones, its technical manual (user guide) does not rely on a specific investigation of factorial structure. The focus of the manual is to provide information on the scoring of all questionnaires, developing and monitoring programs and interventions that can improve child development, among other features. Therefore, the manual content allows to understand ASQ:SE as being composed by a Multidimensional structure that comprises the seven behavioral areas previously mentioned, or a Second-order or a Bifactor model. Figure 1 describes the proposed models.
Regarding the psychometric properties, the original study of the ASQ:SE relies on an evaluation of 3,014 children aged between 6 months and 5 years old to compute indicators of validity, reliability and produce normative tables. The convergent and criterion validity were accessed with part of the sample that was evaluated by ASQ:SE, by the Child Behavior Checklist (CBCL) and by a team of professionals with doctors and psychologists. Hence, we verified their ability to identify true positives (sensitivity) and true negatives (specificity) in each of the evaluated age groups. The sensitivity varied between 0.75 and 0.89, and the specificity had an average of 0.92 (Briggs et al., 2012; Squires & Bricker, 2009). ASQ:SE has had validation studies conducted in the United States, Spain (Squires et al., 2001), and Korea (Heo & Squires, 2012). In the United States, the use of ASQ:SE has been replaced by ASQ:SE 2, which also evaluates the 2-month age group.
In Brazil, only a few instruments have been developed and used to assess development of preschoolers and there is a lack of studies about the dimensional structure of the ASQ:SE. On the other hand, ASQ:SE is becoming frequently used among professionals that work with child evaluation and treatment (Anunciação, Chieh-yu, Squires, & Fernandez, 2018; Filgueiras, Pires, & Landeira-Fernandez, 2014; Fioravanti-bastos, Filgueiras, Lucia, & Moura, 2016), which has generated the need to explore the psychometric properties of the ASQ:SE within the Brazilian sample to ensure the interpretation of its results (AERA, APA, & NCME, 2014).
As such, the present study explored the psychometric aspects of the ASQ:SE using data from 5-year-old Brazilian children. This age was chosen because social and emotional skills are more easily discriminable in 5-year-old children than in 6-month-old infants, as pointed out by specific literature (Gordon & Browne, 2014; Shulman, 2016).
This work integrates a larger study in cooperation with the Secretary of Education of the Municipality of Rio de Janeiro to develop a large-scale assessment to assess the development of children who were enrolled in public daycare centers and preschools, and to measure the effectiveness of educational public policies. This project was stopped in 2013, and one of its main objectives was to explore the psychometric aspects of ASQ:SE in the population of children enrolled in public day care centers in the city of Rio de Janeiro in 2011 and 2012 (N = 76,760).
The dataset was comprised of 23,334 children from 625 public preschools in 10 Education Coordination Regions in the city of Rio de Janeiro, Brazil. The children were evaluated during 2011. The sample included 50.6% boys and 49.4% girls. The mean age was 5 years (SD = 3 months). Each child was only evaluated once. The Research Ethics Committee of Pontifical Catholic University of Rio de Janeiro approved this study.
A 1-day, 8-hour meeting was scheduled by the Secretary of Education of the Municipality of Rio de Janeiro, and all directors of the 468 public child daycare centers were invited to participate. In each meeting, the ASQ:SE was presented by a person who was previously trained, and the directors were responsible for taking the ASQ:SE back to their centers and instructing their teachers how to administer it in their classrooms. Children were evaluated by their teachers during the second semester in 2011, and the data were gathered in an encrypted and specific website that was created for its aim. Each classroom had an average of 35 children and every child was only assessed once.
The ASQ:SE was used in this assessment. ASQ:SE is a series of eight questionnaires that are designed to be completed by parents or caregivers to address the social and emotional competence of young children (6, 12, 18, 24, 30, 36, 48, and 60 months old). Each scale has at least 21 items and scoring should be reversed for some items. All questions can be answered with “Most of time” (0 points) “Sometimes (5 points)”, and “Rarely or never” (10 points).
To be more comprehensive, a fourth column asks respondents to indicate whether the behavior is a concern to them, but it was not used in the present study. High scores suggest problem behaviors. Empirically derived cutoff scores indicate which children should be referred for further evaluation. The results can be used to both identify children with delays and as a monitoring tool for social and emotional development.
In being consistent with recommendations in the literature for cross-cultural procedures, three steps were taken to translate and adapt the ASQ:SE to Brazil. The first step was completed using a parallel back-translation procedure. Each item was translated into Brazilian Portuguese by three independent native Portuguese speakers with professional experience in English-Portuguese translation. All of the translated items were evaluated by a multidisciplinary panel of specialists with a high level of English fluency and different expertise in psychometrics and cross-cultural adaptation instruments, public child daycare systems, child development and education, economics, and public programs for low-income families.
This version was named the ASQ:SE(T), which was then back-translated into English by a native American English speaker who is fluent in Portuguese, thus creating the ASQ:SE:BR. In the second phase, the ASQ:SE:BR was compared with the original ASQ:SE, and minimal changes were made. The third and last step was a pilot test with 120 children from different public child daycare centers.
Firstly, the database was explored to identify missing cases, typing errors and inconsistency of results. Because of the volume of collected data, the listwise deletion was used for missing data, which resulted in the use of the complete dataset. The psychometric analyses were conducted in successive steps, as currently recommended when dealing with Structural Equation Modeling: (1) Specification of the theoretical model, (2) Model identification, (3) Estimation of the parameters of the theoretical model using the Confirmatory Factor Analysis, (4) Estimation of fit indices, and (5) New specification of the model, which can be carried out by Exploratory Factor Analysis and (6) Confirmation of the results obtained in the exploration (Bandalos, 2018, p. 357; Byrne, 2005; Hahs-Vaughn, 2017).
To detail this method, the first step was to perform a Confirmatory Factor Analysis (CFA) of the ASQ:SE by assigning the items according to the test user guide in an oblique multidimensional model. In this step, the two possibilities collaterally exposed - the hierarchical model and the bifactor model - were also modeled.
The second-order is a hierarchical model and is chosen when: (a) there is a hypothesis that a hierarchically superior factor of the first-order factor(s) is responsible for explaining its variability; and (b) the first-order factors are highly correlated with each other, eclipsing a factor in the variance explanation (Canivez, 2016; Chen, West, & Sousa, 2006). Second-order models impose a causal relation of the second order factor on the first factors.
The bifactor model specifies a general factor measured by all test items and the specific factors accounting for the residual variance shared by subsets of items. It is modeled when: (a) there is a theoretical hypothesis of a general factor relating to the commonality of the items; (b) there are specific factors that are responsible for the responses to the items related to its dimension; and (c) a research question relies on the effect of the general factor and the specific factors on the response to the items. Bifactor models are orthogonal in their latent traits and all observed variables are determined by both a general factor and by their specific factors. Bifactor models often better fit the data than correlated factors and higher-order models, but it comes with the price of being less parsimonious (Reise, 2012; Rodriguez, Reise, & Haviland, 2016).
Next, the model identification and the model estimation were conducted. This last step allows to test the model by looking at its fits and checks how well the proposed model fits the data. If the model does not fit the data or when its fit is poor, then the predicted model is not empirically supported by the data. An Exploratory Factorial Analysis (EFA) was subsequently performed via Parallel analysis of 10,000 permutations of the raw data and its results were used to determine the number of factors by comparing the eigenvalues for each factor from the raw data with the 95th percentile of eigenvalues from the permutations, in addition to the visual inspection. This technique is implemented when confirmatory models do not show acceptable adjustments (Barrett, 2007).
After, a sub-sample composed of 25% of the base was created and used to perform a two-dimensional CFA (social and emotional dimensions) with the results obtained from the EFA results. This two-dimensional structure model is conceptually coherent to the ASQ:SE theory. A Student’s t-test was performed in this sub-sample to check for any difference between its data and the complete dataset.
In accordance with current recommendations, the Comparative Fit Index (CFI) ≥ 0.9 and the Root Mean Square Error of Approximation (RMSEA) ≤ 0.08 were considered as an adequate adjustment for the model. The Chi-square goodness-of-fit statistic (χ2) is the most frequently cited index of absolute fit and hypothesis that the population covariance matrix is equal to the model-based estimated covariance matrix. A non-significant result indicates no statistically significant difference between the actual covariance matrix and the proposed model to explain this covariance matrix. However, its results should be interpreted cautiously because it is very sensitive to sample size and model complexity. In small sample sizes, such as when n is equal or less than 200, its results may fail to reject an unfitting model, while it may falsely reject an adequate model in a large sample (Kyriazos, 2018; Marôco, 2010). As this study relied on a large sample and complex models, χ2 results were not considered as a criterion for discarding the model.
The Weighted Least Squares Mean-Adjusted (WLS) using a probit link function and Delta parameterization was used in all analyses. This estimator is best suited for categorical data (nominal or ordinal) that are not normally distributed, such as the case of psychometric analysis. The Delta parametrization is part of probit regression, assuming scale factors as parameters in the model, which can make the interpretation of model parameters simpler and it is the default parameterization when using a WLS estimator (Asún, Rdz-Navarro, & Alvarado, 2016; Grimm & Liu, 2016). Cronbach’s Alpha and McDonald’s Omega reliability coefficients were calculated and their procedures are currently seen as part of the validation process (AERA et al., 2014).
Exploratory and descriptive analyses were conducted using the Statistical Package for the Social Sciences version number 22 (IBM, 2013) and the psychometric procedures were performed in Mplus Version 8 (Muthén & Muthén, 2017), and in R 3.5.1 (R core team, 2017), with tidyverse (Wickham, 2016), lavaan (Rosseel, 2012) and SemTools packages (Jorgensen, Pornprasertmanit, Schoemann, & Rosseel, 2018).
As previously discussed, the ASQ:SE items address seven behavioral areas correlated with each other. This was the first model tested. However, the results showed a non-positive covariance matrix. This can occur for multiple reasons: a set of variables greater than that of respondents, linear dependence, variables with constant responses, polymorphic correlation, missing data, and factors with few items (Costello & Osborne, 2011). In this regard, the dimension “Autonomy” had a correlation above 1 with Self-regulation, Communication, and Adaptive functioning, which indicated its dependence.
The Cauchy-Schwarz inequality ensures that the correlational results are in the range [-1,1]. However, this can occur because Mplus estimates the correlation between dimensions indirectly. As the software violates this limitation and presents estimated results, although inadequate, the adjustment indicators were: χ2 (443) = 45468,210, p < 0.001; RMSEA = 0.067; CFI = 0.828; TLI = 0.807.
The second order hierarchical model also had a negative matrix, with adjustments χ2 (457) = 54260.201, p < 0.001; RMSEA = 0.073; CFI = 0.794; TLI = 0.776. Results in the same direction also occurred in the bifactor model, which was not identified, and therefore did not have its parameters and adjustment indicators estimated.
As previously reported, situations like this call for changes in the models considered. Thus, we chose to perform an EFA after reaching a Kaiser-Meyer-Olkin factor adequacy of 0.91 and a significant Bartlett test of homogeneity of variance (K-squared = 83920, df = 31, p < 0.01). The EFA was modeled up to 7 dimensions, and a Parallel analysis of 10,000 permutations of the raw data was conducted to determine the number of dimensions to extract (Marsh, Morin, Parker, & Kaur, 2014). Image 2 shows the obtained Scree Plot and Table 1 shows the obtained adjustment indicators.
The results suggested the retention of 2 dimensions, indicating this model is acceptable and has a correlation of 0.532 (p < 0.01). Thus, a random sub-sample containing 25% of the original base (n = 6,706) served to perform the CFA of this model, which is exposed in Figure 3.
The adjustments of this model were χ2 (463) = 46363.495, p < 0.001; RMSEA = 0.067; CFI = 0.918; TLI = 0.913 and all items were statistically significant (p <0.01). The labeling of the dimensions were based on key-items. The dimension composed of items such as “Does your child like to play with other children?” And “Do other children like to play with your child?” was labeled as Social, and the dimension composed of the item “Does your child try to hurt other children, adults, or animals (for example, by kicking or biting)?” And “When your child is upset, can he/she calm down within 15 minutes?” was labeled as Emotional.
Cronbach’s Alpha and McDonald’s Omega were calculated for each of the dimensions, as well as for the scale as a whole. The social dimension presented a Cronbach’s Alpha of 0.91 and the McDonald’s Omega of 0.82. The Emotional dimension presented a Cronbach’s Alpha of 0.85 and McDonald’s Omega of 0.82. The scale presented a Cronbach’s Alpha of 0.93 and McDonald’s Omega of 0.87.
Thus, since the model fit indicators meet the necessary criteria, the version shown in Table 2 is considered suitable for use with Brazilian children aged 5-years old.
Psychological characteristics such as social and emotional development are of an abstract or latent nature rather than an observable and tangible. Its hypothetical concepts are never absolutely confirmed, but the psychometric analysis of the responses to test items provides a scientific way to relate these latent traits to the observed behavior. This work aimed to analyze the psychometric aspects of the ASQ:SE construct validity using data from 5-year-old children. After data analysis, it was possible to verify that the ASQ:SE measures the social and emotional domains of children with this age. Some of the main results are discussed below.
One way of monitoring child development and preventing possible delays and difficulties is through continuous evaluation of its progress and the accomplishment of its milestones. The use of instruments with scientific evidence for aspects of validity and reliability are fundamental resources for its use and for interpreting the results generated by the test. In addition to such conditions, it is recommended that such instruments have: (1) easy and context-appropriate application in which persons without specific training are able to apply and correct it, as well as (2) normative tables in which their use improve the practical utility of the tool (Fernald, Kariger, Engle, & Raikes, 2009).
The theoretical model of the ASQ:SE proposes that 7 behavioral areas are evaluated by their items and considers that child development is part of a process that, although dynamic, has some specific markers (Engle & Black, 2008; Squires et al., 2002). The empirical results within the Brazilian context did not allow for confirming this dimensional model. There are different possible explanations for this, with one of the main ones being related to the number of items needed for factorial identification. This occurred in the Autonomy and Self-regulation dimension. Factors with less than three items are unstable and eventually are not identified in confirmatory analyses (Furr & Bacharach, 2008).
Many considerations can be made when the theoretical model does not fit the data and this result often occurs in psychometrics, especially because of the development of computational psychometrics. As data analysis methods have undergone several modifications, current negative results can be positive due to new techniques (Cano, Cano, & Hobart, 2011; Ropovik, 2015). In the present study, this misfit was only true for the model that considers specific items related to the seven behavioral areas, but the empirically obtained two-dimensional structure was psychometrically adequate. In addition to this, it is widely accepted today that validity is not a property of the instrument itself, but rather of the interpretations of its results and depends on several sources of evidence (AERA et al., 2014). As ASQ:SE was developed to be a screening instrument, in which the sum of the results was built to be a quick and sensitive indicator of possible delays in developmental milestones, its use is recommended as a screening tool for infants and young children.
On this subject, the oblique model that considers a social and an emotional dimension reached appropriate indicators of its construct validity. This is also valid for reliability results. This is because the psychometric properties guarantee the results obtained by the ASQ:SE, and are fast and contextually adapted, while also being valid and reliable (Headrick & Sheng, 2013; Higham & Arnold, 2007; Pasquali, 1996).
The Social dimension has brought together items in which aspects of interaction, communication, and affection are present. Interaction with peers as well as children’s ability to communicate with others is evidence which is already known in studies on the development of social aspects in childhood. When brought together, these conditions enable the child to have a sense of confidence and security in the world (McKown, Gumbiner, Russo, & Lipton, 2009).
The Emotional dimension brought together items in which self-regulation and compliance conditions are accessed. It is known that the ability of self-regulation provides the child with the ability to monitor and evaluate their emotional states, aiming to better adapt to the environment (Squires et al., 2002). Studies show that children with Autistic Spectrum Disorder (ASD) have a low capacity in this sense since they have difficulty managing their emotions, as well as a lower ability to postpone possible bonuses and to switch between tasks (Laurent & Gorman, 2018).
The ability of compliance is also a vital trait in child development. It indicates that the child has the ability to assertively follow rules (Squires et al., 2002). In the same way as previously mentioned, children with ASD have a greater limitation regarding the understanding, respect and acceptance of orders and rules (Ekas, McDonald, Pruitt, & Messinger, 2017), which is also evidenced in children with alterations in behavior and Oppositional Defiant Disorder (Fleming, McMahon, & King, 2017).
It is well established in the literature that the first five years of life are fundamental for child development and have long-lasting consequences (Gordon & Browne, 2014). A set of evidence demonstrates that non-cognitive skills, such as social and emotional domains, are important predictors of the development of language skills, mental health competencies and academic success (Saarni et al., 2008). As risk situations during development is analogous to shift the course of an ocean liner two degrees at the beginning of a voyage and try to fix it later, the assessment of child development is the first step to fulfill the learning potential of children by ensuring health and social services when they are needed.
Child development screening tools help professionals to individually monitor the progress each child has made in their developmental markers. In addition, the results help identify those who may be experiencing any difficulty in which a diagnostic evaluation is needed. Child care starts with psychometric assessment of children’s development and growth.
The aim of the present study was to evaluate the psychometric properties of ASQ:SE with data from children enrolled in public daycare centers in the city of Rio de Janeiro in 2011. As ASQ:SE is frequently used, the investigation of its properties is fundamental to ensure the validity of interpretations of the results.
In summary, ASQ:SE had its dimensional structure explored within a SEM framework and the results of a two-dimensional structure (social and emotional) were psychometrically reliable and valid within the Brazilian context. This result is partially convergent to the specific literature of the test, which extends the model considering 7 specific behavioral areas.
It is important to note that this research is not free of limitations. Scales that rely on hetero-assessment data present inaccurate or partial information about the studied object. In addition, external indicators could not be analyzed because the data collection was only done with the ASQ:SE. However, we attempted to reduce some of the problems by cleaning the dataset used, splitting the database when different techniques were performed and using a modern statistical approach.
Despite these limiting conditions, the present study offers psychometric evidence for using the ASQ:SE in Brazilian children aged 5 years old. Thus, it is believed that the study brings important contributions to professionals who need to have direct or indirect measures on child development.
New research in which the investigation of psychometric aspects, such as Differential Item Functioning (DIF), and aspects of child development such as latent growth curve are being carried out in order to amplify the results found herein, as well as to replicate them.
Contato com os autores: Departamento de Psicologia. PUC-Rio Av. Marques de São Vicente, 225 / L 201 Rio de Janeiro-RJ, Brazil CEP: 22451-900
