Introduction

The fast and important development observed by researchers in various areas of study, including psychology, has demanded special interest in how people perceive their worlds, how their memories are formed, and how their thoughts occur. This evolution is due to the dedication of psychologists aided by improved technology through the advent of computer science, artificial intelligence, and neuroscience, which produced a combination of perception, memory, thought, and information processing: human cognition (Casey & Moran, 1989). Cognition refers to the knowledge and study of the brain and behavior of individuals, seeking to unveil the mysteries surrounding the human mind (Eysenck & Keane, 2017).

Understanding how individuals collect, store, and interpret their information and how it is used in the different interactions of human activity is part of cognition (Lachman et al., 2015). In what is related to science problems, a concern emphasized by Maldonado (2017) consists of knowing how information is processed by individuals, given that the way they use their information determines the capacity with which they learn and adapt. With this, the interest in understanding how the human brain and mind work increased considerably (Eysenck & Keane, 2017), inserting cognition in studies from different areas such as language (Anthonissen, 2020), management (Liu & Wang, 2020), and survey methodology, particularly pretesting methods, favoring the recognition of problems in surveys (Ikart, 2018; Lenzner et al., 2024).

The relevance of information processing in data collection and interpretation stems from the fact that, in this process, different problems may lead to serious errors in the survey, such as the deviation of a response from its true meaning (Story & Tait, 2019). Hence, the questions need to be well formulated and understood by both the respondents and the interviewers, minimizing the survey errors (Miller et al., 2014) and producing valid and reliable information. To identify the cognitive difficulties in the questions, pretesting is an important means to offer information on a wide variety of problems that survey questions may contain (Miller et al., 2014; Lenzner et al., 2024; Campoamor et al., 2024).

Presser et al. (2004) highlights a paradox found in the literature, stating that, on the one hand, the pre-test is the only way to assess in advance the problems of a questionnaire, pointing it out as indispensable, and, on the other hand, some literature presents it only as a research step, not providing the due importance and development. Furthermore, the authors mention that there is little literature on which pretest research is based. Willis (2018) corroborates, stressing that the pre-test is an important element for research and its use is widespread, however, in practice, little is known about its real effectiveness and reliability, thus, it is a promising area for research, and these are necessary to analyze best practices on pre-testing.

The literature on cognitive methods in pretesting still lacks studies for the consolidation of the theme (Tontini et al., 2023). Forsyth and Lessler (2004) and Campoamor et al. (2024) pointed out that a continuous assessment of survey methods must occur, verifying it they really identify valid components of the response process to produce results with quality, as well as to ensure that the pre-testing practices incorporate the investigations. Regarding the existing pretesting methods, a new perspective has emerged to contribute to minimizing survey errors: using eye tracking in cognitive interviews. This eye-tracking technique has been obtaining satisfactory results in detecting problems in survey questions (Galesic, 2018). Eye tracking consists of understanding and observing the cognitive process of an individual through visual attention (Luan et al., 2015).

In this sense, this study aimed investigate the effectiveness of eye tracking associated with the cognitive interview in identifying survey pretesting problems. Therefore, a quasi-experiment was carried out using eye tracking and cognitive interviews, and the General Decision-Making Style Inventory (GDMS) questionnaire was employed to collect the data.

This contributes to the advancement of literature and scientific practice on research pre-testing and its methods, such as cognitive interviewing and eye tracking. It is worth noting that many management decisions are made based on research data, therefore, it is essential that this data is of quality, valid, reliable, and legitimate (Lenzner, Hadler, & Neuert, 2023). The pre-test helps by revealing these problems in the questions so that they can be improved and provide quality information. With that, it can minimize of unnecessary research costs through efficient allocation of resources, how to gather redundant data. It also guarantees greater satisfaction for the respondent, research participant, with greater clarity in the questions and relevance of the instrument used.

Furthermore, Campoamor et al. (2024) indicate that a continuous discussion about pre-testing is necessary, demonstrating its concept and guiding application processes, in order to ensure that pre-testing practices incorporate investigations. Therefore, this study favors the literature in question and presents methods for its application.

Review of Literature

The idealization of research is not a simple task, it involves obtaining information, knowledge, and practical evidence. Surveys need clear questions, so that all participants understand it in the same way and provide reliable and reproducible results. For this, researchers must put themselves in the position of respondents, developing quality research (Story & Tait, 2019).

The understanding of the process inherent to survey questions and the way individuals respond to a question comes from social and cognitive aspects. In 1980, groups of psychologists and methodologists began studying and developing strategies to increase the validity of the responses obtained in their surveys, developing methods to pretest the questions before being applied. From the pretests, data are obtained from the respondents about the entire cognitive processing to respond to a question, i.e., how they process the thought, feeling, belief, and experience as they respond to a question (Willis, 2004; Thelk & Hoole, 2006). Pretesting aims to examine if the questions are being understood correctly as planned, emphasizing quality in the data collection method (Hilton, 2017; Ikart, 2018).

The development of survey pretesting methods allowed data collection instruments to be adapted or created to identify and correct existing problems. Such methods comprise cognitive interviews, behavior code, response latency, vignette analysis, and other forms highlighted by different authors (Presser et al., 2004; Beatty & Willis, 2007; Galesic, 2018). Consequently, such methods raised questions about their efficacy in measuring surveys and if a combination of them exists (Presser et al., 2004), moving new studies on the theme, as is the case of this research. The application of a cognitive interview corroborates the decrease in measurement errors or difficulties in survey questions, being considered one of the most relevant and usual methods (Beatty & Willis, 2007). Moreover, the eye-tracking technique has been used recently to detect problems in survey questions (Galesic, 2018).

The initial paradigm of the cognitive interview method is avowedly psychological, originated from an interdisciplinary seminar entitled “Cognitive Aspects of Survey Methodology”. According to Tourangeau (1984), a cognitive interview is a form of semi structured interview focused on the thought processes of the respondents and their responses to the survey questions. There are two more usual cognitive interview techniques denominated by the think-aloud verbal protocol or verbal probing procedures (Willis, 2004).

The think-aloud verbal protocol seeks to promote the verbalization of the participants about their thought processes, through a cognitive interviewer who acts as a facilitator in the generation of this information while the participants are answering the research questions. This method involves asking, for example: “What are you thinking when answering this question?”, encouraging respondents to verbalize what they are thinking (Beatty & Willis, 2007).

Verbal probing is a variation of the think-aloud verbal protocol method, having similar instructions (Willis, 2018). This method proposes the practice of intensive interviews with follow-up, asking, for example, if the respondent can say in his own words what the question was asking, that is, they are interviews formulated by the researchers to obtain additional information about the research questions (Beatty & Willis, 2007).

Moreover, a primordial step is assessing the product of the interviews or the verbal text (Beatty & Willis, 2007). Different approaches exist to encode the data, but most are based on the response process model by Tourangeau (1984), in which the problems found are classified according to the four steps of the response process: comprehension, information retrieval, judgment, and response (Demaio & Landreth, 2004). The four steps that make up the model do not need to be followed in a linear way, the process may involve some attributes and others may be ignored, having different interactions. The cognitive effort employed in the answers (going through all the processes or not) depends on the difficulty and complexity of the question (Groves et al., 2009).

The comprehension stage is related to the interpretation and understanding of what the question means and the underlying answer. After understanding the question, the interviewee moves on to a second step, referring to the retrieval of relevant information. People seek to retrieve information from their memories so that an appropriate response can be generated. Thus, when designing a survey, one should ask what information the interviewee needs to retrieve in order to answer the question (Willis, 2004). The third step in the process is judgment. This step is necessary because the interviewees need to combine all the information acquired through memory to then make a judgment. At the end of the process, it is time for the interviewee to select and inform his/her answer. To make the final judgment, respondents are asked to adjust and edits to the response categories offered by the survey. After the trial, individuals adjust their response to the response options provided (Tourangeau, Rips, & Rasinski, 2000). Thus, the model presented is the most general and complete.

On the other hand, Presser and Blair (1994) highlighted that the existing pretesting methods still need systematic analysis about the reliability, effectiveness, and validity of their procedures. In view of this, from the end of the 20th* century, a new perspective of studies in this segment emerged: interest in eye tracking. In 1967, Yarbus introduced the eye-tracking technique applied to the understanding of different problems. From this, Just and Carpenter (1980) presented the eye-mind hypothesis, assuming that the eye remains fixed on an object or word while the mind processes this image, hence demonstrating an connection between eye movements and cognition, given that eye tracking provides subsidies about the processing of the information and the cognitive effort demanded by people in this process.

Duchowski (2017) points out that eye tracking is important so that a part of the field of view can be visualized with high definition, observing with more detail and precision the object that is seen. Consequently, by tracking the movement of a person's eyes, it is possible to verify the observer's attention to a certain path taken by the eyes. With this, it helps in understanding what most interested the observer and his perception of what he saw.

From this relationship found with cognitive processing, more recently, eye tracking is inserting itself in the methodological field of surveys. Eye tracking favors the observation of the eye movements of people when they are submitted to responding to a research instrument, allowing to verify, for example, where, for how long, and what the respondent was visualizing while responding to a question (Neuert, 2016; Galesic, 2018).

Studies such as that of Neuert (2016) seek to demonstrate the relationship between eye tracking and the survey methodology. On the other hand, Lenzner et al. (2014) investigated which answer options in closed-ended questions lead to an improvement in usability and ease of responses in web surveys. In this sense, we sought to investigate the use of eye tracking associated with cognitive interviews in survey pretesting.

Materials and Methods

From the presented objective, we conducted a piece of explanatory or causal research proposed to respond to the causes of a given event. One of the ways to understand the causes of a given phenomenon is the conduction of an experiment; hence, this study is a quasi-experiment, but the term experiment will be used for better reading fluidity (Sampieri et al., 2013).

Research Model

The following research problem was considered: Is eye tracking associated with cognitive interviews an effective complement in identifying survey pretesting problems? Next, the research model (Figure 1), the variables composed by the model, and the hypotheses are presented.

The experimental subjects were stricto senso graduate students in the field of Administration and Public Management from the Federal University of Santa Maria, the selection of which was carried out by convenience in the manner most accessible to the researcher, using the non-probabilistic sampling method (Hair et al., 2005).

Hence, based on recent studies employing eye tracking (Tagliapietra, 2018; Bender, 2019), forty subjects were surveyed, with twenty participating in the experimental group and twenty in the control group. They were submitted to an experimental task that consisted in responding to a survey pretesting questionnaire described later.

As for the level of treatment presented to the subjects, the inter-subject’s approach was used, where only one group, the experimental one, received the treatment of the independent variable. In this case, the distribution is random and each subject is assigned to only one group. To guarantee the equivalence of the groups and any systematic bias, a simple random distribution was used, where by means of a coin, the group that the individual will perform the task is determined (Cozby, 2003), thus, when tossing a coin and the result is heads, the subject will perform the experimental task, and if the result is tails, the control task is performed.

As an independent variable, there is the use of eye tracking, therefore, a group of participants underwent the experimental treatment with the use of eye tracking and another group without the use of the tool. Regarding the dependent variables, we used the number of problems identified, the types of problems identified, and the number of problematic questions found, which were analyzed after the execution of the experimental task. Such variables stem from the work carried out by Neuert (2016), who used eye tracking and cognitive interviews to pretest survey questions.

The research model exposes that, when responding to a survey pretesting questionnaire while performing the task, the subjects went through a response process composed of four steps with distinct interactions: comprehension, information retrieval, judgment, and response (Tourangeau, 1984). Krosnick (1991) highlighted that precise and reliable responses may only be expected when the respondents advance thoroughly through the four proposed steps, which may demand much cognitive effort depending on the question to be responded.

After the individuals responded to the survey pretesting questionnaire, they underwent cognitive interviews to analyze the other problems found in the resolution of the pretest, besides probing questions. The cognitive interviews took place with all experimental subjects, i.e., for both groups, experimental and control. Probing questions were elaborated in the interview for the respondents to explain in their words further information on their responses, obtaining additional data on the questions. Moreover, cognitive interviews serve to obtain a perception about the cognitive process involved while responding to a survey, upon assessing how the respondents interpret the questions, retrieve relevant information in their memories, carry out judgments, and map internal responses (Willis, 2018).

Regarding applying cognitive interviews to reduce measurement errors or difficulties in survey questions, it is considered one of the most usual methods (Beatty & Willis, 2007). With this, eye tracking has also been used to detect problems in survey questions (Galesic, 2018). Neuert (2016) emphasized that using the two methods is effective in survey pretesting for identifying problems, especially eye tracking, given that, with it, it is possible to obtain an image of the response process and mage more targeted probing questions, besides detecting problems that the respondents may not know how to explain verbally.

The following analysis parameters were used with employing eye tracking and subsequent cognitive interviews: time and number of fixations, regressions on the text and scale (going back to reread some question or term), regressions from the response to the text, and response correction. Regarding the non-use of eye tracking and the application of the same online questionnaire followed by cognitive interviews, the time to respond to a question, response corrections, shown signs of uncertainty, and requests for clarification or some other observations will be analyzed. Hence, from the theoretical background, variables and their relationships to be analyzed were defined and tested through the following hypotheses (Table 1):

The laboratory environment was used for executing the study with the goals of having more considerable control of the variables, having more precision, and maximizing the internal validity of the experiment, producing data properly stemming from the experimental manipulation proposed and constructed by the researcher (Hair et al., 2005).

The experimental task

The experimental task consisted of the individuals responding to a survey pretesting questionnaire after agreeing with the Free and Informed Consent Form. Hence, the task presented itself under two forms of execution: a) using eye tracking; b) not using eye tracking.

Task using eye tracking

The task using eye tracking began after adjusting the eye-tracking equipment to the head of the participant, who was seated on a chair in front of a computer, following the procedures recommended by Pupil Labs, the manufacturer of the Pupil Core Eye-Tracking Headset used in this study. Hence, the detection of the pupil and the calibration of the equipment were carried out for the proper use of the eye tracker. For detecting the pupil, a red circle is formed around the pupil border, and a red dot appears at the center of the pupil, marking an appropriate detection. For calibrating the eye tracker, the subjects were asked to look at five targets displayed on the computer screen (Kassner et al., 2014).

After executing the procedures presented above, the participants in the experimental task responded to an online questionnaire with twenty-five questions through the Google Forms platform as they would in a normal environment; however, if they found any difficulties or problems, they had to point them out to the researchers. While the participant responded to the questionnaire, a follow-up and registration of any observed peculiar reading behavior were carried out on an adjacent screen. These data were used later to ask the participant about the reasons for a given peculiar behavior.

After the participants responded to the twenty-five questions, we proceeded to conduct the cognitive interviews, with general pre-programmed and probing questions, seeking to understand certain peculiar behaviors identified during the previous step. In this phase of the experiment, the participant responded again to seven questions of the questionnaire, also online, i.e., 28% of the total questions of the original questionnaire, chosen randomly through randomization carried out in Microsoft Office Excel®, in a sample generated from random numbers without repetition, with the choice of this selection criterion based on the study by Neuert (2016).

The general pre-programmed questions used to conduct the cognitive interviews sought to understand how the participant responses took place, for example, "how easy or hard was your response?". These questions were devised previously by the researchers. Besides the programmed questions in the cognitive interview, the additional probing questions were made in case any peculiar reading behavior had been identified during the experimental task for any of the eighteen remaining statements of the questionnaire. The use of probing questions made from behaviors observed by the researchers is valid in the sense of maximizing the chances of the participant expressing problems spontaneously or on their own (Conrad & Blair, 2004).

Regarding the probing questions, these were applied according to the identification of each subject's difficulties at the time of the experimental task. In these cases, some probing questions were prepared prior to the application (for example, why did you take so long to answer this question? what reason?), for use in the progress of the task, in addition to probing questions formulated at the time of the task, according to the need.

We emphasize that the procedure of having the participant respond again to seven questions of the questionnaire was used for them to recall their initial thoughts when they responded to the questionnaire during the task. In this context, right after resolving each question, the participant was interviewed about that question to which they had just responded. This procedure was adopted until all seven questions had been responded to.

About the identification of difficulties found by the participants, the indicators used to observe them, denominated peculiar reading patterns/behaviors, were the following: long fixations observed in eye tracking, text reading regressions, and response correction. These peculiar reading behaviors were monitored throughout the resolution of the questionnaire (on the computer screen) in the experimental task and, when observed, were noted for later analysis during the cognitive interview. Hence, these annotations were made while the subject executed the experimental task of responding to the online questionnaire. In the case any peculiar behavior was identified, an interview was also carried out for the question for which this behavior was identified, corresponding to one of the eighteen remaining questions of the questionnaire. After completing the experimental task of responding to the online questionnaire and giving the cognitive interview, the session was closed.

Task not using eye tracking

In the task not using eye tracking, the control condition, the participants responded to the same online questionnaire used in the task using eye tracking, however, instead of monitoring and recording peculiar reading behaviors on a screen adjacent to the computer, the researchers observed the resolution of the questionnaire from a distance, signaling any peculiar response behaviors for the interview.

After the resolution of the online questionnaire, a cognitive interview was carried out from the pre-programmed questions for the seven questions specified in the interview protocol. For the other questions (eighteen), probing interviews were only conducted in the following cases observed by the researchers: if a subject demanded a long time to respond to a given question, made a response correction, demonstrated signs of uncertainty, requested the clarification of a question, or other observed behaviors.

Questionnaire

The execution of the experimental task included the resolution of a survey questionnaire to identify the problems in the survey questions. Hence, we chose to use a questionnaire that verifies the decision-making of the individuals. The questionnaire refers to the General Decision-Making Style Inventory (GDMS) model by Scott and Bruce (1995). Scott and Bruce (1995) stated that the decision-making style of an individual is related to the habits of reacting to a specific and decision. The authors identified five decision-making styles: rational, intuitive, dependent, procrastinator, and spontaneous.

The inventory created by Scott and Bruce (1995) has twenty-five items and uses a five-point scale that varies from totally disagree to totally agree, composing the five decision-making styles. In Brazil, Löbler et al. (2019) conducted the reverse translation of the instrument and a systematic analysis by a specialist in decision-making processes, then performed the pretesting and statistical validation of the model. Beyond the investigations carried out by these authors, the translation and transcultural adaptation to the Brazilian context based on Beaton et al. (2000) and the General Decision-Making Style Inventory were applied in this story, given that, in the model validated by Löbler et al. (2019), the reverse translation and confirmatory data analysis steps were not specified. Hence, with the Translation and Transcultural Adaptation, the final version of the General Decision-Making Style Inventory was obtained (Table 2).

Data collection and analysis

To meet the proposed goals and the listed hypotheses, the data were obtained using basically two instruments: eye tracking and cognitive interviews. Hence, the Pupil Core Eye-Tracking Headset hardware and the Pupil Capture and Pupil Player software were used for the task using eye tracking, while the data were collected through cognitive interviews for the task without eye tracking. A pretesting questionnaire was applied for both tasks.

The data were analyzed descriptively through inferential statistics. Hence, the quantitative data from eye tracking and qualitative data from the cognitive interviews were organized, tabulated, and analyzed using Microsoft Office Excel® and IBM SPSS 20.0. The analyses were based on three main issues: the identification of the number of problems, the types of problems, and the number of problematic questions, as per the study by Neuert (2016).

Relative to the task using eye tracking, from the raw data generated by exporting the videos to the Pupil Player software, we calculated the number of fixations, the duration of the fixations (minimum, maximum, mean, and standard deviation), the number of text and scale regressions, the number of regressions from the response to the text, and the number of response corrections for each of the twenty-five statements of the questionnaire. With this, the criteria calculated based on the upper quartiles were selected for each of the statements. These data were analyzed through basic descriptive statistics (mean, median) according to the particularity of each criterion. With this, it was possible to identify the problematic statements. About the task not using eye tracking, the data were obtained and analyzed through observations made during the execution of the experimental task, i.e., the time to respond to the question, the number of response corrections, the number of signs of uncertainty, the number of requests for clarification, and the number of other observations during the task. In this sense, the problematic statements of the questionnaire were identified the same way in this task.

The cognitive interviews and the probing questions that emerged for each subject were carried out with and without eye tracking. These interviews were considered through interpretative analysis, as per Yin (2016). The data were compiled, decomposed, encoded, labeled, interpreted, and analyzed (conclusion). The interpretation is defined as assigning proper meanings to the data, articulating all steps performed in the qualitative data.

With the analysis of the interviews for the tasks with and without eye tracking, it was possible to identify the number of problems and types of problems in the problematic statements of the pretesting questionnaire. The types of problems were encoded from a problem classification scheme adopted by Neuert (2016) based on several existing schemes (Presser & Blair, 1994; Demaio & Landreth, 2004). This scheme contains twenty-nine problem codes grouped into the four steps of the process of responding to a question as per Tourangeau (1984).

Non-parametric statistical tests were performed to verify the hypotheses. Hence, the chi-square test was applied to compare the observed and expected frequencies, stemming from the hypothesis that there is no association between the analyzed variables (Sampieri et al., 2013). Because the number of cells is small, Fisher's test was applied as it is more precise for all sample sizes (Agresti & Finlay, 2012). With this, the analysis and discussion of the results found are presented next.

Results and Discussions

Identification of the number of problems, types of problems, and problematic questions

To analyze the use of eye tracking associated with cognitive interviews identifies the number and type of problems and problematic questions in pretesting, we based ourselves on the work by Neuert (2016). In this sense, from the conduction of the experiment in tasks with and without eye tracking, it was possible to identify the problematic questions (Table 3).

Through the data in Table 3, it is possible to verify seven statements (S.1, S.2, S.4, S.5, S.9, S.20, and S.25) identified as problematic in the tasks using eye tracking associated with cognitive interviews, with two of them (S.9 and S.25) not coinciding with the previously selected statements through randomization for the cognitive interview performed for all individuals. The five other statements pointed out as problematic coincided with the previously selected statements that received the general pre-programmed questions. In this sense, a higher number of problematic questions was identified using eye tracking than in the control condition without using the tool, given that only one statement (S.9) presented a problem. Moreover, upon analyzing if both methods identified the same questions, it was verified that statement S.9 emerged as problematic in both tasks.

Hence, to verify the number and types of problems in the pretesting, the cognitive interviews conducted through the pre-programmed questions and the questions that emerged as problematic (probing questions) in both experimental tasks were analyzed from their encoding. In this sense, the data are made evident in Table 4, in which 108 problems resulted from eye tracking associated with cognitive interviews. The control group without eye tracking resulted in 81 problems. These problems were also categorized as the statements were related to the interview with pre-programmed questions or selected from peculiar behaviors.

With this, one may once again observe that the number of identified problems is higher in the task using eye tracking associated with cognitive interviews than in the task without the aid of the tool. These problems were classified relative to the types of problems adopted by Neuert (2016) and proposed by other authors (Presser & Blair, 1994; Demaio & Landreth, 2004). The problems were grouped into the four steps of the process of responding to a question as per Tourangeau (1984).

One may notice in Table 5 the types of problems categorized from the task with and without eye tracking. In general, more problems classified as comprehension problems (66.67%) were identified in both tasks, with 67 comprehension problems stemming from eye tracking and cognitive interviews (experimental task) and 59 comprehension problems after only the cognitive interviews carried out in the control group. Next come the problems with response selection (14.81%), information retrieval (13.76%), and judgment (4.76%). These results are important because they demonstrate more specifically the problems of the questions analyzed, enabling their improvement more effectively.

The comprehension problems are related to the content and structure of the question and the duration. From the analysis of the interviews, most of the problems pointed out by the pretesting participants were related to vague/little clear questions, undefined/vague terms, topic inherited from the previous question, objectively wrong response/poorly comprehended question, multiple questions in one/several subjects, long recall period (unknown information), and uncertainty that the category reflects their opinion. This information was analyzed according to the cognitive interview conducted.

As emphasized, the types of problems listed are classified into the steps of the response process; therefore, it is valid to highlight that all steps of the process of responding to a question are important and, at the same time, do not need to be followed linearly (Tourangeau, 1984). On the other hand, this analysis by the researcher in performing pretesting or elaborating questions of an instrument is primordial because the cognitive effort demanded in the responses (going or not through all steps of the response process) depends on the difficulty and complexity of the questions (Groves et al., 2009). Moreover, more precise responses and, consequently, data with more quality occur when the respondents go through the four steps of the process (Krosnick, 1991).

Quantitative analysis - hypothesis tests

According to the objective that sought to investigate the effectiveness of eye tracking associated with the cognitive interview in identifying survey pretesting problems, the hypotheses of this study were presented and tested from a statistical analysis. To test the Hypotheses, we proceeded to the analysis of the chi-square and Fisher's tests.

In H1, the problematic questions initially identified in the tasks with and without eye tracking were considered, as well as the recommendation by Fowler (1992) also used by Neuert (2016) that a given question is problematic if at least 15% of the participants presented some problem with the item. In this sense, if at least three individuals had problems with the statement, it was selected for analysis. Hence, we have:

H1: there is an association between the number of problematic questions identified and using eye tracking.

The Chi-square test is based on the difference between observed and expected frequencies to evaluate the association between variables. Because of the smaller sample size, if any expected frequency is below five, Fisher's assumption is applied (Table 6).

From the data, one cell presented an expected value smaller than five, so the result of Fisher's test was used. In this sense, given that the sigma value (sig=0.008) was lower than 0.05, this result allows concluding that there is an association between the number of problematic questions identified and using eye tracking.

The observed results and those highlighted in the task using eye tracking complement each other, corroborating the statement that using eye tracking identifies a larger number of problematic questions. Moreover, the results agree with the findings by Neuert (2016), who emphasized that using eye tracking allied to another method such as cognitive interviews is beneficial to testing survey questions.

The cognitive interview method and its techniques such as probing and the think-aloud verbal protocol are commonly used and effective methods; however, eye tracking is a discrete and non-reactive way that allows detecting conscious and unconscious reactions from the individuals, providing objective information about the survey questions and the response process (Neuert, 2016). In addition, with eye tracking, it is possible to understand if the respondent read all pieces of information and instructions of the instrument, given that the respondents do not report or demonstrate this data (Galesic, 2018). On the other hand, for survey pretesting, eye tracking alone is not appropriate as an independent technique because it does not provide information on the exact problem and its causes (Neuert, 2016).

To test Hypothesis 2, that “Eye tracking associated with cognitive interviews identifies more problems in survey pretesting”, we also proceeded to the analysis of the chi-square and Fisher's tests. In this analysis, we considered the results obtained in the conducted interviews and the number of problems identified from the problematic questions identified in both experimental tasks. In this sense, to test the hypothesis, we have:

H2: there is an association between the number of problems identified in survey pretesting and the use of eye tracking associated with cognitive interviews.

For this analysis (Table 7), the data were organized through the identification or not of a problem (dichotomous analysis).

It is possible to observe that Statements 5 and 20 have sigma values above 0.05 (S.5: sig=0.521; S.20: sig=0.474), and the other statements (1, 2, 3, 4, and 11) present values lower than 0.05. In addition, Statement 9 is a constant, which is why no statistics were calculated. In this sense, the existence of an association between the number of problems identified in survey pretesting and the use of eye tracking associated with cognitive interviews is confirmed for statements S.1, S.2, S.3, S.4, and S.11.

To test Hypothesis 3 (Table 8), that “Eye tracking associated with cognitive interviews identifies the types of problems in survey pretesting”, we proceeded to the analysis of the chi-square and Fisher's tests. In this analysis, we considered the results obtained by the conducted interviews and the classification of the types of problems from the problematic questions identified in both experimental tasks. Hence, we have:

H3: there is an association between the number of problems identified in survey pretesting considering different types of problems and the use of eye tracking associated with cognitive interviews.

From the data, one may verify that Statements 3 and 5 sigma values were over 0.05 (S.3: sig=0.139; S.5: sig=0.055) and the other statements (1, 2, 4, 11, 20, and 9) the sigma values were lower than 0.05. In this sense, there is an association between the types of problems identified and the use of eye tracking associated with cognitive interviews for statements S.1, S.2, S.4, S.11, S.20, and S.9, i.e., for these statements, it was possible to identify that using eye tracking associated with cognitive interviews allowed identifying more problems considering the types of problems identified in the study carried out.

Aizpurua (2020) pointed out the importance of obtaining valid and reliable data for making comparative estimates between populations because, even if the questionnaires are well elaborated and undergo careful translation or adaptation, the analyzed groups may differ in the way they interpret and respond to the questions, presenting a threat to the validity of the comparisons and results. Hence, pretesting is primordial to identify these potential problems in survey questions (Willis, 2018), and, allied to this, understanding what these problems are and categorizing them into problem types increases the quality of the instrument. With this, it is verified that the incorporation of eye tracking associated with cognitive interviews details these problems in survey questions, directing to the errors and misunderstandings of the questions (Neuert, 2016), as observed throughout the results analyzed in this study.

Conclusion

This study aimed to investigate the effectiveness of eye tracking associated with cognitive interviews is an effective complement in identifying survey pretesting problems. For such, through the experimental design, the subjects executed the task of responding to a survey pretesting questionnaire using eye tracking (experimental group) and without using the tool (control group).

From the conduction of the experiment and the hypothesis tests, it was verified that using eye tracking associated with cognitive interviews in survey pretesting is an important means to analyze the questions of research instruments. Using eye tracking associated with cognitive interviews, seven statements of the questionnaire were considered problematic, with five coinciding with the statements previously selected for the cognitive interviews with the pre-programmed questions and two others presenting as unexpected, an important result, given that they could have failed to be identified if only an interview had been conducted, without using eye tracking. Regarding the analysis without eye tracking (control group), conducting only the cognitive interviews, one statement was considered problematic, i.e., eye tracking associated with cognitive interviews identified more problems in the questions than just the cognitive interviews, thus being an effective complement in identifying problems in survey pretesting.

It is noteworthy that the results obtained about the types of problems identified through the problematic questions, because of the use of eye tracking associated with cognitive interview, provide essential data in the research pre-test. The researcher has the possibility to understand the problems and improve the quality of the instrument before its actual application. It is also worth noting that, even translating and adapting the Inventory of General Decision-Making Styles by Scott and Bruce in this research, in which some questions were analyzed and modified, was not enough to give full quality of understanding of the instrument. In addition, if only a cognitive interview were carried out, as usually occurs in research pre-tests, these problems could not be identified, as individuals may not verbalize these difficulties or still not be aware of this misunderstanding, and eye tracking is adequate. for that identification.

Considering the three hypotheses of this study, one may conclude that Hypothesis 1 was confirmed, given that eye tracking identifies more problematic questions in survey pretesting. Hypothesis 2 was confirmed for five of the seven statements identified as problematics, i.e., eye tracking associated with cognitive interviews allowed identifying more problems in five statements that demonstrated being problematic. Lastly, Hypothesis 3 was also confirmed for five of the seven statements that presented problems considering different types of problems.

In this sense, this study bridges gaps in the literature highlighted by different authors such as Presser et al. (2004), Forsyth and Lessler (2004), Hilton (2017), Willis (2018) and Campoamor et al. (2024), who stressed the importance of new contributions on survey pretesting and its techniques and applications. Moreover, this study corroborates the national literature on survey pretesting, highlighting its relevance not only as a methodological step to be followed in studies but as a valuable mechanism for identifying problems in research instruments. Likewise, using eye tracking associated with cognitive interviews is a recent technique for performing pretesting that maximizes the efficacy of the results, as observed in this study, which also contributes to the dissemination and application of this tool.

Scientific research, through the results and data obtained, provides assistance for making important decisions. Therefore, Lenzner, Hadler and Neuert (2023) reinforce the importance of high-quality research instruments, with greater validity and reliability. On the other hand, even evaluating a questionnaire in different ways, it is not possible to guarantee that it is excellent and free of misunderstandings. Thus, pre-testing is a means of seeking to produce adequate questionnaires, with higher quality.

It is also worth mentioning that, with the advances arising from Generative Artificial Intelligence in scientific research, including in the evaluation and pre-testing of questionnaires, Olivos and Liu (2024) emphasize that human pre-testing is essential, considering the judgment of researchers a fundamental step, adding GPT (Generative Pre-trained Transformer) feedback as an additional stage before human pre-testing. In this sense, the incorporation of advanced techniques provides a comprehensive approach to the evaluation of questionnaires, on the other hand, it reinforces the indispensable role of human judgment in the construction of research instruments.

In summary, the results obtained in this study highlight eye tracking associated with cognitive interviewing as a promising methodological tool for improving research pre-tests, with the potential to positively impact the quality and accuracy of the data obtained. These methodological advance offers researchers a refined approach for building more appropriate and well-structured data collection instruments, allowing a deeper understanding of participants' response processes and, consequently, increasing the rigor of scientific investigations.

This study presents some limitations. Using eye tracking provides some challenges, such as the use of additional equipment with a greater processing capacity for collecting and analyzing the data, also, in some cases, not all eye movements are recorded, due to head movement. Moreover, despite having assistance in international literature, it is still little researched at the national level. Another limitation is related to the type of applied research (experimental), as it is difficult to reproduce real-life situations, in this case the simulation of answering an online questionnaire. Thus, there is the possibility of not achieving full control of the variables.

Hence, this study provides insights to be explored in new approaches on the theme, such as the verification of the efficacy in survey pretesting of eye tracking associated with other techniques of the cognitive interview method such as the think-aloud verbal protocol.

Managerial Implications

Relative to the practical contribution, the use of instruments in different areas is verified, with their quality being primordial to collect reliable data. The techniques presented may be applied in different organizational contexts, both public and private, for analyzing research instruments important in such environments. Thus, these results may be used to verify research instruments in general, as well as instruments not yet validated in specific contexts.

Furthermore, carrying out the pre-test alone provides contributions to making more informed decisions, as, by identifying and correcting potential problems with a research instrument, such as questionnaires or interviews before its implementation, the pre-test Testing helps ensure the quality and reliability of the data collected. On the other hand, it also benefits survey respondents, who will have a well-designed instrument and questions, facilitating their understanding and response.

References

Aizpurua, E. (2020). Pretesting methods in cross-cultural research. The essential role of language in survey research, 129.

Agresti, A., & Finlay, B. (2012). Métodos estatísticos para as ciências sociais. Porto Alegre: Penso.

Anthonissen, L. (2020). Cognition in construction grammar: Connecting individual and community grammars. Cognitive linguistics, 31(2), 309-337. https://doi.org/10.1515/cog-2019-0023

Beaton, D. E., Bombardier, C., Guillemin, F., & Ferraz, M. B. (2000). Guidelines for the process of cross-cultural adaptation of self-report measures. Spine, 25(24), 3186-3191.

Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public opinion quarterly, 71(2), 287-311. https://doi.org/10.1093/poq/nfm006

Bender, C. S. (2019). O processamento da informação online e a decisão de compra em social commerce. Dissertação (Mestrado em Administração) – Universidade Federal de Santa Maria, Santa Maria.

Campoamor, N. B., Guerrini, C. J., Brooks, W. B., Bridges, J. F., & Crossnohere, N. L. (2024). Pretesting Discrete-Choice Experiments: A Guide for Researchers. The Patient-Patient-Centered Outcomes Research, 1-12. https://doi.org/10.1007/s40271-024-00672-z

Casey, G., & Moran, A. (1989). The computational metaphor and cognitive psychology. The Irish Journal of Psychology, 10(2), 143-161. https://doi.org/10.1080/03033910.1989.10557739

Conrad, F. G., & Blair, J. (2004). Data quality in cognitive interviews: the case of verbal reports. Methods for testing and evaluating survey questionnaires, 67-87. https://doi.org/10.1002/0471654728.ch4

Cozby, P. C. (2003). Métodos de pesquisa em ciências do comportamento. São Paulo: Atlas.

DeMaio, T. J., & Landreth, A. (2004). Do different cognitive interview techniques produce different results?. Methods for testing and evaluating survey questionnaires, 89-108. https://doi.org/10.1002/0471654728.ch5

Duchowski, T. A. (2017). Eye tracking: methodology theory and practice. Springer International Publishing AG.

Eysenck, M. W., & Keane, M. T. (2017). Manual de Psicologia Cognitiva-7. Artmed Editora.

Forsyth, B. H., & Lessler, J. T. (2004). Cognitive laboratory methods: A taxonomy. Measurement errors in surveys, 393-418. https://doi.org/10.1002/9781118150382.ch20

Galesic, M. (2018). Use of eye tracking for studying survey response processes 1. In Social and behavioral research and the internet, 349-370, Routledge.

Groves, R. M., Fowler Jr, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology, John Wiley & Sons.

Hair, J., Babin, B., Money, A., & Samouel, P. (2005). Fundamentos de métodos de pesquisa em administração. Bookman Companhia Ed.

Hilton, C. E. (2017). The importance of pretesting questionnaires: a field research example of cognitive pretesting the Exercise referral Quality of Life Scale (ER-QLS). International Journal of Social Research Methodology, 20(1), 21-34. https://doi.org/10.1080/13645579.2015.1091640

Ikart, E. M. (2018). Questionnaire Pretesting Methods: A Comparison of Cognitive Interviewing and Respondent Debriefing Vis-à-vis the Study of the Adoption of Decision Support Systems by Knowledge Workers. International Journal of Business & Information, 13(2). DOI: 10.6702/ijbi.201806_13(2).0001

Just, M. A., & Carpenter, P. A. (1980). A theory of reading: from eye fixations to comprehension. Psychological review, 87(4), 329. https://psycnet.apa.org/doi/10.1037/0033-295X.87.4.329

Kassner, M., Patera, W., & Bulling, A. (2014). Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction. In Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing: Adjunct publication. https://doi.org/10.1145/2638728.2641695

Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5(3), 213-236. https://doi.org/10.1002/acp.2350050305

Lachman, R., Lachman, J. L., & Butterfield, E. C. (2015). Cognitive psychology and information processing: An introduction. Psychology Press. https://doi.org/10.4324/9781315798844

Lenzner, T., Hadler, P., & Neuert, C. E. (2024). Cognitive Pretesting. GESIS, Leibniz Institute for the Social Sciences (GESIS – Survey Guidelines), version 3.0, Mannheim, Germany. DOI: 10.15465/gesis-sg_en_049

Lenzner, T., Hadler, P., & Neuert, C. (2023). An experimental test of the effectiveness of cognitive interviewing in pretesting questionnaires. Quality & Quantity, 57(4), 3199-3217. https://doi.org/10.1007/s11135-022-01489-4

Lenzner, T., Kaczmirek, L., & Galesic, M. (2014). Left feels right: A usability study on the position of answer boxes in web surveys. Social Science Computer Review, 32(6), 743-764. https://doi.org/10.1177/0894439313517532

Liu, M. S., & Wang, M. L. (2020). A multi‐level model of commitment‐based human resource practices on team performance: the team cognition model. Asia Pacific Journal of Human Resources, 58(3), 378-398. https://doi.org/10.1111/1744-7941.12220

Löbler, M. L., dos Reis, E., Nishi, J. M., & Tagliapietra, R. D. (2019). Inventário de estilos de tomada de decisão: Validação de instrumento no contexto brasileiro. Revista de Administração Unimep, 17(1).

Luan, J., Yao, Z., Zhao, F., & Liu, H. (2016). Search product and experience product online reviews: An eye-tracking study on consumers' review search behavior. Computers in Human Behavior, 65, 420-430. https://doi.org/10.1016/j.chb.2016.08.037

Maldonado, C. E. (2017). Positive affirmation of non-algorithmic information processing. Cinta de moebio, (60), 279-285. http://dx.doi.org/10.4067/S0717-554X2017000300279

Miller, K., Chepp, V., Willson, S., & Padilla, J. L. (Eds.). (2014). Cognitive interviewing methodology. John Wiley & Sons.

Neuert, C. (2016). Eyetracking in questionnaire pretesting. Thesis, Universität Mannheim, Baden-Württemberg.

Olivos, F., & Liu, M. (2024). ChatGPTest: opportunities and cautionary tales of utilizing AI for questionnaire pretesting. Field Methods. https://doi-org.ez47.periodicos.capes.gov.br/10.1177/1525822X241280574

Presser, S., Couper, M. P., Lessler, J. T., Martin, E., Martin, J., Rothgeb, J. M., & Singer, E. (2004). Methods for testing and evaluating survey questions. Methods for testing and evaluating survey questionnaires, 1-22. https://doi.org/10.1002/0471654728.ch1

Presser, S., & Blair, J. (1994). Survey pretesting: Do different methods produce different results?. Sociological methodology, 73-104. https://doi.org/10.2307/270979

Sampieri, R. H., Collado, C. F., & Lucio, P. B. (2013). Metodologia de pesquisa. São Paulo: McGraw-Hill.

Scott, S. G., & Bruce, R. A. (1995). Decision-making style: The development and assessment of a new measure. Educational and psychological measurement, 55(5), 818-831. https://doi.org/10.1177/0013164495055005017

Story, D. A., & Tait, A. R. (2019). Survey research. Anesthesiology, 130(2), 192-202. https://doi.org/10.1097/ALN.0000000000002436

Tagliapietra, R. D. (2018). Investigando a influência das recomendações de produtos online no processo de decisão de compra sob a perspectiva Neuro-IS. Dissertação (Mestrado em Administração) – Universidade Federal de Santa Maria, Santa Maria.

Thelk, A. D., & Hoole, E. R. (2006). What are you thinking? Postsecondary student think-alouds of scientific and quantitative reasoning items. The Journal of General Education, 55(1), 17-39. https://doi.org/10.2307/27798035

Tontini, J., Malheiros, M. B., dos Reis Lehnhart, E., & Rossato, V. P. (2023). Pré-teste cognitivo: um panorama da produção científica internacional. Gestión Joven, 24(3), 12-31.

Tourangeau, R. (1984). Cognitive sciences and survey methods. Cognitive aspects of survey methodology: Building a bridge between disciplines, 15, 73-100.

Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. New York: Cambridge University Press.

Willis, G. B. (2004). Cognitive interviewing: A tool for improving questionnaire design. Sage publications.

Willis, G. (2018). Cognitive interviewing in survey design: State of the science and future directions. The Palgrave handbook of survey research, 103-107. https://doi.org/10.1007/978-3-319-54395-6_14

Yarbus, A. L. (1967). Eye movements and vision. New York: Plenum Press.

Yin, R. K. (2016). Pesquisa qualitativa do início ao fim. Porto Alegre: Penso.

IR