In a high-tech society, education becomes a valuable resource for sustainable development. Therefore, the quality assessment of education and the search for ways to improve it are essential tasks both at the national and at the individual pedagogical collectives levels. In the international educational practice, there are various approaches for assessing the quality of university work. So, there are many publications devoted to this problem (for example, GERRITSEN-VAN LEEUWENKAMP et al., 2017; HARVEY; GREEN, 1993; NOVIKOV, 2007; SCHINDLER et al., 2015, TAM, 2001). Essential aspects in determining the quality of university educational activities are assessments of training courses quality and the students’ satisfaction with the quality of education. The analysis of publications concludes that studies on the problems of student evaluation of teaching (SET) are conducted by scientists around the world, which is evidence of their relevance for education theorists and practitioners. The articles of Benton and Cashin (2014), Kulik (2001) and Richardson (2005) presented reviews of such studies.

In Russia, SET has not been widely adopted. During the period of social and economic transformations (the end of the 1980s), the Ministry of Higher and Secondary Education of the Russian Federation introduced the practice of student interviews using the questionnaire “The Teacher in the Eyes of Students”. The appearance of this questionnaire aroused criticism from scientists and educators (GORBATENKO, 1990; ZELENTSOV, 1999; LEVCHENKO, 1990). The publications of that time noted the lack of research on psychometric properties of this questionnaire and incorrect use of questioning results for the adoption of important decisions. Regular surveys in higher education institutions have gradually ceased. At present, in connection with the accession of Russia to the Bologna process, the problem of assessing the quality of education has again become topical. However, now the development of questionnaires and their application are at the level of separate higher education institutions (for example, ZELENEV; TUMANOV, 2012; KUZNETSOVA, 2019).

Lipetsk State Technical University also began to pay attention to students’ satisfaction with the education quality. The main problem of monitoring was the choice of an evaluation tool because due to the lack of broad practice of student evaluations, publications devoted to the analysis of evaluation tools used in Russia are scarce. To assess the students’ satisfaction with the quality of the educational program, we developed the questionnaire “The Learning Process in the Eyes of Students (LPES)” and surveyed senior students. Using a 100-point scale, students assessed the quality of the learning process organization, the quality of teaching and their results of training for the 34 study courses that make up the “Applied Mathematics” educational program. The survey outcomes were investigated through correlation, factor, regression, and cluster analysis. The study results are presented in the article Kuznetsova (2019). Next, a survey was conducted using the Color Selection Method (CSM). This method is based on Max Lüscher ideas, according to which students choose the color that they associate with each academic discipline of the educational program.

The purpose of this article is, using CSM, to study the students’ attitude to the courses that make up the Applied Mathematics curriculum, and to compare the two methods (verbal LPES and imaginative CSM) to reveal their ability to identify bottlenecks in learning and teaching. This article draws on previous research on the factors affecting the satisfaction of undergraduates with the quality of the educational program in applied mathematics.

It is well known that the practice of student surveys has become widespread since the late 1960s (DARWIN, 2016) and is seen as a mean of improving teaching quality and involving students in perfecting the education process (BENTON; CASHIN, 2014; HAMMONDS et al., 2017). As noted by Nilson (2012), students have changed noticeably in recent decades. Therefore, the questions “Are the students telling us the truth?” as well as “How reliable are students’ evaluations of teaching quality?” continue to interest researchers (CLAYSON; HALEY, 2011; FEISTAUER; RICHTER, 2017; MCCLAIN et al., 2018). Indeed, outcomes of SET depend not only on teachers and university administrations but also on the students themselves: their attitude to knowledge and the ways of obtaining this knowledge (O’DONOVAN, 2017), the notion of teaching and its forms (FEISTAUER; RICHTER, 2017), and academic maturity and emotion (LYNAM; CACHIA, 2018). Therefore, it seems essential that along with the Likert-type evaluation tools (such as the Course Experience Questionnaire, or Students’ Evaluation of Educational Quality Questionnaire) there are alternative approaches based on associations and imaginative thinking. For example, Gal and Ginsburg (1994) demonstrated an example of an evaluation tool for measuring students’ attitudes toward the study of mathematics. There is a set of 12-15 cards showing faces with different emotions (“anxious”, “puzzled”, “fearful”, “frozen”, “interested”, “indifferent”, “confused”, and so on). Students must choose one card which most reflects their feelings concerning the academic discipline or situation in the learning process. According to the authors, despite the limitations, this technique can be useful in identifying bottlenecks and problem situations. “But it is useful to break out of the mold for perceiving students’ attitudes as lying along linear paths, and for an “attitude change” as moving students “higher” or “lower” along such paths, as is the case when five-point Likert scales are used (GAL; GINSBURG, 1994).

This characteristic may refer to the Color Selection Method (CSM). The developer of the test's original version (Max Lüscher) postulated that the color choice reflects the mood, functional state, and the most enduring personality traits. The test development is based on an empirical approach and was initially associated with a study of a person's emotional and psychological state (LÜSCHER, 1990). The Russian psychologist Sobchik (2007) characterizes CSM as projective, because, in her opinion, this technique reveals not so much the conscious, subjective attitude of the examinee to the color standards, but their unconscious reactions.

The ideas of diagnosis through color associations appeared in the middle of the 20th century. Since then, there were numerous empirical studies on the effect of color on humans and the possibilities of CSM for the diagnosis of a person's internal state. For example, many researchers believe that this method is not sufficiently reliable as a diagnostic tool in clinical practice (see, CERNOVSKY; FERNANDO, 1988; HOLMES et al., 1985). At the same time, numerous studies show examples of successful use of the test to describe personality and behavior (CARMER et al., 1974; COROTTO; HAFNER, 1980; LANGE; RENTFROW, 2007; NOLAN et al., 1995). Donnelly (1974), in his study of the color preferences of college students, concluded that the reliability of the Lüscher Color Test, “although somewhat low, appears comparable to that reported for other projective techniques”. Sobchik (2007) argued that the test based on Lüscher's ideas has the following essential characteristics: it does not provoke (in contrast to other, especially verbal, tests) reactions of a protective nature, and also is consistent with the concept of a holistic multi-level understanding of the individual.

Specialists of the psycho-diagnostics laboratory of Tomsk Polytechnic University have developed a particular modification of color selection method in order to investigate students’ attitude to teaching. The interpretation of colors is presented in Table 1.

The study of this method was presented in the dissertation research by Maruhina (2003), who is an employee of this university. She saw CSM as an addition to the Likert-type questionnaire to investigate students’ attitudes toward study courses. The joint application of the two methods allows concluding that the students’ color associations correspond to the interpretations presented in Table 1.

Due to this fact, and that many studies also support the validity of the Lüscher color theory (for example, CARMER et al., 1974; COROTTO; HAFNER, 1980), we hypothesized that CSM could be used as a separate methodology for students’ assessment of training courses. Since the further investigation of CSM application in education has not been carried out, we believe that our study will contribute to the disclosure of CSM's capabilities as a substantive student teaching evaluation tool and will help expand the practice of using it.

Sixty-six seniors of the Applied Mathematics undergraduate program took part in the survey. Of these, 39 are men and 27 are women. The survey took place in the last month of their university studies. These students were invited to participate in the survey, because they could evaluate all the courses of study that make up the educational program. As the studies confirm, senior students have academic maturity and experience and therefore can give an adequate assessment of teaching quality (LYNAM; CACHIA, 2018; THEALL; FRANCLIN, 2001).

Also, an essential factor in conducting student interviews is the attitude of students to participate in the survey. According to Hoshower and Chen (2003), McClain et al. (2018), and some other researchers, students give more honest assessments if they believe that their opinion will help improve the courses’ content and teaching rather than serve administrative interests. Benton and Cashin (2014) argue that in order to increase survey validity, “the instructor should take time to encourage students to take the process seriously.” Therefore, not only we discussed with the students the goals of the upcoming survey but also invited them to participate in the questionnaire items discussion and selection of the assessment scale (KUZNETSOVA, 2019).

This study consisted of two consecutive stages. At the first stage, a survey was conducted using the questionnaire “The Learning Process in the Eyes of Students” (LPES). The questionnaire, developed by us, consists of 10 items and reflects the students’ opinions on the following aspects of the learning process: the educational process quality of organization; teaching quality; the results of the process of studying the course. Using the 100-point scale, familiar to them, the students evaluated 34 courses studied by them since their first year. From the survey outcomes, we compiled a summary table, in which each academic discipline corresponds to the average score for each of the ten items.

Table 2 presents the results of the internal consistency analysis of this questionnaire. Student evaluations were analyzed using correlation, factor, regression, and cluster analysis. The results are presented in the article KUZNETSOVA (2019).

In the second stage, which took place two weeks later, we used a test based on Max Lüscher's ideas. The same group of students was asked to choose which color from the ones presented in Table 1 (but without deciphering their meaning) they associated with one or another course. The students evaluated the same 34 academic disciplines as in the first stage. Based on the survey outcomes, a summary table was compiled. Table 3 demonstrates a fragment corresponding to such disciplines as Discrete Math and Sociology.

According to Table 3, the discipline Discrete Math causes associations with blue in 14% of respondents (the value of the blue variable is 0.14), with green in 18% of the students surveyed (the value of the green variable is 0.18), and so on. In general, we can conclude that for most of the 66 students, this course is associated with bright, optimistic colors. The attitude of students to the Sociology course is different. For 23% of students, this course is associated with gray (the value of the gray variable is 0.23). Following Table 1, gray means indifference. At the same time, there is only 9% of association with green (the value of the green variable for Sociology is 0.9). It is evident that this course did not arouse the students’ interest.

The CSM-based summary table for the 34 academic disciplines was researched through correlation and cluster analysis using the procedures implemented in the STATISTICA application package. Further, the CSM test results compared with the LPES test results, performed in the first stage.

Supporting student feedback, an analysis of survey outcomes, and respect for students’ opinion are essential factors in the successful improvement of education quality. Application of CSM for monitoring the educational program quality in applied mathematics has shown its ability not only to reveal the existence of problems but also to define their causes. For example, as a result of the cluster analysis, such courses as Philosophy, Physics, Computer Architecture, and Functional Analysis have been gathered in the group of disciplines uncomfortable for students. The detailed analysis of color associations for each of these courses allows us to conclude that the causes of discomfort are different. It will help us choose the correct strategy for improving teaching and learning.

The research has revealed the most appealing courses for students. They are Economics, Mathematical Methods and Models in Economics I, Mathematical Methods and Models in Economics II, Intelligent Systems, and Application Software. The content of these disciplines has a close connection with the students’ future professional activity as applied and industrial mathematicians. The fact that on average, 44% of students have these courses associated with green (interest, the search for themselves, usefulness, following the Table 1), proves how important the understanding of the connection between learning and practice is for them.

The cluster analysis of the CSM-based survey outcomes has detected problems with the teaching of social sciences and humanities. Such courses as History, Russian Language, Sociology, and Philosophy are not attractive to future mathematicians. An exception to this group is English (Cluster2* – comfortable courses), and Economics (Cluster3* – very interesting courses). At first sight, social sciences and humanities have no connection with the future professional activity of bachelors-mathematicians. However, mathematics expands its scope today. Increasingly mathematical methods are applied in human sciences, for example, Sociology or Psychology. Besides, the Russian mathematician Kolmogorov emphasized that for developing creative abilities in mathematics, it is necessary to go beyond mathematics and develop common cultural interests, in particular, interest in art and poetry (Yurkevich, 2001). Therefore, the problem of improving teaching in social sciences and humanities within the Applied Mathematics educational program also requires care and attention.

We can note other advantages of CSM. First, conducting a survey using this questionnaire does not demand much time from students. This is especially important if we have to estimate not one but several disciplines. Indeed, for the characteristic of a course, a student has to choose a color among the eight offered, instead of responding to numerous Likert-type items. Secondly, as our experience has shown us, CSM has not caused rejection among students; none of them refused to participate in the survey. Students were interested in color associations. This fact is consistent with results obtained by Sobchik (2007). However, this task confused three respondents in the beginning of the survey.

The analysis of the CSM outcomes and their comparison with LPES allows us to conclude that the surveys for each of the two methods do not contradict each other and therefore both questionnaires may be considered as robust. It confirms our hypothesis that CSM can be used as a full-fledged evaluation tool for identifying the problems arising in the process of teaching and learning.

In the future, it would be interesting to continue researching the properties of CSM. For example, further research should investigate the influence of temperament on its results, to understand the reasons why the same situation in studying the course for some students causes a feeling of comfort (association yellow), indifference for others (association with gray), and rejection (association with black).

It is also necessary to note that the results of the CSM, as well as the results of SET in general, need a balanced attitude as one of the education quality indicators and should be considered in conjunction with other indicators of learning effectiveness.