Background

Responsible Research and Innovation (RRI) represents a contemporary view of the connection between science and society. This concept has been developed by the European Union and is the basis of several EU projects. The goal of RRI is to create a shared understanding of the appropriate roles of those who have a stake in the products of science and technology, including governments, businesses, scientists, technologists, educators, the general public and NGOs. The hope is that through the creation of such a shared understanding, mutual trust and confidence will result, along with safe and effective systems, processes and products of innovation (Sutcliffe, 2011).

One way to understand RRI is to see it as

a transparent, interactive process by which societal actors and innovators become mutually responsive to each other with a view to the (ethical) acceptability, sustainability and societal desirability of the innovation process and its marketable products (in order to allow a proper embedding of scientific and technological advances in our society). (Schomberg & Von Schomberg, 2013, p. 19)

Another definition of RRI is built on six dimensions: 1. Engagement, 2. Open Access, 3. Ethics, 4. Science Education, 5. Gender Equality, and 6. Governance. These dimensions were published and recommended by the Horizon 2020 framework of the European commission (2015). More detailed description of the 6 RRI dimensions is provided in an earlier publication (Blonder, Zemler & Rosenfeld, 2016).

RRI is not only a framework to promote responsible scientific research and technological development, but it also has an educational component, whose goal with teachers and students is to develop the skills and attitudes associated with "RRI-based thinking" about past and current scenarios regarding the development of science and technology advances. When working to achieve this goal, an evaluation tool is needed to assess teacher and student attitudes relating to RRI.

Goals

The goals of this article are to trace the development and use of a questionnaire for evaluating teacher and student attitudes regarding RRI. More specifically:

• . To develop and validate a questionnaire attitudes about responsible research and innovation (RRI).

  · To illustrate the use of this questionnaire within an evaluation study.

The article begins with a brief history of the development of the RRI framework, followed by a presentation of how the RRI questionnaire was developed and used, within the context of an EU-funded project called Irresistible (Irresistible, 2015). We close by exploring the possibility that the RRI questionnaire can be used to assess the development of attitudes regarding RRI across diverse populations of teachers, students, scientists, and members of the general public.

The Development of an Educational Framework to Assess Attitudes about RRI

RRI can be seen as a new contract between science and society, a "social innovation" which relates to the roles and responsibilities of the many stakeholders involved in the processes and products of science and technology, including scientists, technologists, businesses, governments, citizens, NGO's, teachers and students (Rip, 2014).

The roots of the educational framework for RRI can be seen in the use of socio-scientific issues (SSI) within the science curriculum. SSI was used as early as 1986 (Fleming, 1986) but its development as a recognizable framework for research and practice in science education emerged only in the late 1990's. SSI can be defined as "social dilemmas linked to science about which citizens have to make decisions" (Molinatti, Girault & Hammond, 2010, p. 513). This definition reflects the view that "all aspects of science are inseparable from the society from which they arise" (Sadler, 2004, p. 513). According to this view, it is important to recognize that there are links between science, politics and business and that there are many different actors in the scientific-technological enterprise (Simonneaux, 2014).

Therefore, it is important for teachers and students to develop "moral-ethical reasoning" so that they will be able to take into account the different points of view of different social groups when considering real-world socio-scientific issues, which by nature are controversial, preliminary and under debate (Sadler & Zeidler, 2005;Zimmerman et al., 2001). In SSI, students are encouraged to understand how different stakeholders have different perspectives, i.e., different ways to perceive and interpret the same issue; in this regard, it is important to distinguish perspectives from positions (where one stands on an issue) and orientations (how one approaches an issue in relation to others) (Kahn & Zeidler, 2016). SSI "entails the examination of competing claims, values, and evidence, thoughtful deliberation and negotiation, and the ability (to) navigate the concept of optimality throughout this process" (Zeidler, 2014, p. 720).

Developing this type of reasoning by teachers and students is one of the educational goals of RRI scenarios, in which teachers and students consider how different stakeholders can cooperate to produce optimal scientific and technological products. One way to evaluate RRI is to assess teacher and student attitudes regarding the above-mentioned 6-dimensions of RRI (Apotheker et al., 2016; Blonder et al., 2016). If we can produce an appropriate tool to assess attitudes relating to each of these dimensions and to the perceived responsibility of the different stakeholders, we may be able to evaluate how well teachers and students internalize the intended educational outcomes of RRI.

The Context of the Study

In order to understand the context in which the questionnaire was developed and used, we need to understand the cultural environment in which it was developed. In the last decade, several EU-sponsored projects have been devoted to integrating RRI into science education. The general approach has been to provide relevant curricular materials to Communities of Teachers (CoLs), in order for them to engage their students in socio-scientific issues via IBSE (Inquiry-Based Science Education) strategies. For example, the "ENGAGE" project offers three kinds of materials: dilemma lessons, problem-solution lessons, and scenario-based topics (Okada, Young & Sherborne, 2015). Another example is that "PARRIS" project offers an integrated approach to Socio-Scientific Inquiry-Based Learning (SSIBL). It collects and shares existing best practices across Europe and develops learning tools, materials and in/pre-service training courses for science teachers based on the SSIBL approach. Other examples are described in Blonder, Zemler, and Rosenfeld (2016).

Next will now describe in more detail one of the EU-funded projects to integrate RRI into science education–the Irresistible project, in which the questionnaire was developed.

The Irresistible project (Irresistible , 2015) is an European project in the FP-7 framework aims to make young people more aware about RRI issues, through curricular materials (the Irresistible modules) to be used both in the classroom and in science centers. Ten European countries participated in the three years project (2014-2016). Each partner country has formed a Community of Learners (CoL). Detailed description of the Irresistible project is provided in several recent publications (Apotheker et al., 2017; Blonder et al., 2017) and in the project Website (Irresistible, 2015). Three important features of the project are described below:

CoL:

The modules were developed by a Community of Learners (CoL) composed of a research scientist, high-school science teachers, a member of the local science center and science educators. Each module was based on the research work of a research scientist at the university.

Module's goals and main topics:

The main goal of the Irresistible modules was to foster positive attitudes towards RRI by both teachers and students. Each module that was developed by the different CoLs has its own scientific topic (e.g. the main topic of the Israeli module was the development of perovskite-based photovoltaic cells (Snaith, 2013) within the context of using alternative energy). The topics of the other modules are presented in the Irresistible project website (Irresistible, 2015).

Pedagogical Approach:

The design of the modules was guided by two approaches: (a) the 6E inquiry model, based on Bybee, et al. (2006) and (b) an effort to bridge between formal and informal science education (Fallik, Rosenfeld & Eylon, 2013), which focused on the production of student-designed exhibits, in the tradition of interactive science exhibits.

The Structure of the RRI Questionnaire

With the above 6-dimension definition of RRI in mind (see Table 1), a RRI questionnaire for teachers and for students was developed and validated according the stages presented in Table 2. The questionnaire includes three parts:

Attitudes towards the RRI dimensionsAttitudes towards the RRI dimensions

This section was included in the teachers' and students' questionnaire. It evaluated their attitudes towards the 6 different dimensions that constitute RRI. The respondents were asked to "determine the degree to which you agree with the following statements (5 = agree a great deal, 1 = do not agree at all.)". Table 1 presents examples of items in this part according the RRI dimension.

Table 1

Sample of items for each RRI dimension in the RRI questionnaire

Responsibility of different stakeholders in the real world and in an ideal worldResponsibility of different stakeholders in the real world and in an ideal world

This section was given only to the teachers. It started by presenting by the following question: "The following groups can each take different degrees of responsibility for the consequences of research and innovation in society and the environment. In an ideal world, what degree of responsibility should each of these groups take?" They were then presented with a list of the following actors: the Government (policy planners), Academic Institutions, Scientists, Educators, Environmental Organizations, Non-profit organizations, Consumers, Businesses, the Printed and Electronic Media. The teachers were asked to rate the degree of responsibility for RRI in an ideal world (1= to a very small degree ; 5 = to a great degree).

Next they were asked to do the same in the real world: "In your country today, what degree of responsibility for RRI does each of these groups take? (1= to a very small degree ; 5 = to a great degree)".

Teacher experience in integrating social issues in science educationTeacher experience in integrating social issues in science education

This section was given only to the teachers. It presented the following four questions: How often have you participated in discussions in science classrooms that deal with ethical issues of science and society? (For example: "Should we pursue new nuclear technologies?" or "What are the risks and benefits of nanotechnology applications?") (1= never ; 5 = often). How often have you participated in classes or workshops that deal with ethical issues of science and society? (1= never ; 5 = often). How often have you taught ethical issues relating to science and society? (1= never ; 5 = often). Which ethical issues in science and society do you think are relevant in regard to developing new technologies for _______? (Each partner was asked to write here the name of the domain topic of the module which was taught)?

In the first three questions, the prior experience of the teachers was collected in a Likert scale (1= never ; 5 = often). These questions track the teachers' personal experiences as participants in discussions which involve SSI (socio-scientific issues) and their professional experience in conducting science lessons that integrate ethical and social aspects with science and technology. The fourth question is an open question in which the respondents are asked to suggest and write social issues that relate to the scientific topic of the module they would learn and teach.

The Development of the RRI Questionnaire

The process of developing the RRI questionnaire included several stages of validation, a test of internal consistency to support its reliability, and its multicultural adaptation in the international community of the Irresistible project. The implementation process is summarized in Table 2 (on the following page), and further elaborated in the text.

Stage 1: Creating item pool

At the first development stage the Weizmann team created a pool of items (in Hebrew) that were based on the RRI literature and covered the 6 RRI dimensions. Forty-four items were gathered in this stage.

Stage 2: First expert validity check

Three experts in science education who were part of the Weizman team's CoL discussed the items' content and validated that the phrasing reflected the meaning of the RRI dimensions that each item represented. The three experts also selected 5 items for each of the 6 RRI dimensions to produce a 30-item questionnaire.

Table 2

The implementation process of the RRI questionnaire

* These stages were conducted only for the first part of the questionnaire attitudes towards the RRI Dimension

Stage 3: Translation to English and inter-translator reliability

Based on Anastasi (1988), the questionnaire was translated to English by two translators. Two translations were compared by the development teams and the meaning of the translated items was compared to the meaning of original items in Hebrew in order to obtain inter-translator reliability.

Stage 4: Second expert validity check and multicultural adaptations

The 30 items were sent to all the partners (10 experts in science education in ten European countries) for expert validation, and modifications were made according the comments the partners sent. For example the item: "A research director that needs 'work around the clock' should not hire women who have young children." (NEGATIVE statement regarding the gender dimension), was modified to: "A research director that needs 'work around the clock' should not hire pregnant women" according the Turkish suggestion to emphasize the dilemma. In another suggestion, the item: "A scientist should be involved in programs to make his/her research accessible to students and their teachers in the science classroom" was modified to "Scientists should be involved in programs to make their research accessible to students and their teachers in the science classroom." This change was made in response to a suggestion of the Finish team that this issue is not about just one scientist. In other items some modifications were made to keep the English simpler and to share the same meaning in the different countries. These modifications helped to clarify the meaning of the items for the international community of teachers and students.

Stage 5: Pilot international implementation

The modified version (pilot version) was sent to all partners and was translated into 10 languages. In the pilot trial, the teachers from all the CoL members in all countries (N=54) filled the questionnaire. Appendix 1 presents the pilot stage of the questionnaire.

Stage 6: Reliability analysis and third expert validity check

The items were again examined by three experts in science education for their valid representation of the RRI dimension and more coherent language was applied. For example, instead of using different terms to describe academic research and researchers (e.g., research institutes, universities, academic institute, and scientists), only one term was chosen (scientists), because this term is clearly understood by students and teachers who are not part of the academic culture. In addition, based on alpha-Cronbach test, items were reduced to 4 for each RRI dimension.

Stage 7: First international implementation

The new version (version 1) of the questionnaire was filled-in by the teachers (N=210) and students (N=1160) in the 10 countries. Appendix 2 presents the questionnaire that was administered in this stage.

Stage 8: Reliability analysis

Alpha-Cronbach values were calculated for each of the 6 RRI dimension (teachers and students separately). We therefore decided to choose for each dimension the two items (as shown in Table 3) showing the highest significant correlation (for both students and teachers) and to proceed with a shorted questionnaire that measures the RRI construct. The alpha-Cronbach for the 12 selected items that composed the RRI construct was 0.76 for students and 0.78 for teachers.

Table 3

The selected items for each RRI dimension and their r-correlation value

* ** p00001

Stage 9: Final scale

The final version questionnaire included three sections (1) 12 items to measure the respondents' attitudes towards the RRI construct' (2) a comparison of the respondents' perspectives regarding the responsibility for RRI of different stakeholders in the real world and in an ideal world (only teachers), and (3) measurement of the ability to find socio-scientific ethical issues related to relevant module's scientific topic (only teachers). The results presented in section 3 of this report used the final scale of the RRI questionnaire.

Stage 10: International implementation of the final scale

In the last stage, the RRI questionnaire was administered in a pre-post procedure in the second round of the CoL (the second phase of the project. The alpha-Cronbach for the 12 selected items was 0.78 for students (N=3117); and 0.79 for teachers (N=224). The r-correlation between the two items that construct each RRI dimensions are presented in Table 4.

Table 4

The r-correlation value for two items represent the same RRI dimension

* ** p00001

Ethical issues

The evaluation was carried out according to the ethical issues and precautions described in the Irresistible Description of Work (Irresistible, 2015). To ensure anonymous analysis of the research data, each surveyed CoL member was represented by a personal code which cannot be tracked back to the respondents’ identity but can be used to connect an individual's responses for the pre- and post-tests. According to EU regulations, participating schools, students and parents returned consent forms, also containing information about the research (Irresistible Description of Work, 2013).

Using the RRI Questionnaire in Educational Research

The RRI questionnaire was used in ten European countries who participated in the Irresistible project. Teachers who were members in the CoL and their students who learned the Irresistible modules filled the questionnaire in a pre-post procedure. In this part we present the results gained from the Israeli teachers and students, and present them in the context of the results obtained by all the Irresistible teachers and students.

Population

The numbers of teachers and students who completed the questionnaire are presented in Table 5.

Table 5

Number of teachers from the different countries who completed the questionnaire

Results

The results will be presented according the 3 parts of the questionnaire: (1) attitudes towards the RRI dimensions, (2) responsibility of different stakeholders in the real world and in an ideal world, and (3) Teacher experience in integrating social issues in science education. The final scale was used in its on-line version (Each country used the translated questionnaire to its own language, see Table 2 for details).

Attitudes towards the RRI dimensions

This part of the questionnaire was administered to teachers who participated in the CoL and to students who studied the Irresistible modules. Tables 6 and 7 present the results of the teachers and students respectively, both regarding the Israeli data as well as the data for the 10 partner countries in the Irresistible project.

Table 6
Pre-post average scores of teachers' attitudes towards RRI and its 6 dimensions, in Israel and in the Irresistible project (Comparison between pre and post values were calculated using two-tailed t-test)
*p<0.05, **p<0.01, ***p<0.001

Table 7
Pre-post average scores of students' attitudes towards RRI and its 6 dimensions, in Israel and in the Irresistible project (Comparison between pre and post values were calculated using two-tailed t-test)
* p<0.05; **p<0.01; ***p<0.001

The findings show that in Israel, the process of teacher professional development in the CoL led to a positive statistically significant difference between the pre- and the post-test attitudes of teachers toward RRI as a general construct and for 4 of the RRI dimensions (the dimensions of engagement and gender equality were not statistically significant), as presented in Table 6. The teachers who participated in the Israeli CoLs used the modules that were developed in the project and positively influenced the development of students' attitudes towards RRI (statistically significant for the general construct, and for all the RRI dimensions, except for gender equality, ethics and governance), as presented in Table 7.

Regarding the Irresistible project, the findings show that the process of teacher professional development in the CoLs led to a positive statistically significant difference between the pre- and the post-test attitudes of teachers toward RRI as a general construct and for each of the 6 RRI dimensions that compose it, as presented in Table 6. The teachers who participated in the Irresistible CoLs used the modules that were developed in the project and positively influenced the development of students' attitudes towards RRI (statistically significant for the general construct, and for all the RRI dimensions), as presented in Table 7.

Responsibility of different stakeholders in the real world and in an ideal world

When the Irresistible teachers were asked: "In your country today, what degree of responsibility does each specific group take (for the consequences of research and innovation in society and the environment)?" only one significant difference between the pre- and post-test was obtained regarding the degree of responsibility of the NGOs (p<0.01). For all the other stakeholders, no differences were found between how teachers perceived the degree responsibility in the real world between the pre- and post-test, as shown in Figure 1. The results of the Israeli teachers are presented in Figure 2. The same trend was obtained by the Israeli teachers: the only statistically significant difference was the NGOs. In both the overall Irresistible data as well as in the Israeli data, the stakeholders considered most responsible for RRI were the scientists and academic institutions, while the stakeholders least responsible for RRI were consumers and educators.

Figure 1
Teachers' perspectives regarding degree of responsibility that each of specific group takes in their country today ("the real world") in the pre- and post-test. This analysis includes all the teachers in the Irresistible project who filled in the questionnaire (N=213)
** p<0.01

Figure 2
Israeli teachers' perspectives regarding the degree of responsibility that each of specific group takes for RRI in Israel today ("the real world") in the pre and post-test (N=25)
** p<0.01.

When the teachers were asked: "In an ideal world, what degree of responsibility should each of specific groups take for RRI (for the consequences of research and innovation in society and the environment)?" significant differences between the pre- and post-test were obtained regarding the degree of responsibility of all the presented stakeholders (p<0.001). A less significant difference was obtained regarding the government (policy makers): p<0.05. For all the other stakeholders, no differences were found between how teachers perceived the degree of different stakeholders take in the real world (in their own country) in the pre- and post-test, as shown in Figure 3. Figure 4 presents the results for the Israeli teachers. In the Israeli sample, an increase was obtained regarding the perceived responsibility of all the stake holders. However only four of them had a significant with P<0.01 (governance, educators, consumers, and NGOs). The last three stakeholders represent roles that can be taken by the teachers who are educators, consumers that can be part of NGOs.

Figure 3
Teachers' perspectives regarding degree of responsibility that each of specific group takes for RRI in ideal word in the pre and post-test. This analysis includes all the teachers in the project who filled in the questionnaire (N=213)
* p<0.05; **p<0.01; ***p<0.001

It is interesting to note that the teachers' perspectives regarding stakeholders with whom they could be identified (such as educators, NGO, and consumers) received the lowest values, especially in the pre-test. However, even though teachers' perspectives regarding these stakeholders significantly improved in the posttest, they were still lower than the responsibility they assigned to the scientists and academic institutions. Teachers still perceived that the major responsibility for RRI rests with these two traditional stakeholders and much less with themselves as teachers or consumers.

Teacher experience in integrating social issues in science education

The third part of the questionnaire recorded the teachers' experience in connecting social issues to scientific content. This section included four questions:

How often have you participated in discussions in science classrooms that deal with ethical issues of science and society? (For example: "Should we pursue new nuclear technologies?" or "What are the risks and benefits of nanotechnology applications?") (1= never ; 5 = often). How often have you participated in classes or workshops that deal with ethical issues of science and society? (1= never ; 5 = often). How often have you taught ethical issues relating to science and society? (1= never ; 5 = often). Which ethical issues in science and society do you think are relevant in regard to developing new technologies for? (Each partner was asked to write here the name of the domain topic of the module which was taught)? Table 8 presents the pre-post average results obtained for all the teachers who completed the questionnaire. For the first three questions, responses could be found between 1-5 in the Likert questionnaire. For the fourth question, the number of the ethical issues suggested by the teachers was counted and the average presents the average number of issues that was suggested.

Table 8
Teachers' experience with SSI issues as indicated in third part if the questionnaire and their ability to suggest ethical issue related to the scientific topic of the module in their country

In order to explain which ethical issues were expressed by teachers we provide some examples, taken from the Israeli sample:

"Are the voices of everyone involved equal in the decision making regarding the innovative solar cells?"; "To what extent do the perovskite-based solar cells development take into account social and environmental damage?"; "Who will supervise the influence of perovskite-based solar cells of children health?"

Regarding the Irresistible teachers, in general, the results of part three show that they reported an increase in their experience to take part in social issues related to scientific and technological topics and to participate in workshops dealing with these socio-scientific issues (SSI). These results are reasonable since the teachers participated in their respective CoLs in which such discussions were part of the CoL activity. The teachers also reported that they tend to conduct more SSI discussions in their classes (question 3). However, when they were asked to suggest ethical issues that are relevant to the scientific topic that was part of their Irresistible module, they were able to suggest more issues but the difference between their pre and post abilities was not significant (Table 8).

Regarding the Israeli teachers, specifically, the results show that they also reported an increase in their experience to take part in social issues related to SSI issues and to participate in workshops dealing with them. However, unlike the general Irresistible teachers, the Israeli teachers showed no statistically significant difference regarding their teaching of SSI. Upon closer examination of the data, the reason for this finding seems to be that the Israeli teachers scored higher than the Irresistible teachers on both the pre-test and on the post-test for this item, but that the difference between these two scores was not statistically significant.

Summary and Discussion

As mentioned above, one way to understand RRI is to see it as a framework to guide scientists and technologists, citizens and consumers, as well as other innovators and societal actors to "become responsive to each other" in terms of the ethical acceptability, sustainability and societal desirability of the marketable products of science and technology (Schomberg & Von Schomberg, 2013). Thus, the effort to implement RRI in society is a broad-ranging goal which relates to many societal groups. Some of the societal groups involved in this effort include science teachers and their students within formal and informal science education settings. In order to assist these groups in implementing RRI, the European Community has established a number of projects focusing on RRI in science education as mentioned earlier.

The educational focus on RRI in these projects has emphasized the development of various curricular interventions for teachers and their students. Our interest in developing a questionnaire to assess attitudes relating to the 6 RRI dimensions has been to provide educators with a way to evaluate how well teachers and students internalize the intended educational outcomes of RRI. Thus, such a questionnaire could provide teachers with a way to obtain feedback about the effectiveness of RRI-based curricula. It could provide science education researchers with a tool to explore various issues relating to RRI in science education.

Using a 10-stage development process, we produced a valid and reliable 3-part questionnaire to evaluate the attitudes of teachers and students regarding the RRI construct which includes 6 dimensions. We then used this questionnaire to explore various issues relating to RRI in science education, within the context of the Irresistible project. We used the questionnaire in a pre-post design with teachers and students to evaluate to what extent their attitudes were changed during the course of the project, using curricular modules produced and taught in each of the 10 countries.

There were three parts of the questionnaire that were used to explore the development of RRI attitudes in teachers and their students, before and after teaching and learning the various modules developed within the context of the Irresistible project. The first part explored the RRI attitudes of teachers and their students in the project. The results demonstrate statistically significant gains in positive attitudes regarding the RRI construct for both teachers and their students. We can therefore conclude that the process of professional development in the Communities of Learners (CoLs) in the project led to significant gains in the teachers' attitudes and that the teachers used the modules developed in the Irresistible project to positively promote students' attitudes towards the RRI construct, across all 10 partner countries. It is interesting to note that when we examine the attitudes of teachers and students regarding the RRI in one country (for example Israel) the change was not significant for all 6 RRI dimensions. The accumulation of all 10 countries provides a variety of Irresistible modules, each emphasize different RRI dimensions, and together create a significant change in the attitudes of teachers and students of the whole project regarding the RRI.

The second part of the questionnaire explored how teachers viewed the respective responsibility for RRI taken by potential stakeholders, in the real world as well as in an ideal world. While the teachers developed stronger attitudes regarding RRI during the project, their view of their degree of responsibility, as educators and consumers in the real world, remained relatively low before and after the project. At the same time, their post-test assessment of the degree of responsibility that educators should take for RRI in an ideal world increased significantly. This finding leads to the conclusion that the Irresistible project empowered teachers to begin to expand their ideal role as science teachers regarding their teaching of RRI in their classrooms. Teachers also developed higher expectations for all potential stakeholders to take responsibility for developing RRI in an ideal world—especially NGOs, consumers and educators—which leads to the conclusion that the project expanded their view of the importance of RRI and the importance of its implementation by a variety of stakeholders.

Based on the results of the third part of the questionnaire, we can conclude that the Irresistible project increased the teachers' experiences with ethical issues in science education. However, teachers' inability to significantly improve their ability to suggest multiple examples of such SSIs may be explained by the conclusion that they do not have enough experience in working with ethical issues in the classroom. Another explanation for the same data is that these findings are module-dependent, i.e., it was more difficult for teachers to think of multiple examples of ethical issues for the domain topics of some modules than for others.

One implication of the questionnaire's findings with a sample drawn from 10 countries is that the use of socio-scientific issues (SSI) in science education has not yet become mainstream in science teaching practice. Yet the findings of the study also provide a reason for optimism. Although teachers originally expressed the attitude that educators and consumers have a low responsibility for RRI, after teaching the RRI-based modules the teachers felt that educators and consumers have a much greater responsibility for RRI, in an ideal world.

This attitudinal shift—as well as the increase in positive attitudes about RRI as reported above—may mirror an epistemological shift, based on a pedagogical strategy that engages teachers and their students into a "knowledge inquiry" (Simonneaux, 2014). According to this line of thinking, epistemological stances are fostered by pedagogical strategies. For example, a "scientistic" epistemological stance, where science is understood to be essential to progress and the researcher is accepted as the essential actor, is supported by a "doctrinal" pedagogical strategy, where the teacher's authority leaves little room for interaction with the students. Alternatively, an epistemological stance of "skepticism," which understands that scientific research produces controversies and risks (as well as breakthroughs) and therefore may be guided by political and economic choices, is supported by pedagogical strategies such as "problematising" and assessing uncertainties and risks relating to complex socioscientific issues (Simonneaux, 2011, cited in Simonneaux, 2014).

More specifically, since the pedagogical approach of the Irresistible Project included these latter two pedagogical strategies, by raising questions relating to each RRI dimension (See Table 9), this approach could have fostered an epistemological stance of "skepticism," first by the participating teachers and afterwards by their students.

Table 9
Student Questions Relating to Each RRI Dimension (from Blonder et al., 2016)

Science education needs to expose teachers and their students not only to the facts, principles and discoveries of science, but also to the challenge of how to navigate a reality in which science and technology produces consumer products, when universities and research institutes, as well as the military and commercial sponsors of research and innovation operate with vested interests (Hodson, 2011; Ziman, 1998). While these vested interests might try to promote "the cultural production of ignorance" for consumers (Proctor & Schiebinger, 2008) Clearly engaging teachers and students in actively assessing complex socioscientific issues, using something like the RRI-related questions presented in Table 9, could act to counter this possibility.

Clearly more work needs to be invested in developing and implementing curriculum that develop RRI attitudes in science classrooms, as well as in developing and using tools to evaluate these attitudes in teachers and their students. We suggest that the RRI questionnaire presented here is one such tool.

We also suggest that the RRI questionnaire can be used to assess the development of attitudes regarding RRI across other stakeholders involving RRI, such as scientists, consumers and other members of the general public.

References

Anastasi, A. (1988). Psychological Testing. Englewood Cliffs, NJ: Macmillan Publishing Company.

Apotheker, J., Blonder, R., Akaygun, S., Reis, R., Kampschulte, L., & Laherto, A. (2016). Responsible Research and Innovation in secondary school science classrooms: Experiences from the project Irresistible. Pure and Applied Chemistry, 1-9. doi: 10.1515/pac-2016-0817

Apotheker, J., Blonder, R., Akaygun, S., Reis, P., Kampschulte, L., & Laherto, A. (2017). Responsible Research and Innovation in secondary school science classrooms: experiences from the project Irresistible Pure and Applied Chemistry, 89, 211.

Blonder, R., Rosenfeld, S., Rap, S., Apotheker, J., Akaygün, S., Reis, P., . . . Laherto, A. (2017). Introducing responsible research and innovation (RRI) into the secondary school chemistry classroom: the irresistible project. Daruna, 44, 36-43.

Blonder, R., Zemler, E., & Rosenfeld, S. (2016). The story of lead: A context for learning about responsible research and innovation (RRI) in the chemistry classroom. Chemistry Education Research and Practice, 17, 1145-1155. doi: 10.1039/C6RP00177G

Bybee, R., Taylor, J. A., Gardner, A., Van Scotter, P., Carlson, J., Westbrook, A., & Landes, N. (2006). The BSCS 5E instructional model: Origins and effectiveness. Retrieved from Colorado Springs.

Fallik, O., Rosenfeld, S., & Eylon, B.-S. (2013). School and out-of-school science: a model for bridging the gap. Studies in Science Education, 49(1), 69-91. doi: 10.1080/03057267.2013.822166

Fleming, R. (1986). Adolescent reasoning in socio-scientific issues. Part I: Social cognition. Journal of Research in Science Teaching, 23, 677-687.

Hodson, D. (2011). Looking to the future: Building a curriculum for social activism. Rotterdam: Sense.

Horizon 2020 (2015). Retrieved from http://ec.europa.eu/programmes/horizon2020/

Kahn, S., & Zeidler, D. (2016). Using our heads and HARTSS*: Developing perspective-taking skills for socioscientific reasoning (*Humanities, ARTs, and Social Sciences). Journal of Science Teacher Education, 27(3), 261-281. doi:10.1007/s10972-016-9458-3

Molinatti, G., Girault, Y., & Hammond, C. (2010). High school students debate the use of embryonic stem cells: The influence of context of decision-making. International Journal of Science Education, 32(16), 2235-2251.

Okada, A., Young, G., & Sherborne, T. (2015). Innovative teaching of responsible research and innovation in science education. Open Education Europa Journal - eLearning Papers, 44(1), 1-14. Retrieved from file:///D:/my%20doc/downloads/epaper%202%20Okada%20(2).pdf

Proctor, R., & Schiebinger, L. (2008). Agnotology: The making of unmaking of ignirance. Stanford, CA: Stanford University Press.

Rip, A. (2014). The past and future of RRI. Life Sciences, Society and Policy, 10, 17. doi: 10.1186/s40504-014-0017-4

Sadler, T. D. (2004). Informal reasoning regarding socioscientific issues: A critical review of research. Journal of Research in Science Teaching, 41(5), 513-536.

Sadler, T. D., & Zeidler, D. (2005). The significance of content knowledge for informal reasoning regarding socioscientific issues: Applying genetics knowledge to genetic engineering issues. Science Education, 89(1), 71-93. doi: 10.1002/sce.20023

Schomberg, V., & Von Schomberg, R. (2013). A vision of responsible research and innovation. In R. Owen, M. Heintz, & J. Bessant (Eds.), Responsible Innovation (pp. 51-74). London: John Wiley & Sons, Ltd.

Snaith, H. J. (2013). Perovskites: The emergence of a new era for low-cost, high-efficiency solar cells. The Journal of Physical Chemistry Letters, 4, 3623−3630. doi: dx.doi.org/10.1021/jz4020162

Sutcliffe, H. (2011). A report on Responsible Research and Innovation for the European Commission. Retrieved from http://ec.europa.eu/research/science-society/document_library/pdf_06/rri-report-hilary-sutcliffe_en.pdf

Zeidler, D. (2014). Socioscientific issues as a curriculum emphasis: Theory, research and practice. In N. G. Lederman & S. K. Abell (Eds.), Handbook of Research on Science Education (pp. 697-726). New York, NY: Routledge.

Zeidler, D. L., & Nichols, B. H. (2009). Socioscientific issues: Theory and practice. Journal of Elementary Science Education, 21(2), 49-58. doi: 10.1007/bf03173684

Zimmerman, C., Bisanz, G. L. , Bisanz, J., Klein, J. S., & Klein, P. (2001). Science at the supermarket: A comparison of what appears in the popular press, experts’ advice to readers, and what students want to know. Public Understanding of Science, 10, 37-58.

Ziman, J. (1998). Why must scientists become more ethically sensitive than they used to be? Science, 282(5395), 1813-1814. doi: 10.1126/science.282.5395.1813