Reflection Questionnaires
Responsible AI does not only focus on technical aspects of AI development and focuses on how to align this new technology with human values and needs.
Thus, it is important that students critically reflect on the learned topics such as transparency, replicability and fairness. To measure whether students really thought and reflected about a topic is an important part of the learning experience.
Besides methods such as reflection essays (see resources), surveys and questionnaires can be utilized. In the following we provide a list of questionnaires, that may be useful. We tested one of them in detail – the AT-EAI.
Beware: One thing to bear in mind is that such attitudes are highly personal. To ensure that students answer honestly, a personal code can be created to anonymise the answers (see materials). Whether or not students answer these questionnaires should be optional.
AT-EAI Questionnaire
The ET-EAI was developed by Jang, Choi and Kim in their paper “Development and validation of an instrument to measure undergraduate students’ attitudes toward the ethics of artificial intelligence (AT-EAI) and analysis of its difference by gender and experience of AI education”. The questions encompass key categories of AI ethics: fairness, transparency, non-maleficence, privacy, and responsibility, making them an ideal fit if these topics are part of your curriculum. The questions can be found here: https://doi.org/10.1007/s10639-022-11086-5.
The questions are specifically designed to assess students’ attitude towards AI ethics. The goal of Jang et al. was to provide a tool that helps to prepare and evaluate AI ethics curricula. The AT-EAI questionnaire is validated and reliable for measuring undergraduate students’ attitude towards AI ethics. For each category (e.g., fairness, transparency) one scenario is given and participants can check the digit that fits their opinion on a Likert scale.
Recommended use: We recommend having students complete the questionnaire at both the start and end of the term to compare their attitudes toward AI over time. In our course, there was little change in students’ attitudes, which may be due to factors such as prior knowledge of the topic or a tendency to select socially acceptable answers. However, different contexts may yield different results.
Advantages: Focuses on known key topics and provides scenarios to judge. The format is easy to translate into surveys and could be printed. The questionnaire is validated.
Disadvantages: There are only five scenarios, which may not capture many nuances of opions. If students have to assess them at the start and end of the term, they may still remember the scenarios. Further, general attitudes may be more stable.
Other Possible Questionnaires
Below you find a list of questionnaires related to various constructs that may provide insides into students reflection, learning and their ethical positions. However, all of these questionnaires are very specific. Thus, it is important to check in how far they fit to the courses contents.
Reflection Questionnaires
We have identified 7 questionnaires that focus on reflection:
| Name | Description | Number of items | Dimensions | Reference |
|---|---|---|---|---|
| Collaborative Reflection Coding Scheme | Coding schema for deductive coding of reflection depth based on stated reflective activities mapped to ordered reflection levels. | n.a. | n.a. | Prilla, Michael, und Bettina Renner. “Supporting Collaborative Reflection at Work: A Comparative Case Analysis.” Proceedings of the 18th International Conference on Supporting Group Work – GROUP ’14, ACM Press, 2014, pp. 182–93. https://doi.org/10.1145/2660398.2660400 |
| Reflective Questionnaire (RQ) | Goal: Examine the extent to which students engage in reflective thinking in professional preparation courses; measuring the level of reflective thinking. | 16 Items; 5-Point Likert Scale | Constructs: habitual action, understanding, reflection, critical reflection | Kember, D., Leung, D. Y., Jones, A., Loke, A. Y., McKay, J., Sinclair, K., … Yeung, E. (2000). Development of a questionnaire to measure the level of reflective thinking. Assessment & Evaluation in Higher Education, 25(4), 381–395. |
| reflective practice questionnaire (RPQ) | Designed to investigate experiences, benefits, and potential pitfalls of reflective practice and supervision; applicable across service industries. | n.a. | n.a. | Download |
| Private Self-Consciousness Scale (PrSCS) | Distinguishes rumination and reflection dispositions; explains the “self-absorption paradox” where higher self-consciousness links to both self-knowledge and distress. | 24 items (12 each); 5-Point Likert Scale | Rumination (e.g. “I often find myself reevaluating something I’ve done.”), Reflection (e.g. “I love analyzing why I do things.”) | Trapnell, P. D., & Campbell, J. D. (1999). Private self-consciousness and the five-factor model of personality: Distinguishing rumination from reflection. Journal of Personality and Social Psychology, 76, 284–304. |
| Self-Reflection and Insight Scale (SRIS) | Measures three factors in the self-regulation cycle: need for reflection; engagement in reflection; insight. | 20 Items; 6-Point Likert Scale; 3 categories | Need for reflection; engagement in reflection (e.g. “I don’t often think about my thoughts.”); insight | Download Grant, A. M. (2001). Rethinking psychological mindedness: Metacognition, self-reflection and insight. Behaviour Change, 18(1), 8–17. Grant, A. M. (2002). The Self-Reflection and Insight Scale: A new measure of private self-consciousness. Social Behaviour and Personality, 30(8), 821–836. |
| Groningen Reflection Ability Scale Questionnaire (GRAS) | Goal: Measure personal reflection ability of medical students. | 23 Items; 5-Point Likert Scale (1 = totally agree to 5 = totally disagree) | e.g. “I want to understand myself.” | Questionnaire Aukes, L. C., Geertsma, J., Cohen-Schotanus, J., Zwierstra, R. P., & Slaets, J. P. (2007). The development of a scale to measure personal reflection in medical practice and education. Medical Teacher, 29(2–3), 177–182. doi:10.1080/01421590701299272 |
| Moral Attitudes Scale (MAS) | Items on European values: lying in your own interest, abortion, euthanasia. | 20 Items; scale from 1 = “Never” to 10 = “Always” | e.g. “Please tell me for each whether you think it can always be justified, never be justified, or something in between, using this card.” | Download Remizova, A., & Rudnev, M. (2020). Understanding the justifiability scale in international surveys: An exploratory study. International Journal of Social Research Methodology, 23(3), 291–309. doi:10.1080/13645579.2019.1683135 |
* deductive coding = top-down approach, i.e., the coders use an already existing codebook or framework and decide whether the code or concept is mentioned accordingly in a text
Ethical Positition Questionnaires
We have identified 5 questionnaires that focus on ethical positions, morals and values:
| Name | Description | Number of items | Dimensions | Reference |
|---|---|---|---|---|
| Ethics Position Questionnaire | The Ethics Position Questionnaire to measure individual differences in moral thought. | n.a. | Statements | Questionnaire |
| Short Ethics Position Questionnaire (EPQ-5) | Measures variations in sensitivity to harm (idealism) and to moral standards (relativism). | 10 Items; 5-Point Likert Scale (1 – strongly agree to 5 – strongly disagree) | Statements (e.g. “One should never psychologically or physically harm another person.”); English | O’Boyle, E. H., & Forsyth, D. R. (2021). Individual differences in ethics positions: The EPQ-5. PLoS ONE, 16(6), e0251989. https://doi.org/10.1371/journal.pone.0251989 |
| Epos – Ethische Position | Captures the individual ethical position of researchers, developers and users with regard to interactive software. | 17 Items; 5-Point Likert Scale (1 – strongly agree to 5 – strongly disagree) | 1) Regularien; 2) Privatsphäre; 3) bereichsspezifisches Wissen; 4) gesellschaftliche Verantwortung; 5) organisationale Verantwortung | Questionnaire Brandenburg, C., Schott, C., & Minge, M. (2018). Zur Erfassung der ethischen Position in der Softwareentwicklung. In R. Dachselt & G. Weber (Hrsg.), Mensch und Computer 2018 – Workshopband (Dresden, 02.–05. September 2018). GI. doi:10.18420/muc2018-ws06-0517 |
| Measurements for Values | This website presents three questionnaires to measure values: Valued Living Questionnaire (VLQ) Portrait Values Questionnaire (PVQ) Personal Values Assessment (PVA) | n.a. | n.a. | Questionnaire |
| Moral Sensitivity Questionnaire (MSQ) | Goal: Measure moral sensitivity of nursing students. | 11 Items | n.a. | Questionnaire Kuilman, J., Jansen, D., Mulder, M., Middel, L., & Roodbol, P. (2020). Re-assessing the validity of the Moral Sensitivity Questionnaire (MSQ): Two new scales for moral deliberation and paternalism. Journal of Evaluation in Clinical Practice, 26(2), e13353. https://doi.org/10.1111/jep.13353 |
