Problems with Multiple Choice Questions

When multiple choice questions have issues, they usually have to do with Cognitive Load and/or Validity issues. Cognitive Load refers to the amount of thinking it takes to process a question. A question stem is effective when it can deliver the problem to the test taker simply. If a stem can be simple, logical, limited in the use of negatives, and meaningful, it can help reduce the cognitive load on a student. Cognitive load theory recommends avoiding elements of instruction or assessment that will overload students' capacity to consciously process the immediate task on which they are working (Sweller 1998).

A Validity issue occurs when the results of the question don't actually assess the desired outcome. This can be simply because the question is poorly worded, or more often because of Rival Explanations. A Rival Explanation is when there is more than one possible reason why a student selected the wrong answer. As Jason mentioned in his video, a good question will tell you not only if they got it right or wrong, but also why they got it wrong, and also identify what common mistake or misunderstanding they've made.

The example questions below illustrate some common mistakes in writing multiple choice questions that increase cognitive load, and cause validity issues by introducing rival explanations.

What makes up a Multiple Choice Question?

Multiple choice questions have specific terminology used to describe the different parts of a question.

Stem: A question or statement followed by a number of choices or alternatives that answer or complete the question or statement

Choices: All the possible choices or responses to the stem

Distractors (foils): Incorrect choices

Correct answer (key): The correct choice!

Effective Stems

Keep the Stem Simple

Reduce the clutter in the stem and reduce non-essential information. Too much information in a stem can confuse students.

Avoid Negative Phrasing

Unless the learning outcomes are associated with negative phrasing in questions, avoid negatives when possible. Students often have difficulties understanding and processing negative questions.

Make the Stem a Question or Partial Sentence

Reduce the cognitive load of a student by asking a direct question. In the better example, the question is direct and the student can focus on answering the question. In the first example, the student has to hold the partial sentence in memory and complete the questions with each possible answers. If you can't find a way to write a direct question, try to move the blank to the end of the sentence.

Effective Distractors

Use Plausible Distractors

Choose distractors with a purpose. Implausible distractors don't help you identify whether or not your student understands the topic being assessed. In the extreme example below, students can easily remove questions A and D, leaving them with only two possible choices. The number of distractors can vary as long as they are plausible. Don't feel as if every question requires 4-5 choices.

implausible distractors.png

Keep Choice Lengths Consistent

Try to keep a consistent length with the choices you have as distractors. Students can easily guess the correct answer when one choice is particularly wordy and specific while other distractors are brief.

wordy distractor.png

Placement of the Correct Answer

Keep from consistently placing the correct answer as one specific choice. If students find a pattern in which answers are the correct ones (e.g., "c" is usually the right answer or "d" is never the right answer) then they can increase their chances of correctly guessing, providing a rival explanation. When distractors consist of a series of values on a continuum, place the choices in order logically (e.g. numerically, or chronologically).

Keep Grammar Consistent

Good test takers can notice clues to the correct answer when grammar isn't consistent between the stem and the distractors. In the example below, proteins is listed in its plural form, while the stem is asking for a singular substance.

grammatical hints-1.png

Avoid "All the Above" or "None of the Above" Options

When “all of the above” is used as a choice, test-takers who can identify more than one alternative as correct can select the correct answer even if unsure about other alternative(s). When “none of the above” is used as a choice, test-takers who can eliminate a single option can thereby eliminate a second option. In either case, students can use partial knowledge to arrive at a correct answer, thus not truly assessing whether they knew the whole answer.

The main point

Some of you, while reading the above examples probably thought to yourself, "Yeah, but the students should be smart enough to figure that out." You're right, they should - but that's irrelevant at this point. If your stem or distractor is confusing and the students choose the wrong option, we don't know if they didn't know the material or just got confused, this introducing a validity issue. Remember, the value of the assessment comes from it's ability to accurately tell us whether the students have reached the desired outcomes.

Click Next to learn from some other experts in this field.