This article was previously published in our August 2013 Newsletter

We have had a certain request come up several times since our last newsletter and I’d like to take a few minutes and share our thoughts on the concept of test randomization. If we could all sit together and just talk you would find a mix of factual statements and some just raw guts feelings. Every time our team has had this discussion we end up with the same conclusion, we feel that randomizing the question order and/or key and distractors may negatively impact the reliability and/or validity of a test.

I would like to point out just a couple of items from the National Guidelines for Educating EMS Educators that directly impact our stance. Module 12 section VIII states the exam needs to be organized in a logical manner and like items should be grouped together. EMSTesting is designed to make it very easy for an instructor to create a test using a blue print model while keeping like content together. Randomizing questions would directly impact this guideline. The guidelines further state that the key and distractors need to be positioned appropriately. What this simply means is that if they are letter choices they need to be in alphabetical order and number options need to be in ascending or descending order. Therefore, randomizing the key and distractors is also suspect. Randomizing the key and distractors also allows for possible discrimination towards students with dyslexia.

We feel that it is important that every students experience be identical so common reliability variables are not a factor when judging the exam. Any inconsistencies in this could have an impact on cut scores, discrimination and difficulty numbers.

One of our gut feelings can be explained with a scenario. Take a class and split it right down the middle. Use the same 100 questions for the entire class. Arrange the test in ascending order by cut score to one half of the class and descending order by cut score for the other half. This would allow half the class to build their confidence while progressing through the test to the most difficult questions at the end. The remaining half has the opportunity to have their confidence greatly impacted because they are receiving the most difficult questions first. Now we could really mess with things and scramble the key and distractors.

We understand the idea that this could help with potential cheating. Although this is a possibility, we feel that making it harder to cheat simply makes the cheaters better at cheating. In the end this is an attempt to try and fix an affective issue with a cognitive solution. If this is a sincere concern and you have a situation where you feel a group of students may be cheating just ask. We can show you how to export your test results and sort by two columns to find potential cheats.

We are not a free service and feel it is our responsibility to offer our customers the best possible solution. EMSTesting has developed into the great tool it is today because our customers provide feedback to us and we do listen and improve. There are instances where things come up that we have to just thank our customers for the idea and stand firm in what we are being paid to do, provide the very best in valid and reliable testing solutions.

We are grateful to have an incredible relationship with a growing group of customers and friends. Please do keep the suggestions coming and challenge us to become better at what we do. We appreciate every idea we receive either by email, phone call or in person at conferences across the country. We would not exist or be such a valuable solution for our customers without this open communication.

If you promise to keep the questions coming so will we. Thanks for your continued support of our solutions!