Preventing misleading outcomes in online assessments

Not all online assessments are created equal. Some yield trustworthy results with valuable insights, while others generate unreliable outcomes that should be avoided. Ensuring the production of dependable results should be the foremost priority in all assessment processes. When the obtained results lack reliability, it becomes impossible to make informed decisions about individuals. Consequently, all the time, effort, and resources invested go to waste, and poor decisions can have lasting negative consequences for all parties involved.

Outline

1. Dual nature of online assessments
2. How to obtain dependable results
3. When do online assessments fail to produce reliable results?
1. Bad practices that fail to prevent cheating
2. Creating unreliable assessments

Dual nature of online assessments

Online assessments possess a dual nature. They wield significant power and efficiency in gathering valuable information about individuals, offering insights into their strengths, weaknesses, and orientations in a highly effective, objective, and accurate manner. However, when they fail to deliver reliable results, they can precipitate ill-advised decisions with lasting, irreversible consequences. This dual nature underscores the critical importance of meticulously implementing the assessment process, from crafting tests and exams thoughtfully to safeguarding their integrity during delivery.

Obtaining dependable results from online exams or tests

When using test or exam results for decision-making, such as selecting and placing individuals, it is crucial to ensure the reliability of the outcomes. There are two fundamental pillars for achieving dependable results from an online exam or test. 

• First, the test should be appropriately structured, well-balanced, and measure various aspects, incorporating a fair scoring algorithm.
• Second, test-takers should not have prior knowledge of the test questions, and cheating should not be accessible during the test administration.

Only when a well-designed assessment is delivered while safeguarding its integrity, the results can be trusted and provide valuable insights

Online assessments failing to produce reliable results

Despite the considerable efforts and time invested, many organizations still struggle to obtain dependable results from their online assessments, leading to wasted resources and inaccurate decision-making about individuals, which can have significant negative consequences for all parties involved.

Online assessments often produce misleading results, either due to cheating or because they are inadequately structured to provide accurate measurements and valuable insights into individuals.

Bad practices that fail to prevent cheating

Numerous ineffective organizational practices fail to deter cheating attempts:

Placing trust in test-takers to refrain from cheating

When the integrity of an assessment is compromised, some test-takers, regardless of their background, may resort to cheating due to the ease of access to cheating methods. When a portion of the participants cheats while others do not, the organization ends up with untrustworthy results, particularly when the assessment's purpose is to select or place individuals based on their performance.

Administering a test with identical questions on different dates or at various times

When a test or exam consists of the same questions, and test-takers have the opportunity to take the same test on different occasions or dates, some individuals may gain access to prior knowledge of both the questions and their answers, enabling them to achieve high scores with ease.

Organizations should always be aware that some test-takers may collaborate with others, and test questions may be shared or published online, allowing many test-takers to become aware of the questions in advance, potentially rendering the assessment efforts futile.

Implementing ineffective anti-cheating measures

Numerous organizations implement anti-cheating measures, but their effectiveness often falls short. One prevalent approach involves utilizing a test-taker's webcam to capture photos at intervals, which, while a reasonable measure, proves most effective when combined with other anti-cheating safeguards. For instance, locking the device and recording the screen during the test ensures greater integrity. This is crucial because even when a test-taker appears to be alone in front of the webcam, they might still engage in communication through the device or seek answers from external resources and search engines, undermining the assessment's integrity.

Creating assessments that yield shallow or inaccurate outcomes

When tests or exams are not structured in accordance with best practices, their results lack valuable insights, resulting in inaccurate decision-making.

Examinations providing a single overall score for the entire test

Some exams are designed to provide only a single overall score for the entire test, without any breakdown of details. Relying solely on a single score may not offer a comprehensive picture, as it fails to pinpoint the strengths and weaknesses of individuals across a range of skills.

Tests with a single time control

When an exam is designed to assess multiple aspects of individuals, time controls should be applied to each aspect to ensure that individuals allocate their time appropriately to each area. Failing to apply time controls based on dimensions can lead to misleading results, where an individual may appear significantly better in some areas and much worse in others due to unequal time allocation.

Randomizing questions without a systematic approach

Some exams employ question randomization by drawing from a large pool of questions, but the exam engine may not guarantee a balanced and equitable test for all test-takers. The test could contain varying numbers of questions in different subjects and differing levels of difficulty, resulting in unequal measurement of individuals.