Validity, to give a short and non-technical definition, means “the extent to which a test measures exactly what the test maker intends it to measure, nothing more, nothing less.” (definition courtesy of Donald E. Powers, ETS).

Why is understanding a test's validity important? It's simple--not knowing how valid a test is means you don't know to what extent results are meaningful and actually provide the information you need.

How can you determine how valid a test is? As a test user, you can simply ask the testing company for test development background and validity statistics. For the test's creator, however, ensuring validity is a huge challenge.

Proven superior validity levels are the primary reason to consider TOEIC tests, more important even than the tests’ worldwide recognition, the high level of support provided for administration, and the ease of score interpretation for varied work levels and environments. ETS therefore makes validity its priority.

Step one in building a valid test is defining what exactly the test should measure. TOEIC’s goal is to measure a non-native English speaker’s ability to communicate in English in the context of the global workplace environment using key expressions and common, everyday vocabulary.

Test developers and language/assessment experts break this goal down as precisely as possible. What skills are most important for communicating effectively in this context and how important is each? 

TOEIC is researched and designed so that the most important components in communicating in English, such as listening for purpose, use of vocabulary, task completion, etc. are being measured and are included in the test in proportion to their real life importance for effective global English workplace communication.

A second important part of achieving test validity during test design involves eliminating to the greatest extent possible “construct-irrelevant components,” elements outside what the test was designed to measure that can impact test taker results. For example, overly complex test questions may make test takers anxious and unable to show what they can normally do. A test may take too long and measure test taker ability to answer questions quickly as much as their English proficiency. Questions may yield significantly different results when used with similarly capable individuals of different genders or from different cultures.

A few of the methods used to achieve validity during TOEIC test design:

• Detailed test “blueprints” created by world-class test developers are used in designing tests, so that each form is highly similar to every other one and so that the right content is covered in the same proportion in each form.
• World-class psychometricians design TOEIC “test items” (questions and answer choices).
• Each individual test item goes through numerous reviews, being inspected by some 20 test reviewers before pilot testing. Some items never make it through this process, while others must undergo extensive revisions.
• Pilot test results undergo statistical analysis at the item level to examine such elements as test question difficulty, how well the item distinguishes high from low ability, how incorrect options behave, and whether the test behaves consistently with test takers at the same ability level but from different subgroups.

Thise brief discussion gives only a taste of what’s involved in creating a quality test, and the challenge for TOEIC extends to every step of the testing process and across each test form’s lifecycle, from test administration and scoring to security and limiting form exposure.

Results? Top of class performance in validity, internal test consistency, fairness, and cross-form reliability.

We’re proud to distribute TOEIC tests for ETS in the United States as well as in many countries in the Middle East and North Africa. Contact us to learn how you can offer TOEIC at your site.

—Lia Nigro, TOEIC USA Team