Ah, the noble pilgrims. As you gather ‘round the table with family and friends this Thursday to give thanks, we direct your attention to the difficult tests faced by the Pilgrims leading up to that very first Thanksgiving.  Sure, there were the physical tests that go along with making a life for oneself from scratch in the new world.  There were the mental tests that they faced during the long, treacherous voyage across the Atlantic.  But, a little known fact, the Pilgrims faced another, even more challenging test!  For a passenger to even be granted entry upon the Mayflower, they had to pass a written test!  True story!

You might be thinking that such a test would have been simple. Perhaps a one-item, yes or no assessment along the lines of:

Dost thou wish to emancipate thyself from the tyrannical yoke of the Church of England?
Circle one:
Yes         No

Sadly, the test was not that simple—quite the opposite, in fact. This test set the standard for what we now refer to as a Job Knowledge Test. Although having slightly changed over the centuries, Job Knowledge Tests (JKTs) are one of many mechanisms that organizations use to evaluate candidates for a job opening or to measure a job incumbent’s knowledge about the job they are currently performing. Both uses of a JKT are as essential for assessing employee-job fit now as they were for assessing Mayflower-passenger fit back then. 

The JKT is a critical component of the Competency Model we’ve discussed in previous blogs over the years and is illustrated in this graphic. While it officially falls under the “Assessment” heading, the JKT is really applicable in all four sections of the model.  For example, in the Selection process, the purposes of JKTs range from simple tests designed to help managers determine the examinee’s level of understanding for a particular task to tests that determine critical employment decisions, such as hiring, firing, promoting, or demoting.  Such tests used to determine critical employment decisions must adhere to the Uniform Guidelines on Employee Selection Procedures, as established by the U.S. Equal Employment Opportunity Commission (EEOC)—or in the case of the Pilgrims, the “Selection, Hiring, and Induction Protocol” (SHIP).

The more common use of JKTs is to efficiently and accurately measure the level of mission-critical knowledge required by an employee to successfully fulfill the essential requirements of a job position. This application of JKTs has received much attention in the organizational research literature and has been well-studied since the late 80’s (that’s 1980’s, not 1580’s.)  Many large organizations have adopted job knowledge testing as a critical part of the ongoing development and coaching programs to ensure that employees are performing at the highest levels possible.  Interestingly, job knowledge testing received the most attention around jobs associated with direct customer contact as these jobs were seen to have the most impact on retaining customers—not unlike the disproportionate amount of attention given to the “Wayste and Sanitaytion Managemynt” section of the Mayflower JKT due to its impact on ... well, everyone on the ship.

The Mechanics of Job Knowledge Testing
JKTs across different organizations look pretty much the same.  They typically consist of multiple choice/multiple answer items, sometimes incorporating free-response essay-type tests.  Regardless of format, the items that make up a JKT must be reviewed regularly to ensure the items accurately reflect the functions of the job for which they purportedly test.

JKT Steps
There are established best practices associated with developing and maintaining a job knowledge test. Those steps include the following:

1) Job Analysis
2) Test Construction
3) Pilot Test
4) Quantitative Analysis of Test Results
5) Ensuring Test Maintenance

Job Analysis
An in-depth review of the skills, knowledge, tasks, competencies, etc. required to successfully perform a job is essential to ensuring the validity of the JKT.  A comprehensive job analysis will yield a JKT that logically matches the skills, knowledge, tasks, and competencies of the job.  This is not unlike the position analysis discussed in our earlier blogs regarding succession planning.  In addition to the skills required to carry out the position, a job analysis will also give insight about the demographics of the audience who will be taking the JKT, including age, gender, race, and the likelihood that an examinee is a spy for King James I, bent on sabotaging the entire venture ... just to name a few.

Test Construction
Following the job analysis, the next step is building the JKT. Test items should be developed based on those skills, knowledge, tasks, and competencies identified as the most critical during the job analysis. Again, think “Wayste and Sanitaytion” here.  Determine the medium of delivery (administered electronically, paper and pencil, quill and scroll, etc.) and the test format.  During the construction phase, consider that the JKT will be scrutinized and updated on a regular basis.  JKTs that are easily customizable will save the organization time and effort in the future.

Pilot Test
Testing the items before they are used on the general public is essential to understanding how the items “behave.”  Specific characteristics about the items are determined and used in the mathematical calculations that are ultimately used in the pass/fail rubric.  Characteristics about the items like item difficulty, item discrimination, and chance-guess levels can be determined and help the test developers create a test that measure knowledge very precisely.  A measurement specialist or “psychometrician” typically conducts these activities. If you don’t have access to either, ensure the test developer is handy with a musket, familiar with crop rotation techniques, and can speak conversational Wampanoag at the very least.

Quantitative Analysis of Test Results
Assuming that you have developed and delivered a JKT that is reliable in the psychometric sense, there is much that your organization (or new settlement) can learn with a quantitative analysis of the test results.  Your ultimate objective for administering the JKT will dictate how you approach the results analysis.  For example, if you’re assessing an examinee’s level of understanding of a position to help you make better decisions for future training opportunities, you may look for patterns where the majority of examinees responded incorrectly.

Ensuring Test Maintenance
The validity of the JKT’s content is of the utmost importance.  Periodic reviews of the test items should be performed throughout the life of the JKT, particularly in dynamic industries, such as information technology.  JKTs administered electronically may also require technological maintenance to ensure optimal performance of the testing environment.

Conclusion
Around the Orgwide offices, we have much to be thankful for, and you can bet the Pilgrims are on that list.  Seriously, it’s amazing when you think about all the things that we have to thank the Pilgrims for, isn’t it?  A food-centered holiday, awesome hats, and JKTs!  From the Orgwide family to yours, we wish all of our readers a wonderful (happy) thanksgiving!  As always—take care of your customers, take care of each other, and take care of yourself!

Jim Hartigan, Chief Business Development Officer and Partner joined OrgWide Services, a Training/e-Learning, Communications, Surveys and Consulting firm in April 2010 after nearly 30 years experience in the hospitality industry, including the last 18 as a senior executive with Hilton Worldwide. Jim’s last position was that of Senior Vice President – Global Brand Services where he provided strategic leadership and business development and support to the $22B enterprise of 10 brands and more than 3,400 hotels in 80 countries around the world. His team was responsible for ensuring excellence in system product quality, customer satisfaction, market research, brand management, media planning, and sustainability.