News for the Hospitality Executive
Beyond Awesome Hats - The Pilgrim's ultimate contribution...Job Knowledge Testing
|By Jim Hartigan
November 20, 2012
Ah, the noble pilgrims. As you gather ‘round the table with family and friends this Thursday to give thanks, we direct your attention to the difficult tests faced by the Pilgrims leading up to that very first Thanksgiving. Sure, there were the physical tests that go along with making a life for oneself from scratch in the new world. There were the mental tests that they faced during the long, treacherous voyage across the Atlantic. But, a little known fact, the Pilgrims faced another, even more challenging test! For a passenger to even be granted entry upon the Mayflower, they had to pass a written test! True story!
You might be thinking that such a test would have been simple. Perhaps a one-item, yes or no assessment along the lines of:
Dost thou wish to emancipate thyself from the tyrannical yoke of the Church of England?
Sadly, the test was not that simple—quite the opposite, in fact. This test set the standard for what we now refer to as a Job Knowledge Test. Although having slightly changed over the centuries, Job Knowledge Tests (JKTs) are one of many mechanisms that organizations use to evaluate candidates for a job opening or to measure a job incumbent’s knowledge about the job they are currently performing. Both uses of a JKT are as essential for assessing employee-job fit now as they were for assessing Mayflower-passenger fit back then.
The JKT is a critical component of the Competency Model we’ve discussed in previous blogs over the years and is illustrated in this graphic. While it officially falls under the “Assessment” heading, the JKT is really applicable in all four sections of the model. For example, in the Selection process, the purposes of JKTs range from simple tests designed to help managers determine the examinee’s level of understanding for a particular task to tests that determine critical employment decisions, such as hiring, firing, promoting, or demoting. Such tests used to determine critical employment decisions must adhere to the Uniform Guidelines on Employee Selection Procedures, as established by the U.S. Equal Employment Opportunity Commission (EEOC)—or in the case of the Pilgrims, the “Selection, Hiring, and Induction Protocol” (SHIP).
The more common use of JKTs is to efficiently and accurately measure the level of mission-critical knowledge required by an employee to successfully fulfill the essential requirements of a job position. This application of JKTs has received much attention in the organizational research literature and has been well-studied since the late 80’s (that’s 1980’s, not 1580’s.) Many large organizations have adopted job knowledge testing as a critical part of the ongoing development and coaching programs to ensure that employees are performing at the highest levels possible. Interestingly, job knowledge testing received the most attention around jobs associated with direct customer contact as these jobs were seen to have the most impact on retaining customers—not unlike the disproportionate amount of attention given to the “Wayste and Sanitaytion Managemynt” section of the Mayflower JKT due to its impact on ... well, everyone on the ship.
The Mechanics of Job Knowledge Testing
JKTs across different organizations look pretty much the same. They typically consist of multiple choice/multiple answer items, sometimes incorporating free-response essay-type tests. Regardless of format, the items that make up a JKT must be reviewed regularly to ensure the items accurately reflect the functions of the job for which they purportedly test.
There are established best practices associated with developing and maintaining a job knowledge test. Those steps include the following:
1) Job Analysis
2) Test Construction
3) Pilot Test
4) Quantitative Analysis of Test Results
5) Ensuring Test Maintenance
An in-depth review of the skills, knowledge, tasks, competencies, etc. required to successfully perform a job is essential to ensuring the validity of the JKT. A comprehensive job analysis will yield a JKT that logically matches the skills, knowledge, tasks, and competencies of the job. This is not unlike the position analysis discussed in our earlier blogs regarding succession planning. In addition to the skills required to carry out the position, a job analysis will also give insight about the demographics of the audience who will be taking the JKT, including age, gender, race, and the likelihood that an examinee is a spy for King James I, bent on sabotaging the entire venture ... just to name a few.
Following the job analysis, the next step is building the JKT. Test items should be developed based on those skills, knowledge, tasks, and competencies identified as the most critical during the job analysis. Again, think “Wayste and Sanitaytion” here. Determine the medium of delivery (administered electronically, paper and pencil, quill and scroll, etc.) and the test format. During the construction phase, consider that the JKT will be scrutinized and updated on a regular basis. JKTs that are easily customizable will save the organization time and effort in the future.
Testing the items before they are used on the general public is essential to understanding how the items “behave.” Specific characteristics about the items are determined and used in the mathematical calculations that are ultimately used in the pass/fail rubric. Characteristics about the items like item difficulty, item discrimination, and chance-guess levels can be determined and help the test developers create a test that measure knowledge very precisely. A measurement specialist or “psychometrician” typically conducts these activities. If you don’t have access to either, ensure the test developer is handy with a musket, familiar with crop rotation techniques, and can speak conversational Wampanoag at the very least.
Quantitative Analysis of Test Results
Assuming that you have developed and delivered a JKT that is reliable in the psychometric sense, there is much that your organization (or new settlement) can learn with a quantitative analysis of the test results. Your ultimate objective for administering the JKT will dictate how you approach the results analysis. For example, if you’re assessing an examinee’s level of understanding of a position to help you make better decisions for future training opportunities, you may look for patterns where the majority of examinees responded incorrectly.
Ensuring Test Maintenance
The validity of the JKT’s content is of the utmost importance. Periodic reviews of the test items should be performed throughout the life of the JKT, particularly in dynamic industries, such as information technology. JKTs administered electronically may also require technological maintenance to ensure optimal performance of the testing environment.
Around the Orgwide offices, we have much to be thankful for, and you can bet the Pilgrims are on that list. Seriously, it’s amazing when you think about all the things that we have to thank the Pilgrims for, isn’t it? A food-centered holiday, awesome hats, and JKTs! From the Orgwide family to yours, we wish all of our readers a wonderful (happy) thanksgiving! As always—take care of your customers, take care of each other, and take care of yourself!
Jim Hartigan, Chief Business Development Officer and Partner joined OrgWide Services, a Training/e-Learning, Communications, Surveys and Consulting firm in April 2010 after nearly 30 years experience in the hospitality industry, including the last 18 as a senior executive with Hilton Worldwide. Jim’s last position was that of Senior Vice President – Global Brand Services where he provided strategic leadership and business development and support to the $22B enterprise of 10 brands and more than 3,400 hotels in 80 countries around the world. His team was responsible for ensuring excellence in system product quality, customer satisfaction, market research, brand management, media planning, and sustainability.
Chief Business Development Officer & Partner
165 N. Main Street, Suite 202
Collierville, TN 38017
office: 901.850.8190 Ext. 230
Up Success: Waiting Tables Taught Me Customer Service? / Stephanie
Hartigan / October 2012
Similarities Between Marketing and Training – Start, Stop, and Continue
/ Jim Hartigan / October 2012
Loyalty in as Little as 17 1/2 Minutes! / Bob Taylor and Jim
Hartigan / August 2012
Things First - How to Start a Successful Mentoring Program / Bob
Taylor / May 2012
Can't We All Just Get Along? (It turns out, we're not supposed to!)
/ Bob Taylor / August 2011
Do Hotel Leaders Really Do? / Bob Taylor / February 2011
Management – What’s Love got to do with it? / Jim Hartigan /
Hotel Team Members the OSKAR Way - Not like training a dog! / Jim
Hartigan / January 2011
Employee Selection and the Venus Fly Trap – A Study in Floral Business
Acumen / Jim Hartigan / December 2010
Right Tool for the job makes all the difference in the world! / Jim
Hartigan / October 2010
Honor to Your Organization: Succession Planning…the Ninja Way / Jim
Hartigan / October 2010
to Manage – The importance of KPIs ... and vampires / Jim Hartigan
/ September 2010
|For Everything There is a Season - Including Budgets / Jim Hartigan / August 2010|
|Say What? Listen, Learn, and Act–Why Guess, When You Can Know Redux / Jim Hartigan / August 2010|
Guess When You Can Know - Maximizing the Effectiveness of Your
Workplace Surveys / Jim Hartigan / August 2010
Member Segmentation in the Workplace…or “If everyone brought potato
salad to the picnic – it wouldn’t be much of a picnic!” / Jim
Hartigan / July 2010
Power of Effective Communication in the Workplace (and our Founding
Fathers’ unrivaled Tweeting abilities) / Jim Hartigan / July 2010