Content Validity Report for the Customer Service Representative Job Profile



FOR THE CUSTOMER SERVICE

REPRESENTATIVE

JOB PROFILE

GCOMM

Anywhere, Minnesota

December 1, 2017

Content Validity Report

Authorized

Profilers to reproduce this document for training or marketing use.

Table of Contents

Table of Contents ....................................................................................................................... i

Executive Summary .................................................................................................................. 1

Purpose .............................................................................................................................. 1

WorkKeys Job Profiling Procedure .................................................................................. 1

Results and Recommendations ......................................................................................... 4

Section 1.................................................................................................................................... 6

User, Location, and Dates of Study ...................................................................................... 6

Section 2.................................................................................................................................... 7

Problem and Setting .............................................................................................................. 7

A. Setting ......................................................................................................................... 7

B. Current Selection Procedures ...................................................................................... 7

C. Purpose of the Study ................................................................................................... 7

Section 3.................................................................................................................................... 8

Job Analysis – Content of the Job......................................................................................... 8

A. Analysis of the Job Using the WorkKeys

Job Profiling Procedure .......................... 8

B. A Complete Description of the Work Behaviors and Their Associated Tasks ........... 9

C. Method of Determining the Measure of Importance ................................................. 10

D. Collecting Data and Calculating Ratings for the Work Behaviors ........................... 10

E. Operational Definitions of the WorkKeys Skills....................................................... 11

F. Method Used to Determine the Relationship Between Each of the Skills and the

Tasks Required on the Job .............................................................................................. 11

G. Relationships Between Each of the Skills and the Final Task List for the Job......... 11

Section 4.................................................................................................................................. 16

Selection Procedure and its Content ................................................................................... 16

A. Title, Form, And Publisher ....................................................................................... 16

B. Validity In General .................................................................................................... 16

C. WorkKeys Content Validity Evidence ...................................................................... 19

Section 5.................................................................................................................................. 29

Relationship between the Selection Procedure and the Job ................................................ 29

A. Evidence Demonstrating that the Selection Procedure is a Representative Sample of

a Skill Used as Part of a Work Behavior and Necessary for that Behavior .................... 29

B. Identification of the Work Behavior That Each Part of the Selection Procedure Is

Intended to Measure ........................................................................................................ 32

C. Comparison of the Manner, Setting, and Level of Complexity of the Selection

Procedure with Those of the Work Situation .................................................................. 33

Section 6.................................................................................................................................. 43

Alternative Procedures Investigated ................................................................................... 43

A. Testing and Performance .......................................................................................... 43

B. WorkKeys Tests ........................................................................................................ 43

C. Scope of Investigation ............................................................................................... 44

D. Method of Investigation ............................................................................................ 44

E. Investigation Findings ............................................................................................... 44

F. Investigation Conclusions .......................................................................................... 45

Section 7.................................................................................................................................. 46

Uses and Applications......................................................................................................... 46

A. Methods Considered For Use as Part of the Selection Procedure ............................. 46

B. Rationale for Choosing the WorkKeys System......................................................... 46

C. Purpose of Assessments ............................................................................................ 47

D. Determination of the Normal Expectations of the Workforce for the Cutoff Scores 47

Section 8.................................................................................................................................. 48

Contact Person .................................................................................................................... 48

Section 9.................................................................................................................................. 49

Accuracy and Completeness ............................................................................................... 49

Appendix A ............................................................................................................................. 50

References ........................................................................................................................... 50

Appendix B ............................................................................................................................. 53

Skill Descriptions ................................................................................................................ 53

Appendix C ............................................................................................................................. 88

WorkKeys Terminology ..................................................................................................... 88

Appendix D ............................................................................................................................. 91

Group A Final Task List ..................................................................................................... 91

Appendix E ............................................................................................................................. 98

Group B Final Task List ..................................................................................................... 98

Executive Summary

PURPOSE

This content validation report presents the results of an ACT

WorkKeys

profile (job

analysis) of the Customer Service Representative (CSR) job at the GCOMM site in

Anywhere, Minnesota. The profile was conducted by Filerville Community College’s

ACT authorized WorkKeys Profiler Margaret Allen, to establish task lists and identify

the WorkKeys skills (and corresponding assessments) and skill levels necessary for

selection into and effective performance of the CSR job at GCOMM. The results of this

project and review of its findings will also help to inform GCOMM’s use of WorkKeys

to improve the employee selection process for the CSR job.

When developing personnel programs such as selection procedures, it is important to

establish that they are appropriate to use for this purpose. Validation is the process of

collecting evidence to demonstrate that an employment selection procedure is related to

the context of the job. The WorkKeys system uses a content validation strategy. A

selection measure has content validity when evidence shows that it representatively

samples the important aspects of the job for which the measure will be used.

In developing the WorkKeys Program, including WorkKeys assessments and the job

profiling system, ACT has been and will continue to be guided by professional

documents such as the Standards for Educational and Psychological Testing (2014;

developed by the American Education Research Association, American Psychological

Association and National Council for Measurement in Education), the Principles for the

Validation and Use of Personnel Selection Procedures (2018, Society for Industrial and

Organizational Psychology); and the Uniform Guidelines on Employee Selection

Procedures (1978), which have been adopted by the Equal Employment Opportunity

Commission (EEOC) and various other federal agencies (Ref: 29 C.F.R. Part 1607).

This report is designed to address the requirements set forth in Section 15c of the

Uniform Guidelines. The report is organized in a manner to permit direct evaluation of

the validity of the WorkKeys assessments. Each section heading corresponds to the

section in the Uniform Guidelines (e.g., Section 15c(1) of the Uniform Guidelines is

User(s), location(s), and date(s) of study and the title of Section 1 of the report is the

same).

WORKKEYS JOB PROFILING PROCEDURE

WorkKeys job profiling is conducted by Profilers who have been trained and authorized

by ACT Industrial/Organizational Psychologists. The profiling procedure is designed to

systematically develop accurate profiles and establish content validity through a task

analysis that is used to select the tasks most important to a job, and a skill analysis that is

used to identify the on-the-job behaviors associated with the WorkKeys skills (and

corresponding assessments) under consideration and to identify the skill levels necessary

for entry and effective performance on the job (i.e., cut or passing scores).

Task Analysis

The first step in conducting the profile session was to develop a Final Task List showing

the critical tasks of the job. Dr. Allen first developed an Initial Task List using the

Department of Labor’s O*NET database, GCOMM’s job descriptions, resources from

similar job profiles, information gathered from the tour of the facility, job shadowing,

and interviews. Dr. Allen toured GCOMM's Filerville, Arkansas, location with Jane

Phillips, Customer Service Supervisor, on October 16, 2017. Dr. Allen job shadowed

Shelley Fisk and Josie Barr, Customer Service Representatives, for four hours the same

day. CSRs speak with customers and potential customers by phone or in-person,

interview applicants for telephone service, respond to customer questions or complaints

(such as billing inquiries), and assist customers in determining the calling plan which

would best suit their needs. They receive and process the paperwork for services

including installation, turn-on, discontinuance, or changes to service. They also place

sales calls to current or potential customers to explain new products or services.

Dr. Allen then met with two groups of Subject Matter Experts (SMEs) to tailor the Initial

Task List to make sure that the resulting Final Task Lists accurately and completely

described the job. Each group worked to develop one list. The Profiler met with the

first group (Group A) of six SMEs on October 18, 2017, and met with the second group

(Group B) of seven different SMEs on October 19, 2017. The SMEs in each group

worked to add, delete, consolidate, and change the descriptions of tasks, as needed, to

make sure they accurately depicted their job as it is performed at their company. Then

they independently rated each task in terms of its Importance. The average Importance

rating for each task was used to sort the task statements and list them in order, with the

most important (or critical) tasks placed at the beginning of the list. Tasks that received

an average rating of 2 (i.e., low importance) or lower were grayed out, italicized, and

moved to the bottom of the list. The SMEs then reviewed the list to see the final order

of the tasks. The Final Task List, with the tasks listed in order of Importance, can be

found at the end of this report.

Skill Analysis

A skill analysis to identify the on-the-job behaviors associated with the WorkKeys skills

under consideration and to identify how the skills are used on the job was then

conducted. The SMEs reviewed four WorkKeys skills and determined that four skills

are required. The skills required are: Applied Math, Graphic Literacy, Workplace

Documents, and Workplace Observation. The skills were considered one at a time, and

the SMEs completed their discussion of one skill before going on to the next. The

Profiler gave each SME a copy of the WorkKeys skill definition, read the definitions

aloud, and then answered any questions the SMEs had. Once the SMEs understood the

definition of a skill and had determined its relevance to the job, they independently

identified the important tasks on their Final Task List that require the skill and explained

how they use the skill to complete the identified task. For a task to be considered in the

next step to set the skill level (i.e., cut or passing score), the majority of the SMEs had to

agree that the task required the skill (see Appendix D for a table linking tasks of the job

to the WorkKeys skills).

The Profiler then presented detailed descriptions of the WorkKeys skill levels to the

SMEs (see Appendix B) which included examples of problems or situations employees

deal with at each level. Next, the SMEs determined which skill levels are necessary at

job entry and for effective performance. The profile results shown in Table 1 indicate

skill requirements for job entry and the profile results shown in Table 2 indicate the skill

requirements for effective performance. Following the Uniform Guidelines on Employee

Selection Procedures (1978), entry into the CSR job at GCOMM was defined as an

employee's first day in the job. Employees should be expected to come into the job with

the skills shown; they are not expected to learn these skill levels while in the job. The

final entry-level skill requirements reported here are recommended as cutoff (or passing)

scores on the WorkKeys assessments for entry into the job. Effective performance is the

point at which an employee performs competently without continuous supervision.

GCOMM defines this as being when a CSR has completed the four weeks of classroom

training followed by five weeks of on the-job training with a mentor, can handle an

average of 15 calls per hour, and receives ratings of “good” from the supervisor on all

monitored calls. This level of performance is typically achieved after completing the

required training, mentoring, and a total of three months in the job. The final results

shown in Table 2 indicate skill requirements for effective performance and may be used

for training purposes.

Table 1. Entry – Level Skill Requirements for the Job

WorkKeys Skill

Skill Level

Range Group A Group B

Final Entry

Level

Applied Math

3–7 4 4 4

Graphic Literacy

3–7 4 4 4

Workplace Documents

3–7 4 4 4

Workplace Observation

1–5 2 2 2

Table 2. Effective Performance Level Skill Requirements for the Job

WorkKeys Skill

Skill Level

Range Group A Group B

Final Effective

Level

Applied Math

3–7 4 4 4

Graphic Literacy

3-7 4 4 4

Workplace Documents

3–7 4 4 4

Workplace Observation

1–5 2 2 2

RESULTS AND RECOMMENDATIONS

There are several issues to consider before using the skill levels established by the

profile to set expectations for employees and potential employees in the CSR job.



The WorkKeys assessment scores should be used in conjunction with other criteria

as determined by GCOMM (e.g., interviews, employment history, application

reviews) when making selection or other high-stakes employment-related decisions.



GCOMM is not required to administer WorkKeys assessments for all of the skills

included in the profile. Administering WorkKeys assessments along with using other

selection measures (e.g., structured interviews) for the screening of applicants into

the profiled job should provide sufficient information. GCOMM may want to

consider administering assessments only for those skills most relevant to the job.



For selection and promotion purposes, GCOMM should consider using WorkKeys

assessments for the following skills: Workplace Observation, and Graphic Literacy.

If GCOMM chooses to administer additional assessments, the following skills should

be considered: Workplace Documents or Applied Math. This recommendation is

based on the number of important/critical tasks identified by the SMEs as requiring

each skill. If GCOMM would like to use different WorkKeys assessments, the

Profiler should be consulted regarding the number of important tasks associated with

each related skill. The reasons for assessing individuals on those skills should then

be documented.



For training and development purposes, GCOMM should consider using WorkKeys

assessments for the following skill areas: Workplace Observation, Graphic Literacy,

Workplace Documents, and Applied Math. Training on skills at the beginning of the

list may provide more impact than training on skills at the end of the list because the

skills at the beginning of the list are more relevant to performance of the job.



If GCOMM would like to use the profile results for hiring applicants at any

additional locations, then GCOMM should consider conducting one or more profile

sessions across these locations. The purpose of this is to have adequate

representation of incumbents and of the work performed at the additional location(s).

If the job does not yet exist at a location (e.g., a new facility), but the jobs are

anticipated to be the same (i.e., using similar equipment), then these profile results

can be used for selection purposes until subject matter experts at the new facility

become available. At that time, profiling should be conducted at the new facility to

confirm that the job requirements are the same.



The use of assessment scores for making decisions regarding reductions in force

should be carefully considered. Use of actual measures of employee performance

such as performance appraisal results and attendance records should be taken into

account when making decisions of this nature.



If the selection system does not yield enough qualified candidates, modification of

the selection system or a component of the selection system may increase the size of

the applicant pool. Recruiting efforts could be increased (e.g., by increasing

advertising efforts to cover a larger geographic area), requirements of the job could

be modified, or training could be adjusted to address skill gaps. If the job is

modified, the Profiler should be consulted to reevaluate the profile.



For the WorkKeys assessments to be as useful as possible, their placement within the

selection process should be considered carefully. In general, selection tools should

be used in an order that allows the most economical and efficient measures to be

placed at the beginning of the process. For example, if an employer's selection

battery consists of (a) one-on-one interviews and (b) WorkKeys assessments, the

employer may wish to administer the assessments prior to conducting interviews.

One-on-one interviewing of candidates may take significantly more staff time, so

administering assessments to groups of candidates first may be more efficient

because it can reduce the number of interviews.

Section 1

User, Location, and Dates of Study

This content validation report presents the results of an ACT WorkKeys® job profile

(job analysis) of the Customer Service Representative (CSR) job at the GCOMM

(GCOMM) site in Anywhere, Minnesota. The profile was conducted by Filerville

Community College’s ACT authorized Profiler, Dr. Allen.

Dr. Allen met with the first group (Group A) of six subject matter experts (SMEs) on

October 18, 2017, and met with the second group (Group B) of seven different SMEs on

October 19, 2017.

Section 2

Problem and Setting

A. SETTING

The focus of this study is on GCOMM’s CSR job in Anywhere, Minnesota.

B. CURRENT SELECTION PROCEDURES

To hire applicants into the CSR job, GCOMM reviews applications and conducts

interviews. New hires attend four weeks of classroom training and then complete

comprehensive on-the-job training for the next five weeks. During the on-the-job

training, trainees are placed with an experienced worker and continue to learn the

responsibilities of the job as coached, demonstrated, and critiqued by their peer trainer.

At the end of three months in the job, CSRs are evaluated by their supervisor and must

receive ratings of “good” on all monitored calls in order to retain their position.

GCOMM anticipates adding WorkKeys assessments to this procedure.

C. PURPOSE OF THE STUDY

GCOMM intends to implement the WorkKeys system, which includes the job profiling

procedure documented here and the job-related WorkKeys skill assessments. The

WorkKeys system is used to determine an individual’s levels of proficiency in specific

skill areas and to identify pools of qualified applicants who have achieved the levels of

proficiency needed to perform a specific job, as determined through a job analysis using

the WorkKeys job profiling procedure. GCOMM can use the results of this study to

support the inclusion of the WorkKeys assessments in the selection procedure for the

CSR job. The WorkKeys system can also be used to identify skill gaps among

applicants and incumbents, and to design training to eliminate such gaps. GCOMM may

use the effective performance skill levels as training goals for the CSR job.

Section 3

Job Analysis – Content of the Job

A. ANALYSIS OF THE JOB USING THE WORKKEYS

JOB PROFILING PROCEDURE

The WorkKeys job profiling procedure is a method of job analysis designed to help

businesses identify the WorkKeys skills (and corresponding assessments) and the skill levels

employees need in order to perform a particular job effectively as well as to establish

content validity. It also gives individuals a clear picture of the skill levels they need if they

are to qualify for and be successful in the jobs they want. When combined with the

assessments, instructional support, and reporting, job profiles allow employers to make

appropriate hiring and training decisions and allow individuals to make appropriate

decisions about jobs and identify areas they need to strengthen as they pursue their

education and career goals. The WorkKeys job profiling procedure is designed to

systematically develop accurate profiles through a task analysis that is used to identify the

tasks most critical to a job, and a skill analysis that is used to identify the skills and skill

levels required for entry-level and for effective performance of that job (i.e., cut or passing

scores).

Dr. Allen met with the first group (Group A) of six subject matter experts (SMEs) on

October 18, 2017, and met with the second group (Group B) of seven different SMEs on

October 19, 2017. During the profile sessions, the groups developed task lists that

accurately and completely describe the job. The SMEs reviewed four WorkKeys skills and

determined that four are required: Applied Math, Graphic Literacy, Workplace Documents,

and Workplace Observation.

The results from the task analysis (including the development of the Final Task List) and the

skill analysis (including the identification of the WorkKeys skills required for the CSR job)

are described in this section of the report. The discussion and identification of the

appropriate skill levels are presented in Section 5 of this report. The SME demographic

information for the profile and incumbent population totals for the job are provided below.

Job Status Population Group A Group B

Incumbent 100 6 7

Gender Population Group A Group B

Female 58 3 4

Male 42 3 3

Racial/Ethnic Group Population Group A Group B

African American/

Black, Non-Hispanic

24 2 1

Asian-American or

Pacific Islande

8 0 1

Caucasian/White,

Non-Hispanic

42 3 3

Mexican-American/

Chicano

16 1 1

Othe

10 0 1

Years in Profiled Job Group A Group B

Avera

e 20 13

hest 35 17

Lowest 2 1

Years with Company Group A Group B

Avera

e 22 14

hest 35 20

Lowest 3 1

Age Group A Group B

Avera

e 54 41

hest 60 49

Lowest 28 20

B. A COMPLETE DESCRIPTION OF THE WORK BEHAVIORS AND THEIR ASSOCIATED

TASKS

Creating an Initial Task List

To profile the job, the Profiler first developed a comprehensive Initial Task List using the

Department of Labor’s O*NET database, GCOMM’s job descriptions, resources from

similar job profiles, and information gathered from a tour of the facility.

Conducting the task list review

To develop the Final Task List, the Profiler met with the SMEs to edit the Initial Task List to

make sure the task statements accurately and completely described the work required of

incumbents. The SMEs then evaluated each task on the task list in terms of its Importance.

The average Importance rating for each task was used to sort the task statements and list

them in order, with the most important tasks placed at the beginning of the list. The SMEs

then reviewed the list to see the final order of the tasks. The Final Task List(s) are shown at

the end of this report.

C. METHOD OF DETERMINING THE MEASURE OF IMPORTANCE

Importance refers to the significance of the task to overall job performance. The SMEs are

asked to consider what may happen if the task is not performed properly (excluding the

effect of gross negligence or intentional sabotage). Each SME is asked to rate each task on a

6-point scale from 0 (This task is not performed) to 5 (This task is critical/extremely

important to the job I perform). The Importance Rating Scale shown below is given to the

SMEs when they make their Importance ratings. The average Importance rating for each

task was used to sort the task statements and list them in order, with the most important

tasks placed at the beginning of the list. Tasks that received an average rating of 2 or lower

were grayed out, italicized, and moved to the bottom of the list. The SMEs then reviewed

the list to see the final order of the tasks.

Importance Ratin

Scale

Step 1 Read each task statement.

Step 2 If the task is NOT performed as part of your job, write a zero (0) in the box next

to the task statement. If the task is performed as part of your job, think about how

important the task is to your job.

Importance refers to the significance of the task to overall job performance. In

evaluating Importance, consider what may happen if the task is not performed

properl

. (Exclude the effects of

ross ne

ence or intentional sabota

e.)

Step 3 Choose the Importance statement that best describes how important the task is to

your job. Write the number of that Importance statement in the box next to the

task statement.

Importance Level

0 This task is not performed.

1 This task is not ver

important to the

ob I perform.

2 This task is of low importance to the

ob I perform.

3 This task is important to the

ob I perform.

4 This task is ver

important to the

ob I perform.

5 This task is critical/extremel

important to the

ob I perform.

D. COLLECTING DATA AND CALCULATING RATINGS FOR THE WORK BEHAVIORS

After carefully examining and editing the Initial Task List, the SMEs rated each task

according to its Importance. The average Importance rating was used to sort the tasks,

placing the most important tasks at the top of the list. The SMEs then reviewed the list to

see the final order of the tasks. The Final Task List(s) are shown at the end of this report.

E. OPERATIONAL DEFINITIONS OF THE WORKKEYS SKILLS

The WorkKeys skill definitions are provided in Appendix B.

F. METHOD USED TO DETERMINE THE RELATIONSHIP BETWEEN EACH OF THE

SKILLS AND THE TASKS REQUIRED ON THE JOB

The SMEs completed a skill analysis to identify the on-the-job behaviors associated with the

WorkKeys skills under consideration and to describe how the skills are used on the job. The

skills were reviewed one at a time, and the SMEs finished the analysis for one skill before

going on to the next. The Profiler gave each SME a copy of the WorkKeys skill definition,

read the definition aloud, and then answered any questions the SMEs had. Once the SMEs

understood the definition of a skill and had determined its relevance to the job, they

independently identified the important tasks (i.e., those tasks that received ratings of

3/important or higher) on their Final Task List that require the skill. Then, the SMEs

discussed how they use the skill to complete the identified task. Based on this discussion,

the Profiler documented the Important/Critical tasks for which a majority of the SMEs

agreed that the skill is required for task performance. See Appendix D for a table linking

tasks of the job to the WorkKeys skills.

G. RELATIONSHIPS BETWEEN EACH OF THE SKILLS AND THE FINAL TASK LIST FOR

THE

JOB

Applied Math

WorkKeys Applied Math is the skill people use when they use mathematical reasoning and

problem-solving techniques to solve work-related problems. Employees may use

calculations and conversion tables to help with the problems, but they still need to use math

skills to think them through.

The two groups of SMEs indicated that Applied Math skills are used to perform 30% (Group

A) and 26% (Group B) of the critical tasks on their Final Task Lists, respectively.

According to the SMEs, CSRs use Applied Math skills when they:



handle customer complaints by making an adjustment to their monthly bills, such as to

subtract an errant charge from the total of the bill and explaining the corrected total of

the bill ($155 total bill - $10 errant charge = $145 corrected total);



help customers understand their bills by explaining the calculation of charges on the bill

such as taxes, fees, toll charges, subtotals ($155 + $8 taxes = $163);



sell services or upgrades to business or residential clients and use one or two operations

of mathematics to determine the estimated monthly charge for one option compared to

another (such as to compare a voice-only plan of $50, to a voice plus data plan of $75, or

to a voice plus equipment plan of $80);



quote prices for customized plans by adding and subtracting the price of services and/or

equipment and then adding to obtain the new total charges;



determine cost differences between GCOMM’s prices and competitor’s prices by adding

up the prices entered on the GCOMM Price Quote worksheet and subtracting total from

current amount paid by customer to competitor ($185 - $175 = $10 saved); and



train new CSRs by listening to their customer calls, completing the same steps of

mathematics used by the CSR during the call, and then comparing their answers with the

trainee’s answers.

Graphic Literacy

The WorkKeys Graphic Literacy skill is the skill people use when they work with workplace

graphics such as tables, graphs, charts, digital dashboards, flow charts, timelines, forms,

maps, and blueprints. Employees use this skill when they find, summarize, compare, and

analyze information to make decisions using workplace graphics to solve work-related

problems.

The two groups of SMEs indicated that WorkKeys Graphic Literacy skill is required for

performing 100.0% (Group A) and 100.0% (Group B) of the critical tasks on their Final

Task Lists, respectively.

According to the SMEs, CSRs use Graphic Literacy skills when they:



communicate with customers to select and purchase telephone services using pricing

charts and Price Quote Worksheets;



update customer accounts by updating or deleting information from various fields of

computer screens (e.g., to make address, service plan, or other changes on Client

Information, Services, or Payment History screens);



open new customer accounts by entering information (e.g., name, address, service(s)

requested, whether customer has previous history with GCOMM, employment

information) into fields of GCOMM’s New Accounts System or by referring to a

completed Application for Service and transferring data from the application into the

system;



process customer refunds over phone with customer or by referring to Refund Request

Form and entering approval/denial for refund by filling in pertinent fields such as to

enter checkmarks (e.g., reason for refund/denial) or filling in applicable approval code

(e.g., “021” for charges less than $5) in the customer’s account;



sell services to new/existing clients using computer printouts (containing tables of

prospect data) and service plan information (e.g., pricing charts, Price Quote

worksheets), and completing Follow-up Contact Form at conclusion of call;



share new product and service information with existing and potential customers using

marketing materials (tables, charts) and the Sales Goals Projection Report;



handle billing complaints using troubleshooting diagrams, Call Charge Logs, scripts

(graphics sections);



set-up payment schedule for delinquent account by entering data into sequential fields of

Delinquent Account Catch-Up (DACU) screens to arrange for customer payment

including customer data (name, account number, address), amount in arrears, beginning

date for late payment, special circumstances; and



train/coach new Customer Service Representatives (using both the training version and

then actual versions of the graphics listed above).

Workplace Documents

WorkKeys Workplace Documents is the skill employees use when they read and use written

text in order to do a job. The written texts include messages, emails, letters, directions,

signs, notices, bulletins, policies, websites, contracts, and regulations. It is often the case

that these workplace communications are not necessarily well written or targeted to the

appropriate audience. Workplace Documents materials do not include information that is

presented graphically, such as in charts, forms, or blueprints.

The two groups of SMEs indicated that the WorkKeys Workplace Documents skill is

required for performing 33.3% (Group A) and 27.0% (Group B) of the critical tasks on their

Final Task Lists, respectively.

According to the SMEs, CSRs use reading skills when they use the following materials:



CSR Procedure Manual – which is read and the information is then applied to perform a

variety of tasks including resolving customer inquiries or complaints, processing

requests for refunds, initiating work orders, and selling new/additional products and

services to clients;



prepared scripts (which outline typical conversations with customers) – which are read

and the content understood so that the CSR can convey the content convincingly to the

customer and be able to respond to customer questions;



help screens – the content of which has been read and understood by the CSR so that

during a call they can quickly scan the relevant help screen and then provide an answer

to the customer (in his/her own words preferable to reading answer to customer);



product brochures or notices, marketing materials, and company update memos – which

are read and the information applied in order to prepare for sales calls or to recommend

additional services to customers during routine calls; and



training materials – which are provided during the 4 weeks of classroom training (and

include print and online resources) that describe the job and the resources used on the

job including procedures, job aids (e.g., help screens, glossaries) marketing materials,

scripts, and forms.

Workplace Observation

The WorkKeys Workplace Observation skill is an employee’s skill in visually observing a

workplace event, noticing details, and remembering instructions, procedures, processes, and

demonstrations in order to generalize to workplace situations that may be similar or very

different from what was observed.

The two groups of SMEs indicated that Workplace Observation skills are used to perform

100.0% (Group A) and 100.0% (Group B) of the critical tasks on their Final Task Lists,

respectively.

According to the SMEs, tasks that CSRs perform using Workplace Observation skills can be

grouped into the categories of learning the tasks of the CSR job, performing the tasks of the

CSR job, and mentoring the tasks of the job.

Specifically, CSRs use Workplace Observation skills when they:



Learn the job

– CSRs use Workplace Observation skills when they complete their 4-weeks of classroom

training.

– CSR tasks are demonstrated via video-based scenarios, online step-by-step instructional

software, and role-play activities.

– Trainees learn the job by observing demonstrations and then practicing, either individually

or in group settings, and receiving constructive feedback from the trainer(s) and their

peers.

– Group activities involve observing one’s peers take practice calls.

– Individual activities include taking practice calls with the Simulator by observing

variables such as lights (such as those that indicate incoming calls), timers (which indicate

the length of the call), and panel displays (such as those which indicate whether the caller

is a current customer).

– CSRs also use observation skills as they mimic the steps shown in online tutorials in order

to learn the functions of the Customer Service software.



Perform the job

– CSRs work at workstations where they use information displayed through numerous

computer screens and following established procedures to accomplish their tasks.

– They have to watch for error messages, incomplete record messages, and incorrect codes.

– CSRs find information on one computer screen and remember that information in order to

take action on a different screen (such as to open the customer’s account, notice the

account has been ‘flagged,’ and take action by documenting a payment plan (on the

DACU screen of their account) before completing the customer’s request to add any

additional services.

– They use Workplace Observation skills as they handle complicated customer complaints

which may require them to retrieve information from multiple screens (e.g., monthly

statement, payment history, services list), incorporate what they’ve seen on those screens,

and take action such as by removing a charge(s) or by telling the customer that because of

their inconsistent payment history, that they will have to return their account to good

standing before adding any new services.

– Workplace Observation is used to take action by documenting Incidents on the Follow-up

Contact form based on observations made during a call.



Mentor other CSRs to perform the tasks of the job

– GCOMM has a 5-week mentoring program for new CSRs once they have completed the

4-week training program. Experienced CSRs complete training so that they can participate

as mentors.

– Over the 5-weeks, the mentors guide their mentees through planned activities designed to

grow their skills as a CSR.

– Mentors observe new CSRs and take action based on those observations (e.g., to watch

new CSRs while they are on the phone with customers and make sure that the new CSRs

enter the information correctly into forms and computer screens and take action by

discussing the aspects of the task that were performed incorrectly).

Skill definitions and skill levels

The Final Task List(s) for the CSR job are shown at the end of this report in the appendix.

The mean Importance ratings and skill requirements are also shown. A checkmark in a skill

column means that, according to the SMEs in the profile session, the task in that row

requires that skill. Tasks shown in gray italics are of low importance (i.e., they received

averages ratings below 3) and they did not influence the skill levels set for the profile. The

skill and skill level definitions are shown in Appendix B. WorkKeys terms are defined in

Appendix C.

Section 4

Selection Procedure and its Content

A. TITLE, FORM, AND PUBLISHER

Titles and Forms

ACT recommends using individual assessments as specified in the Executive Summary.

Representative titles and forms are listed in the table below, but are updated regularly.

Contact ACT Customer Service for information on the most current forms at all times.

WorkKeys

Assessment

Paper and Pencil Internet

Applied Math

J91AH J92AH

J93AH J94AH

J01AH (Retest)

WKAM02101CD0

WKAM02102CD0

WKAM02103CD0

WKAM02104CD0

Applied Technology

G10FG A08FF A30911

A40311

Business Writing

N/A J01BW

H01BW

Graphic Literacy

J91AI J92AI

J93AI J94AI

J01AI (Retest)

WKGL00101CD0

WKGL00102CD0

WKGL00103CD0

WKGL00104CD0

Workplace Documents

J91AJ J92AJ

J93AJ J94AJ

J01AJ (Retest)

WKWD00101CD0

WKWD00102CD0

WKWD00103CD0

WKWD00104CD0

Workplace

Observation

NA A01WO

C01WO

Publisher

ACT, Inc. with its headquarters in Iowa City, Iowa.

B. VALIDITY IN GENERAL

When the word “validity” is used in a testing context, it frequently refers to the three-part

model of validity: content validity, construct validity, and criterion-related validity (Guion,

1980). Most of what has been written and reported with respect to the valid use of test

scores has used these three concepts. Almost every measurement textbook uses these

concepts as the basis for its organizational structure (Anastasi, 1982; Hopkins, Stanley, &

Hopkins, 1990), and they are a central feature of the Uniform Guidelines on Employee

Selection Procedures (1978): “For the purposes of satisfying these guidelines, users may

rely upon criterion-related validity studies, content validity studies, or construct validity

studies” (CFR 41, 60-3).

The Standards for Educational and Psychological Testing (AERA, et al., 2014) uses these

concepts when it describes methods of collecting data for validating the uses of test scores,

but it describes validity as a unitary concept supported by evidence. Instead of focusing on

types of validity, it discusses the same concepts in terms of the accumulation of validity

evidence. The Principles for the Validation and Use of Personnel Selection Procedures

(SIOP, 2018) also embraces this view. So while the Guidelines talks about “criterion related

validity,” the Standards talks about “evidence of the relation of test scores to a relevant

criterion” (p. 17). While the Guidelines talks about “content validity,” the Standards talks

about “evidence based on test content” (p. 14). And when it comes to “construct related

validity,” the Standards explains that this is established by “the validity argument” (p. 21),

which “integrates various strands of evidence into a coherent account of the degree to which

existing evidence and theory support the intended interpretations of test scores for specific

uses.” (p. 21). Evidence may be accumulated in a number of ways. What is relevant is that

validity is established as a whole. The value of each way of collecting evidence is

determined by its appropriateness to the situation, not by any inherent value of its own.

The Standards (pp. 11–14) explains that the need for validity evidence is based on the

assumption that a test is going to be used for a purpose, and it is necessary to provide

evidence showing that it is appropriate to use the test for that purpose. When a test is

administered and a score is reported, it is necessary to determine what the score means

within the context in which it is being used. This is done by accumulating evidence that

shows what the test score means. Validity refers to the degree to which the evidence

supports the interpretation of the scores for proposed uses of tests. This is what is validated,

not the test itself. Therefore, the validation process involves accumulating relevant evidence

to provide a sound scientific basis for the proposed score interpretations. Validation is

achieved when a scientifically sound validity argument has been presented. Such an

argument is one that supports the intended interpretation of the test scores by showing what

they mean within a specific context. The validation process is unitary in that when the

evidence is collected and analyzed, the results will be described in terms of one concept,

“validation,” and not in terms of the three concepts mentioned above. Any or all of the three

may be used as evidence, but none should be viewed as an end in itself. In this, the

Standards differs from the Guidelines, the requirements of which can be met by establishing

validation using any one of the three types of validity it specifies.

Different uses of the WorkKeys system

Education and businesses can use WorkKeys scores for a variety of purposes. For example,

potential educational uses of WorkKeys scores include identification of proficiency status,

identification of educational needs, program evaluation, certification, and guidance and

vocational counseling. The examinees in this setting include learners of all kinds in settings

such as general education, vocational education, college preparation, and special

certification programs. Potential business score uses include identification of training needs,

training program evaluation, certification, and selection of individuals to be hired, promoted,

or transferred. The examinees in this setting include a wide range of potential job applicants

as well as incumbent employees. These individuals may range greatly in such

characteristics as age, socioeconomic status, language background, and educational history.

Specifically, the WorkKeys scores can be used by education and businesses as support for

their decisions. The figure below provides a depiction of how the WorkKeys system can be

used to provide a rationale that supports a variety of judgments made with the assessments.

Essentially, the figure illustrates the process from identification of which tests to use, to

setting standards of test performance, administration of the tests to examinees, and making

decisions based upon whether or not the individuals meet the standards. For example,

assume that a business is interested in using the WorkKeys tests for selection of applicants

into a job. To begin, they identify the WorkKeys skills that are required for the job (identify

relevant tests) as well as the required level of performance of that skill (set standards) using

the results of a job profile. Then, using the WorkKeys tests for the required skills, the

employer administers tests to job candidates and compares the scores to the standards set by

the profile. Candidates who meet the standard may be selected for employment, or if the

selection system were structured in hurdles, the employer would move the candidates to the

next stage of the selection process. Candidates who do not meet the standards would be

removed from the employment selection process and could be referred to an organization

that provides skill remediation.

Using the WorkKeys System for Decision-Making Support

C. WORKKEYS CONTENT VALIDITY EVIDENCE

When developing personnel programs such as selection procedures, it is important to

establish that they are appropriate to use for this purpose. Validation is the process of

collecting evidence to demonstrate that an employment selection procedure is related to the

context of the job. The WorkKeys system primarily uses a content validation strategy. A

selection measure has content validity when evidence shows that it representatively samples

the important aspects of the job for which the measure will be used. The use of WorkKeys

assessments, as specified in the Executive Summary, has been validated in accordance with

the EEOC’s Uniform Guidelines on Employee Selection Procedures (1978).

According to the Standards, important validity evidence can be obtained from an analysis of

the relationship between a test’s content and the constructs it is intended to measure. Test

content includes themes, wording, and the format of the items, tasks, or questions on the test

(p. 14). Each WorkKeys assessment has been built by first defining a hierarchical skill scale

and then creating test items to be representative of both the skill area and each skill level

within that skill area. This process is described below.

Selection of skills

In developing the WorkKeys system, ACT consulted with employers, educators, and labor

organizations to develop a list of generalizable workplace skills that are used in a wide range

of jobs, are teachable in a reasonable period of time, and facilitate job analysis (ACT, 1992).

In 1992, the initial WorkKeys skills were selected on the basis of a review of the literature

relating to employer-identified skill needs (Agency, 1989; ACT, 1987; Bailey, 1990;

Carnavale, Gainer, & Meltzer 1990; Center, 1990; Conover, 1991; Educational Testing

Service, 1975; Electronic Selection Systems, 1992; Greenan, 1983; SCANS, 1990) and a

survey of employers and educators who participated in the design of the WorkKeys system.

The latter were from seven states (Illinois, Iowa, Michigan, Ohio, Oregon, Tennessee and

Wisconsin) and a network of community colleges in California. They assisted in the design

and review of plans and materials and provided examinees for the prototype and field-test

phases of assessment development. Twelve skills were selected for initial development

based on those identified by educators and employers (McLarty, 1992). It was anticipated

that this list would be modified over time in response to changing employer needs.

ACT released the first WorkKeys assessments, Applied Mathematics, Reading for

Information, Listening, and Writing in 1992. Applied Technology, Locating Information,

and Teamwork were released the next year, followed shortly by Observation. The Business

Writing assessment was released in 2002; Workplace Observation was released in 2009; and

Listening for Understanding was released in 2011. In 2017, the Applied Mathematics

assessment was renamed Applied Math and the Reading for Information assessment was

renamed Workplace Documents. The Graphic Literacy assessment was introduced in 2017.

New items are generated and new forms are released periodically. In response to changing

employer demands, ongoing development of new skill areas has continued over the years.

Skill scale development

In assessment development, it is common for the assessment to be developed first and for

the development of score scales to follow (Crocker & Algina, 1986; Nunally & Bernstein,

1994). This is necessary for any score scale that depends on examinee data for its

construction (e.g., norm-referenced score scales) because in that case, the skill scale and

score scale are necessarily identical. Each WorkKeys skill scale can be conceptualized as an

independent definition of the construct to be measured, a definition that is not based on the

psychometric characteristics of the assessment. The score scales reflect the characteristics

of each assessment and can be evaluated with respect to how well the assessment scores

represent the designated skills and skill levels. More than one assessment approach can

conceivably use the same skill scale. In the development of the WorkKeys system, the need

to link job analysis with the assessment of individuals argued for separating the skill scales

from the score scales so that both assessment of jobs (job profiling) and the assessment of

individuals could be described by the same set of skill scales. It was therefore necessary to

develop the skill scales before developing the tests and their score scales.

Several skill scale criteria were identified by WorkKeys staff as critical for the operational

system (McLarty, 1992). WorkKeys skill levels would have to (a) be readily interpretable

as a description of what the examinee can do and of the skills required by the job; (b) be

appropriate for large-scale use and for validation as part of a system for selecting qualified

job applicants; and (c) provide information that could be useful to a person wishing to

improve skills in order to meet job requirements, an educator or trainer wishing to assist

people in improving their job-related skills, and an employer wishing to select well-qualified

employees. A primary goal was that the skill scales should be able to communicate to test

takers, teachers, trainers, employers, labor union leaders, and other audiences, the levels of a

generalizable skill that a job requires and that a person possesses. Based on their content,

the WorkKeys skill scales would have to communicate clearly and concisely to people

making decisions using the assessment results.

The WorkKeys skill scales are, therefore, intentional scales, created to ensure that the

resulting score scales will be both meaningful and hierarchical. Cognitive and content-

related aspects of a skill are analyzed to identify and combine their component strands in

order to generate hierarchical scales that will be meaningful to individuals, educators, and

employers. Aspects of content knowledge and cognitive skills that do not contribute to the

hierarchical scale are excluded from it. The quality of the resulting skill scales is then

judged by the degree to which they serve as a common metric to link the job analysis to the

assessments and by their usefulness in identifying the skill levels required by jobs and

possessed by individuals. Clustering approaches (identifying skills perceived as different

but equally valuable) are intentionally excluded from the scale-building process, although

such clusters could become the basis of separate skill scales.

Thus, the WorkKeys skill levels are designed to be standardized, but particular to each skill.

That is, Level 6 in Workplace Documents is not the same as Level 6 in Graphic Literacy.

Level 6 in Workplace Documents does, however, mean the same skill level whether it is

used to describe a job or a person, just as Level 6 in Graphic Literacy means the same thing

when it is an individual’s assessment score that it means when it is part of a job profile. The

common metric, then, forges a link between assessments results and job analysis, but does

not refer to any relationship between the skill areas.

Currently, six WorkKeys assessments are available for operational use: Applied Math,

Applied Technology, Business Writing, Graphic Literacy, Workplace Documents, and

Workplace Observation. In defining the basic or foundational skill domain to be covered in

each WorkKeys assessment, ACT applies the following criteria:



The way a skill is assessed is generally congruent with the way the skill is used in the

workplace.



The lowest level assessed approximates the lowest level for which an employer would be

interested in setting a standard.



The highest level assessed approximates the level beyond which specialized training

would be required.



The score scales are designed to have practical value in documenting workplace skills.



The assessments are sufficiently reliable for high-stakes decision making such as

selecting candidates for jobs.



The assessments can be validated against empirical criteria.



The assessments are feasible with respect to cost, administration time, and complexity.

The development process for a WorkKeys assessment consists of five phases: skill

definition, test specification development, prototyping, pretesting, and construction of

operational forms.

Phase 1: Skill Definition. Working with an expert panel of employers and educators, ACT

first defines the content domains and develops hierarchical WorkKeys skill descriptions.

The panel is asked to develop a broad definition of a skill area and to identify the lowest and

highest levels of the skill to be measured. The panel then identifies examples of tasks within

this broadly defined skill domain and narrows that domain to those examples that are

important for job performance across a wide range of jobs. Next, the tasks are organized

into subtasks (sometimes called “strands”), which are aspects of the general skill domain or

skill area to be measured. The subtasks assessed in Workplace Documents, for example,

include “choosing main ideas or details,” “understanding word meanings,” “applying

instructions,” “and applying information and reasoning.”

Each subtask is also divided into levels based on the variables thought to make it more or

less difficult. In general, at the lower levels of difficulty, the employee has few variables to

attend to. However, at the higher levels of difficulty, the employee must be able to process

information to apply it to more complex situations. For example, at the lower levels of the

subtask “applying instructions” in Workplace Documents, the employee need only apply

instructions to clearly described situations. At the higher levels, however, employees must

not only understand instructions in which the wording is more complex, meanings are more

subtle, and multiple steps and conditionals are involved, but they must also apply these

instructions to new situations.

Phase 2: Test Specification Development. Using the skill definitions developed in

Phase 1 for multiple-choice tests, the ACT WorkKeys development team prepares test

specifications, which detail the skills the assessment will measure and how the items will

become more complex as the skill levels increase. Each level is defined in terms of its

characteristics, and exemplar multiple-choice items (test questions) are created to illustrate

it. Each item is comprised of a stimulus, a stem, and a set of response options. A stimulus

may be a recorded voice, reading passage, video scenario, described situation, or one or

more graphics, depending on the test. The stem is the part of the item for which an answer,

or response option, must be selected. In most cases the complexity of the stimulus and the

stem determines the skill level to which a particular test item is assigned.

The WorkKeys Business Writing assessment uses a constructed-response (essay) format.

For this assessment, the WorkKeys test development staff works together with professionals

from a wide variety of job backgrounds to identify and refine prompt ideas that require a

written response and have meaning to a broad audience. Next, they develop and revise

prompts that are realistic, do not require prior knowledge, are specific and substantive

enough to hold the examinee’s interest, and are representative of situations that could arise

in a wide variety of jobs. Prompts that meet all of these criteria are edited into a form that is

clear, direct, and concise.

New prompts for all WorkKeys tests are reviewed for realism, accuracy, and fairness by a

number of people including other members of WorkKeys development staff, staff members

from other ACT departments, and business and academic experts outside of ACT from

diverse cultural and ethnic backgrounds.

Phase 3: Prototyping. In Phase 3, the ACT WorkKeys development team recruits content

experts to write items for prototyping. These experts draft items designed to meet the test

specifications and correspond to the respective skill levels. ACT staff then edits the items,

producing a number sufficient to create one full-length test form for the skill area.

The prototype test form is administered to convenience samples, which typically are at least

two groups of high school students and two groups of employees. Usually, one group of

students and one group of employees will be from the same city. The second groups of

students and employees will be found in another state with different demographic

characteristics. For example, if the first groups are from a suburban setting, the second may

be from an urban area. The number of examinees varies according to the test format, with

more being used for multiple-choice tests than for constructed-response tests. Typically, at

least 200 students and 60 employees are divided across the two administration sites for each

multiple-choice prototype test form. In 2001, ACT conducted a prototype administration of

the first forms of the Business Writing assessment to approximately 2,640 Midwestern high

school and community college students.

During the prototype process, interviews or surveys of the examinees are used to gather their

reactions to the test instrument. Educators and employers are also invited to review and

comment on the test. This information helps ACT evaluate the functioning of the test

specifications. Questions such as whether the prototype items were too hard, too easy, or

tested skills outside the realm of the specifications must be answered before development

can move to the pretesting stage. Based on all the information from the prototype testing,

the test specifications are adjusted if necessary, and additional prototype studies may be

conducted. When the prototype process is completed satisfactorily, a written guide for item

writers is prepared.

Phase 4: Pretesting. For the pretesting phase of item development, ACT contracts with

numerous freelance content experts who write a large number of items. WorkKeys item

writers must be familiar with various work situations and have insight into the use of a

particular skill in different employment settings to ensure content and contextual accuracy.

ACT staff edits the submitted items to meet content, cognitive, and format standards. A test

item containing inaccurate content may be distracting even if the specific content does not

affect the examinee’s ability to respond correctly to the skill portion of the item. For this

reason, inaccurate facts, improbable circumstances, or unlikely consequences of a series of

procedures or actions are excluded from the content of the items. An examinee who knows

about a particular workplace should not recognize any of the assessment content,

circumstances, procedures, or keyed (correct) responses as unlikely, inappropriate, or

otherwise inaccurate.

Given the wide range of employability skills assessed, verifying content accuracy for

WorkKeys is challenging. To help WorkKeys staff detect any possible problems, the item

writers write a justification for the keyed response and for each incorrect response for each

test item. ACT editors check both the items and the justifications, and make modifications,

if needed.

After the items have been edited, they are submitted to external consultants for content and

fairness reviews. Qualified experts in the specific skill area being assessed, usually persons

utilizing the skills regularly on the job, check for content and contextual accuracy. Members

of minority groups review the items to make sure they will not be unfair or offensive to any

racial, ethnic, and gender groups. ACT provides the reviewers with written guidelines

(ACT, 1995) and receives written evaluations from them. ACT staff responds to every

concern the reviewers raise, and make needed adjustments to the test items before pretesting.

The items are then administered in pretest forms constructed to be as similar as possible to

the final test. Examinees for the pretest are drawn from schools and businesses in sufficient

numbers to provide stable data on which to base final decisions.

Statistical studies of the pretest data help ACT staff identify problem items. These items are

then reevaluated by staff and, if necessary, by outside experts. The tests are subjected to

both classical and item response theory-based (IRT-based) statistical analyses to evaluate the

psychometric properties of each item and the test forms. In addition, when sample sizes

permit, statistical differential item functioning (DIF) analyses of the multiple-choice

assessment items are carried out to determine whether items function differently for various

groups of individuals matched on ability. Items that show DIF are eliminated from the item

pool. Based on data collected during pretesting and periodically after operational

administration, no items in the multiple-choice WorkKeys tests show DIF. As statistical

procedures for detecting DIF in constructed-response items are not yet in common use,

analytical reading of the prompts by expert reviewers serves as the basis for detecting

possible bias.

Phase 5: Construction of Operational Forms. Pretest item statistics are considered

carefully when the forms for operational testing are constructed. Test forms for each

assessment are developed from the pool of items that meet all the content, statistical, and

fairness criteria for items and for test forms. The items are presented in approximate order

of their complexity. That is, the least complex items are at the beginning of the test and the

most complex items are at the end. ACT staff construct at least one base form and one

alternate test form for each assessment, making sure that the two forms are similar in terms

of their psychometric characteristics, the test characteristics as a whole, and the within-level

characteristics with respect to content and complexity.

Integral to the test development procedures described above is an effort to minimize the

likelihood of adverse impact resulting from use of the WorkKeys tests. The assessments are

designed to be job-related and fair through careful screening prior to their being made

available to employers. Specifically, the tests and test development must meet these criteria:



The assessments are criterion-referenced, which means that scores are interpreted in

relation to job requirements rather than population norms.



The test specifications are well defined.



Items tap a domain of workplace skill because people who have experience in the

workplace write them.



Items measure a particular workplace skill.



Content and fairness experts review the items to determine possible differences in

responses among racial and ethnic groups and genders.



Statistical analyses at the item and test level are conducted to monitor the performance

of various subgroups.

The WorkKeys system as a whole provides a well-designed procedure that businesses,

schools, and job seekers can use to facilitate and improve career transitions—from applicant

to employee, from school to work, and from job to job.

Job profiling

Collecting evidence to demonstrate content validity is also important in the business setting.

According to the Uniform Guidelines on Employee Selection Procedures (1978), validation

should demonstrate that the content of the selection instrument “is representative of

important aspects of performance on the job for which the candidates are to be evaluated”

(section 16D). Establishing a well-defined content domain for each test is part of the

process of establishing the content validity of WorkKeys tests for selection purposes. In

addition, the use of a given test for selection purposes should be validated for the applicable

job. The WorkKeys job profiling component addresses this aspect of the process.

It is common to conduct a job analysis to see what tasks are required for a particular job and

to then use the results to build a content valid test for that job (Cascio, 1982; Dunnette &

Hough, 1990). This approach works best where a test will be built specifically for each job.

This is not the case with the WorkKeys system, where the same test is designed to assess

generalizable skills and skill levels associated with many different jobs. Thus, the primary

uses of job analysis by WorkKeys are to (a) establish the content-relatedness between a

specific job and the existing WorkKeys skill area and, if the job and skill are found to be

content related, to (b) establish the WorkKeys skill level required by the job. This latter task

in effect establishes a skill standard for that job.

As with other judgment-based standard-setting methods, job profiling (job analysis relative

to WorkKeys skill scales) must address the key issues of who should make the judgments,

on what basis, and by what method. Because those setting the standards must be familiar

with the job, the decision was made that the subject matter experts (SMEs) would, wherever

possible, be job incumbents. Although some companies choose to include supervisors and

trainers as SMEs, ACT recommends that, when feasible, the standard-setting group be

comprised predominately of individuals who are doing or have recently done the job that is

being profiled (ACT, 1995b; Anastasi, 1982).

The SMEs make their judgments based on their knowledge of the job under consideration

and on knowledge of the WorkKeys skills and skill levels. The latter information is

provided by WorkKeys Profilers during job analysis. Because of the depth of understanding

that must be gained by the SMEs before they can accurately align the skills they perform in

their jobs with the WorkKeys skill scales, the job profiling process is carried out in small

group settings, much like focus groups. Each group of SMEs is guided by an authorized

WorkKeys Profiler. The Profiler is selected based on relevant previous training and

experience and is authorized by ACT after completing training in the job profiling process

and in the WorkKeys skills and skill levels.

Job profiling is conducted in two phases: task analysis and skill analysis. In task analysis,

the SMEs identify the tasks they complete on the job and then rate them for Importance.

The scale used for these ratings is provided in Section 3. The average Importance rating for

each task is used to sort the task statements and list them in order, with the most important

tasks placed at the beginning of the list.

In skill analysis, the SMEs first link the WorkKeys skills (such as Workplace Documents) to

the important tasks on the list (i.e., those with ratings of 3/important and higher), identifying

each task that requires the particular skill and discuss why the task requires the skill. For the

task to be considered in the next step to set the skill level (i.e., cut or passing score), the

majority of the SMEs had to agree that the task requires the skill. Once the set of important

tasks requiring that skill has been identified, the SMEs, using successive approximation,

determine which level of the skills is required for that set of tasks. The SMEs begin with a

description of a skill level that the Profiler believes is just below the level needed on the job.

They determine whether their job requires skills that are above, below, or about the same as

the level described. If they determine that the skills they must have are higher, they are

shown the next higher level; if they determine the skills they must have are lower, they are

shown the next lower level. If they determine that the skills they must have are about the

same, they are shown both the next higher and the next lower levels. No decision is reached

until the SMEs have considered a range of skill levels: the one they have identified as the

required level and at least one level above it and one level below it (unless they have chosen

the highest or lowest level available). SMEs sometimes find that the level required is below

or above the levels measured by WorkKeys.

Establishing the link between assessments and the job

Careful attention has been given to developing both WorkKeys assessments and the job

profiling procedure in a manner consistent with the standards for content validity established

in the Uniform Guidelines (1978), Standards (2014), and Principles (2018). The essential

element of content validation is a rational demonstration of linkage between test content and

important job performance requirements. In WorkKeys, this linkage is normally

accomplished in two steps: (1) test items are linked to skill and skill level definitions by the

assessment development process, and (2) skill level definitions are linked to job

requirements by the judgment of SMEs, who are generally job incumbents (and may

sometimes be supervisors or others familiar with the job).

Items written and selected for WorkKeys assessments go through a series of screens in an

attempt to ensure job-relatedness and fairness. For example, both minority review (a

judgmental process) and DIF analysis (a statistical procedure) are used to determine possible

differences in responses among racial groups and between men and women prior to

construction of the released assessments. All aspects of the test development process are

conducted to ensure that items pertain to the workplace and that they tap a domain of

workplace skill.

In the case of the WorkKeys job profiling process, the validation procedure involves several

steps to ensure that a link is established between the skill level definitions and the

requirements of a particular job. Ideally, a representative pool of SMEs should be identified.

It would include the variety of races, genders, regions, locations, and so on that is

represented by employees in the job to be profiled. The pool of qualified SMEs then

develops a list of tasks critical/important to the job. A thorough review of each skill level

definition is made and then the SMEs are asked to come to a consensus on the levels of

WorkKeys skills that are required for performing the job as a whole. This comprehensive

and systematic analysis of jobs helps employers identify the important tasks as well as the

skills and skill levels needed to perform those tasks.

During the job profiling procedure, incumbent workers who are knowledgeable about the

job set the skill level requirement for entry into the job and for effective performance (the

level that is expected to be acquired through training). At the end of the skill analysis, the

SMEs complete the following statements, which appear at the bottom of the skill rating

form:

Skill Level _____ is necessary for effective performance of this job.

Skill Level _____ is required for entry into this job. Employees should be expected to come

into the job with this skill level; they are not expected to learn this skill level on the job.

The job profile established by employees who are knowledgeable about the job is based on

entry level job requirements, not on personal capabilities or “wish list” expectations. The

participation of supervisors in determining the skill requirements is recommended only

when the supervisors have had experience in the job.

It is possible that SMEs may identify a higher skill level as the requirement to perform the

job effectively (after initial entry). However, selection decisions should be based on the

entry level requirements.

Section 5

Relationship between the Selection Procedure and the Job

A. EVIDENCE DEMONSTRATING THAT THE SELECTION PROCEDURE IS A

REPRESENTATIVE SAMPLE OF A SKILL USED AS PART OF A WORK BEHAVIOR

AND

NECESSARY FOR THAT BEHAVIOR

Selection of skills

In developing the WorkKeys system, ACT consulted with employers, educators, and

labor organizations to develop a list of foundational workplace skills that are used in a

wide range of jobs, are teachable in a reasonable period of time, and facilitate job

analysis (ACT, 1992). In 1992, initial WorkKeys foundational skills were selected on

the basis of a review of the literature relating to employer-identified skill needs (Agency,

1989; ACT, 1987; Bailey, 1990; Carnavale, Gainer, & Meltzer, 1990; Center, 1990;

Conover, 1991; Educational, 1975; Electronic, 1992; Greenan, 1983; Secretary’s, 1990)

and a survey of employers and educators who participated in the design of the

WorkKeys system. The latter were from seven states (Illinois, Iowa, Michigan, Ohio,

Oregon, Tennessee and Wisconsin) and a network of community colleges in California,

all of which served as charter members of the WorkKeys development effort. These

charter members assisted in the design and review of plans and materials and also

provided examinees for the prototype and field-test phases of assessment. Twelve skills

were selected for initial development based on those identified by educators and

employers (McLarty, 1992). It was anticipated that this list would be modified over time

in response to changing employer needs.

Skill scale development

In assessment development, it is common for the assessment to be developed first and

development of score scales to follow (Crocker & Algina, 1986; Nunally & Bernstein,

1994). This is necessary for any score scale that depends on examinee data for its

construction (e.g., norm-referenced score scales) because in that case the skill scale and

score scale are necessarily identical. Each WorkKeys skill scale can be conceptualized

as an independent definition of the construct to be measured, a definition that is not

based on the psychometric characteristics or the assessment. The score scale reflects the

characteristics of the assessment and can be evaluated with respect to how well the

assessment scores represent the designated skills and skill levels. More than one

assessment approach can conceivably use the same skill scale. In the development of the

WorkKeys system, the need to link job analysis with the assessment of individuals

argued for separation of the skill scale from the score scale so that both the assessment

of jobs (job profiling) and the assessment of individuals could be described by the same

skill scale. This circumstance required the development of skill scales before the

development of the tests and their score scales.

Several skill scale criteria were identified by WorkKeys staff as critical for the

operational system (McLarty, 1992). WorkKeys skill levels would have to (a) be readily

interpretable as a description of what the examinee can do and the skills required by the

job; (b) be appropriate for large-scale use and for validation as part of a system for

selecting qualified job applicants; and (c) provide information useful for an examinee

wishing to improve skills in order to meet job requirements, an educator or trainer

wishing to assist examinees in improving their job-related skills, and an employer

wishing to select well-qualified employees. A primary goal was that the skill scale

metric should communicate the level of a generalizable skill that the job requires, and

that a person possesses, to the test taker, the teacher, the trainer, the employer, the labor

union leader, and other audiences. Based on their content, the WorkKeys skill scales

would have to communicate clearly and concisely to people making decisions using the

assessment results.

The WorkKeys skill scales are, therefore, intentional scales, created to ensure that the

resulting score scale will be both meaningful and hierarchical. Cognitive and content-

related aspects of a skill are analyzed to identify and combine their component strands in

order to generate hierarchical scales that will be meaningful to individuals, educators,

and employers. Aspects of content knowledge and cognitive skills that do not contribute

to the hierarchical scale are excluded from it. The quality of the resulting skill scales is

then judged by the degree to which they serve as a common metric to link the job

analysis to the assessments and by their usefulness in identifying the skill levels required

by jobs and possessed by individuals. Clustering approaches (identifying skills

perceived as different but equally valuable) are intentionally excluded from the scale-

building process, although such clusters could become the basis of separate skill scales.

Thus, the WorkKeys skill levels are designed to be arbitrary but standardized, and

particular to each skill. To explain, while a hat size of six is arbitrary but standardized, it

is not expected to be comparable to a shoe size of six. A woman who wants to purchase

a hat and shoes will need to measure both her head and her feet using scales that are

appropriate for those parts, and no one would suggest that her feet are better than her

head if she needs size six shoes and a size six hat. Similarly, just as a person who needs

size six shoes does not automatically need a size six hat, a Level 6 in Workplace

Documents is not the same as Level 6 in Graphic Literacy. However, Level 6 in

Workplace Documents does mean the same skill level whether it is used to describe a job

or a person, and Level 6 in Graphic Literacy means the same thing when it is an

individual’s assessment score that it means when it is part of a job profile. The common

metric, then, forges a link between assessment results and job analysis, but does not

indicate a relationship between skills.

Currently, six WorkKeys assessments are available for operational use: Applied Math,

Applied Technology, Business Writing, Graphic Literacy, Workplace Documents, and

Workplace Observation. In the process of developing the skill scales, the WorkKeys

development team has refined the following procedure for establishing hierarchical skill

scales. Each WorkKeys skill scale is developed initially by a panel of employers and

educators. The panel first develops a broad definition (such as “workplace observation”)

of the skill area for which a scale is to be developed, identifies examples of tasks within

this broadly defined skill domain, and narrows that domain to those examples which are

important for job performance across a wide range of jobs (by excluding things like “the

observation of microscopic samples”). Next, they organize the remaining tasks into

strands (such as “observation for the purpose of maintaining quality control”). Within

each strand, they order the tasks into a series of difficulty levels, with the lowest being

the simplest and the highest being the most complex.

The number of skill levels is determined iteratively on the basis of the number of

separate levels that appear to best fit the task groupings. The panel then abstracts the

variables it believes cause a task to be more or less difficult. For example, less difficult

observation for quality control involves (a) directed attention to one or a very small

number of features which are (b) easy to differentiate from the standard and for which

(a) no specific direction as to what to attend to, (b) a large variety of features to be

inspected simultaneously, (c) distractions, (d) a short time in which the determination

must be made, and (e) very fine distinctions between the item inspected and the

standard. This conceptual analysis process is repeated for each strand identified.

Finally, to facilitate combining the strands into a single test, the panel suggests which

levels of each strand are at approximately the same levels of difficulty as given levels of

other strands (the strands do not necessarily have the same number of levels). The

exemplar tasks, the identification of elements of difficulty, and the suggested common

levels across strands form the basis for creating a description of the skill area and its

levels. This description is then reviewed by panel members and others and refined until

it is as conceptually clear as possible. It should be noted here that strand refers to tasks

and content that pertain to a singular concept to be measured.

Once each skill scale is defined, it is necessary to construct a test to measure individuals’

skills relative to it. The WorkKeys system tests were designed to meet the following

criteria:



The way in which the skill is assessed is generally congruent with the way the skill is

used in the workplace.



The lowest level assessed is at approximately the level for which an employer would

be interested in setting a standard.



The highest level assessed is at approximately the level beyond which specialized

training would be required.



The steps between the lowest and highest levels are large enough to be distinguished

and small enough to have practical value in documenting workplace skills.



The assessments are sufficiently reliable for high-stakes decision making.



The assessments can be validated against empirical criteria.



The assessments are feasible with respect to administration time and complexity, as

well as cost.

Skill definitions and skill levels

The Final Task List(s) for the CSR job are shown at the end of this report. The

aggregate importance ratings and skill requirements are also shown. A checkmark in a

skill column means that, according to the SMEs, the task on that row requires that skill.

The skill and skill level definitions are shown in Appendix B and the skill levels required

for performing the tasks are given below. Tasks shown in gray italics are of low

importance (i.e., they received average importance ratings below 3) and they did not

influence the skill levels set for the profile.

B. IDENTIFICATION OF THE WORK BEHAVIOR THAT EACH PART OF THE

SELECTION PROCEDURE IS INTENDED TO MEASURE

The WorkKeys job profiling procedure is designed to systematically develop accurate

profiles through a task analysis that is used to select the tasks most important to a job

and a skill analysis that is used to identify the skills and skill levels required at entry into

the job and for effective performance on that job. A description of the procedure used is

given below.

Conducting the task list review

Dr. Allen toured selected work areas of GCOMM facilities on October 16, 2017, with

Jane Phillips, Customer Service Supervisor. The Profiler then met with the groups of

Subject Matter Experts (SMEs) to tailor the Initial Task List to make sure that the

resulting Final Task Lists accurately and completely described the job. Each group

worked to develop one list. The Profiler met with the first group (Group A) of 6 SMEs

on October 18, 2017, and met with the second group (Group B) of 7 different SMEs on

October 19, 2017. The SMEs in each group worked to add, delete, consolidate, and

change the descriptions of tasks, as needed, to make sure they accurately depicted their

job as it is performed at their company. Then they independently rated each task in

terms of its Importance. The average Importance rating for each task was used to sort

the task statements and list them in order, with the most important (or critical) tasks

placed at the beginning of the list. Tasks that received an average rating of 2 (i.e., low

importance) or lower were grayed out, italicized, and moved to the bottom of the list.

The SMEs then reviewed the list to see the final order of the tasks. The Final Task Lists,

with the tasks listed in order of Importance, can be found at the end of this report.

Collecting the Importance ratings

After carefully examining this list of tasks, the SMEs rated each task according to its

Importance. Importance refers to the significance of the task to overall job performance.

The average Importance rating for each task was used to sort the task statements and list

them in order, with the most important tasks placed at the beginning of the list. Tasks

that received an average rating of 2 or lower were grayed out, italicized, and moved to

the bottom of the list.

Identifying on-the-job behaviors associated with each skill as it is used on

the job

Once the SMEs understood the definition of a WorkKeys skill and had determined its

relevance to the job, they independently identified the important tasks on the Final Task

List that required the skill and they identified how the tasks specifically use that skill.

For example, the Workplace Documents skill may be identified by SMEs as necessary

for reading such documents as Standard Operating Procedures or company policies.

After discussing the relationship between the skill and the tasks, only those tasks which

the majority of SMEs agreed require the skill were included in the subsequent

discussion, and only those tasks were used to determine the level of skill required for the

job.

Determining the WorkKeys skill levels of the job

The Profiler presented detailed descriptions of the WorkKeys skill levels to the SMEs

and showed them examples of problems or situations employees deal with at each level.

For each WorkKeys skill, the SMEs decided which skill level is necessary at job entry

and for effective performance of the job.

Prioritizing the WorkKeys skills

The WorkKeys skills are prioritized in terms of their importance to the CSR job. This is

based on a systematic examination of the number of important (or critical) tasks

identified by the SMEs as requiring each skill and the Importance rating of those tasks.

The result is an ordering of the WorkKeys skills that are most important to the

performance of the job.

C. COMPARISON OF THE MANNER, SETTING, AND LEVEL OF COMPLEXITY OF THE

SELECTION PROCEDURE WITH THOSE OF THE WORK SITUATION

Section 3 of this report documents the need for specified WorkKeys skills by identifying

the important tasks that require those skills. This section (Section 5) summarizes the

SMEs’ discussions of the skill levels and documents their reasons for finding that the

specified levels are required for both job entry and effective performance of the job. The

skill levels required for job entry and for effective performance are presented in Tables 1

and 2 in the Executive Summary.

Applied Math

WorkKeys Applied Math is the skill people use when they use mathematical reasoning

and problem-solving techniques to solve work-related problems. Employees may use

calculators and conversion tables to help with the problems, but they still need to use

math skills to think them through.

In evaluating the level of Applied Math skill necessary for the tasks of the job, the SMEs

considered the types of mathematical operations (including single-step or multiple-step

mathematical operations and conversions either within or between systems of

measurement); how the information in the problem is presented (i.e., the information is

presented in the order in which it is needed or it must be reordered); and whether all the

information employees need for solving problems is provided or if they must derive

some necessary information. The SMEs evaluated their work situation in comparison to

WorkKeys Applied Math skill levels 3 through 5.

The SMEs in both groups determined that Level 4 Applied Math skills are needed for

entry into, and for effective performance of, the CSR job. New CSRs need this level

because they must perform the same mathematical operations when they begin the job as

they do when they are experienced in the job, and no training in mathematics is

provided.

At Applied Math Level 4, tasks may present information out of order and may include

extra, unnecessary information. One or two operations may be needed to solve the

problem. A chart, diagram, or graph may be included. When employees use Level 4

Applied Math skills they can solve problems that require one or two operations. They

may add, subtract, or multiply using positive or negative numbers, and they may divide

positive numbers. They can figure out an average or mean of a set of numbers using

whole numbers and decimals. They can figure out simple ratios, simple proportions, or

rates. At Level 4 employees can add commonly known fractions, decimals, or

percentages and add or subtract fractions that share a common denominator. They can

multiply a mixed number by a whole number or decimal and they can put the

information in the right order before they perform calculations.

According to the SMEs, Level 4 Applied Math skills are needed for the CSR job because

the calculations that they perform can require them to figure out averages, use one or

two operations of mathematics, and put information in the right order before performing

calculations.

 To illustrate, when they help a customer understand their bill they have to figure

the cost of each individual call, add the prices of calls to get a total, determine the

taxes for the total and then get a grand total price. For each call they multiply the

number of minutes (e.g., 20) by the rate (e.g., .06/minute) to determine the price

of each call (20 x .06 = $1.20), after noting the time of the call to make sure that

they use the correct rate. So the CSR first has to figure out how to set up the

problem by identifying the correct rate to use and then completing the

calculations.

CSRs use multiple operations and/or steps of mathematics to suggest (i.e., sell)

additional services to customers during a routine call with a customer or when asked to

generate a customized price estimate for a customer.

 For example, the CSR can enter the Pricing Screen to enter the total amount that

the customer would like to spend each month and then the system presents the

pricing plans that would accommodate that estimate, along with an estimated

number of calls (e.g., for $49.95/month the Friends Forever plan allows

approximately 30 30-minute nights/weekends calls), but if the customer wants to

know how many calls can be made during the day with just 4 calls made during

nights/weekends, the CSR re-figures the estimate by hand, multiplying the day

rate per minute (e.g., 20 cents) by 30 minutes by 7 calls (.20 x 30 x 7 = $42) and

then figuring the nights/weekends rate (e.g., 5 cents) by 30 minutes by 4 calls

(.05 x 30 x 4 = $6) to get a revised total ($48).

 Similar calculations are used when the CSR is asked to develop an estimate for a

customized plan (such as the basic Friends Forever plan plus a texting option).

Level 4 Applied Math skills are also used when training new CSRs.

 Trainers have to be able to check the accuracy of the trainee’s math such as in a

test of the scenario described above.

 This level of skill is also used when CSRs generate an average when the

customer requests a price comparison of the average monthly costs for

GCOMM’s service as it compares to a competitor’s rates.

 And when troubleshooting the software for new pricing plans, each CSR is

required to spend a minimum of one hour testing the software including the

mathematical functions built into the software, by creating test customer

scenarios, and computing the required mathematics by hand and comparing with

the mathematics generated by the software tool (such as to use addition or

subtraction to correct a bill or to use multiple operations to generate pricing for

service plans).

Graphic Literacy

The WorkKeys Graphic Literacy skill is the skill people use when they work with

workplace graphics such as tables, graphs, charts, digital dashboards, flow charts,

timelines, forms, maps, and blueprints. Employees use this skill when they find,

summarize, compare, and analyze information to make decisions using workplace

graphics to solve work-related problems.

To determine the level of Graphic Literacy skill needed for the tasks employees

complete in their job, the SMEs considered the difficulty of the graphics and how hard it

is for employees to use the information to complete the tasks. The SMEs compared the

tasks of the job to WorkKeys Graphic Literacy skill levels 3 through 6. The SMEs in

both groups stated that Level 4 Graphic Literacy skills are necessary for job entry and

for effective performance.

At Graphic Literacy Level 4, workplace graphics are common and of low to high

moderate difficulty. Characteristics of low moderate graphics include a moderate amount

of data; more than one level of data, but no nesting; several variables; one or two axes, if

there are axes; and if two simple graphics are required to solve the problem, they should

be considered a low moderate graphic. At Graphic Literacy Level 4, employees can use

one or two low moderate graphics at a time to locate information in a graphic using

information found in another graphic; compare two or more pieces of information;

identify a trend/pattern/relationship; make an inference or decision; and identify the

graphic that accurately represents the data. High moderate graphics may be less common

at Graphic Literacy Level 4 and have characteristics which include a moderate amount

of data; more than one level of data and it may be nested; many variables such as types

of wood, drill speeds, hole diameter, and type of bit; one or two axes (such as an x

and/or y axis), if there are axes; and if a low moderate graphic and a simple graphic are

required to solve the problem, they should be considered a high moderate graphic. At

Level 4, employees can use one high moderate graphic to locate and find information

and identify the next or missing step in a process.

According to the SMEs, CSRs need Level 4 Graphic Literacy skills to set-up new

customer accounts and make changes to current customer accounts because they must

enter several pieces of information (e.g., name, street address, transaction codes, service

plans) into a computerized form while referring to tables for the information they need.

The forms and tables are simple. The screens usually contain five to fifteen fields with

drop-down lists and the CSR will need to reference a simple table to find the correct

code to insert into the form. When two simple graphics are used together, they are

considered a low moderate graphic. The CSRs are required to locate information in one

graphic using information found in another graphic.

If a current customer is calling because they want to switch to a competitor’s plan, the

CSR can look at graphics that show their usage for different services and use that

information to determine if a different GCOMM plan might meet the customer’s needs

better than their current plan (i.e., switch to another GCOMM plan rather than

discontinuing their service). Level 4 Graphic Literacy skills are required because the

usage graphics are simple line graphs or bar charts with two axes. The CSR needs to

review the information in these graphics and decide which GCOMM plan to recommend

to the customer. The CSRs perform similar tasks with potential customers, when they

assist them to select a plan.

The CSRs use a troubleshooting diagram to guide the content of a customer call that is

of high moderate difficulty because there can be many variables that require the CSR to

follow different paths. However, the CSR is just finding the next or missing step so

Level 4 is required overall. They also have diagrams for processing requests for refunds

and issuing work orders. Although, they are of only low moderate difficulty and only

require Level 3 skills.

All of the SMEs agreed that level 4 skills are required for entry. While three of the six

SMEs in Group A and all of the SMEs in Group B (i.e., 9 of 12 total SMEs) agreed that

Level 4 skills are also required for performing the job effectively because CSRs are

required to perform all of the tasks in the 4-weeks of training and 5 weeks of mentoring

that they are expected to perform at the conclusion of training (i.e., after 3 months in the

job when effective performance is expected). Neither the classroom training nor the

mentoring period are expected to grow the CSR’s Graphic Literacy skills.

Consequently, individuals must bring the skills required to the job and that same level of

skill is sufficient for effective performance.

The remaining three SMEs in Group A stated that to be effective, Level 5 Graphic

Literacy skills are required. At Graphic Literacy Level 5, workplace graphics may be

less common and of low moderate, high moderate, or difficult complexity.

Characteristics of low moderate graphics include a moderate amount of data; more than

one level of data, but no nesting; several variables; one or two axes, if there are axes; and

if two simple graphics are required to solve the problem, they should be considered a

low moderate graphic. At Level 5 Graphic Literacy, employees can use a low moderate

graphic to compare two or more pieces of information; interpret a

trend/pattern/relationship; make a reasonable inference or decision based on one graphic

after finding information in another graphic; justify a decision or inference based on

information; identify the most effective graphic for the task; and justify the most

effective graphic for the task. High moderate graphics may be less common at Graphic

Literacy Level 5 and have characteristics which include a moderate amount of data;

more than one level of data and it may be nested; many variables; one or two axes if

there are axes; and if a low moderate graphic and a simple graphic are required to solve

the problem, they should be considered a high moderate graphic. At Level 5 Graphic

Literacy, employees can use one high moderate graphic to locate information in a

graphic using information found in another graphic; compare two or more pieces of

information; identify a trend/pattern/relationship; make an inference or decision; and

identify the graphic that accurately represents the data. Difficult graphics at Graphic

Literacy Level 5 are likely to be less common or a composite of graphics. Data presented

is dense; more than one level of data and nesting is likely; there are many variables such

as types of wood, drill speeds, hole diameter, and type of bit; three or more axes, such as

an x, y, and z axis, if there are axes; and if a high moderate graphic and a low moderate

graphic are required to solve the problem, they should be considered a difficult graphic.

At Level 5, employees can use one difficult graphic to locate and find information and

identify the next or missing step in a process.

The three SMEs indicated that Level 5 skills are required because while effective

employees may perform the same tasks as new employees, they use more complex

materials. These SMEs stated that CSRs can receive an individual reward for retaining

customers and for selling new services to customers in addition to the rewards given to

sales teams. They maintained that customers can read marketing literature themselves so

CSRs have to provide them with more convincing information in order to make a sale.

These SMEs said that they gain this information by reading additional literature that is

made available to all employees (such as sales reports, new product development

reports, and product manuals). This literature contains tables, graphs, and diagrams that

are of high moderate difficulty.

The majority of the SMEs in the two groups (i.e., 9 of 12) agreed that Level 4 Graphic

Literacy skills are required to perform the job effectively. The three SMEs who stated

that Level 5 skills are required gave examples of Graphic Literacy resources that match

the complexity of Level 5 that are used by CSRs who are attempting to increase their

rewards for high sales. These SMEs did report that they had achieved the highest sales

records for the previous year, but conceded that Level 5 skills are not necessary to

effectively perform the tasks of the job.

GCOMM may consider providing voluntary training to CSRs who are interested in

increasing their sales; however, Graphic Literacy Level 5 skills are not required for

effective performance.

Workplace Documents

WorkKeys Workplace Documents is the skill employees use when they read and use

written text in order to do a job. The written texts include messages, emails, letters,

directions, signs, notices, bulletins, policies, websites, contracts, and regulations. It is

often the case that these workplace communications are not necessarily well written or

targeted to the appropriate audience. Workplace Documents materials do not include

information that is presented graphically, such as in charts, forms, or blueprints.

To determine the level of Workplace Documents skill needed for the tasks employees

complete on the job, the SMEs considered the difficulty of the reading materials and

how hard it is for employees to find the information they need and make use of it. The

SMEs evaluated their work situation as it compares to WorkKeys Workplace Documents

skill levels 3 through 6. The SMEs in both groups agreed that Level 4 Workplace

Documents skills are required for entry into the CSR job and for effective performance.

At Workplace Documents Level 4, reading materials include policies, procedures, and

notices. Materials are straightforward with some long sentences and contain a number of

details. These materials use common words, but do have some harder words, too. They

describe procedures that include several steps. When following procedures, employees

must think about changing conditions that affect what they should do. For example, they

can follow directions that include if-then statements. When employees use Level 4 skills

they can identify the main idea and details that may not be clearly stated, use the reading

material to figure out the meaning of words that are not defined for them (not jargon or

technical terms), apply information/instructions to a situation that is the same as the

situation in the reading materials, and choose what to do when changing conditions call

for a different action.

According to the SMEs, a CSR needs Workplace Documents Level 4 skills because the

reading materials used during CSR training (e.g., training manual descriptions that

explain procedures or script content) contain passages that have a number of procedural

steps and associated details along with longer sentences explaining specific customer

service problems and how to resolve them. For example, the CSR must verify the

identity of the caller before making any changes to the account. The script includes the

following content:

CSR: May I have the spelling of your first and last name please? And your phone

number is____?

 If the account is found, verify at least 2 other pieces of information in the account,

such as account number, address, or pin #.

 If the customer’s name is not found in the database, ask for an alternate name or

address that could be associated with the account.

The vocabulary includes common words and some harder words (e.g., accommodate,

resolve, status). CSRs must be able to apply instructions (some of which involve several

steps) to situations that are the same as those described in the materials. Reading

materials that fit this definition include scripts of typical conversations with customers

and marketing materials that incumbents use with customers to explain plan options

(e.g., calling plans such as Friends Forever; text and voice plans) or equipment

specifications (e.g., equipment sales or rental options). CSRs must determine what

information from the reading materials applies to the current situation, such as to read

the three suggested responses to a typical customer question and choose the statement

that best fits the situation.

Other resources that require Level 4 Workplace Documents skills (i.e., the CSR

Procedure Manual, help screens) contain straightforward “if-then” scenarios for the CSR

to apply to an appropriate situation. Product brochures and company memos also match

the Level 4 description because their text is straightforward, but contains a number of

details. For example: The new GCOMM Innovator is a 5G mobile computing device

and is mobile hotspot capable. Or Approval to attend a non-GCOMM sponsored

training event will be based on the relevance of the seminar/workshop to the staff

member’s job.

When shown the definition and examples of Level 5 Workplace Documents, the SMEs

in both groups agreed that skills at this level are not required to perform the CSR job. At

Workplace Documents Level 5, reading materials include policies, procedures,

announcements, legal, and multiple related documents that have many details. The

information that employees need is generally stated directly, but it is hard to find

because there are so many details and some may not be needed for the task being

performed (extraneous information). The materials include technical terms, jargon, and

acronyms, or words that have several meanings. The documents may have complex

sentences and/or contain conditional situations. When employees use Level 5 skills they

can figure out the appropriate meaning of a word based on how the word is used. They

can identify the appropriate meaning of technical term, jargon, or an acronym that is

defined in the document. They can apply technical terms and jargon to stated situations.

Employees can apply information/instructions to a new situation that is similar to the one

described in the material while considering changing conditions. At Level 5, employees

can apply complex information/instructions that include conditionals to situations

described in the materials. Employees may need to make some inferences to accomplish

their goal.

While some of the materials used by CSRs (such as procedures) contain technical terms,

all technical terms are clearly defined during the course of the 4-week training and there

are resources available after the training (such as help screens and mentor assistance) so

that terminology used in print materials is clearly understood. Other characteristics of

the Level 5 definition are not presented in any reading materials, consequently, the

SMEs agreed that Level 5 skills are not required for job performance.

Workplace Observation

The WorkKeys Workplace Observation skill is an employee’s skill in visually observing

a workplace event, noticing details, and remembering instructions, procedures,

processes, and demonstrations in order to generalize to workplace situations that may be

similar or very different from what was observed. Employees must pay careful attention

to steps that are followed, safety procedures, and quality-control standards.

In determining the level of skill necessary for the tasks of the job, the SMEs considered

the following five characteristics:

the complexity of the procedures;

how likely there are to be extra details presented;

how likely there are to be distractions;

how difficult is it to detect differences, discrepancies, or changes; and

the action the employee must take.

The SMEs were shown Workplace Observation skill Levels 1 through 3. According to

the SMEs, new CSRs need Level 2 Workplace Observation skills at job entry to

successfully complete the four-week training. Level 2 Workplace Observation skills are

sufficient for effective performance, as well, because the tasks of the job do not change

and no training in observational skills is provided.

At Workplace Observation Level 2, employees interpret a straightforward procedure that

involves a condition. With a condition, several possible things may happen, and a

specific response is provided for each one. A Level 2 situation includes obvious and

easily disregarded distractions, and there are a few extra details and differences. When

employees use Level 2 Workplace Observation skills on the job, they can recognize

cause and effect in a straightforward demonstration, process, pattern, or procedure; can

filter out obvious distractions; and can identify the cause of a particular effect.

Additionally, employees can recognize what to do next in a situation given a single

condition. At Level 2, employees can also indicate action to be taken when there is an

incorrect step identified.

According to the SMEs, Level 2 Workplace Observation skills are used in the CSR job

in the following ways:

Complexity: Trainees observe and CSRs perform straightforward processes with if-then

conditions (e.g., if the caller says they had a problem with their monthly bill, that means

the CSR should retrieve the customer’s background information and monthly statement

screens; but if the caller says they want to add services, then a different combination of

screen retrieval is appropriate). CSRs learn which combination of screens to retrieve

based on observing and then participating in training exercises and mentoring routines

that simulate customer call situations. Level 2 skills are also needed to perform the job

after training is complete because the scenarios played out during the 9 weeks of training

(i.e., classroom and mentoring phases) are comparable to the types of experiences that

occur after training (i.e., after the first 3 months in the job).

Details – There are few extra details that are a part of the workplace situation being

observed by CSRs; most details are an integral component of the situation. Extra details

can include a blinking light on the console indicating that another caller is waiting.

Distractors – Distractors are possible, but they can be easily disregarded, such as the

call timer which can be turned off by the CSR when handling a complex, lengthy call, so

that the ‘ticking of the timer’ does not interfere with handling the call appropriately.

Differences – Differences are not difficult to discern and are generally pointed out, such

as during the training when the differences between the purposes of the system computer

screens are explained or the differences between the online scripts for a customer

complaint and the script for a customer sale are explained.

Action – The action taken by the CSR based on observations is generally

straightforward. During the training/mentoring process, CSRs observe multiple

situations and the appropriate responses, and then apply that information to similar

situations. CSRs learn the steps to handle customer calls so that they understand which

step ‘comes next’ in the process. Even if a caller asks a question that is out of sequence

(from the script), the CSR has the training to understand which step to apply next. The

CSR doesn’t need to draw conclusions based on what has been observed because the

conclusion is already evident.

Both SME groups also determined that Level 2 skills are required for effective

performance of the job. According to both groups, the complexity, details, distractions,

differences in details they must notice, and actions they are required to take based on

their observations do not differ from the time they begin the job to the time they become

effective (i.e., by the conclusion of the first 3 months in the job). This is because the 4

weeks of classroom training and 5 subsequent weeks of mentored on-the-job training

expose the CSR to all the tasks required of an effective CSR. Therefore, a higher skill

level is not required for effective performance. Further, when shown the definition and

examples of Workplace Observation Level 3, the SMEs indicated that there are no tasks

that require that level of skill.

At Workplace Observation Level 3, employees observe complex procedures that include

several tasks that may occur at the same time, interact, and change from one situation to

another. More than one condition may be present. Several important details are

presented, but a few are not clearly prompted; and some distractions may make

remembering details difficult. And, the employee may be asked to apply information

observed to other similar situations. Steps may seem similar, but differ based on varying

factors, and a few differences may be present that are not clear. When employees use

Level 3 Workplace Observation skills on the job, they can identify a course of action to

take given more than one condition; distinguish steps that seem similar but are different

based on varying factors; and maintain attention to significant details with little

prompting. Employees are able to recognize when steps can be combined and when they

must be kept distinct; and can combine steps to achieve desired result. At Level 3,

employees identify differences and/or details that are not clear; select, interpret, and

integrate the steps, in the correct order, within a complex process; and apply information

to a similar situation.

The SMEs indicated that in the rare instances that a call gets this complex, the conditions

of the call are beyond the training of the CSR, so the appropriate step is to forward the

caller to a Supervisor to resolve the issue(s). Consequently, Level 3 Workplace

Observation skills are not required for the CSR job.

Prioritizing the WorkKeys skills

The skills were prioritized according to their importance to the job based on the number

of important tasks identified by the SMEs as requiring the skills. Skills with the largest

percentage of important tasks requiring the skill and with the highest importance ratings

for these tasks are given the top ranking. GCOMM should consider using assessments

for the following skill areas: Workplace Observation and Graphic Literacy. If GCOMM

chooses to administer additional assessments, the following skill areas should be

considered: Workplace Documents or Applied Math.

Section 6

Alternative Procedures Investigated

A. TESTING AND PERFORMANCE

It is widely accepted among professionals in the area of personnel selection that

cognitive ability can be used to predict performance. This conclusion is born out in the

cumulative results of an extensive body of research on the validity of performance

predictors (Gottfredson, 1986; Hogan and Hogan, 1990; Hunter and Hunter, 1984). This

body of research reveals that paper-and-pencil tests constitute the most effective method

of measuring cognitive abilities or knowledge for personnel selection. Although other

selection procedures (such as interviews, reference checks, and experience ratings) can

assess cognitive ability and knowledge, cognitive ability tests have more validity than

these and other alternative methods of predicting applicant job performance. In their

large scale meta-analysis of validity research results, Hunter and Hunter (1984) revealed

that commonly used alternatives to ability tests, such as interviews, college GPA, and

biodata had validities that ranged from .10 to .37. Today, most researchers and

practitioners continue to agree with the Hunters’ central contention:

Meta-analysis of the cumulative research on various predictors of job performance

shows that for entry level jobs [those for which current job performance is not

available] there is no predictor with validity equal to that of ability, which has a mean

validity of .53. For selection on the basis of current job performance the work sample

test, with mean validity of .54 is slightly better. (Hunter and Hunter, 1984, p. 72)

The cumulative research suggests that for most jobs, there are currently no selection

procedures for hiring new employees that approach the validity of ability tests.

B. WORKKEYS TESTS

The assessments in the WorkKeys system which map to profiled skills are cognitive

ability tests. They each quantify individual knowledge or ability in specifically defined

content areas. As with many other cognitive ability tests, the developers of the

WorkKeys assessments addressed the potential for adverse impact by having the items

that make up the tests reviewed by minorities and other experts for bias and

offensiveness. The tests were also investigated using Differential Item Function

Analysis for ethnic and gender differences in item responses, and revised accordingly.

Unlike other cognitive ability tests, the WorkKeys assessments further reduce the

potential for adverse impact through the nature of their construction and use. The tests

are criterion referenced. That is, they are constructed to yield specific information about

a person’s performance relative to an established criterion standard without reference to

other test-takers or norms (Buck, 1975). WorkKeys job profiling establishes the passing

score, and then the assessments are used to identify a pool of applicants who have

reached the established criterion level. The tests are not used to rank applicants from

highest to lowest, so there is none of the adverse impact that typically results from such

ranking procedures (Nathan, 1995).

C. SCOPE OF INVESTIGATION

ACT’s investigation of alternatives to the WorkKeys assessments comprised a search for

testing or screening instruments of similar skills and an evaluation of their validity,

adverse impact, and relationship to work. The skills of interest are Applied Math,

Applied Technology, Graphic Literacy, Workplace Documents, Workplace Observation,

and Business Writing. The search for other assessments of these skills began with a

broad search for other procedures that have been validated for use in personnel selection.

Three comprehensive references for assessments, Tests in Print (Murphy, Spies, &

Plake, 2006), Mental Measurements Yearbook (Conoley & Impara , 1995), and Tests

(Maddox, 1997), proved to be valuable sources of information and were used to identify

alternative measures of the knowledge and abilities of interest. The descriptions and

reviews published in these references indicate whether a given assessment was

developed for use in personnel selection, and they provide some information on each

instrument’s construction, content, and validity. In some cases, additional information

was obtained from technical reports provided by a test’s publisher and from published

research articles. Finally, the search for alternatives was rounded out with a review of

the catalogs and Internet Web pages of test publishers known to provide assessments for

personnel selection, and by searching the assessment research literature.

D. METHOD OF INVESTIGATION

The EEOC (1985), in its interpretation and clarification supplement to the Uniform

Guidelines on Employee Selection (1978), identified two criteria for evaluating

alternative selection procedures: validity and adverse impact. In evaluating the

alternatives identified in the search described above, validity, adverse impact, and

relationship to work were used as evaluative criteria. Relationship to work is the extent

to which an assessment’s stimulus materials reflect materials or situations found on jobs,

or the intended behavioral overlap between the assessment and jobs. This criterion was

included to supplement explicit validity information. When considering the assessments

included in this study, “adverse impact” refers to the extent to which attempts were made

to enhance test fairness and eliminate discriminatory language and racial, cultural, or

gender bias from the assessments. As such, it is an indicator of an assessment

publisher’s dedication to eliminating adverse impact and it is not always an actual

measure of the existence of adverse impact.

E. INVESTIGATION FINDINGS

In general, there were several alternatives to the WorkKeys Applied Math and

Workplace Documents to consider. There were only two alternatives to WorkKeys for

assessing applicants on their Graphic Literacy skills. The search revealed six

alternatives for the Applied Technology assessment. There was only one assessment

similar to the WorkKeys Observation and Workplace Observation assessments and three

alternatives for the Business Writing assessment. The search revealed no other civilian

alternatives designed for these purposes, although there are several observation tests

specific to police work available to law enforcement agencies. Despite the variety of

sources used to compile this section of the report, information needed for the evaluation

of the various alternatives was at times incomplete.

F. INVESTIGATION CONCLUSIONS

No evidence was found to suggest that any of the alternatives investigated exceed the

WorkKeys assessments in validity or attention to reducing adverse impact. While a

number of the alternatives appear work related in content, only the WorkKeys

assessments have an explicit link or reference back to the requirements of a specific job.

Like WorkKeys, a number of the alternative assessments attempt to address fairness and

adverse impact through a careful consideration of test content, but only the WorkKeys

assessments combine careful attention to content with a criterion-based method of test

construction that in itself reduces the potential for adverse impact (Nathan, 1995). Based

upon the results of this investigation, the combination of job specific content validity

evidence and safeguards against adverse impact make the specific WorkKeys

assessments recommended in the Executive Summary a viable selection alternative in

this situation. The WorkKeys assessments recommended in the Executive Summary

should be used in conjunction with their respective entry-level cut scores listed in both

Table 1 of the Executive Summary and in the write-up of Section 5 of this report. Final

decisions about the use of these assessments are entirely within the purview of the

company including whether or not to obtain additional evidence about alternative

procedures prior to making those decisions.

Section 7

Uses and Applications

A. METHODS CONSIDERED FOR USE AS PART OF THE SELECTION PROCEDURE

The WorkKeys assessments will be used as a part of the GCOMM selection procedure

for hiring employees into the CSR job at the facility in Anywhere, Minnesota.

Applicants will be required to meet the entry-level profile, as set by the subject matter

experts (SMEs), for the skills being assessed. They will either meet or not meet (pass or

fail) the requirements. They will not be ranked by their performance levels as

established by the assessments.

Based on the relevance of the skills to the critical tasks of the job, GCOMM should

consider building a WorkKeys assessment battery for applicants for the CSR job using

assessments for the following skills: Workplace Observation and Graphic Literacy. If

additional assessments are desired, these skill areas should be considered: Workplace

Documents or Applied Math.

B. RATIONALE FOR CHOOSING THE WORKKEYS SYSTEM

The WorkKeys system provides job-related, content-valid cutoff scores by determining

the level of cognitive skills necessary for specific jobs. The job profiling component of

the system requires a trained Profiler to work with one or more groups of SMEs to

develop a thorough list of tasks performed on the job and to provide information about

the level of skill necessary for performing the tasks or job upon job entry and/or for

effective performance. During a job profile session, SMEs are able to make direct

comparisons between the content of their job and the content of WorkKeys skill levels.

The SMEs evaluate and discuss whether job incumbents need a certain skill level (as

defined and described through the use of examples in the profiling session) to enter the

job. The SMEs are asked to decide how a skill description and the accompanying

examples compare to the skill needed on the job: is the skill needed higher, lower, or

about the same as the description and examples? (These are paired comparison

judgments.) The examples used for these judgments are consistent with the ordinal

scaling of the empirically tested skill levels of the tests.

An important issue in selection testing is that of adverse impact. The EEOC considers

adverse impact to be evidence of discrimination. However, showing that the selection

device is job-related despite its adverse impact is the employer’s response to a claim of

adverse impact. Demonstration of a test’s validity for use in selecting qualified

employees is the typical measure of a test’s job-relatedness. A common misconception

is that tests are valid or invalid and that valid tests do not have adverse impact.

However, as Nathan (1995) explains, the reality is that all competently developed

cognitive ability tests are valid (see, for example, validity generalization results) and also

have adverse impact. The use of information obtained from valid tests must still be

supported by evidence that the tests and the cutoff scores are appropriate to the situation.

WorkKeys addresses the issue of adverse impact in several ways. ACT has developed

the WorkKeys assessments to be job-related and fair by putting the items through a

series of screens prior to their being included in tests that are used for selection

decisions. These are:



the assessments are criterion-referenced (they use job requirements as the reference);



the test specifications are well defined;



items are written by people who have job experience in the workplace, so the items

tap a domain of workplace skill;



items measure a particular workplace skill;



content and fairness experts review the items to identify possible differences in

responses by people in different racial groups and between men and women prior to

construction of the released assessment; and



statistical analyses at the item and test level are conducted to monitor the

performance of various subgroups and of the items and tests, themselves. For

example, differential item functioning (DIF) analysis, a statistical procedure for

identifying bias for or against such things as race and gender is run for each item.

The job profiling process provides validation evidence by establishing a link between

tasks performed on the job, the skills needed to perform the tasks, and the skills and skill

levels measured with the WorkKeys assessments. The job profiling procedure requires

the participation of SMEs who are knowledgeable about the job and who, together,

constitute a representative sample of the incumbent employees. This group of SMEs

must convene to describe, generally with consensus, the skill requirements for their job.

Another group of SMEs also constituting a representative sample will confirm the task

analysis results and the skill requirements in a replication session, as needed.

C. PURPOSE OF ASSESSMENTS

The WorkKeys assessments will be used as part of the selection procedure for the CSR

job at GCOMM in the Anywhere, Minnesota facility and for training purposes.

D. DETERMINATION OF THE NORMAL EXPECTATIONS OF THE WORKFORCE FOR

THE

CUTOFF SCORES

See Section 3A for a complete description of the SMEs, and Section 3G for a discussion

of which skills are required. The entry skill levels for the most critical skills may be

used as hiring cutoff scores on the relevant WorkKeys assessments as recommended in

the Executive Summary.

Section 8

Contact Person

For further information regarding this validity study, contact:

Dr. Margaret Allen

Job Developer/ACT– authorized Profiler

Filerville Community College

801 University Avenue

Filerville, AR 55555

Phone: 555-555-5555

Email: [email protected]

Fax: 555-555-5555

Section 9

Accuracy and Completeness

In order to ensure completeness and accuracy of the collection and analysis of data and

the reporting of the results, the following procedures were followed:

Margaret L. Allen, Ph.D., who conducted the profiling for the CSR job and prepared this

report, is an ACT– authorized WorkKeys Profiler for Filerville Community College.

She holds a doctorate in Human Resources Management, performs tasks for customers

including new/existing job development, and conducts profiling activities for clients.

In developing the WorkKeys system, including the WorkKeys assessments and the job

profiling system, ACT has been, and will continue to be, guided by the Uniform

Guidelines on Employee Selection Procedures (1978), which have been adopted by the

Equal Employment Opportunity Commission (EEOC) and various other federal agencies

(Ref: 29 C.F.R. Part 1607), as well as the Standards for Educational and Psychological

Testing (AERA, et al. 2014), and the Principles for the Validation and use of Personnel

Selection Procedures (SIOP, 2018). The profile for the CSR job at GCOMM adheres to

the requirements of the WorkKeys system.

Appendix A

References

Agency for Instructional Technology. (1989). Workplace readiness: Education for employment.

Personal behavior, group effectiveness and problem solving for a changing workplace.

Unpublished report, Bloomington, IN: Author.

American College Testing. (1987). Study power [Series]. Iowa City, IA: Author.

American College Testing. (1992). A strategic plan for the development and implementation of the

Work Keys system. Unpublished report, Iowa City, IA: Author.

American College Testing. (1995). Work Keys guide for reviewers. Iowa City, IA: Author.

American Educational Research Association, American Psychological Association, and National

Council on Measurement in Education. (2014). Standards for educational and psychological

testing. Washington, DC: Author.

Anastasi, A. (1982). Psychological testing (5th ed.), New York: Macmillan.

Bailey, L. J. (1990). Working: Skills for a new age. Albany, NY: Delmar.

Buck, L. S. (1975). Use of criterion-referenced tests in personnel selection: A summary status report.

Springfield, VA: U.S. Department of Commerce National Technical Information Service.

Carnevale, A. P., Gainer, L. E., & Meltzer, A. S. (1990). Workplace basics: The essential skills

employers want. San Francisco: Jossey-Bass.

Cascio, W. F. (1982). Applied psychology in personnel management (2nd ed.). Reston, VA: Reston

Publishing.

Center for Occupational Research and Development. (1990). Applied mathematics [Curriculum

product]. Waco, TX: Author.

Conoley, J. C. & Impara, J. C. (Eds.). (1996). The supplement to the twelfth mental measurements

yearbook. Lincoln, NE: University of Nebraska Press.

Conover Company. (1991). Workplace literacy system [Brochure]. Omro, WI: Author.

Crocker, L. M., & Algina, J. (1986). Introduction to classical and modern test theory (pp. 68-83).

New York: Holt, Rinehart, and Winston.

Dunnette, M. D., & Hough, L. M. (Eds.). (1990). Handbook of industrial and organizational

psychology (2nd ed., Vol. 1). Palo Alto, CA: Consulting Psychologists Press.

Educational Testing Service. (1975). Cooperative assessment of experimental learning. Interpersonal

learning in an academic setting: Theory and practice. Princeton, NJ: Author.

Equal Employment Opportunity Commission (1985). Uniform employee selection guidelines:

Interpretation and clarification. Washington, DC: Commerce Clearing House Inc.

Electronic Selection Systems Corporation. (1992). Accuvision systems for personnel selection and

development [Assessment]. Maitland, FL: Author.

Gottfredson, L. S. (Ed.). (1986). The g factor in employment [Special issue]. Journal of Vocational

Behavior, 29, 293–296.

Greenan, J. P. (1983). Identification of generalizable skills in secondary vocational programs

[Executive summary]. Urbana, IL: Illinois State Board of Education.

Guion, R. M. (1980). On trinitarian doctrines of validity. Professional Psychology, 11, 385–398.

Hogan, J. & Hogan, R. (Eds.). (1990). Business and industry testing: Current practice and test

reviews. Austin, TX: Pro-Ed.

Hopkins, K. D., Stanley, J. C., & Hopkins, B. R. (1990). Educational and psychological measurement

and evaluation (7th ed.). Englewood Cliffs, NJ: Prentice Hall.

Hunter, J. E. & Hunter, F. H. (1984). Validity and utility of alternative predictors of job performance.

Psychological Bulletin, 96, 72–98.

Impara, J. C. & Conoley, J. C. (Eds.). (1995). The twelfth mental measurements yearbook. Lincoln,

NE: University of Nebraska.

Maddox, T. (Ed.).(1997). Tests: A Comprehensive reference for assessments in psychology,

education, and business (4th ed.). Austin, TX: Pro-Ed.

McLarty, J. R. (1992, August). Work Keys: Developing the assessments. In J. D. West (Chair), Work

Keys supporting the transition from school to work. A symposium presented at the one

hundredth annual meeting of the American Psychological Association, Washington, DC.

Murphy, L. L., Spies, R. A. & Plake, B. S. (Eds.). (2006). Tests in print VII: An index to tests, test

reviews, and the literature on specific tests. Lincoln, NE: University of Nebraska.

Nathan, B. R. (1995, June). Tests great, less filing: Reducing adverse impact through job analysis.

Paper presented at the annual meeting of the International Personnel Management Association

Assessment Council, New Orleans, LA.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw Hill.

Secretary’s Commission on Achieving Necessary Skills (SCANS), Employment and Training

Administration, U.S. Department of Labor. (1990). Identifying and describing the skills

required by work.

Society for Industrial and Organizational Psychology (2018). Principles for the validation and use of

personnel selection procedures (4th ed.). Bowling Green, OH: Author.

U.S. Department of Labor, Employment and Training Administration (2010). Occupational

Information Network (O*NET) [Data file]. Retrieved from O*NET Resource Center web site:

http://www.onetcenter.org/database.html?p=2

U.S. Equal Employment Opportunity Commission (1978). Uniform guidelines on employee selection

procedures, Federal Register, 43, 38290–38315.

Appendix B

Skill Descriptions

APPLIED MATH SKILL

WorkKeys

Applied Math is the skill people use when they use mathematical reasoning and

problem-solving techniques to solve work-related problems. Employees may use calculators

and conversion tables to help with the problems, but they still need to use math skills to think

them through.

There are five levels of difficulty. Level 3 is the least complex and Level 7 is the most

complex. The levels build on each other, each incorporating the skills assessed at the

previous levels. For example, at Level 5, employees need the skills from Levels 3, 4, and 5.

Examples are included with each level description.

When deciding what level of the Applied Math skill employees need for the tasks they do at

work, consider the following questions:



How is the information presented? That is:



Is it presented in the same order that it is needed?



Is it necessary to change the order that the information is in before the math can be

performed?

Is all the information needed for solving the problems provided? That is:



Is all the information presented in the right form?



Is it necessary to do some calculations to get some of the important information?



Does the problem require a formula?



Does the information need to be taken from a graphic?

What kind of mathematical operations do employees perform? That is:



Can the math problem be completed in one step?



Does the problem need to be done in several steps?



Is it necessary to convert measurements from one form to another, either within or

between systems of measurement?

Applied Math Level 3

Level 3 problems can easily be translated from a word problem to a math equation requiring

a single type of math operation. All the needed information is presented in a logical order

and there is no extra information given.

When employees use Level 3 Applied Math skills on the job, they can:



Solve problems that require a single type of mathematical operation. They add or

subtract either positive or negative numbers (such as 10 or -2). They multiply or

divide using only positive numbers (such as 10).



Convert a familiar fraction (such as ½ or ¼ to a decimal) and convert from a decimal

to a common fraction; OR convert between decimals to percentages (such as 0.75 to

75%).



Convert between familiar units of money and time (for example, one hour equals 60

minutes or ½ of a dollar equals $0.50)



Add the prices of several products to reach a total, and they can make the correct

change for a customer.

Applied Math Level 4

At Level 4, tasks may present information out of order and may include extra, unnecessary

information. One or two operations may be needed to solve the problem. A chart, diagram,

or graph may be included.

When employees use Level 4 Applied Math skills on the job, they can use the skills

described at Level 3, and they can:



Solve problems that require one or two operations. They may add, subtract, or

multiply using positive or negative numbers (such as 10, -2), and they may divide

positive numbers (such as 10).



Figure out an average or mean of a set of numbers (such as

(10 11 12 )



). For this they

use whole numbers and decimals.



Figure out simple ratios (such as

), simple proportions (such as

100

cases), or rates

(such as 10 mph).



Add commonly known fractions, decimals, or percentages (such as ½, .75, or 25%).



Add or subtract fractions that share a common denominator (such as



Multiply a mixed number (such as 12

) by a whole number or decimal.



Put the information in the right order before they perform calculations.

For example, at this level, employees can figure out sales tax or a sales commission on a

previously calculated total, and they can find out rates of use or business flow.

Applied Math Level 5

In Level 5 problems, the information may not be presented in logical order; the item may contain

extraneous information; it may contain a chart, graph or diagram; and the mathematical set-up

may be complicated. In solving, the test taker may need to perform multiple operations. For

example, at this level employees may complete an order form by totaling an order and then

computing tax.

When employees use Level 5 Applied Math skills on the job, they can use the skills described at

Levels 3 and 4, and they can:



Decide what information, calculations, or unit conversions to use to find the answer to a

problem.



As part of a multiple step problem, the employee may have to find one value and use it

to find another value that answers the question.



Add and subtract fractions with unlike denominators (such as ½ - ¼).



Convert units within or between systems of measurement (e.g., time, measurement, and

quantity) where the formula is provided such as converting from ounces to pounds or

from centimeters to inches.



Solve problems that require mathematical operations Calculate using mixed units, such

as adding 3.50 hours and 4 hours 30 minutes or subtracting 3 feet and 10 inches from 6

feet and 4 inches.



Identify the best deal by doing one- and two-step calculations and then comparing the

results to determine the solution that meets the stated conditions.



Calculate perimeters, circumference, and areas of basic shapes like rectangles and

circles.



Calculate a given percentage of a given number and then use that percentage to

determine the solution (e.g., find the total cost of a product after calculating discount,

markup or tax.



Identify where a mistake occurred in a calculation (such as identifying the row in a

spreadsheet where a problem occurred).

Applied Math Level 6

Level 6 tasks may require considerable translation from verbal form to mathematical expression.

They generally require considerable setup and involve multiple-step calculations.

When employees use Level 6 Applied Math skills on the job, they can use the skills described at

Levels 3, 4, and 5, and they can:



Use fractions with unlike denominators and calculate reverse percentages.



Convert units within or between systems of measurement (e.g., time, measurement, and

quantity) where multiple-step conversions are required and the formulas are provided

such as converting from kilometers to meters to feet.



Identify why a mistake occurred in a solution.



Find the best deal and use the result for another calculation.



Find the area of basic shapes (rectangles and circles) when it may be necessary to

rearrange the formula, convert units of measurement in the calculations, or use the result

in further calculations.



Find the volume of rectangular solids.



Calculate rates, productions rates, rate by time (such as, production rate is 59 cups

produced per hour, how many will be produced in an 8 hour shift).



Identify the correct equation for solving a problem

Applied Math Level 7

At Level 7, the task may be presented in an unusual format and the information presented may be

incomplete or require the employee to make an assumption. Tasks often involve multiple steps

of logic and calculation, and multiple operations.

When employees use Level 7 Applied Math skills on the job, they can use the skills described at

Levels 3, 4, 5, and 6, and they can:



Solve problems that include ratios, rates, or proportions with at least one of the

quantities related to a fraction



Identify the reason for a mistake.



Convert between units of measurement that involve fractions, mixed numbers, decimals,

or percentages.



Find the area of multiple shapes or find the area of a composite shape.



Calculate volumes of spheres, cylinders, or cones



Calculate the volume when it may be necessary to rearrange the formula, convert units

of measurement in the calculations, or use the result in further calculations



Set up and manipulate ratios, rates or proportions where at least one of the quantities is a

fraction.



Determine the better economic value of several alternatives by using graphics or by

finding a percentage difference or a unit cost.



Apply basic statistical concepts for example calculate the weighted mean, interpret

measures of central tendency, or interpret measure of spread and tolerance.

APPLIED TECHNOLOGY SKILL

WorkKeys Applied Technology is the skill people use when they solve problems with machines

and equipment found in the workplace. This skill includes four areas of technology: electricity,

mechanics, fluid dynamics, and thermodynamics. Employees need to know the basic principles

of each area, as described below.

Electricity involves the flow of electrons. Employees need to know how electrical current

moves through a circuit or a system and how electricity affects a circuit or system. For example,

they need to know how to control current and resistance.

Mechanics involves the way solid things move and how leverage, force, friction, and momentum

affect that motion. Employees need to solve problems with simple machines, complex machines,

and mechanical systems.

Fluid dynamics involves the way fluids (liquids and gases such as water and air) move through

systems. Employees need to use this knowledge to solve problems with plumbing, hydraulics, or

pneumatics (compressed gas).

Thermodynamics involves the movement of heat. Employees need to know which substances

warm up quickly when heated and which ones warm up more slowly. For example, air heats

faster than water. They also need to know how specific heat works. That is, they need to know

how different materials hold heat for different amounts of time. They need to solve problems

with refrigeration, heating, air conditioning, and phase changes.

Applied Technology Skill Continued

The Applied Technology skill focuses on reasoning, not math. Therefore, employees do not

need to make calculations or use formulas to solve problems.

When employees use the Applied Technology skill, they may:



Analyze a problem by identifying the problem and its parts.



Decide which parts of a problem are important.



Decide on the order to follow when dealing with the parts of the problem.



Apply existing tools, materials, or methods to new situations.

There are four levels of difficulty. Level 3 is the least complex and Level 6 is the most complex.

The levels build on each other, each incorporating the skills assessed at previous levels. For

example, Level 5 includes the skills used at Levels 3, 4, and 5. Individual situations may involve

only one area of technology, but each skill level requires employees to know the basic principles

of all four areas at that skill level.

When considering the level of Applied Technology skill needed for the tasks employees

complete on the job, you should think about the complexity of the system and the physical

principles involved. You might consider the following questions:

How complex is the system? That is:



Is the system one simple object or several objects working together?



Are tools or test equipment involved?



How much extra information is included?

How complicated is the troubleshooting task? That is:



How many components of the system are involved?



Are the underlying principles elementary or complex?



Is more than one area of technology involved?

Applied Technology Level 3

Level 3 tasks are straightforward because the situations include the following characteristics:



They involve one simple system (such as a sprinkler) that generally has two to five

components.



They show clear physical symptoms and usually have only one variable.



They include all the needed information.



They use only basic technical terms (such as loosen, temperature, and flow).

When employees use Level 3 Applied Technology skills on the job, they can:



Identify how basic tools (such as hammers and wrenches) work.



Identify how simple machine parts work (such as how turning a screw clockwise drives

it into wood).



Apply basic principles to solve problems involving a simple system (for example,

friction creates heat).



Solve basic problems (for example, using heat to loosen a metal nut stuck to a bolt or

using a rubber washer to stop a leak at a hose connection).



Identify the clear physical symptom (such as an oil leak, a stopped shaft, or warm coils)

that points to the potential source of a problem.



Identify the best solution after getting rid of clearly unsuitable options.

Applied Technology Level 4

Level 4 tasks are moderately complex because the situations include the following

characteristics:



They can involve two or more simple systems that work together (such as a timer and

sprinkler) or one moderately complex system (such as plumbing for a sink). These

systems have up to ten components.



They can involve one or two variables.



They present all the information that is needed but may contain extra information.



They include less common technical terms that are defined (such as heat exchanger and

circuit breaker).

When employees use Level 4 Applied Technology skills on the job, in addition to the skills at

Level 3, they can:



Understand the operation of moderately complex tools (such as gear pullers) and

diagnostic equipment (such as continuity testers).



Understand the operation of moderately complex machines and systems. These can

include appliances, pulley-driven equipment, or piping systems that carry more than one

fluid. For example, they can find the drive wheel shown on a diagram of a tape player.



Apply less obvious basic principles to solve problems within physical systems (moving

air cools a warm object).



Solve moderate problems such as installing a valve to manage oil flow or using a block

to improve the leverage of a pry bar.



Eliminate physical symptoms that do not point to the source of a problem by choosing

which information is important and ignoring information that is clearly extra.



Identify the best solution after getting rid of other unsuitable options.

Applied Technology Level 5

Level 5 tasks can be moderately complex or of advanced difficulty because the situations include

the following characteristics:



They can involve two or more simple tools or systems that affect each other or a

complex system (such as the plumbing in a house) that includes several components

(such as drains, hot and cold water lines, and valves). These systems perform somewhat

complex operations and generally have more than ten components.



They can involve two or three variables.



They may require the use of technical knowledge (for example, condensing coils should

be warm and evaporating coils should be cool) and often include extra information.



They use technical terms, which may be clearly defined or whose meaning may be

implied in the context and illustrations.

When employees use Level 5 Applied Technology skills on the job, in addition to the skills at

Levels 3 and 4, they can:



Understand the operation of moderately complex tools (such as cutting torches and drill

presses) and diagnostic equipment (such as ohmmeters, micrometers, or thermometers)

and choose the best tool for the task.



Understand the operation of complex machines and systems, such as gasoline engines,

dishwashers, freezers, and building electrical systems.



Apply two or more principles of technology (within one area or in two or more areas) as

they interact in moderately complex systems (heat from friction causes components to

expand).



Solve moderate and advanced problems, such as how to change the speed of pulley-

driven machines, protect electric circuits from overload, place fans in the best position,

or find a problem in a hydraulic cylinder.



Eliminate physical symptoms that do not lead to the source of a problem by choosing

which information is important and ignoring extra information. To do this, it is

necessary to use clues to find the source of a problem. For example, the smell of hot

rubber near a conveyor might indicate a loose drive belt.



Identify the best solution after getting rid of other suitable options.

Applied Technology Level 6

Level 6 tasks are advanced because the situations include the following characteristics:



They involve complex tools or systems (such as the water system of a new subdivision)

with more than ten components and they include large amounts of information.



They present a variety of possible sources of problems that are subtle and difficult to

diagnose.



They may involve many variables at the same time.



They require the use of technical, although not job-specific, knowledge (for example,

how an electromagnetic relay operates) and contain considerable extra information.



They use technical terms, which may be clearly defined or whose meaning may be

implied in a complex context and illustrations.

When employees use Level 6 Applied Technology skills on the job, in addition to using the skill

at Levels 3, 4, and 5, they can:



Understand the operation of complex tools (such as an arc welder or compound miter

saw) and diagnostic equipment (such as an oscilloscope) and choose the best tool for the

task.



Understand the operation of complex machines and their components (for example,

hydraulic lifts, automobiles, and water treatment facilities).



Apply two or more principles of technology (within one area or in two or more areas) as

they interact in complex systems. For example, fluids, like refrigerant, are often

subjected to phase changes in order to move greater amounts of heat.



Solve advanced problems where a variety of mechanical, electrical, thermal, or fluid

faults could be the reason for the problem.



Eliminate physical symptoms that do not lead to the source of a problem by choosing

which information is important and ignoring extra information. To do this, it is

necessary to use less obvious clues to find the source of a problem. For example,

discolored oil may mean a faulty gasket is allowing moisture into an engine.



Test possible hypotheses to ensure the problem is diagnosed correctly and the best

solution is found. This is necessary when systems have difficult problems that could be

caused by a variety of mechanical, electrical, thermal, or flow faults.

BUSINESS WRITING SKILL

WorkKeys Business Writing is the skill people use when they write an original response to a

work-related situation. Components of the Business Writing skill include sentence structure,

mechanics, grammar, word usage, tone and word choice, organization and focus, and

development of ideas.

The main requirement of workplace writing is clarity. Employers want their employees’ written

communications to be direct, grammatically correct, and easy to read. Careless errors may lead

the reader to believe there are also errors in the facts, and the writer loses credibility and

trustworthiness. WorkKeys Business Writing requires standard business English, defined as

writing that is direct, courteous, grammatically correct, and not overly casual.

There are five skill levels. Level 1 is the least complex and Level 5 is the most complex. At

each new level, employees demonstrate more competent writing skills than those used at the

previous levels. For example, Level 3 builds upon the skills used at Levels 1 and 2. With the

increased skills at each level, the writing that employees produce communicates more clearly and

smoothly as they move to Level 5. At the same time, errors become less frequent and less

serious.

When you evaluate a job to see what skill level employees need for completing their tasks,

consider the following questions:



To what extent do the ideas need to be developed? Should the writing include relevant

supporting examples and details, or should there just be a focus on the main ideas

themselves?



How important is the writing style? Does the writing have to flow smoothly, or can it be

choppy as long as the writer’s thoughts are successfully communicated?



Is it necessary to have a professional tone? Does the writing need to be formal, or is it

all right to be casual?



How important are the writing mechanics and grammar? Is it important for all the

grammar, punctuation, and spelling to be correct, or are errors acceptable as long as the

ideas get across?

BUSINESS WRITING SKILL Continued

When measuring the WorkKeys Business Writing skill, we focus on the writing a person can

produce without help from a dictionary, a spelling or grammar checker, another person, or any

other aids. As you decide which writing skills employees need on the job, you need to consider

what kind of writing they must produce.



The assigned skill level will be higher if the writing must be polished and well

developed, but the writer does not have the chance to get help from a dictionary, a

spelling or grammar checker, another person, or any other aids.



The assigned skill level will be lower if the final product must be polished and well

developed, but the writer has the chance to use writing aids.



The assigned level will also be lower if the important thing is to convey the information,

and what the writing looks like is not as important.

After this skill description, the skill levels are described individually. Sample responses are also

provided, and each sample is followed by an explanation of why the response is at the specified

level.

Business Writing Level 1

Employees with Level 1 skills can write responses in English. However, a large number of

errors make the response very difficult to understand.

When employees use Level 1 Business Writing skills on the job, they produce writing:



That consists of incomplete sentences the majority of the time.



That includes a large number of major grammatical, mechanical, and word usage errors

that interfere with communication.



That has rude or overly casual language, tone, and style that may be inconsistent with

standard business English.



That has no organization.



That attempts to communicate their ideas but provides little or no development or

support.

Business Writing Level 2

Employees with Level 2 skills can write responses that are generally understandable. When

employees use Level 2 Business Writing skills on the job, they can produce writing:



That has some complete sentences, although some sentences may be simple or repetitive.



With enough correct mechanics, word usage, and grammar to convey an idea, although

many errors may somewhat interfere with comprehension.



That has rude or overly casual language, tone, and style that may be inconsistent with

standard business English.



With some organization, but with an unclear focus. They use few or no transitions.



With ideas that are generally understandable but that are not expanded.

Business Writing Level 3

Employees with Level 3 skills can write clear responses. The writing may include incorrect

structure, some errors in grammar and punctuation, and adequate development, although the

ideas may be limited.

When employees have Level 3 Business Writing skills on the job, they produce writing:



With most of the sentences complete. Some variety and complexity are attempted.



With few mechanical, grammatical, and word usage errors so that the response is

adequately conveyed, but may be repetitive. Spelling is generally correct.



With style and tone that are generally consistent with standard business English. The

writing may have somewhat casual language, but does not contain rude language.



That exhibits some organization but that may lose focus at some point. Transitions are

simple.



That has adequate development of ideas, but which may be limited in depth and

thoroughness. Supporting examples tend to be general and details are relevant, but they

may be repetitive.

Business Writing Level 4

At Level 4, employees write responses that are clear, with almost no errors. When employees

have Level 4 Business Writing skills on the job, they produce writing:



With all sentences complete and generally varied in length and complexity.



With very few mechanical, grammatical, and/or word usage errors. These do not

interfere with communication. Word usage is precise and varied.



With style, tone, and language that are consistent with standard business English (the

writing contains no rude or overly casual language).



That is organized and maintains consistent focus. Transitions are effective, if not

especially varied.



That has most of the ideas developed well with relevant supporting examples and details.

Business Writing Level 5

At Level 5, employees write responses that are clear, precise, and generally free of errors. The

writing communicates in a professional, courteous manner.

When employees have Level 5 Business Writing skills on the job, they produce writing:



That uses complete sentences that are varied in length and complexity.



With few or no errors in grammar and/or mechanics. Any errors present do not interfere

with communication. They write with word usage that shows considerable precision and

variety.



With style, tone, and language that are consistent with standard business English (the

writing contains no rude or overly casual language).



That is smoothly organized and that maintains clear and consistent focus from beginning

to end. Transitions are varied and effective, creating a seamless flow of ideas.



That has well-developed ideas elaborated on with relevant supporting examples and

specific details. The writing shows insight, perception, and depth.

GRAPHIC LITERACY SKILL

The WorkKeys Graphic Literacy skill is the skill people use when they work with workplace

graphics such as tables, graphs, charts, digital dashboards, flow charts, timelines, forms, maps,

and blueprints. Employees use this skill when they find, summarize, compare, and analyze

information to make decisions using workplace graphics to solve work-related problems.

There are five levels. Level 3 is the least complex and Level 7 is the most complex. At each new

level, employees need more demanding skills in addition to the skills used at the previous levels. For

example, Level 5 includes the skills used at Levels 3, 4, and 5. At the lower levels, employees may

need to locate or find information in a simple graphic. At the higher levels, employees may use

information in one or more difficult graphics to draw conclusions and make decisions. The

complexity can also increase as the quantity and/or density of the information increases.

Skill levels depend on two things: the complexity of the graphic and the task that the employee is

asked to perform. When you consider what skill level is needed for the tasks that employees

complete on the job, think about the following things:

How complex is the workplace graphic?



Is the graphic simple or difficult, common or uncommon?



Is the content familiar or unfamiliar?



How many graphics are there? Is there one graphic, two graphics multiple graphics, or a

composite graphic (such as a bar chart with a line graph over it)?



How many pieces of information are presented? Is there a lot of data presented or not

very much?



How many variables are there? Are there one or two variables such as weight and age or

are there many variables such as height, weight, age, gender, and body mass index?



If there are axes, how many are there (such as X, Y, and Z)?



How many levels of data are there? Is the data nested such as major cities within states?

How complicated is the employee’s task when using the graphics? That is:



Is it only necessary locate, find, or compare information in a single graphic, or is it

necessary to use the information in another graphic?



Does the next step in a process or procedure need to be identified?



Do trends, patterns, or relationships in a graphic need to be identified, compared, or

interpreted?



Is the information in the graphic used to make inferences or decisions? Does the

inference or decision need to be justified?



Is it necessary to identify the graphic that accurately represents the data or is the most

effective? Does the choice need to be justified?

Graphic Literacy Level 3

At Level 3, workplace graphics are common and of simple or low moderate difficulty.

Characteristics of simple graphics include:



A limited amount of data (i.e., less than twenty data points/fields)



One level of data such as number of items in inventory



One or two variables such as day of the week and number of items in inventory



If there are axes, there will be one or two, such as an x and/or y axis

Characteristics of low moderate graphics include:



A moderate amount of data



More than one level of data, but no nesting



Several variables



If there are axes, there will be one or two



If two simple graphics are required to solve the problem, they should be considered a low

moderate graphic.

At Level 3, employees use one simple or low moderate graphic at a time to perform the

following tasks:



Locate and find information



Identify the next or missing step in a process

Graphic Literacy Level 4

At Level 4, workplace graphics are common and of low to high moderate difficulty.

Characteristics of low moderate graphics include:



A moderate amount of data



More than one level of data, but no nesting



Several variables



If there are axes, there will be one or two



If two simple graphics are required to solve the problem, they should be considered a low

moderate graphic.

At Level 4, employees have demonstrated all of the skills defined at Level 3 and they can use

one or two low moderate graphics at a time to perform the following tasks:



Locate information in a graphic using information found in another graphic



Compare two or more pieces of information



Identify a trend/pattern/relationship



Make an inference or decision



Identify the graphic that accurately represents the data

High moderate graphics may be less common and have the following characteristics:



A moderate amount of data



More than one level of data and it may be nested



Many variables such as types of wood, drill speeds, hole diameter, and type of bit



If there are axes, there will be one or two such as an x and/or y axis.



If a low moderate graphic and a simple graphic are required to solve the problem, they

should be considered a high moderate graphic.

At Level 4, employees have demonstrated all of the skills defined at Level 3 and they can use

one high moderate graphic to perform the following tasks:



Locate and find information



Identify the next or missing step in a process

Graphic Literacy Level 5

At Level 5, workplace graphics may be less common and of low moderate, high moderate, or

difficult complexity.

Characteristics of low moderate graphics include:



A moderate amount of data



More than one level of data, but no nesting



Several variables



If there are axes, there will be one or two.



If two simple graphics are required to solve the problem, they should be considered a low

moderate graphic.

At level 5, employees have demonstrated all of the skills defined at Levels 3 and 4, and they can

use a low moderate graphic to perform the following tasks:



Compare two or more pieces of information



Interpret a trend/pattern/relationship



Make a reasonable inference or decision based on one graphic after finding information

in another graphic



Justify a decision or inference based on information



Identify the most effective graphic for the task



Justify the most effective graphic for the task

High moderate graphics may be less common and have the following characteristics:



A moderate amount of data



More than one level of data and it may be nested



Many variables



If there are axes, there will be one or two.



If a low moderate graphic and a simple graphic are required to solve the problem, they

should be considered a high moderate graphic.

Graphic Literacy Level 5 Continued

At Level 5, employees have demonstrated all of the skills defined at Level 3 and 4, and they can

use one high moderate graphic to perform the following tasks:



Locate information in a graphic using information found in another graphic



Compare two or more pieces of information



Identify a trend/pattern/relationship



Make an inference or decision



Identify the graphic that accurately represents the data

Difficult graphics are likely to be less common or a composite of graphics and have the

following characteristics:



Data presented is dense.



More than one level of data and nesting is likely



Many variables such as types of wood, drill speeds, hole diameter, and type of bit



If there are axes, there will be three or more such as an x, y, and z axis.



If a high moderate graphic and a low moderate graphic are required to solve the problem,

they should be considered a difficult graphic.

At Level 5, employees have demonstrated all of the skills defined at Level 3 and 4, and they can

use one difficult graphic to perform the following tasks:

 Locate and find information



Identify the next or missing step in a process

Graphic Literacy Level 6

At Level 6, workplace graphics may be less common and of high moderate or difficult

complexity.

High moderate graphics may be less common and have the following characteristics:



A moderate amount of data



More than one level of data and it may be nested



Many variables



If there are axes, there will be one or two



If a low moderate graphic and a simple graphic are required to solve the problem, they

should be considered a high moderate graphic.

At level 6, employees have demonstrated all of the skills defined at Levels 3, 4 and 5, and they

can use a high moderate graphic to perform the following tasks:



Compare two or more pieces of information



Interpret a trend/pattern/relationship



Make a reasonable inference or decision based on one graphic after finding information

in another graphic



Justify a decision or inference based on information



Identify the most effective graphic for the task



Justify the most effective graphic for the task

Difficult graphics are likely to be less common or a composite of graphics and have the

following characteristics:



Data presented is dense.



More than one level of data and nesting is likely



Many variables



If there are axes, there will be three or more.



If a low moderate graphic and a high moderate graphic are required to solve the problem,

they should be considered a difficult graphic.

Graphic Literacy Level 6 Continued

At Level 6, employees have demonstrated all of the skills defined at Level 3, 4 and 5, and they

can use one difficult graphic to perform the following tasks:



Locate information in a graphic using information found in another graphic



Compare two or more pieces of information



Identify a trend/pattern/relationship



Make an inference or decision



Identify the graphic that accurately represents the data

Graphic Literacy Level 7

At Level 7, workplace graphics may be less common and of difficult complexity.

Difficult graphics are likely to be less common or a composite of graphics and have the

following characteristics:



Data presented is dense.



More than one level of data and nesting is likely



Many variables



If there are axes, there will be three or more.



If a low moderate graphic and a high moderate graphic are required to solve the problem,

they should be considered a difficult graphic.

At level 7, employees have demonstrated all of the skills defined at Levels 3, 4, 5 and 6, and they

can use a difficult graphic to perform the following tasks:



Compare two or more trends/patterns/relationships



Interpret a trend/pattern/relationship



Make a reasonable inference or decision based on one graphic after finding information

in another graphic



Justify an inference or decision based on information



Identify the most effective graphic for the task



Justify the most effective graphic for the task

WORKPLACE DOCUMENTS SKILL

Employees read and use workplace documents in order to do a job. The documents include, but

are not limited to, messages, emails, letters, directions, signs, notices, bulletins, policies,

websites, contracts, and regulations and are based on materials that reflect the actual reading

demands of the workplace.

It is often the case that these workplace communications are not necessarily clearly written or

targeted to the appropriate audience. These documents do not include information that is

presented graphically, such as in charts, forms, or blueprints.

There are five levels of difficulty. Level 3 is the least complex and Level 7 is the most complex.

The levels build on each other, each incorporating the skills assessed at the preceding levels. For

example, at Level 5, employees need the skills from Levels 3, 4, and 5. The reading materials at

Level 3 are short and direct. The material becomes longer, denser, and more difficult to use as

readers move toward Level 7. The tasks also become more complex as readers move from

Level 3 to Level 7. At Level 3, readers begin by finding very obvious details and following short

instructions. At the more complex levels, tasks can also involve more application and

interpretation.

When you consider what level of skill is needed for the tasks employees complete on the job,

you might consider the following questions:

How difficult are the materials? For example:



Are the sentences short, simple, and clear; or are they complex and possibly even

confusing?



Do the materials use only common words; or do they include difficult words, jargon, and

words used in unfamiliar ways?



How much extra information is included?

How complicated is the task? For example:



Is it only necessary to use information that is stated clearly?



Is it necessary to make inferences based on the reading materials before using the

information?



Do the employees need to apply the information to a situation exactly like the one

described in the materials or to one that is quite different?

Workplace Documents Level 3

Level 3 reading materials include basic work related policies, procedures, and announcements

with the following characteristics:



They are short, with no extra information.



Employees read the materials to find out what they should do.



All the information within the document is stated clearly and directly.



Short sentences and common, everyday, and workplace words (such as employee,

timecard, office) are used.



The document contains a small number of clearly stated details.

When employees use Level 3 skills on the job, they can:



Find the main ideas and clearly stated details.



Choose when to perform each step in a series of short steps.



Apply information/instructions to a situation that is the same as the one they are reading

about (such as knowing what button to push first after reading instructions on how to run

a copy machine).

Workplace Documents Level 4

Level 4 workplace documents include policies, procedures, and notices with the following

characteristics:



They are straightforward with some long sentences and contain a number of details.



These materials use common words, but do have some harder words, too.



They describe procedures that include several steps.



When following the procedures, employees must think about changing conditions that

affect what they should do. For example, they can follow directions that include “if-then”

statements.

When employees use Level 4 skills on the job, in addition to using Level 3 skills, they can:



Identify the main idea and details that may not be clearly stated.



Use the reading material to figure out the meaning of words that are not defined for them

(not jargon or technical terms).



Apply information/instructions to a situation that is the same as the situation in the

reading materials.



Choose what to do when changing conditions call for a different action.

Workplace Documents Level 5

At Level 5, workplace documents include policies, procedures, announcements, legal, and

multiple related documents that have many details with the following characteristics:



The information that employees need is generally stated directly, but it is hard to find

because there are so many details and some may not be needed for the task being

performed (extraneous information).



The materials include technical terms, jargon, and acronyms, or words that have several

meanings.



The documents may have complex sentences and/or contain conditional situations.

When employees use Level 5 skills on the job, in addition to using the skills described at Levels 3

and 4, they can:



Figure out the appropriate meaning of a word based on how the word is used.



Identify the appropriate meaning of technical term, jargon, or an acronym that is defined

in the document.



Apply technical terms and jargon to stated situations.



Apply information/instructions to a new situation that is similar to the one described in

the material while considering changing conditions.



Apply complex information/instructions that include conditionals to situations described

in the materials.



They may need to make some inferences to accomplish their goal.

Workplace Documents Level 6

At Level 6, workplace documents include policies, informational, instructional (procedures),

legal, and multiple related documents with the following characteristics:



They use mostly complicated sentences.



Documents may be long and/or complex and/or contain conditional situations.



There are implied and/or extraneous details with difficult words, jargon, and technical

terms.



Most of the information is not clearly stated.



Meanings may need to be determined from context.

When employees use Level 6 skills on the job, in addition to using the skills at Levels 3, 4, and

5, they can:



Infer implied details.



Infer the meaning of an acronym, jargon, or technical term from context.



Apply information/instructions to a situation not directly described or to a completely

new situation.



Apply principles inferred in a passage to a situation not directly described or to a

completely new situation.



Identify the rationale behind a procedure, policy, or communication.

Workplace Documents Level 7

At Level 7, workplace documents include policies, informational, instructional (procedures),

legal, and multiple related documents with the following characteristics:



The documents contain a lot of details, and the concepts are complicated.



May cover uncommon topics (concepts) and/or contain conditional situations.



There are implied and extraneous details.



Advanced, unfamiliar, and/or uncommon words, technical terms, and jargon;

meanings must be determined from context.



Not clearly stated, pieces of information may be spread throughout the document and

may be extraneous.

When employees use Level 7 skills on the job, in addition to using the skills at Levels 3, 4, 5,

and 6, they can:



Infer the meaning of an acronym, jargon or technical term from context.



Apply principles inferred from the materials to a situation not directly described or to

a completely new situation.



Identify the rationale behind an entire document or a section of a document.



Infer implied details.

WORKPLACE OBSERVATION SKILL

WorkKeys

Workplace Observation is the skill that employees use to visually observe a

workplace event, notice details, and remember instructions, procedures, processes, and

demonstrations in order to generalize to workplace situations that may be similar or very

different from what was observed. Employees must pay careful attention to steps that are

followed, to safety procedures, and to quality-control standards.

There are five levels. Level 1 is the least complex and Level 5 is the most complex. The

levels build on each other, each incorporating the skills assessed at the previous levels. For

example, Level 5 includes the skills used at Levels 1, 2, 3, 4, and 5. The skill level is

determined by the complexity of the situation being observed and the task(s) that employees

are asked to do based on their observations. At Level 1, employees must be able to recall

information from a short, straightforward sequence with few details, no distractions, and

obvious differences from the standard. For example, they may need to identify the next step

in a series of steps. At Level 5, employees must be able to generalize information from a

complex situation to new situations in order to make accurate predictions or anticipate

changing variables. Distractions and differences are difficult to recognize.

Video examples of situations that belong at each skill level are provided. When you consider

what skill level is needed for the tasks that employees complete on the job, think about the

following things:

How complex is the procedure being observed and remembered?



Is it logical or illogical, familiar or new, commonplace or unique, straightforward or

complicated?



How much information is involved, and are the procedure’s parts independent or

interactive?



How much distracting information is there?



Are extra details likely?



How difficult is it to detect differences, discrepancies, or changes?

How difficult is the task that employees are asked to do?



How much generalizing is the employee required to do? Are they identifying the

next step in a straightforward process or are they determining how a change will

affect an outcome?



How subtle are the details or differences in the procedure to be noticed by

employees? Are these differences significant?



Are employees required to apply instructions, demonstrations, procedures, or

processes to other situations?



Do employees need to take changing conditions into account to choose the best

course of action?

Workplace Observation Level 1

At Level 1, employees follow a short, straightforward, and simple procedure with each step

clearly shown. The task is performed in a routine and predictable manner. There are no

distractions and differences are obvious. There are a few details, but no unnecessary details.

When employees use Level 1 Workplace Observation skills on the job, they can:



Repeat a short, straightforward demonstration, process, pattern, or procedure



Recognize an incorrect step (wrong order or not in process)



Identify the next step in a series of steps



Put steps in correct order



Identify a missed or incorrect step



Match placement or identify misplacement of components (e.g., follow instructions

for putting phone cord in proper location when packing a box)

Workplace Observation Level 2

At Level 2, employees interpret a straightforward procedure, but there is a condition (if –

then or cause-effect). Several possible things may happen and a specific response is

provided for each one. There are obvious and easily disregarded distractions. There are a

few extra details and differences.

When employees use Level 2 Workplace Observation skills on the job, they can:



Recognize cause and effect in a straightforward demonstration, process, pattern, or

procedure



Filter out obvious distractions



Identify the cause of a particular effect (e.g., alarm sounds when pressurized air is

released)



Recognize what to do next in a situation given a single condition



Indicate action to be taken when there is an incorrect step identified

Workplace Observation Level 3

At Level 3, employees watch complex procedures that include several tasks that may occur at

the same time, interact, and change from one situation to another. More than one condition

(if-then or cause-effect) may be present. Several important details are presented, but a few

are not clearly prompted. Some distractions may make remembering details difficult. The

employee may be asked to apply information observed to other similar situations. Steps may

seem similar, but differ based on varying factors. A few differences may be present that are

not clear.

When employees use Level 3 Workplace Observation skills on the job, they can:



Identify course of action to take given more than one condition



Distinguish steps that seem similar but are different based on varying factors



Maintain attention to significant details with little prompting



Recognize when steps can be combined and when they must be kept distinct



Combine steps to achieve desired result



Identify differences and/or details that are not clear



Select, interpret, and integrate the steps, in the correct order, within a complex

process



Apply information to a similar situation

Workplace Observation Level 4

At Level 4, employees must analyze and determine the basic principles before a process can

be generalized to a new situation. Several conditions are present that may influence the

course of action. Strong distractions compete for attention. Some steps may not be

demonstrated (i.e., inferred). Some of the differences are difficult to notice.

When employees use Level 4 skills on the job, they can:



Make inferences from situational cues in a demonstrated process or procedure



Derive steps that are missing from a partial or non-explicit pattern, process or

procedure



Apply complicated instructions to new situations



Decide which conditions apply to a new situation



Determine the general principle underlying the condition, process, or procedure



Determine what comes next (e.g., tiling a floor and figuring out the next placement

in the pattern)



Break down a given process and apply results to unfamiliar processes to complete a

task or diagnose a problem



Use situational cues to determine steps to be taken

Workplace Observation Level 5

At Level 5, employees evaluate a new situation with multiple conditions and then choose the

best course of action. General principles may be difficult to determine and may need to be

applied differently in order to achieve a desired outcome. The situation requires innovation

and the level of abstraction may be high. Distractions are present that appear to be relevant

but are not. Differences are difficult to recognize and/or evaluate.

When employees use Level 5 skills on the job, they can:



Determine the best course of action by applying principles to a new situation and/or

when the information is not complete.



Make accurate predictions based on what has been observed (e.g., what is the likely

result).



Consider the implications of a process or procedure and how they will affect

outcomes.



Prioritize appropriately (identify aspects of process that should be given priority

under certain circumstances, identify parts of the process that can be omitted in

different circumstances)



Identify ways to improve the process



Evaluate whether something is or is not a distraction

Appendix C

WorkKeys Terminology

Assessment A test used to evaluate individuals' performance in a skill area. Scores

on the WorkKeys assessments can be compared to the WorkKeys skill

levels identified in a profile. The difference between the profiled level

and a score indicates the need for training.

Content Validity

Report A summary generated by the Profiler that includes the Final Task List,

detailed descriptions of the session discussions of each skill, and

recommendations for using the results.

Content Validity

Report Addendum When a previous profile has been conducted and a client wishes to use

a WorkKeys skill assessment that was not included in the profile, the

WorkKeys skill will be profiled and a Content Validity Report

Addendum will be written to document the results.

Effective Level Effective performance is the point at which an employee performs

competently without continuous supervision. Effective performance

Level levels are provided for use as training goals.

Entry-Level Following the Uniform Guidelines on Employee Selection Procedures

(1978), WorkKeys defines entry as an employee’s first day performing

the job. The entry-level skill requirements are recommended for use

as cutoff scores on the related WorkKeys assessments.

Final Task List A list specifying the critical/important tasks for a job in statements that

have been reviewed and edited by SMEs and then placed in order

using SME Importance rating averages.

Importance The importance of the task to the job. Importance is represented by

the mean Importance rating for each task.

Initial Task List Prior to the profiling session, the Profiler develops an Initial Task List

using information compiled from databases (e.g., O*NET), job-related

documentation (e.g., job descriptions, resources from similar job

profiles, training materials), and information gathered from the tour of

the facility.

Job Profile The result of conducting one or more job profiling sessions which

shows the most critical/important tasks for a job and the WorkKeys

skills and skill levels required to perform a job.

Job Profiling A procedure to determine the most critical tasks for a job and to

determine the WorkKeys skills and skill levels required to perform

these tasks. Variations on this model include curriculum profiling –

which analyzes WorkKeys-related curriculum requirements through a

review of the curriculum objectives and a skill analysis that identifies

the WorkKeys skills and skill levels required for entry into, and exit

from, a program, and occupational profiling – which identifies the

WorkKeys skill levels required for an occupation across jobs,

companies, or industries; or an employer.

Profiler An individual who has completed ACT's WorkKeys Profiling training

program successfully. An ACT authorized Profiler has been trained to

facilitate the job profiling process while using the SkillPro

software

and write a report of the profile results

Profiling Session A focus group meeting facilitated by an ACT authorized Profiler. The

Profiler meets with SMEs to perform a task analysis and skill analysis.

Reconciliation When SME groups do not agree on skill requirements (generally for

job entry) the Profiler meets with representative SMEs from each

group to resolve the differences in a reconciliation session.

Replication Replication sessions are additional profiling sessions with different

groups of SMEs. Replication sessions are used to make sure that the

results are consistent from one group to another, when there are a large

number of incumbents on the job.

Skill Analysis A skill analysis occurs after a task analysis is completed and consists

of two parts:

 The SME group identifies the critical/important on-the-job

behaviors (i.e., tasks from the Final Task List) that are associated

with the WorkKeys skills under consideration.

 The SME group compares detailed descriptions of the WorkKeys

skill levels to the critical/important tasks that require the specified

skill. The Profiler seeks to bring the group to a consensus

regarding the skill levels required at job entry and for effective

performance.

Skill Gap When the profiled skill level is higher than the assessment score, the

difference is referred to as a “skill gap.”

SME Subject matter experts are employees currently performing the job or

people knowledgeable about the job being profiled (e.g., supervisors or

people who have been recently promoted from the job).

Task Analysis A task analysis consists of three parts:

 The Profiler meets with the SME group to tailor the Initial Task

List (i.e., add, edit, and delete tasks), making sure that the Final

Task List accurately and completely describes the job.

 The SMEs independently rate each task for Importance.

 The Profiler calculates the importance of each task using the SME

Importance ratings and sorts the task statements by placing the

most important tasks at the beginning of the list. The SMEs

review and confirm the order of the tasks. The product of the task

analysis is the Final Task List.

WorkKeys Skills Applied Math, Applied Technology, Business Writing, Graphic

Literacy, Workplace Documents, Workplace Observation.

Appendix D

Group A Final Task List

The Final Task List is shown in the table below. The mean importance ratings and skill requirements are also shown. An “X” in a

skill column means that, according to the SMEs in the profile session, the task on that row requires that skill. Tasks are presented in

order, from those most critical to job performance to those least critical. The tasks in gray italics received mean ratings of Importance

below 2 (i.e., low importance) and were not included in the skill analysis. The Importance Sum (i.e., sum of importance ratings for the

skill) and Skill Percent (i.e., percentage of important tasks requiring the skill) for each skill are shown at the end of the table. The total

Importance Sum possible is 132.

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Assists customers in the selection and purchase of specialized telephone services, such as

long distance plans by using plan descriptions and text from relevant scripts to review and

est options and to answer questions.

Calls prospective new customers to explain new services and products by using the

appropriate script, marketing materials, and pricing charts based on the client's current plan

and usa

e information as shown in the computer record.

Checks for accuracy of customer name and mailing address each time a customer calls or

writes by asking them for this information and comparing it to the information already in the

database.

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Handles customer complaints concerning billing by identifying the nature of the problem,

correctin

minor billin

errors, and forwardin

other requests.

Prevents customers from discontinuing their service if their reason is due to poor service or a

competitor's better deal by offering special Retention Plans, beginning with the least

expensive options.

Processes customer requests for refunds that are due to a move or change of special services

in the middle of a billing cycle by locating the customer's record in the database to enter the

appropriate debit or credit code in their billin

screen.

Sells telephone services to preferred business accounts by offering special plans and

rades outlined in the plan descriptions and scripts.

Arranges for payment of delinquent accounts by calling customers, getting them to agree on

a payment plan, setting dates and amounts of payments, documenting in the database, and

sendin

a form letter with the a

reed upon terms.

Assists customers in placing long-distance phone calls by asking them the nature of the

problem and either accessing their account and dialing the call for them, or connecting them

to the appropriate service carrier compan

Determines if customer problems are due to hardware (e.g., boxes, cables, or telephone) by

finding the problem on the Troubleshooting Chart and reading the action to take to the

customer and issuin

a work order if indicated b

the test.

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Opens accounts for customers by adding a new record in the computer database and

completing the required fields such as name, street address, billing address and service plan

option(s).

Prepares for a sales call by finding the correct Prospect Screen that shows the services

currently subscribed to by the customer, and using this information to determine the

additional services to recommend and the script to follow.

Prepares for a sales call to a non-GCOMM customer by reviewing the printout showing the

prospect's current phone company, number of people in household, and annual income to

determine the appropriate script to follow.

Processes orders (e.g., installation, discontinuance, or change) by inserting the appropriate

transaction code in the client's computer record along with any special instructions (e.g.,

requested service date/time, dan

erous do

ard).

Quotes prices for customized plans to non-GCOMM customers by entering all charges

included in the quote and charges for current plan onto the electronic Price Quote worksheet

and then usin

the information to explain wh

GCOMM's plan is best.

Recommends additional services to customers by reviewing the customer's record in the

database for existing service plan and current services and then offering services such as

additional phone lines, internet service, or caller ID.

Reports irregularities in the database by contacting the shift supervisor and telling them

and/or showing them the problem (e.g., a recent sale or service was not documented

appropriatel

or when a work order was not issued properl

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Activates phone to receive calls b

clickin

#44 on phone and lo

into computer s

stem. 3

Completes Information Requested order form by entering prospect or customer name and

contact information, indicating which company products and services brochures should be

mailed, and submittin

electronicall

Completes special installation or repair requests for unusual problems by locating the screen

for special requests, entering customer information and special request, and submitting

electronicall

Contacts customers to obtain more information about their accounts by using the up-to-date

form letter and completing the Follow-up Contact form after five unsuccessful attempts to

reach the customer b

phone.

Deactivates phone b

clickin

on #43 on phone and lo

out of computer s

stem. 3

Determines if team is meeting sales goals by reading the Sales Goals Projection Report

distributed to each team.

Documents customer report of interrupted service by checking the Reported Outages screen

to see if their area is listed, apologizing for the service disruption, telling them when it is

expected to return to normal, and thankin

them for the call.

Enters additional account information upon customer request by locating the correct screen

in the computer database and entering the information as the customer provides it (e.g., name

of famil

members added to account).

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Investigates errors found in customer accounts by gathering information from the customer

or their record, comparing it for consistency, using the correct transactions screen to correct

the account, and notin

a description of the action taken.

Investigates potential fraud or misuse of calling cards by checking accounts that have been

flagged by the computer for unusual use (such as calls exceeding $100 in a one-day period)

and attemptin

to contact the customer b

phone and U.S. Mail.

Issues calling cards by verifying the caller's identity, obtaining and entering the required

information (e.g., name, number of cards), and using the appropriate transaction code to

submit the request.

Notifies customers that installation is completed by printing the completed service list and

sendin

the customers a form email or postcard.

Notifies other departments of customer's problem by sending an email request for service

outlining the problem to the appropriate department if the reported problem came in after

hours or the

didn't pick up a transferred call.

Participates in monthly team meetings by attending and participating in discussions about

sales

oals and wa

s to increase sales, usin

information provided b

marketin

Participates in training workshops by attending, participating in activities, and reviewing all

materials.

Provides customer with the price for additional services that are not included in their plan by

usin

pricin

charts, Price Quote worksheet, and a calculator to calculate the price.

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Suggests improvements in work methods and procedures by role playing behaviors during

training sessions, such as suggestive selling, and verbally conveying them to the supervisor

for consideration.

Trains newcomers by allowing them to shadow calls (i.e., observe experienced CSR & their

monitor), and then switching roles to have trainees answer calls and navigate through the

customer account software while bein

supervised.

Updates prospect information during a phone solicitation call by entering or updating

prospect information (e.g., names, addresses, purchases, and reactions or additional notes)

into the sales database.

Pages employees on the floor by operating the in-house paging system to get their attention

and indicatin

which line the call is on.

Performs recommended exercises to prevent carpal tunnel by doing the finger-wrist

exercises twice each shi

t and on breaks.

Transfers customers to the billing department when billing adjustments exceed $100 or if the

customer is more than two months delinquent in their payments by either conveying verbally

or in an email about the situation and previous actions taken.

Prepares a verbal report on daily sales activities by summarizing the types of products or

services sold and reasons wh

customers bou

ht or declined an o

er.

Recommends telephone models for purchase/rent to customers by using the marketing

materials and identi

the customer's needs.

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Records toll charges made by writing down the name of the person called, phone number,

reason

or the call, and the emplo

ee's identi

ication number in the Call Char

e Lo

Updates "No Call" list by entering the names and telephone numbers of prospects contacted

who request to be added to the list, and adding the "no call" code to the record for the

rospect in the sales database.

Importance Sum 30 132 50 132

Skill Percent 19.4 100.0 33.3 100.0

Appendix E

Group B Final Task List

The Final Task List is shown in the table below. The mean importance ratings and skill requirements are also shown. An “X” in a

skill column means that, according to the SMEs in the profile session, the task on that row requires that skill. Tasks are presented in

order, from those most critical to job performance to those least critical. The tasks in gray italics received mean ratings of Importance

below 2 (i.e., of low importance) and were not included in the skill analysis. The Importance Sum (i.e., sum of importance ratings for

the skill) and Skill Percent (i.e., percentage of important tasks requiring the skill) for each skill are shown at the end of the table. The

total Importance Sum possible is 156.

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Assists customers in the selection and purchase of specialized telephone services, such as

long distance plans by using plan descriptions and text from relevant scripts to review and

est options and to answer questions.

Calls prospective new customers to explain new services and products by using the

appropriate script, marketing materials, and pricing charts based on the client's current plan

and usa

e information as shown in the computer record.

Handles customer complaints concerning billing by identifying the nature of the problem,

correctin

minor billin

errors, and forwardin

other requests.

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Investigates errors found in customer accounts by gathering information from the customer

or their record, comparing it for consistency, using the correct transactions screen to correct

the account, and notin

a description of the action taken.

Opens accounts for customers by adding a new record in the computer database and

completing the required fields such as name, street address, billing address and service plan

option(s).

Prepares for a sales call by finding the correct Prospect Screen that shows the services

currently subscribed to by the customer, and using this information to determine the

additional services to recommend and the script to follow.

Prepares for a sales call to a non-GCOMM customer by reviewing the printout showing the

prospect's current phone company, number of people in household, and annual income to

determine the appropriate script to follow.

Prevents customers from discontinuing their service if their reason is due to poor service or a

competitor's better deal by offering special Retention Plans, beginning with the least

expensive options.

Processes orders (e.g., installation, discontinuance, or change) by inserting the appropriate

transaction code in the client's computer record along with any special instructions (e.g.,

requested service date/time, dan

erous do

ard).

Provides customer with the price for additional services that are not included in their plan by

usin

pricin

charts, Price Quote worksheet, and a calculator to calculate the price.

100

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Quotes prices for customized plans to non-GCOMM customers by entering all charges

included in the quote and charges for current plan onto the electronic Price Quote worksheet

and then usin

the information to explain wh

GCOMM's plan is best.

Sells telephone services to preferred business accounts by offering special plans and

rades outlined in the plan descriptions and scripts.

Trains newcomers by allowing them to shadow calls (i.e., observe experienced CSR & their

monitor), and then switching roles to have trainees answer calls and navigate through the

customer account software while bein

supervised.

Arranges for payment of delinquent accounts by calling customers, getting them to agree on

a payment plan, setting dates and amounts of payments, documenting in the database, and

sendin

a form letter with the a

reed upon terms.

Assists customers in placing long-distance phone calls by asking them the nature of the

problem and either accessing their account and dialing the call for them, or connecting them

to the appropriate service carrier compan

Checks for accuracy of customer name and mailing address each time a customer calls or

writes by asking them for this information and comparing it to the information already in the

database.

Completes Information Requested order form by entering prospect or customer name and

contact information, indicating which company products and services brochures should be

mailed, and submittin

electronicall

101

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Completes special installation or repair requests for unusual problems by locating the screen

for special requests, entering customer information and special request, and submitting

electronicall

Contacts customers to obtain more information about their accounts by using the up-to-date

form letter and completing the Follow-up Contact form after five unsuccessful attempts to

reach the customer b

phone.

Determines if customer problems are due to hardware (e.g., boxes, cables, or telephone) by

finding the problem on the Troubleshooting Chart and reading the action to take to the

customer and issuin

a work order if indicated b

the test.

Determines if team is meeting sales goals by reading the Sales Goals Projection Report

distributed to each team.

Documents customer report of interrupted service by checking the Reported Outages screen

to see if their area is listed, apologizing for the service disruption, telling them when it is

expected to return to normal, and thankin

them for the call.

Enters additional account information upon customer request by locating the correct screen

in the computer database and entering the information as the customer provides it (e.g., name

of famil

members added to account).

Investigates potential fraud or misuse of calling cards by checking accounts that have been

flagged by the computer for unusual use (such as calls exceeding $100 in a one-day period)

and attemptin

to contact the customer b

phone and U.S. Mail.

102

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Notifies other departments of customer's problem by sending an email request for service

outlining the problem to the appropriate department if the reported problem came in after

hours or the

didn't pick up a transferred call.

Participates in monthly team meetings by attending and participating in discussions about

sales

oals and wa

s to increase sales, usin

information provided b

marketin

Participates in training workshops by attending, participating in activities, and reviewing all

materials.

Processes customer requests for refunds that are due to a move or change of special services

in the middle of a billing cycle by locating the customer's record in the database to enter the

appropriate debit or credit code in their billin

screen.

Recommends additional services to customers by reviewing the customer's record in the

database for existing service plan and current services and then offering services such as

additional phone lines, internet service, or caller ID.

Reports irregularities in the database by contacting the shift supervisor and telling them

and/or showing them the problem (e.g., a recent sale or service was not documented

appropriatel

or when a work order was not issued properl

Updates "No Call" list by entering the names and telephone numbers of prospects contacted

who request to be added to the list, and adding the "no call" code to the record for the

prospect in the sales database.

103

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Updates prospect information during a phone solicitation call by entering or updating

prospect information (e.g., names, addresses, purchases, and reactions or additional notes)

into the sales database.

Issues calling cards by verifying the caller's identity, obtaining and entering the required

information (e.g., name, number of cards), and using the appropriate transaction code to

submit the request.

Notifies customers that installation is completed by printing the completed service list and

sendin

the customers a form email or postcard.

Recommends telephone models for purchase/rent to customers by using the marketing

materials and identif

the customer's needs.

Records toll charges made by writing down the name of the person called, phone number,

reason for the call, and the emplo

ee's identification number in the Call Char

e Lo

Transfers customers to the billing department when billing adjustments exceed $100 or if the

customer is more than two months delinquent in their payments by either conveying verbally

or in an email about the situation and previous actions taken.

Pages employees on the floor by operating the in-house paging system to get their attention

and indicatin

which line the call is on.

Performs recommended exercises to prevent carpal tunnel by doing the finger-wrist

exercises twice each shi

t and on breaks.

104

Tasks

Importance

Applied Math

Graphic Literacy

Workplace Documents

Workplace Observation

Prepares a verbal report on daily sales activities by summarizing the types of products or

services sold and reasons wh

customers bou

ht or declined an o

er.

Suggests improvements in work methods and procedures by role playing behaviors during

training sessions, such as suggestive selling, and verbally conveying them to the supervisor

or consideration.

Importance Sum 41 156 47 156

Skill Percent 24.3 100.0 27.0 100.0