Job Demands Final Rpoert - Disability Research Institute

Social Security Job Demands Project
Methodology to Identify and Validate Critical Job Factors
Deliverable 9 Submitted by
Disability Research Institute at
The University of Illinois at Champaign-Urbana Northwestern University
Rehabilitation Institute of Chicago November 3, 2002 The research reported herein was performed pursuant to a grant from the
U.S. Social Security Administration (SSA) and funded as part of the
Disability Research Institute (DRI). The opinions and conclusions expressed
are solely those of the authors and should not be construed as representing
the opinions or policy of SSA or any agency of the Federal Government.
Table of Contents
Table of Contents 1 Executive Summary 7 Chapter 1 11
Introduction 11
Overview of SSA Transition from DOT to O*Net 11
SSA's Concerns Regarding O*NET 12
Data Aggregation Issues 12
Job Demands and Measures 13
Data Collection Issues 13
Activities to Address Concerns about O*NET and Obsolescence of DOT 14
Defining the Conceptual Model and Domains of Interest for Identifying
Optimal Job Demands 15
Disability Determination within SSA 15
The SSA Five-Step Sequential Evaluation Process 16
1. Is the individual engaging in substantial gainful activity? 16
2. Does the individual have an impairment or combination of
impairments that significantly limits his or her physical or mental
ability to do basic work activities? 16
3. Does the individual's impairment(s) meet or equal a listed
impairment in Appendix 1 to Subpart P of the Regulations Part 404
(Listing of Impairments)? 16
4. Does the individual's impairments prevent the claimant from
performing his or her past relevant work? 17
5. Does the individual's impairment(s) prevent him or her from
performing other work? 18
The Current Project 20
Methodology 22
Deliverables 23
Overview of the Proposed Procedures for Identifying Optimal Job
Demands 24 Chapter 2 28
Review of Job Analysis Methodologies 28 Level 3 Review of Job Materials 29
Direct Observation 30
Performance 30
Interview 30
Subject Matter Experts (SME) panels 31
Critical incidents 31
Functional Job Analysis (FJA) 32
Task inventories 32
Standardized job analysis systems/questionnaires 32 Review of Job Analysis Systems/Questionnaires 33
Common Metric Questionnaire (CMQ) 33
Fleishman Job Analysis Survey (F-JAS) 34
Occupational Analysis Inventory (OAI) 36
Occupational Information Network (O*NET) 38
Position Analysis Questionnaire (PAQ) 39
A Critique of Existing Job Analysis Systems 40 Chapter 3 43
A Methodology to Identify Optimal Job Demand Variables 43
Survey Development 43
Survey Participants 44
Survey Administration and Analysis 45
Automated Concept Mapping 45
Step 1: Data Collection from a Large Group. 46
Step 2: Identification of Core Concepts from the Large Group Data. 47
Step 3: Sorting and Prioritizing the Job Demand Variables. 47
Step 4: Identification of Dimensions of Job Demands. 47
Step 5: Clustering of Job Demand Variables. 48 Chapter 4 50
Procedures for Validating Job Demand Variables 50
Need for Verification/Initial Validation 51
Replication/Cross-Validation: 51
SME Evaluation: 52
Convergent/Content Validation: 52
Procedure A. Replication/Cross-Validation 53
Procedure B: Evaluation by Subject Matter Experts 53
Procedure C: Convergent/Content Validation (Comparison with Existing
Lists) 54
Summary and Recommendations 55
Table 4-1. Comparative Analysis of Recommended Validation Procedures
55 Chapter 5 57
Procedures for Developing a Job Demands Questionnaire 57 Questionnaire Development 58
Selecting Rating Scale Response Categories 58
Figure 5.1. Anchors for frequency and importance 59
Selecting the appropriate range of response categories 59
Additional considerations 60
Table 5.1. Rating Scales: Job Analysts versus Incumbents 60
Types of Ratings 61
Frequency 62
Figure 5.3. Frequency Ratings: Physical strength (behaviorally-
specific) 62
Figure 5.4. Frequency Ratings: Psychosocial job demands (global) 62
Figure 5.5. Frequency Ratings: Oral Communication (behaviorally-
specific) 63
Importance 63
Figure 5.7. Importance Ratings: Physical strength (behaviorally-
specific) 64
Figure 5.8. Importance Ratings: Psychosocial demands (global) 64
Figure 5.9. Importance Ratings: Information exchange (behaviorally-
specific) 65
Proficiency 65
Figure 5.10. Proficiency Ratings: Physical Job Demands (global) 65 Ability to lift and move less than 10 pounds 66
Figure 5.11. Proficiency Ratings: Psychosocial Job Demands (global) 66
Choosing the appropriate response format 66
Missing Data 66
Observable Job Demand Items 67
Additional issues for items development 68
Cognitive Interviewing 68 Pilot Test of the Questionnaire 69
Number of Jobs for the Field Test 69
Number of Participants per Job 70
Training of Questionnaire Participants 70 Chapter 6 70
Construct Validation: Dimensionality Assessment and Calibration of Job
Demand Variables 70
Selecting Optimal Job Demands for a Minimum Data Set 71
Analysis of Ratings of Job Demands 72
Traditional Item Analysis 72
Factor Analysis 73
Item response theory - The Role of One Parameter and Two Parameter
Models 74
2-Parameter Models 75
Unidimensionality and Fit 77
Application to the Validation of Job Demands 77 Chapter 7 80
Summary 80
Conclusions of the Job Demands Project 80 References 84 Appendix A: 93
Research Approaches to Validation of SSA'S Job Demands Job Demands
Validation Criteria 93
Introduction 93
Current Approaches to the Evaluation of Job Demands 93
a. Fleishman's Job Analysis Survey 94
b. O*NET: Occupational Information Network 94
c. Functional Job Analysis 95
d. Position Analysis Questionnaire 96
e. Common-Metric Questionnaire - CMQ 97
Reliability 97
Internal Consistency 98
Measurement Validity 99
a. Content-Related/Translational Approaches to Measurement Validity
99
Face validity 100
Content validity 101
b. Construct-Related Validity 103
Hierarchical Order and Fit 103
Convergent and Discriminant (Divergent) Validity 106
Factor Analysis 107
c. Criterion-Related Validity 109
Concurrent validity 109
Predictive validity 109
Issues in Applying Validation Criteria 110
References 112 Table 1: Summary of Selected Job Analysis Systems 117 Definitions 119 Appendix B: 121
Report on Job Analysis Databases, Systems and Software 121
Introduction 121 Background 122 Programs or Systems Offered by Public and Private Companies 123
Job-Matching Programs 123
General Methodology Used By Job-Matching Programs 124
Identification of Job-Matching Programs 126
Utilization of Job-Matching Programs for Determining Disability
134
Job Analysis Programs 136
International Systems 137 Summary and Conclusions 140
Table 1. 141
Summary of Job-Matching Systems and Programs 141 Bibliography 143
Systems & Software 143
Job Analysis & Classification Methodologies 173
Job Matching Methodology 185
Vocational Evaluation 193
Changing Nature of the Work Environment 204 Appendix C: 213
Annotated Bibliography 213
Introduction 213 1. The Changing Nature of the Work Environment 214 Defining Human Performance 216
Vocational Evaluation 216
Evaluation of Human Performance 221 Describing Job Demands 222
Job Analysis 223
Job Classification Systems 231 Methods of Identifying and Validating Job Demands 232 Identifying Job Demands 232 Validating Job Demands 234 Current Applications 237
Impact of the Social/Political Context on the Assessment of Disability
237 Appendix D: 242
Content Review of Job Demands Contained Within Current Job Analysis
Systems 242 Table1. Functional Job Analysis 242 Table 2. Common Metric Questionnaire (CMQ) 243 Table 3. Fleishman - Job Analysis Survey 244 Table 4. Occupational Analysis Inventory 246 Table 5. Occupational Information Network (O*NET) 248 Table 6. Position Analysis Questionnaire 249 Appendix E 251
Table 1. Examples of Common Job Demands Classified by Domain 251 Table 2. Example of Critical Physical Factors 252 References 253
Executive Summary
This monograph serves as a final report and ninth deliverable to the
Social Security Administration (the Agency or SSA) on one of the major
Disability Research Institute (DRI) projects. The DRI was asked to develop
a methodology to develop the optimal job demand variables that the Agency
needs to compare the residual functional capacity (RFC) of claimants for
disability benefits with the occupational demands of work. The Disability
Research Institute was asked to recommend one or more approa