Downloads - Council of Europe

[pic]
January 2009
Relating Language Examinations to the Common European Framework of
Reference for Languages: Learning, Teaching, Assessment (CEFR)
A Manual
Language Policy Division, Strasbourg
www.coe.int/lang Contents
List of Figures Page iii List of Tables Page v List of Forms Page vii Preface Page ix Chapter 1: The CEFR and the Manual
Page 1 Chapter 2: The Linking Process
Page 7 Chapter 3: Familiarisation Page
17 Chapter 4: Specification Page
25 Chapter 5: Standardisation Training and Benchmarking
Page 35 Chapter 6: Standard Setting Procedures
Page 57 Chapter 7: Validation Page 89 References Page 119 Appendix A Forms and Scales for Description and Specification (Ch. 1 & 4)
Page 122
A1: Salient Characteristics of CEFR Levels (Ch. 1)
Page 123
A2: Forms for Describing the Examination (Ch. 4)
Page 126
A3: Specification: Communicative Language Activities (Ch. 4)
Page 132
A4: Specification: Communicative Language Competence (Ch. 4)
Page 142
A5: Specification: Outcome of the Analysis (Ch. 4)
Page 152 Appendix B Content Analysis Grids (Ch.4)
B1: CEFR Content Analysis Grid for Listening & Reading
Page 153
B2: CEFR Content Analysis Grids for Writing and Speaking Tasks
Page 159 Appendix C Forms and Scales for Standardisation & Benchmarking (Ch. 5)
Page 181 Reference Supplement: Section A: Summary of the Linking Process
Section B: Standard Setting
Section C: Classical Test Theory
Section D: Qualitative Analysis Methods
Section E: Generalisability Theory
Section F: Factor Analysis
Section G: Item Response Theory
Section H: Test Equating List of Figures
Figure 2.1: Validity Evidence of Linkage of Examination/Test
Results to the CEFR Page 8 Figure 2.2: Visual Representation of Procedures to Relate
Examinations to the CEFR Page 15 Figure 6.1: Frequency Distributions of Test Scores in Two Contrasting
Groups Page 67 Figure 6.2: Logistic Regression
Page 73 Figure 6.3: Panel Member Recording Form for Bookmark Method
Page 78 Figure 6.4: Items with Unequal Discrimination
Page 82 Figure 6.5: Item Map, Indicating Difficulty and Discrimination
Page 83 Figure 7.1: Empirical Item Characteristic Curve for a Problematic Item
Page 101 Figure 7.2: A Test Characteristic Curve
Page 105 Figure 7.3: Bivariate Decision Table Using Nine Levels
Page 112 Figure 7.4: Bivariate Decision Table Using Five Levels
Page 113 Figure 7.5: Item Map with Test Items and "Can Do" Statements
Page 116
List of Tables
Table 3.1: Time Management for Familiarisation Activities
Page 23 Table 3.2: Documents to be Prepared for Familiarisation Activities
Page 23 Table 4.1: Forms and Scales for Communicative Language Activities
Page 32 Table 4.2: CEFR Scales for Aspects of Communicative Language
Competence Page 32 Table 5.1: Time Management for Assessing Oral Performance Samples
Page 45 Table 5.2: Time Management for Assessing Written Performance Samples
Page 46 Table 5.3: Documents and Tools to be prepared for Rating Writing
Page 47 Table 5.4: Reference Sources in the CEFR
Page 49 Table 5.5: Standardisation and Benchmarking: Summary
Page 55 Table 6.1: Overview of the Methods Discussed
Page 61 Table 6.2: Basic Data in the Tucker-Angoff method
Page 62 Table 6.3: Computing the Expected Score of 100 Borderline Persons
Page 66 Table 6.4: Frequency Distribution Corresponding to Figure 6.1
Page 68 Table 6.5: Decision Tables for Five Cut-off Scores
Page 68 Table 6.6: Summary of the Rangefinding Round
Page 71 Table 6.7: Results of the Pinpointing Round (partially)
Page 73 Table 6.8: Example of an ID Matching Response Form (abridged)
Page 75 Table 6.9: Bookmarks and Achievement Levels Page
80 Table 6.10: Estimated Theta Page
81 Table 7.1: Balanced Incomplete Block Design with Three Blocks
Page 93 Table 7.2: Balanced Incomplete Block Design with Seven Blocks
Page 93 Table 7.3: Example of High Consistency and Total Disagreement
Page 98 Table 7.4: Bivariate Frequency Table using Four Levels
Page 99 Table 7.5: Frequencies of Allocation of a Single Item to Different
CEFR Levels Page 101 Table 7.6: Summary of Disagreement per Item Page
102 Table 7.7: Outcome of a Tucker-Angoff Procedure
Page 102 Table 7.8: Variance Decomposition Page 103 Table 7.9: Decision Accuracy Page 107 Table 7.10: Decision Consistency Page 108 Table 7.11: Marginal Distributions Across Levels (Frequencies)
Page 110 Table 7.12: Marginal Distributions Across Levels (Percentages)
Page 111 Table 7.13: Design for a Paired Standard Setting
Page 115 Table A1: Salient Characteristics: Interaction & Production
Page 123 Table A2: Salient Characteristics: Reception
Page 124 Table A3: Relevant Qualitative Factors for Reception
Page 143 Table A4: Relevant Qualitative Factors for Spoken Interaction
Page 148 Table A5: Relevant Qualitative Factors for Production
Page 149 Table C1: Global Oral Assessment Scale
Page 184 Table C2: Oral Assessment Criteria Grid
Page 185 Table C3: Supplementary Criteria Grid: "Plus levels"
Page 186 Table C4: Written Assessment Criteria Grid Page
187
List of Forms
Form A1: General Examination Description
Page 126
Form A2: Test Development
Page 127
Form A3: Marking Page
129
Form A4: Grading Page
130
Form A5: Reporting Results
Page 130
Form A6: Data Analysis
Page 131
Form A7: Rationale for Decisions
Page 131
Form A8: Initial Estimation of Overall Examination Level
Page 28 / 132
Form A9: Listening Comprehension
Page 132
Form A10: Reading Comprehension Page
133
Form A11: Spoken Interaction
Page 134
Form A12: Written Interaction
Page 136
Form A13: Spoken Production
Page 137
Form A14: Written Production
Page 138
Form A15: Integrated Skills Combinations
Page 139
Form A16: Integrated Skills
Page 139
Form A17: Spoken Mediation
Page 140
Form A18: Written Mediation
Page 141
Form A19: Aspects of Language Competence in Reception
Page 142
Form A20: Aspects of Language Competence in Interaction
Page 145
Form A21: Aspects of Language Competence in Production
Page 146
Form A22: Aspects of Language Competence in Mediation
Page 150
Form A23: Graphic Profile of the Relationship of the Examination to
CEFR Levels Page 33 / 152
Form A24: Confirmed Estimation of Overall Examination Level
Page 34 / 152
Form C1: Training Record Form
Page 181
Form C2: Analytic Rating Form (Swiss Project)
Page 182
Form C3: Holistic Rating Form (DIALANG)
Page 182
Form C4: Collation Global Rating Form (DIALANG)
Page 183
Form C5: Item Rating Form (DIALANG)
Page 183
These forms are also available on the website www.coe.int/lang
Preface The Council of Europe wishes to acknowledge with gratitude all those who
have made it possible to develop this Manual, and in particular the
contributions by: . The Finnish authorities who provided the forum in Helsinki to launch the
initiative in July 2002. . The "Sounding Board" of consultants for the pilot edition (Prof. Charles
Alderson, Dr Gergely A. David, Dr John de Jong, Dr Felianka Kaftandjieva,
Dr Michael Makosch, Dr Michael Milanovic, Professor Günther Nold,
Professor Mats Oscarson, Prof. Günther Schneider, Dr Claude Springer and
also Mr Josef Biro, Ms Erna van Hest, Mr Peter Lenz, Ms Jana Pernicová,
Dr Vladimir Kondrat Shleg, Ms Christine Tagliante and Dr John Trim) for
their important feedback in the early stage of the project. . The Authoring Group, under the leadership of Dr Brian North: |Dr Neus Figueras |Departament d'Educació, Generalitat de |
| |Catalunya, Spain |
|Dr Brian North |Eurocentres Foundation, Switzerland |
|Professor Sauli Takala|University of Jyväskylä, Finland |
| |(emeritus) |
|Dr Piet van Avermaet |Centre for Diversity and Learning, |
| |University of Ghent, Belgium |
| |Association of Language Testers in |
| |Europe (ALTE) |
|Dr Norman Verhelst |Cito, The Netherlands |
. Dr Jay Banerjee (University of Lancaster) and Dr Felianka Kaftandjieva
(University of Sofia) for their contributions to the Reference Supplement
to the Manual. . The institutions