A Study on the Washback Effects of the Test for English Majors (TEM) PDF

This book adopts a multi-method and multi-phase approach to investigate the washback effects of Test for English Majors (TEM) on program administrators, teachers and students, shedding new light on TEM reform and the reform of English teaching and learning in China. TEM, a nationwide test used to measure the language proficiency of undergraduate English majors in China, is a major standardized test taken by nearly 400,000 students every year. The book’s key features include: an in-depth discussion of the nature of washback and a framework for investigating it; a multi-method and multi-phase approach, employing both the quantitative method of questionnaire surveys and the qualitative methods of interviews and classroom observations; large-scale questionnaire surveys conducted among experts, program administrators, teachers and students, and involving over 30,000 participants; detailed assessments of TEM’s washback effects on stakeholders’ perceptions, classroom teaching practice, students learning activities, etc.; and essential insights into testing and teaching reforms.

118 downloads 6K Views 5MB Size

Report

Download pdf

Recommend Stories

Empty story

Idea Transcript

Qian Xu · Jun Liu

A Study on the Washback Effects of the Test for English Majors (TEM) Implications for Testing and Teaching Reforms

A Study on the Washback Effects of the Test for English Majors (TEM)

Qian Xu • Jun Liu

A Study on the Washback Effects of the Test for English Majors (TEM) Implications for Testing and Teaching Reforms

Qian Xu Shanghai International Studies University Shanghai, China

Jun Liu Shanghai University of Electric Power Shanghai, China

ISBN 978-981-13-1962-4 ISBN 978-981-13-1963-1 (eBook) https://doi.org/10.1007/978-981-13-1963-1 Library of Congress Control Number: 2018951416 © Springer Nature Singapore Pte Ltd. 2018 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

Acknowledgements

This book is funded by the Fundamental Research Funds for the Central Universities in China (Research Grant No. KY01X0222017082).

v

Contents

1 TEM: A High-Stakes Standardized Test in the Chinese Context�� 1 1.1 The Educational System in China�� 1 1.2 The History of TEM�� 3 1.2.1 The Launch of TEM�� 3 1.2.2 The Development of TEM�� 4 1.2.3 The Soaring Test Population �� 6 1.2.4 Research on TEM �� 6 1.3 TEM Test Characteristics�� 9 1.3.1 TEM Test Purpose�� 9 1.3.2 TEM Test Content and Format�� 9 1.3.3 TEM Administration and Marking�� 11 1.3.4 TEM Test Score Report and Interpretation �� 12 1.4 TEM Test Characteristics vs. the Teaching Syllabus�� 13 1.4.1 The Teaching Syllabus�� 13 1.4.2 A Comparison Between the Teaching Syllabus and TEM Characteristics�� 14 1.5 Summary �� 16 2 Washback: Definitions and Dimentions�� 17 2.1 Definitions of Washback �� 17 2.1.1 Washback or Backwash�� 17 2.1.2 Various Terminology�� 18 2.1.3 Washback and Validity�� 18 2.2 Dimensions of Washback�� 19 2.2.1 Direction �� 19 2.2.2 Extent�� 20 2.2.3 Intensity�� 20 2.2.4 Intentionality �� 21 2.2.5 Length �� 21 2.2.6 Specificity �� 21 2.3 Summary �� 22 vii

viii

Contents

3 Theoretical Frameworks for Washback Studies �� 23 3.1 Frameworks for Washback Studies�� 23 3.1.1 Alderson and Wall’s Washback Hypotheses �� 23 3.1.2 Hughes’ Washback Trichotomy�� 24 3.1.3 Bailey’s Basic Model of Washback�� 25 3.1.4 Green’s Washback Model �� 25 3.2 Empirical Studies on Washback�� 26 3.2.1 Summary of Empirical Studies �� 27 3.2.2 A Critique of Approaches in Washback Studies�� 31 3.3 Summary �� 33 4 Research Framework for TEM Washback Study�� 35 4.1 Framework for TEM Washback Study�� 35 4.1.1 Contributing Factors to TEM Washback�� 35 4.1.2 TEM Washback Mechanism �� 36 4.1.3 Several Terms Redefined�� 37 4.1.4 Research Design�� 39 4.2 Participants�� 40 4.3 Instruments�� 41 4.3.1 Research Design for the Questionnaire Surveys�� 41 4.3.2 Classroom Observations �� 94 4.3.3 Interviews�� 98 4.4 Considerations for Validity and Reliability�� 98 4.4.1 The Development of Research Instruments�� 99 4.4.2 Data Collection and Analysis�� 101 4.4.3 Triangulation�� 104 4.5 Summary �� 105 5 Stakeholders’ Perceptions of TEM and Its Washback �� 107 5.1 Experts’ Perceptions �� 107 5.1.1 Demographic Information�� 108 5.1.2 Experts’ Evaluation of TEM �� 108 5.1.3 Experts’ Perception of TEM Washback�� 109 5.1.4 Factors Influencing Teaching�� 110 5.2 Program Administrators’ Perceptions �� 110 5.2.1 Demographic Information�� 110 5.2.2 Program Administrators’ Evaluation of TEM�� 110 5.2.3 Program Administrators’ Perception of TEM Washback�� 116 5.2.4 Washback on Course Design�� 118 5.2.5 Individual Differences�� 120 5.2.6 Factors Influencing Teaching�� 123

Contents

ix

5.3 Teachers’ Perceptions �� 123 5.3.1 Demographic Information�� 123 5.3.2 Teachers’ Evaluation of TEM �� 124 5.3.3 Teachers’ Perception of TEM Washback�� 128 5.3.4 Washback on Classroom Teaching�� 129 5.3.5 Individual Differences�� 132 5.3.6 Factors Influencing Teaching�� 133 5.4 Students’ Perceptions�� 134 5.4.1 Demographic Information�� 134 5.4.2 Students’ Evaluation of TEM �� 135 5.4.3 Students’ Perception of TEM Washback�� 139 5.4.4 TEM Preparation�� 140 5.4.5 Learning Effects�� 145 5.4.6 Individual Differences�� 146 5.4.7 Factors Influencing Learning�� 147 5.5 Triangulation�� 148 5.5.1 Data Triangulation�� 148 5.5.2 Methodological Triangulation�� 152 5.6 Summary �� 154 6 Classroom Practice and the Washback of TEM �� 157 6.1 The Teachers Observed �� 157 6.2 The English Classes Observed�� 158 6.3 Characteristics of English Classroom Practice �� 159 6.3.1 Teaching Content�� 160 6.3.2 Teaching Methods�� 162 6.4 TEM Preparation Classes vs. Non-preparation Classes�� 165 6.4.1 Teaching Content�� 166 6.4.2 Teaching Methods�� 166 6.5 Classroom Practice Affected by Other Factors �� 170 6.5.1 English Used in Class �� 170 6.5.2 Features of Teacher-Student Interaction �� 171 6.5.3 Classroom Atmosphere �� 174 6.6 Triangulation�� 175 6.6.1 Classroom Observation Findings vs. Questionnaire Findings�� 175 6.6.2 Classroom Observation Findings vs. Interview Findings�� 176 6.6.3 Classroom Observation Findings vs. Document Analysis�� 177 6.7 Summary �� 177

x

Contents

7 Conclusions and Implications �� 179 7.1 Summary of the Major Findings �� 179 7.1.1 Washback Effects Perceived by Experts �� 179 7.1.2 Washback Effects Perceived by Program Administrators�� 179 7.1.3 Washback Effects Perceived by Teachers �� 180 7.1.4 Washback Effects Perceived by Students�� 180 7.1.5 Washback Effects on English Classrooms�� 181 7.2 Implications for the Reform�� 181 7.2.1 Implications for TEM Reform�� 181 7.2.2 Implications for the Reform of Teaching and Learning �� 185 Appendices�� 187 Appendix I Survey Questionnaires�� 187 Questionnaire for Experts (EQ) �� 187 Appendix II Classroom Observations�� 215 Bibliography �� 245

List of Acronyms

A Advanced stage AQ-4 Questionnaire for TEM4 related administrators AQ-8 Questionnaire for TEM8 related administrators C-E Chinese to English CET College English Test E-C English to Chinese EQ Questionnaire for experts ESOL English for speakers of other languages F Foundation stage Gra Grammar HKCEE Hong Kong Certificate of Education Examination in English HKEA Hong Kong Examinations Authority IELTS International English Language Testing System KMO Kaiser-Meyer-Olkin MCQ Multiple choice question MET Matriculation English Test NACFLT National Advisory Committee for Foreign Language Teaching NMET National Matriculation English Test PhD Doctor of philosophy PL2000 Progetto Lingue 2000 SPSS 17.0 Statistical Product and Service Solutions 17.0 SQ-4 Questionnaire for TEM4 related students SQ-8 Questionnaire for TEM8 related students TEM Test for English Majors TEM4 Test for English Majors Grade 4 TEM4-Oral Test for English Majors Grade 4 Oral Test TEM8 Test for English Majors Grade 8 TEM8-Oral Test for English Majors Grade 8 Oral Test The Teaching Syllabus The National Teaching Syllabus for English Majors xi

xii

TOEFL Test of English as a Foreign Language TQ-4 Questionnaire for TEM4 related teachers TQ-8 Questionnaire for TEM8 related teachers Voc Vocabulary Wpm Words per minute

List of Acronyms

List of Figures

Fig. 1.1 Educational hierarchy of English teaching for English majors........... 2 Fig. 4.1 TEM washback model......................................................................... 39

xiii

List of Tables

Table 1.1 A comparison between the previous TEM and new TEM............. 5 Table 1.2 TEM studies published over the years........................................... 7 Table 1.3 Research focus of the journal articles and PhD dissertations........ 8 Table 1.4 Test content and format of TEM.................................................... 10 Table 1.5 A comparison between the teaching syllabus and TEM characteristics................................................................. 15 Table 3.1

Overview of the empirical studies on washback............................ 28

Table 4.1 A multi-phase, multi-method research design for TEM washback study............................................................... 40 Table 4.2 The seven questionnaires............................................................... 43 Table 4.3 The number of questionnaires distributed, returned and valid........................................................................... 45 Table 4.4 Reliability test for AQ-4, TQ-4, SQ-4, AQ-8, TQ-8 and SQ-8........................................................................................ 47 Table 4.5 KMO and Bartlett’s tests................................................................ 48 Table 4.6 Rotated component matrix for the questionnaires......................... 49 Table 4.7 Inter-coder reliability (first round)................................................. 103 Table 4.8 Inter-coder reliability (second round)............................................ 103 Table 4.9 Intra-coder reliability..................................................................... 104 Table 5.1 Table 5.2 Table 5.3 Table 5.4 Table 5.5 Table 5.6 Table 5.7 Table 5.8 Table 5.9 Table 5.10

Demographic information of program administrators................... 111 Familiarity with TEM.................................................................... 112 Evaluation of TEM......................................................................... 113 Information to be provided in the test report................................. 114 Use and interpretation of TEM scores........................................... 115 Influence of TEM scores................................................................ 115 Unintended use of TEM scores...................................................... 116 Attitudes towards unintended uses of TEM scores........................ 116 Views on TEM washback............................................................... 117 Washback exerted by different aspects of TEM............................. 117 xv

xvi

Table 5.11 Table 5.12 Table 5.13 Table 5.14 Table 5.15 Table 5.16 Table 5.17 Table 5.18 Table 5.19 Table 5.20 Table 5.21 Table 5.22 Table 5.23 Table 5.24 Table 5.25 Table 5.26 Table 5.27 Table 5.28 Table 5.29 Table 5.30 Table 5.31 Table 5.32 Table 5.33 Table 5.34 Table 5.35 Table 5.36 Table 5.37 Table 5.38 Table 5.39 Table 5.40 Table 5.41 Table 5.42 Table 5.43 Table 5.44 Table 5.45 Table 5.46 Table 5.47 Table 5.48 Table 5.49 Table 5.50 Table 5.51

List of Tables

Courses introduced by TEM participating universities.................. 119 Washback from TEM revisions...................................................... 120 TEM preparation courses............................................................... 121 TEM washback perceived by different program administrators................................................................................. 121 Factors affecting English language teaching.................................. 122 Demographic information of teachers............................................ 124 Familiarity with TEM.................................................................... 125 Evaluation of TEM......................................................................... 126 Information to be provided in the test report................................. 126 Use and interpretation of TEM scores........................................... 127 Influence of TEM scores................................................................ 127 Unintended uses of TEM scores.................................................... 127 Attitudes towards unintended uses of TEM scores........................ 128 Views on TEM washback............................................................... 128 Washback exerted by different aspects of TEM............................. 129 TEM preparation experience.......................................................... 130 TEM preparation classes................................................................ 131 Difference between TEM preparation and non-preparation classes........................................................... 131 Washback from TEM revisions...................................................... 132 TEM washback on teaching perceived by different teachers......... 133 Factors affecting English teaching................................................. 134 Sources of pressure for teachers..................................................... 134 Demographic information of students............................................ 135 Students’ familiarity with TEM..................................................... 136 Students’ evaluation of TEM......................................................... 137 Information to be provided in TEM test report.............................. 138 Use and interpretation of TEM results........................................... 139 Views on TEM washback............................................................... 139 Positive and negative washback of TEM....................................... 140 TEM preparation courses............................................................... 141 Activities conducted in TEM preparation classes.......................... 141 Difference between TEM preparation and non-preparation classes........................................................... 142 Students’ preference for classroom activities................................ 143 After-class preparation efforts........................................................ 143 Learning activities conducted for TEM preparation after class........................................................................................ 144 Difference between TEM preparation and non-preparation stages.............................................................................................. 145 Washback of TEM on learning effects........................................... 146 Progress in fundamental/advanced stage....................................... 146 TEM washback on students with different characteristics............. 147 Factors affecting English learning.................................................. 148 Major aims for learning.................................................................. 148

List of Tables

Table 6.1 Table 6.2 Table 6.3 Table 6.4 Table 6.5 Table 6.6 Table 6.7 Table 6.8 Table 6.9 Table 6.10 Table 6.11 Table 6.12 Table 6.13 Table 6.14 Table 6.15 Table 6.16 Table 6.17 Table 6.18 Table 6.19 Table 6.20

xvii

Background information of the teachers observed......................... 158 No. of classes taught by each teacher............................................ 159 No. of classes observed from different schools.............................. 159 No. of TEM preparation and non-preparation classes observed............................................................................. 159 Teaching content of all the classes observed.................................. 160 Teaching materials used in class.................................................... 162 Classroom activities organized in all the classes observed............ 163 English used in classrooms observed............................................. 164 Features of teacher-student interaction.......................................... 164 Frequency of students’ laughter..................................................... 165 No. of skills (knowledge) covered in each class............................ 166 Mean time devoted to each activity (percentage of time).............. 167 English used by teachers in TEM preparation and non-preparation classes........................................................... 168 English used by students in TEM preparation and non-preparation classes........................................................... 169 English used by teachers in classes of different grades................. 170 English used by different teachers in class ................................... 171 English used by teachers in classes from different schools........... 171 Communicative features of classes delivered by different teachers....................................................................... 172 Communicative features of classes from different schools............ 173 Laughter in classes of different grades........................................... 174

Chapter 1

TEM: A High-Stakes Standardized Test in the Chinese Context

1.1 The Educational System in China In China, quite a few universities offer a four-year program for undergraduate English majors, who will be conferred a bachelor’s degree in English Language and Literature if they have met all the requirements set by their universities, such as credits, degree thesis and TEM certificates. The following figure demonstrates the educational hierarchy of college English teaching for English majors. As illustrated in Fig. 1.1, teaching for English majors is under the influence of many factors, which can be mainly divided into three levels. The top level is the national level, which is composed of policy-makers, the Teaching Syllabus designers and the TEM test developers. Policy makers are the officials in the Ministry of Education who supervise and evaluate the overall quality of college education, particularly those in the National Advisory Committee for Foreign Language Teaching (NACFLT), who are responsible for establishing policies in English education. The policies they have formulated will be implemented by the Teaching Syllabus designers and the TEM test developers, who, although from two separate sections, are in close relation, and will jointly decide how the policies should be managed and how the teaching and testing syllabuses should incorporate the basic principles of the policy. The second level is the school level—the administrators. Based on the Teaching Syllabus, the administrators will set the teaching aims and design the curriculum accordingly, such as the courses to be introduced, the textbooks to be used, and so on. However, different curricula will be developed in each school due to the following reasons. First is regional difference. China is a vast country, and the east coastal regions far surpass west inland areas in educational level, so the language requirements set for English majors differ from region to region. Second is the school level. In China, universities are divided into key universities, ordinary universities and colleges, and the admission standard differs a lot among them, with key universities recruiting the top students. Therefore, English teaching aims and curricula differ © Springer Nature Singapore Pte Ltd. 2018 Q. Xu, J. Liu, A Study on the Washback Effects of the Test for English Majors (TEM), https://doi.org/10.1007/978-981-13-1963-1_1

1

2

1 TEM: A High-Stakes Standardized Test in the Chinese Context

Ministry of Education

Department of Higher Education

National Advisory Committee for Foreign Language Teaching

Teaching Syllabus Designers

TEM Testing Committee TEM

Teaching Syllabus School Administrators Teachers Students

Fig. 1.1 Educational hierarchy of English teaching for English majors

markedly among schools of different levels. Third, schools of different type may vary in their English curriculum. For instance, foreign language universities may focus more on language and literature while technology institute may introduce more science-related English courses. Last, the bottom level is the classroom level, including teachers and students who are responsible for the implementation of the school curriculum and the Teaching Syllabus. Based on the school curriculum, teachers may design their respective course syllabus, and their previous knowledge and available resources will affect their teaching practice. Moreover, university students are from different regions and background, who may perform differently in the same classroom. All the people involved in this hierarchy are TEM stakeholders, and the higher the level, the fewer the stakeholders, but the greater the power they have. Besides, stakeholders from each level may overlap: some members in the NACFLT have participated in the development of the Teaching Syllabus, some experts in the TEM testing committee are also school administrators, and some school administrators are English teachers at the same time. Therefore, in the current study, although the research focus is the washback of TEM on English teaching and learning, opinions of stakeholders from each level will be sought, with teachers and students as the major participants. In this view, a top down approach will be adopted when investigating the washback effects of TEM, which will be carried out in three phases. First, opinions of

1.2 The History of TEM

3

stakeholders from the national level will be investigated, through the analysis of related documents and a semi-open questionnaire survey among the experts, who are either members of the NACFLT or syllabus designers, or both. Then, a questionnaire survey will be conducted among school administrators, to find out different practices in all the TEM participating schools. Last, teachers and students will be investigated, through large-scale questionnaires and case study classroom observations. As a supplementary instrument, interviews will be conducted in all the three phases, to obtain follow-up data.

1.2 The History of TEM The history of TEM can be traced back to the early 1990s, and in its two-decade development, TEM has undergone several revisions. The following part will introduce the launch and development of TEM, its soaring test population, as well as the related research on TEM.

1.2.1 The Launch of TEM Before reviewing the history of TEM, we will first introduce the formulation and publication of the Teaching Syllabus. After the Cultural Revolution, foreign language education in China has entered a new era. The early 1980s witnessed a rapid increase in the demand for English majors in every field, because of our government adoption of the policy of reform and opening up to the outside world. As a result, many universities set up English departments and offered English programs. However, due to various reasons, huge gap existed among different universities in curriculum design, teaching material selection and course evaluation, which led to an urgent demand for a national teaching syllabus. Hence, the first two national teaching syllabuses for college English majors, foundation stage and advanced stage, were published in 1989 and 1990 respectively. The two teaching syllabuses serve as guidelines for the development of curriculum and teaching materials for English majors in various universities throughout China. They have established specific teaching requirements and English proficiency standards for undergraduate English majors, and thus, it is possible and necessary to develop a nation-wide test to check the implementation of the Teaching Syllabus. In line with this demand, two groups of specialists started working on TEM, based in two different universities. Shanghai International Studies University was responsible for TEM4, a test for English sophomores to measure their proficiency in the foundation stage; while Guangdong University of Foreign Studies was responsible for TEM8, a test for English majors in their fourth year to measure their proficiency in the advanced stage. The first TEM4 and TEM8 were launched in 1990 and 1991 respectively. Commissioned by NACFLT, Shanghai International Studies

4

1 TEM: A High-Stakes Standardized Test in the Chinese Context

University has assumed responsibility for both tests since October, 1992. To standardize the test, the first TEM Syllabuses were drafted in 1993 and published in 1994, which marked the official beginning of TEM. Following the introduction of TEM4 and TEM8 written tests, Nanjing University was given responsibility to conduct studies on the feasibility of TEM4-Oral and TEM8-Oral in 1994 and 1998 respectively. After several years’ of trialing, TEM4- Oral and TEM8-Oral were officially launched in 1999 and 2003, and the Oral Test Syllabuses were published in 2005.

1.2.2 The Development of TEM It has been more than 20 years since TEM was first implemented, and in the course of its development, TEM has undergone several revisions. After its publication in 1994, the TEM Syllabuses (TEM4 and TEM8) has undergone two major revisions, one in 1997 and the other in 2004. A comprehensive validation study of TEM was conducted from 1993 to 1996, and based on its findings, a revised edition of TEM Syllabuses were published in 1997. The second major revision took place in 2004 as a response to the revision of the Teaching Syllabus in 2000. The revisions of TEM Syllabuses aim at improving test quality, standardizing test development and administration, and achieving fairness of test result interpretation. (Jin and Fan 2011; Zou 2010). In 2005, the new versions of TEM4 and TEM8 were introduced, employing more integrative tasks and using lengthier and more authentic materials for listening and reading. The test content and format of the previous TEM and new TEM are listed in Table 1.1. From the above table, three major changes can be identified. First and foremost, the new versions of TEM set a higher requirement on the test takers. The listening and reading speed has been increased, with faster listening materials and longer reading passages; short listening materials like statements have been replaced by longer ones like passages; a lengthier composition is required for the writing part; and the passages for cloze and proofreading also lengthened. Second, speed reading has been dropped from both TEM4 and TEM8. Last, a new section of general knowledge has been included in TEM8. As for the oral tests, a major reform was adopted in 2008, supplementing the tape-mediated format with computer-based spoken English tests (Jin and Fan 2011). Further discussions on these revisions will be provided in Sect. 1.4.

1.2 The History of TEM

5

Table 1.1 A comparison between the previous TEM and new TEM Task TEM4 (previous version) Writing 1) composition 2) note-writing Dictation Listening Cloze Grammar & vocabulary Reading 1) careful 2) speed TEM4 (new version) Writing 1) composition 2) note-writing Dictation Listening Cloze Grammar & vocabulary Reading TEM8 (previous version) Listening

Proofreading and error correction Reading 1) careful 2) speed Translation 1) C-E 2) E-C

Input

Format

A topic, chart or graph

A text of about 150 15 35 words A note of 50–60 words 5 10 Word-for-word dictation 15 15

Written prompts A text of about 150 words, listen 4 times, 120 wpm Statements, conversations and news broadcasts, 120 wpm A text of about 250 words 25 sentences 4–5 texts of about 1500 words altogether 4–5 texts of about 900 words altogether A topic, chart or graph

% Time

25 MCQs

15 20

15 MCQs 25 MCQs

10 15 15 15

15 MCQs

15 25

10 MCQs

10 5

A text of about 200 15 35 words A note of 50–60 words 10 10 Word-for-word dictation 15 15

Written prompts A text of about 150 words, listen 4 times, 120 wpm Conversations, passages and news 30 MCQs broadcasts, 120 wpm A text of about 250 words 20 MCQs 30 sentences 25 MCQs

15 20 10 15 15 15

4–5 texts of about 1800 words altogether

15 MCQs

20 25

Talks, conversations and news broadcasts, 140–170 wpm Mini-lecture, 700 words, 140 wpm A text of about 200 words

15 MCQs

15 15– 20 10 20

4–5 texts of about 2500 words 6–7 texts of about 3000 words A text of about 150 Chinese characters, 250 wpm A text of about 150 words, 250 wpm

Note-taking and gap-filling (10 blanks) Error identification and correction 15 MCQs 10 MCQs Translation

15 30 10 10 10 30

Translation

10 30

10 15

(continued)

6

1 TEM: A High-Stakes Standardized Test in the Chinese Context

Table 1.1 (continued) Task Writing TEM8 (new version) Listening

Reading General knowledge Proofreading Translation 1) C-E 2) E-C Writing

Input A topic and prompts

Format A text of 300 words

% Time 20 60

Mini-lecture, 900 words, 150 wpm Conversations and news broadcasts, 150 wpm 4–5 texts of about 3000 words 10 sentences A text of about 250 words

Note-taking and gap-filling (10 blanks) 10 MCQs

10 10

20 MCQs 10 MCQs Error identification and correction Translation

20 30 10 10 10 15 10 30

Translation

10 30

A text of 400 words

20 45

A text of about 150 Chinese characters, 250–300 wpm A text of about 150 words, 250–300 wpm A topic and prompts

10 25

Note: %: weighting, with the overall percentage being 100; Time is given in minutes; Wpm = words per minute; MCQ = multiple choice question; C-E: Chinese to English; E-C: English to Chinese

1.2.3 The Soaring Test Population After more than two decades of development, TEM has grown into one of the predominant English tests in China, winning extensive recognition from test takers, relevant institutions and society at large (Jin and Fan 2011). The number of test takers and participating universities has increased rapidly and significantly. As for the written tests of TEM4 and TEM8, from 1992 to 2015, the number of participants increased respectively from around 10,000 to 270,000 and from 5000 to 210,000. The number of participating schools also increased from around 100 to more than 800 for both TEM4 and TEM8. As for TEM oral tests, the number of test takers and participating schools has also increased significantly, but its test taking population is no more than one-tenth of its written counterpart, around 18,000 and 10,000 for TEM4 and TEM8 respectively in 2015. (data released by TEM test office). With its substantial improvement and increasing test population, TEM is playing a more and more important role in English teaching and learning at the tertiary level, exerting profound washback effects on the students and teachers, the educational system and even the society as a whole.

1.2.4 Research on TEM Ever since the implementation of TEM in the early 1990s, quite a few studies have been conducted to investigate its feasibility, validation, reform, implication for teaching, and so on. The research findings have contributed to the reform of TEM and

1.2 The History of TEM

7

Table 1.2 TEM studies published over the years CSSCI journal article PhD dissertation

1991–1995 11 0

1996–2000 7 0

2001–2005 33 4

2006–2010 35 9

2011–2015 13 5

English teaching and learning, which can also be considered an aspect of washback, according to Bailey (1996). Therefore, in this part, research findings on TEM will be introduced and analyzed, to summarize their impact on English teaching and learning. Due to the limited space and time, as well as the vast number of studies on TEM, only influential studies are selected here. The studies selected are: (1) articles published in CSSCI journals in linguistics, since CSSCI journals are well established in the academic field which are widely received and which may exert influence on future research; (2) PhD dissertations, because they are often based on in-depth research in the field, and should be of high quality and will give us original insight into this issue. A search in the CSSCI journals1 and CNKI2 Corpus of PhD and Master Dissertations, using the key word “TEM”, shows that up till the end of 2015, 99 articles and 18 PhD dissertations have been published. Table 1.2 briefly summarizes the number of research published over the years. The first journal article on TEM was published in 1991 on the feasibility of using dictation in TEM4 (Lian 1991), and the first PhD dissertation was completed in 2004 on the authenticity of listening comprehension questions in TEM (Dang 2004). From the above table, we can find an apparent increase in the number of studies conducted on TEM since 2001. In terms of the research focus, the 99 journal articles and 18 PhD dissertations cover almost every skill assessed in TEM (Table 1.3). From Table 1.3, it can be noted that more journal articles focus on TEM in general, such as the test report of TEM in a certain year, or comment on the revision of TEM Syllabus, while PhD dissertations are more focused, usually concentrating on one skill in TEM. Moreover, writing and translation seem to be studied more than the rest of the language skilled measured in TEM, which is probably because of the subjective nature of these two parts—test development, marking criteria and produced texts are all subjects for research. These researches on TEM may contribute to English teaching and learning in the following ways: 1. Test reports on TEM analyzed the test items and students’ performance in each item, which would provide useful feedback information on the strengths and weaknesses of students; 1 Eleven journals on linguistics (foreign language) included in CSSCI (2013–2014) are searched, which are Foreign Language Teaching and Research, Foreign Language World, Journal of Foreign Languages, Modern Foreign Languages, Chinese Translators Journal, Foreign Languages in China, Foreign Language Research, Foreign Language Education, Foreign Languages and Their Teaching, Computer-Assisted Foreign Language Education, and Foreign Language Learning Theory and Practice. 2 CNKI: China National Knowledge Infrastructure, an online resource sharing platform which has a collection of published articles.

CSSCI PhD

General 28 1

Listening 11 3

Reading 5 3

Writing 16 6

Table 1.3 Research focus of the journal articles and PhD dissertations Speaking 11 1

Translation 22 2

Culture 2 1

Gra& voc 2 1

Cloze 1 0

Dictation 1 0

8 1 TEM: A High-Stakes Standardized Test in the Chinese Context

1.3 TEM Test Characteristics

9

2. Validation of a certain item in TEM investigated the construct of that section, which would assist teaching and learning. For instance, validation of the reading part in TEM may analyze the strategies needed to accomplish the reading tasks; while the training of these strategies may also help students improve their reading ability; 3. Some researches focused on development and use of the TEM-based corpus, both of written and oral English. Analysis of these corpora may help us better understand the language proficiency of the students so that proper teaching can be offered.

1.3 TEM Test Characteristics TEM is a set of criterion-referenced English tests administered by NACFLT on behalf of the Higher Education Department, Ministry of Education, People’s Republic of China (Jin and Fan 2011). The following part will introduce TEM test purpose, TEM test content and format, TEM test administration, and TEM test score report and interpretation.

1.3.1 TEM Test Purpose As stated in the Test Syllabus, the purpose of TEM is twofold: (1) to measure the English proficiency of undergraduate English majors, to check whether they have met the required levels specified in the Teaching Syllabus; (2) to check the implementation of the Teaching Syllabus and to bring about reform on language teaching.

1.3.2 TEM Test Content and Format In the Teaching Syllabus, the four-year undergraduate program for English majors is divided into the foundation stage (freshman and sophomore year) and the advanced stage (junior and senior year). Accordingly, TEM4 and TEM4-Oral are administered at the end of the second year to assess candidates’ English proficiency in the foundation stage and TEM8 and TEM8-Oral are administered at the end of the fourth year to assess candidates’ English proficiency in the advanced stage. The test content of TEM4 includes listening, reading and writing, and two more skills of translation and general knowledge (linguistics, culture and literature) are assessed in TEM8. Basic language knowledge of grammar and vocabulary are measured in both tests, but in different formats, multiple-choice questions in TEM4 and error correction in TEM8. As for oral tests, TEM4-Oral focuses on speaking, while

10

1 TEM: A High-Stakes Standardized Test in the Chinese Context

TEM8-Oral assesses not only speaking but also interpretation. Test content and format of the four tests are listed in Table 1.4. Table 1.4 shows that various item types are employed in TEM, both objective ones and subjective ones, both discrete-point items and integrative tasks. In TEM4 and TEM8, objective items of multiple-choice questions account for 60% and 40% of the test, and the rest are all subjective items, such as writing and translation. Discrete-point items are used, such as multiple-choice questions of grammar and vocabulary in TEM4 and general knowledge in TEM8, and integrative tasks include dictation and cloze in TEM4 and mini-lecture and proofreading in TEM8. Table 1.4 Test content and format of TEM Task TEM4 Writing 1) composition 2) note-writing Dictation Listening Cloze Grammar & vocabulary Reading TEM8 Listening

Reading General knowledge Proofreading Translation 1) C-E 2) E-C Writing TEM4-oral Retelling

Input

Format

% Time

A topic, chart or graph

A text of about 200 words

15 35

Written prompts A text of about 150 words, listen 4 times, 120 wpm Conversations, passages and news broadcasts, 120 wpm A text of about 250 words 30 sentences

A note of 50–60 words Word-for-word dictation

10 10 15 15

30 MCQs

15 20

20 MCQs 25 MCQs

10 15 15 15

4–5 texts of about 1800 words altogether

15 MCQs

20 25

Mini-lecture, 900 words, 150 wpm Conversations and news broadcasts, 150 wpm 4–5 texts of about 3000 words 10 sentences

Note-taking and gap-filling (10 blanks) 10 MCQs

10 10

20 MCQs

20 30

10 MCQs

10 10

10 25

A text of about 250 words A text of about 150 Chinese characters, 250–300 wpm A text of about 150 words, 250–300 wpm A topic and prompts

Error identification and correction 10 15 Translation 10 30 Translation

10 30

A text of 400 words

20 45

A story of about 300 words, listen twice

Retelling for 3 min, no time for preparation

20

9

(continued)

1.3 TEM Test Characteristics

11

Table 1.4 (continued) Task Talking on a given topic Role-playing

Input A topic related to the above story A topic for discussion

Format % Time Talking for 3 min after 3-min 20 9 preparation 4-min discussion between two test 20 7 takers, after 3-min preparation

TEM8-oral Listen to the speech once, listen Interpreting (E-C) A speech of about 300 words, with about 150 words again and interpret sentence by sentence, 5 sentences for for interpretation interpretation Listen to the speech once, listen Interpreting (C-E) A speech of about 400 again and interpret sentence by characters, with about 200 characters for interpretation sentence, 5 sentences for interpretation A topic for presentation Talk for 3 min after 4-min Making a preparation comment on a given topic

20

9

20

9

20

7

Note: %: weighting, with the overall percentage being 100; Time is given in minutes; the overall score of TEM4-Oral and TEM8-Oral is 100, composed of 5 parts: the three tasks, pronunciation and intonation, and grammar and vocabulary

1.3.3 TEM Administration and Marking All the four tests of TEM are administered once a year: TEM4 and TEM4-Oral in May for second-year students, TEM8 in March and TEM8-Oral in December for fourth-year students. The total test time is 135 min for TEM4 and 195 min for TEM8, and the two oral tests take approximate 25 min each. For the written tests of TEM4 and TEM8, the test papers are sent directly to the participating schools before administration, together with detailed guidelines, providing instructions on how the test should be administered, such as when to distribute test papers or answer booklets. The test is administered at the same time throughout China and the listening section is broadcast from a specified radio channel. After administration, used test papers and answer booklets will be sent back to TEM Test Office in Shanghai International Studies University. As for the oral tests, they are administered in the audio laboratories of the participating schools, and candidates’ responses will be recorded either in tapes or in discs, which will be sent to Nanjing University, Guangdong University of Foreign Studies, or Dalian University of Foreign Languages, as assigned by the TEM Oral Test Office in Nanjing University. In TEM4 and TEM8, the objective items of multiple-choice questions are marked by machines, and there is no penalty for wrong answers. The subjective items are scored according to carefully-prepared marking criteria by well trained raters. Detailed marking schemes are developed for TEM4 dictation, TEM8 mini lecture and proofreading, to determine the marks deducted for each type of mistakes.

12

1 TEM: A High-Stakes Standardized Test in the Chinese Context

Moreover, analytic marking is adopted for writing and translation. Writing performance is judged against the criteria of ideas and argument (50%, including content, effectiveness and organization), language use (40%, including correctness, richness, appropriateness and fluency) and mechanics (10%, including spelling, punctuation, neatness, and so on); while translation performance is evaluated against the criteria of faithfulness (60%, faithfulness to the original text, both in content and in style) and fluency (40%, correctness in language and elegance in expression). TEM4-Oral and TEM8-Oral adopt a mixed marking approach. The three tasks are marked separately, 20% for each, according to different marking schemes developed for each task, and the rest 40% is the quality of language, 20% for pronunciation and intonation and 20% for grammar and vocabulary. The performance of each candidate will be marked by two raters independently.

1.3.4 TEM Test Score Report and Interpretation The pass score for all the four tests of TEM is 60, and candidates scoring above 60 will receive a certificate from NACFLT, on which their level of performance will be reported, including excellent (above 80 for TEM4 and TEM8, above 90 for TEM4- Oral and TEM8-Oral), good (70–79 for TEM4 and TEM8, 80–90 for TEM4-Oral and TEM8-Oral), and pass (60–69 for TEM4 and TEM8, 60–79 for TEM4-Oral and TEM8-Oral). Candidates scored below 60 will receive no certificate. The test scores are not reported to the test takers, but they can check their composite test scores of TEM4 and TEM8 through the Academic Affairs Office in their schools. Scores of TEM4-Oral and TEM8-Oral are not reported. Of the four tests in the battery, only TEM4 is required of all undergraduate English majors, while TEM8 and the two oral tests are optional. Decisions concerning penalties for those failing TEM4 are left in the hands of the participating universities (Jin and Fan 2011). As stated in the Test Syllabus, TEM results can be used to evaluate English teaching and learning in the light of the Teaching Syllabus, and to provide feedback to teaching and learning. However, due to the increasing social recognition of TEM as an important benchmark of English proficiency, more and more organizations in China are using TEM results to select and recruit employees; some schools do not confer bachelor’s degree to those who have failed to obtain TEM certificates; and teachers’ evaluation is connected with students’ performance in TEM in some schools. (Jin and Fan 2011). The increasing social stakes of TEM will have double effects. On one hand, more importance will be attached to TEM, which may enhance its facilitative role in English teaching and learning. On the other hand, the extensive social weighting of TEM may lead to overly test-oriented English teaching and learning among undergraduate English majors, which might be harmful.

1.4 TEM Test Characteristics vs. the Teaching Syllabus

13

1.4 TEM Test Characteristics vs. the Teaching Syllabus Since the purpose of TEM is to evaluate English teaching and learning in the light of the Teaching Syllabus, TEM test characteristics should comply with the Teaching Syllabus. Therefore, this section will first introduce the Teaching Syllabus, and then make a comparison between the Teaching Syllabus and TEM test characteristics.

1.4.1 The Teaching Syllabus The Teaching Syllabus is designed to standardize education for undergraduate English majors throughout China, which may serve as guidelines for the development of English program, curricula and textbooks. As stated in the Teaching Syllabus, the aim of college education is to cultivate students with solid language knowledge and profound cultural awareness who can work as teachers, translators, researchers and so on in the field of foreign affairs, education, economy and trade, culture, etc. To meet this end, courses introduced in undergraduate English program include three main categories: courses on language skills (phonetics, grammar, listening, speaking, writing, translation, oral interpretation, fundamental English, advanced English, and so on), courses on general knowledge of English (linguistics, English literature, English culture, and so on), and courses on related knowledge (foreign affairs, economics and trade, law, and so on). Compulsory courses recommended for the foundation stage and advanced stage are: 1. Foundation stage: fundamental English, phonetics, listening, speaking, reading, writing, grammar, and culture of English speaking countries; 2. Advanced stage: advanced English, writing, oral interpretation, translation, linguistics, British literature, American literature, and academic writing. Based on the two stages, the requirements on English teaching and learning are further divided into eight grades, one grade each semester, and English proficiency is divided into ten subcategories: pronunciation, grammar, vocabulary, listening, speaking, reading, writing, translation (including translation and oral interpretation), use of reference books (including dictionaries, encyclopedias, Internet resources, and so on), cultural awareness (familiarity with Chinese culture and culture of English spoken countries). TEM4 and TEM8 correspond to the required levels at the end of the fourth semester and eighth semester, and the ten subcategories outline the construct of TEM.

14

1 TEM: A High-Stakes Standardized Test in the Chinese Context

1.4.2 A Comparison Between the Teaching Syllabus and TEM Characteristics TEM is designed to measure the implementation of the Teaching Syllabus, and the four tests in the TEM battery assess nine out of the ten subcategories of language abilities listed in the Teaching Syllabus. The subcategory of “cultural awareness” has been assessed since 2005, when a new section of “General Knowledge” has been included in TEM8. The subcategory of “use of reference books” is not included because this skill aims at cultivating students’ ability to learn by themselves and do research, which can be measured by the writing tasks of term papers and bachelor’s degree degree paper. Table 1.5 lists the required language abilities in the Teaching Syllabus and how they are measured in TEM. From Table 1.5, it can be concluded that most language requirements in the Teaching Syllabus are measured in TEM with several items and various test methods in their corresponding stages. Direct testing methods are employed to measure writing, speaking, translation and oral interpretation. Listening, grammar and vocabulary are assessed with various testing methods, both objective multiple- choice questions and subjective methods of dictation, gap-filling, error-correction, and so on. Pronunciation, though not tested directly in any item, is adopted as an essential marking criterion which accounts for 20% of the composite score of TEM4-Oral and TEM8-Oral. Reading and cultural awareness are the only two abilities employing multiple-choice questions as the sole testing method. From the above analysis, it can be inferred that TEM characteristics, generally speaking, overlap with its construct stated in the Teaching Syllabus. However, possible mismatches lie in the following aspects: 1. Using multiple-choice questions as the sole testing method to measure language abilities such as reading and cultural awareness will lead to construct-irrelevant variance, which is caused by the possible guessing factor involved in multiple- choice questions. Such variance will contaminate the interpretation of the test score and may encourage test-taking technique training. 2. The small test population of TEM oral tests may lead to construct under- representation. The number of TEM4-Oral and TEM8-Oral test takers is around 20,000 and 10,000 respectively, which is only 10% of the candidates of TEM4 and TEM8 written tests. However, speaking is a required ability in the Teaching Syllabus, which should be of equal weighting as the other abilities. The lack of oral tests for most candidates will perhaps lead to less importance attached to speaking, and thus, the candidates’ low ability in speaking. 3. The misuse of TEM results is also a possible cause of construct irrelevant variance. As mentioned in Sect. 1.3.4, TEM results, as stated in the test syllabus, is meant to measure the English proficiency of undergraduate English majors, to find out whether they have met the required level specified in the Teaching Syllabus. However, with its increasing social weighing, TEM scores are used by some organizations and schools to recruit employees, rank schools and evaluate teachers. These uses of TEM results have not been justified, and will possibly lead to negative washback effects.

15

1.4 TEM Test Characteristics vs. the Teaching Syllabus Table 1.5 A comparison between the teaching syllabus and TEM characteristics Category Courses Pronunciation Phonetics

Stage TEM item F TEM4-oral,

Grammar

F

Grammar, fundamental English, advanced English

A

Vocabulary

Fundamental English, advanced English

F A

Listening

Listening, fundamental English, advanced English

F A

Speaking

Speaking, fundamental English, advanced English

F

A Reading

Reading, fundamental English, advanced English

Writing

Writing, fundamental English, advanced English

Translation

Translation, oral interpretation, fundamental English, advanced English

Culture

F A F A F A

English culture, fundamental F English, advanced English A

TEM8-oral TEM4: Grammar and vocabulary TEM8: Proofreading

TEM4-oral, TEM8-oral TEM4: Grammar and vocabulary TEM8: Proofreading

Test method A marking criterion Multiple-choice Error identification and correction A marking criterion Multiple-choice

Error identification and correction TEM4-oral, A marking TEM8-oral criterion TEM4: Dictation, Dictation, listening multiple-choice TEM8: Mini-lecture, Note-taking and listening gap-filling TEM4-oral: Retelling, Retelling presentation, conversation TEM8-oral: Presentation Presentation Conversation TEM4: Reading Multiple-choice TEM8: Reading TEM4: Note writing, Writing composition writing TEM8: Writing TEM8: Translation Translation (C-E, E-C) TEM8-oral: Oral Oral interpretation interpretation (C-E, E-C)2 TEM8: General Multiple-choice knowledge

Note: F=Foundation stage; A = Advanced stage; E-C = from English to Chinese; C-E = from Chinese to English.

16

1 TEM: A High-Stakes Standardized Test in the Chinese Context

1.5 Summary This chapter provides an overview of TEM, its context, history and characteristics. Moreover, a comparison is made between TEM characteristics and the Teaching Syllabus. Several key points are summarized below. First, in the past 20 years, TEM has developed into a high-stakes national test with extensive social recognition. The test results are used not only to examine the implementation of the Teaching Syllabus, but also to recruit employees, rank schools and evaluate teachers. Therefore, it is imperative to conduct a research on the washback effects of TEM. Next, there seems to be an endless list of TEM stakeholders, but in the educational system, TEM stakeholders can be roughly divided into three levels: policy makers, Teaching Syllabus designers and TEM constructors in the national level, the administrators in the school level and teachers and students in the classroom level. These are the major participants of the current study. Last, as a criterion-referenced test, TEM is not based on any set of textbooks or some particular language theory, but adheres to the Teaching Syllabus which outlines the construct of TEM. A comparison between TEM characteristics and the Teaching Syllabus shows that TEM characteristics generally overlap with its construct, which may lead to positive washback. However, mismatch also exists in the test content, method and use, which might be the source of negative impact.

Chapter 2

Washback: Definitions and Dimentions

2.1 Definitions of Washback Tests influence teaching and learning. A test, if well designed and appropriately employed, may not only help candidates develop positive attitude and strong motivation for learning, but also provide them with timely information on their strengths and weaknesses in learning, thus accelerating their learning process. Teachers can also benefit from the test through the feedback it offers, which makes it possible for them to adjust their teaching accordingly. However, if poorly designed, a test may not only provide false information, but also lead teaching and learning astray. Such influence is termed washback.

2.1.1 Washback or Backwash In the field of applied linguistics, washback and backwash are both used and generally considered interchangeable (Alderson 2004; Hughes 2003). However, a search in the dictionaries shows that “washback” is rarely seen; while only “backwash” can be found, which means either “a backward flow of water, caused by an oar, wave, etc.” or “the bad situation that remains after something bad has happened” (Longman Dictionary of Contemporary English). The negative connotations of these two definitions seem to suggest that the effects of testing on teaching and learning are more likely to be negative than positive. According to Spolsky (1996), the word “backwash” is often applied to unintended and negative effects, and until the 1980s it generally carried a negative meaning in the field of applied linguistics. However, the word “washback” has gained in currency and is now more commonly used in applied linguistics (Cheng et al. 2004; Green 2007; Hawkey 2006). Attitudes towards it have also become neutral, with many studies aiming at promoting beneficial washback (Alderson and Wall 1993; Gu 2007; Hughes 2003). In this sense, © Springer Nature Singapore Pte Ltd. 2018 Q. Xu, J. Liu, A Study on the Washback Effects of the Test for English Majors (TEM), https://doi.org/10.1007/978-981-13-1963-1_2

17

18

2 Washback: Definitions and Dimentions

the term “washback” will be used throughout in this dissertation, except when quoting from other authors.

2.1.2 Various Terminology In applied linguistics, washback is generally defined as the effect of testing on teaching and learning, which can be harmful or beneficial (Hughes 2003). Messick (1996: 241) further specified it as “the extent to which the introduction and use of a test influences language teachers and learners to do things they would not otherwise do that promote or inhibit language learning”. The effect or influence can be so powerful that it may drive teaching and learning, thus leading to the terms “teaching to the test” (Madaus 1988) and “measurement-driven instruction” (Popham 1987). Consequently, “curriculum alignment” (Smith 1991; Resnick and Resnick 1992) was proposed, because it is essential for the test content and format to overlap with that of the curriculum, if a test is to drive teaching along the desirable direction. To make better use of the guiding role of testing in teaching, a new or revised test can be introduced, with the aim that teaching and learning will improve accordingly, which is termed “systemic validity” (Fredericksen and Collins 1989). The above terms all focus on the role of testing in teaching, while “impact” tries to differentiate the influence at the micro level from the macro level. Impact refers to the influence of testing on society, educational systems and the individuals within these systems; while washback occurs at the micro level, only affecting the individuals, mainly teachers and students (Bachman and Palmer 1996; Hamp-Lyons 1998; McNamara 1996; Shohamy 2001). Later, impact was termed “consequence” in the Assessment Use Argument developed by Bachman and Palmer (2010). In this regard, the term “washback” shall be used in the present study, because the research mainly focuses on the effects of TEM on English teaching and learning, which is in the micro level.

2.1.3 Washback and Validity Washback is such an essential criterion for evaluating a test that some suggested it should be considered an aspect of validity, coining the term “washback validity” (Frederiksen and Collins 1989; Morrow 1986; Weir 1990). Messick (1989, 1996) further specified it by proposing “six distinguishable aspects of construct validity”, locating washback within the theoretical concept of consequential validity. The consequential aspect of construct validity includes evidence and rationales for evaluating the intended and unintended consequences of score interpretation and use in both the short- and long-term, especially those associated with bias in scoring and interpretation, with unfairness in test use, and with positive or negative washback effects on teaching and learning. (1996: 251)

2.2 Dimensions of Washback

19

However, this definition is disputed. Ferman (2004) suggested that it is difficult to see a direct connection between washback and validity because validity is not a property of the test, but rather the meaning of the test scores. Alderson and Wall (1993) pointed out that the complexity of washback phenomenon makes it impossible to be directly related to validity. Davies (1997: 335) shared the same view, “the apparent open-ended offer of consequential validity goes too far…it is not possible for a tester as a member of a profession to take account of all possible social consequences.” Messick (1996: 251) also recognized the complexity, since washback involves “a function of numerous factors in the context or setting and in the persons responding as well as in the content and form of the test”. Therefore, he advised that “rather than seeking washback as a sign of test validity, seek validity by design as a likely basis for washback” (1996: 252). However, despite the conflicting views, there seems to be little disagreement over the importance of investigating washback when evaluating a test. In conclusion, the above discussion on washback definitions can be summarized as follows: 1 . Washback refers to the effects of testing on teaching and learning; 2. Washback can be so powerful that it may direct teaching and learning; 3. A test may affect not only teaching and learning, but also the educational system, and even the society as a whole; 4. Washback is an essential aspect to consider when evaluating a test.

2.2 Dimensions of Washback As many studies on washback attest, one of its key features is that it is highly complex (Bailey 1996; Cheng 1997; Watanabe 1996, etc.). Therefore, to better analyze this phenomenon, it is necessary to investigate washback from different dimensions, so that we may look into its nature from various aspects. Based on the previous research (Bachman and Palmer 2010; Green 2007; Hawkey 2006; Watanabe 2004), six main dimensions are to be introduced here, namely, direction, extent, intensity, intentionality, length and specificity.

2.2.1 Direction Washback can be either positive or negative. A test, if well designed and appropriately used, may facilitate teaching and learning, providing useful information and generating strong motivation. This is positive washback. On the other hand, if poorly designed and inappropriately used, a test may lead to negative washback.

20

2 Washback: Definitions and Dimentions

Whether a test will generate positive or negative washback is the focus of most washback studies, with many researchers proposing various ways to promote positive washback (Bailey 1996; Gu 2007; Hughes 2003; Kellaghan and Greaney 1992), which can be summarized into the following three main aspects: 1. To improve the test, such as using direct and authentic testing, sampling widely, reflecting the full curriculum, employing various testing formats, and so on; 2. To facilitate the stakeholders, ensuring that they have adequate knowledge and resources to meet test demands, as well as taking a desirable attitude towards the test; 3. To enhance communication between test designers and stakeholders, such as providing timely test report, offering teacher training, and so on.

2.2.2 Extent A test may influence students, teachers, school administrators, textbook writers, etc. Therefore, in a washback study, it is important to first determine the research extent, whether it is the school context or the educational system as a whole. Generally speaking, the more important a test is, the greater its washback effects will be. A classroom achievement test may affect a certain class only; while a high-stakes test, such as a nation-wide selection exam, the National Matriculation Test, may affect not only the teachers and students involved, but also the related parents, textbook writers, and even the whole society. Hence, in a washback study, the extent of the washback effects shall be investigated, so that a better overview of the study can be achieved.

2.2.3 Intensity Washback can be either strong or weak. If the impact is strong enough a lot of classroom teaching activities can be determined by it, such as what to teach and how to teach, even stakeholders’ attitude to teaching and learning. The intensity of washback is often related to the stakes of a test; that is, the more important the test, the more intense its washback. Undoubtedly, an important test like the National Matriculation Test will affect teaching and learning more than a classroom achievement test. Therefore, a test with more intense washback tends to attract more attention from the researchers and most of the washback studies are conducted on high-stakes tests like TOEFL, IELTS and National Matriculation Test (Cheng 2005; Green 2007; Alderson and Hamp-Lyons 1996; Hawkey 2006; Qi 2004; Wall 2005; Watanabe 1996).

2.2 Dimensions of Washback

21

2.2.4 Intentionality Washback can be either intended or unintended. Intended washback refers to the effects that test designers can predict or want to encourage. For example, the intended washback effects of National Matriculation Test are to motivate students to learn, to provide feedback information, and so on. Moreover, if some new test items are introduced, the intended washback may be to improve teaching and learning with the guiding effects of the test. However, a test may also bring about some unintended effects, such as the anxiety among students, long-term test-taking technique training, etc. Therefore, in a washback study, extra efforts should be made to investigate the unintended washback that may have occurred in actuality.

2.2.5 Length Washback may exist for a short time or last for a long term. The effects like motivating students to learn may disappear shortly after the test is over, while the learning habits developed when preparing for the test may last a life time. Thus, longitudinal study should be carried out to investigate not only the immediate effects of a test, but also its influence in the long run.

2.2.6 Specificity Washback can be either general or specific. General washback refers to the effects that may be generated by any test, such as motivating students to learn and providing feedback information; while specific washback relates only to a certain test or test type, such as the washback effects of multiple-choice phonetics questions. Therefore, when investigating washback, attention should be paid to exploring the specific effects of a test, apart from reviewing the general washback. Among the dimensions listed above, the most frequently studied is the nature of washback, whether it is positive or negative; while the rest of the dimensions are seldom looked into. However, they can also provide researchers with valuable information and a fresh insight into the role of testing in teaching and learning. Therefore, to view washback from different dimensions may be a necessary supplement to washback definition and an important step forward in washback studies, which shall contribute to the improvement and systematization of research in this field.

22

2 Washback: Definitions and Dimentions

2.3 Summary This chapter has reviewed the definitions and dimensions of washback effects, which may suggest that different dimensions shoud be investigated in a washback study. It is essential to look into its direction, whether it is positive or negative, or whether its positive washback outweighs its negative washback. However, besides direction, other dimensions can also be explored, such as whether a test has realized its intended washback, whether the washback effects will last a long time or disappear when the test is over, and so on. Therefore, in a washback study, more efforts can be made to probe into the other dimensions of washback effects, to obtain a more comprehensive picture of the phenomenon.

Chapter 3

Theoretical Frameworks for Washback Studies

3.1 Frameworks for Washback Studies The early 1990s witnessed a great step forward in washback studies—a shift from making assertions to establishing theoretical frameworks (Gu 2007). Alderson and Wall (1993) were the first to develop a framework—Fifteen Washback Hypotheses. Following them, many other scholars have proposed different models. Here, some of the significant ones will be analyzed in detail, namely, Hughes’ (1993) Washback trichotomy, Bailey’s (1996) basic model of washback and Green’s (2007) model of washback, incorporating intensity and direction.

3.1.1 Alderson and Wall’s Washback Hypotheses In “Does Washback Exist?”, Alderson and Wall (1993) provided a thorough and comprehensive framework to illustrate the washback effects of language testing. Based on the most general Washback Hypothesis “a test will influence teaching”, they further developed fourteen hypotheses:

1. A test will influence teaching. 2. A test will influence learning. 3. A test will influence what teachers teach; and 4. A test will influence how teachers teach; and therefore by extension from 2 above: 5. A test will influence what learners learn; and 6. A test will influence how learners learn. 7. A test will influence the rate and sequence of teaching; and 8. A test will influence the rate and sequence of learning. 9. A test will influence the degree and depth of teaching; and 10. A test will influence the degree and depth of learning. 11. A test will influence attitudes to the content, method, etc. of teaching and learning. 12. Tests that have important consequences will have washback; and conversely 13. Tests that do not have important consequences will have no washback.

© Springer Nature Singapore Pte Ltd. 2018 Q. Xu, J. Liu, A Study on the Washback Effects of the Test for English Majors (TEM), https://doi.org/10.1007/978-981-13-1963-1_3

23

24

3 Theoretical Frameworks for Washback Studies

14. Tests will have washback on all learners and teachers. 15. Tests will have washback effects for some learners and some teachers, but not for others. (Alderson and Wall 1993: 120–121. original text in bold as indicated.)

The hypotheses proposed by Alderson and Wall provide us with a detailed description of washback effects, clarifying this vague concept. When classifying the Fifteen Washback Hypotheses, Alderson and Wall tried to make a distinction between teaching (1, 3, 4, 7, 9 and 11) and learning (2, 5, 6, 8, 10 and 11), content (3 and 5) and method (4 and 6), and practice (3–10) and attitude (11). Furthermore, the last four hypotheses (12–15) were raised to acknowledge differences between tests and participants. According to Alderson and Hamp-Lyons (1996: 282), the purpose for raising these hypotheses is not to argue for or against any of them, but to “lay out the territory” so that the questions for further studies on washback can be more clearly seen. Moreover, these Fifteen Washback Hypotheses constitute an open and adaptable framework, since it is highly likely that the research context is more complex than the fifteen hypotheses proposed, so the nature of the test and the educational context should be taken into account (Alderson and Wall 1993). Therefore, in their 1996 article, Alderson and Hamp-Lyons (1996: 296) revisited the Washback Hypotheses, suggesting an expansion of the Fifteenth Washback Hypotheses, “Tests will have different amounts and types of washback on some teachers and learners than on other teachers and learners.” Moreover, the amount and type of washback will vary according to the status of the test, the extent to which the test is counter to current practice, the extent to which teachers and textbook writers think about appropriate methods for test preparation and the extent to which teachers and textbook writers are willing and able to innovate.

3.1.2 Hughes’ Washback Trichotomy While Alderson and Wall (1993) tried to “lay out the territory” of washback questions to be addressed in future research, Hughes’ (1993) attempt was to construct a model of washback mechanism. Hughes (1993: 2) proposed, “In order to clarify our thinking on backwash, it is helpful, I believe, to distinguish between participants, process and product in teaching and learning, recognizing that all three may be affected by the nature of a test”. According to Hughes (1993:2), participants are “all of whose perceptions and attitudes towards their work may be affected by a test”, such as teachers, students, school administrators and material writers. Process refers to “any actions by the participants which may contribute to the process of learning”, such as syllabus design, material development, the selection of teaching content and methods, etc. Last, product is “what is learned (facts, skills, etc.) and the quality of learning (fluency, etc.)”. He further states: The trichotomy into participants, process and product allows us to construct a basic model of backwash. The nature of a test may first affect the perceptions and attitudes of the

3.1 Frameworks for Washback Studies

25

p articipants towards their teaching and learning tasks. These perceptions and attitudes in turn may affect what the participants do in carrying out their work (process), including practicing the kind of items that are to be found in the test, which will affect the learning outcomes, the product of that work. (1993:2)

Besides, Hughes added that the following five conditions were necessary for the washback effects to occur:

1. Success on the test must be important to the learners; 2. Teachers must want their learners to succeed; 3. Participants must be familiar with the test and understand the implications of its nature and content; 4. Participants must have the expertise which is demanded by the test (including teaching methods, syllabus design and materials writing expertise); 5. The necessary resources for successful test preparation must be available. (1993: 2–3)

3.1.3 Bailey’s Basic Model of Washback Combining Alderson and Wall’s Washback Hypotheses with Hughes’ Washback Trichotomy, Bailey (1996) put forward a basic model of washback. In the basic mode, Bailey not only listed the possible participants and products, but also illustrated the direction of influence. Washback effects are from tests to participants, and then, to products; while “washforward” effects are possible influences from the participants on the test. Moreover, she further distinguished between “washback to the learners” and “washback to the program”, with “the program” covering not only teachers, but all the other participants apart from the students. “Washback to the learners” refers to the effects of “test-derived information provided to the test-takers and having a direct impact on them” (Bailey 1996: 263), which includes five of Alderson and Wall’s Washback Hypotheses (2, 5, 6, 8 and 10). Moreover, Bailey (1996: 264) listed 10 processes that might be involved, such as practicing items similar to those on the test and applying test-taking strategies. On the other hand, “washback to the programme” is “results of test-derived information provided to teachers, administrators, curriculum developers, counselors, etc.” (1996: 264), which are directly related to Washback Hypotheses 1, 3, 4, 7 and 9. However, no processes are specified, maybe because of the complicated nature of the programme, which seems to cover an unlimited range.

3.1.4 Green’s Washback Model In his washback study on IELTS, Green (2007) proposed a model, incorporating two dimensions of washback—direction and intensity. In his model, Green pointed out that washback direction is mainly determined by the quality of the test, “the more closely the characteristics of the test reflect the

26

3 Theoretical Frameworks for Washback Studies

focal construct as understood by course providers and learners (the greater the overlap), the greater the potential for positive washback” (2007: 14). Moreover, test use and test stakes, or the perceived consequences of test scores will also affect the direction of washback. Meanwhile, test stakes and test difficulty, as perceived by the stakeholders, will influence the intensity of washback, with important and challenging (but attainable) tests exerting the most intense washback. However, the washback effects of the same test may vary from participant to participant because of their different experiences and beliefs, such as their knowledge and understanding of test demands, resources to meet test demands and acceptance of test demands. In sum, from the above models, it can be concluded that when investigating washback, three questions are to be answered—what, how and why. First, what will be affected by a test? According to Alderson and Wall (1993), a test will influence both teaching and learning, such as the content and method, the rate and sequence, and the degree and depth. Next, how will washback effects work? According to Hughes (1993) and Bailey (1996), the washback mechanism consists of three parts—participants, processes and products. The participants, such as teachers, students, school administrators and material writers, will first be affected by the test— their perceptions of and attitudes to the test will influence their concept of language teaching and learning. Consequently, the teaching and learning processes will change, and the learning products will differ. Last, why would a test exert washback effects? According to Green (2007), the direction of washback is related to the overlap between test construct and test characteristics; while the intensity of washback effects is determined by test stakes and test difficulty. Moreover, the washback effects of the same test may vary from participant to participant because of their different experiences and beliefs. In a word, washback effects are determined by both the test, such as its validity, stakes and difficulty, and the stakeholders, such as their values and characteristics.

3.2 Empirical Studies on Washback Although washback has long been a concept under discussion (Cronbach 1963; Fredericksen 1984; Latham 1877), empirical studies on the washback effects of language testing have only a short history of no more than 30 years. Since the late 1980s, washback has gained importance in language testing and there has been a rapid increase in the number of empirical studies conducted. As a result, Language Testing, a renowned academic journal in the field, published a special issue on washback effects in 1996, reporting the latest developments in washback theory and practice at that time, such as Alderson and Hamp-Lyons’ washback study on TOEFL, Watanabe’s study on the University Entrance Exam in Japan, Wall’s study on O-Level exam in Sri Lanka and Shohamy’s study on Arabic and English tests in Israel. Moreover, in the 2000s, Cambridge University Press published four books on

3.2 Empirical Studies on Washback

27

washback studies (Cheng 2005; Green 2007; Hawkey 2006; Wall 2005), which indicates that research on washback has advanced into a new era, becoming systematic and comprehensive. In mainland China, the first washback study on language testing might be the one conducted by Li (1990) on Matriculation English Test (MET). In the next two decades, more research has been done on high-stakes nation-wide tests, such as MET, CET and TEM, with CET being the most widely investigated, as indicated by a search in the CSSCI1 journals (Cai 2009; Jin 2000, 2006; Xiao et al. 2014; Ye 1998; etc.). Moreover, two books (Gu 2007; Qi 2004) have been published on CET washback and MET washback respectively. Comparatively, TEM washback is less studied, with only three articles (Qin 2012; Xu 2012; Zou and Xu 2014) published in CSSCI journals till the end of 2015. Therefore, it is necessary to conduct more empirical research on TEM washback, to better understand the role of TEM in English teaching and learning, so that positive effects can be achieved.

3.2.1 Summary of Empirical Studies To better clarify the research content and process of washback studies, some of the large-scale and influential empirical studies on the washback of language testing are to be introduced, with the following table summarizing the basic information such as research participants, methods and findings. The following six studies are selected because books have been published on them, providing detailed information on the implementation of these studies. From Table 3.1, it can be concluded that most of the tests studied are large-scale, high-stakes tests, which mainly fall into two categories—selection tests and proficiency tests. 1. Selection tests: NMET (National Matriculation English Test), HKCEE (Hong Kong Certificate of Education Examination in English), O-level English Exam. These tests play such important roles as determining whether candidates will be allowed to enter higher education. 2. Proficiency tests: IELTS (International English Language Testing System), PL2000 (PL2000 is a reform project to encourage external certification of learners’ language proficiency through the examinations of providers such as Cambridge ESOL), CET (College English Test). These are large-scale proficiency tests, either national or international, which assess candidates’ language proficiency so that important decisions can be made. 1 Eleven journals on linguistics (foreign language) included in CSSCI (2014–2015) are searched, which are Foreign Language Teaching and Research, Foreign Language World, Journal of Foreign Languages, Modern Foreign Languages, Chinese Translators Journal, Foreign Languages in China, Foreign Language Research, Foreign Language Education, Foreign Languages and Their Teaching, Computer-Assisted Foreign Language Education, and Foreign Language Learning Theory and Practice.

Hawkey IELTS and Teachers; students; 2006 PL2000 in Italy receiving institution administrators; school heads; parents

Questionnaires; interviews; classroom observations; textbook analysis

Interviews; questionnaires; classroom observations; document analysis

O-level test in Sri Lanka

Wall 2005

Teachers; teacher advisors

Policy makers; textbook Questionnaires; classroom observations; interviews; publishers; teachers; document analysis students

Revised HKCEE in Hong Kong

Cheng 2005

Methods Interviews; classroom observations; questionnaires; document analysis

Participants Test constructors; English inspectors; teachers; students

Author Test Qi 2004 NMET in China

Table 3.1 Overview of the empirical studies on washback Research findings NMET has largely failed to achieve its intended washback effects; A high-stakes test is a powerful spur to trigger pedagogical responses, but it tends to be ineffective to bring about desirable changes. The revision of a test will lead to efficient washback on teaching materials; Teaching content will be affected, but teachers’ attitudes and behaviors will not change much; A change in the test only can hardly fulfill the intended goal of improving teaching and learning. Tests have washback on what teachers teach, but not how they teach, and there is both positive and negative washback; Many factors contribute to or inhibit washback, implying that the nature of curricular innovation is much more complex. IELTS is perceived as a reasonable direct communicative test, using authentic and appropriate content; it is hard but fair and reading and writing are more difficult modules; it is a high-stakes test, causing both anxiety and motivation; PL2000 appears to have positive effect on communicative teaching from teachers’ views, but students’ views differ; It has positive washback on motivation and attitudes, assessment, exam candidacies and performances; It helps school management and provides teachers with professional support.

28 3 Theoretical Frameworks for Washback Studies

IELTS

CET in China

Green 2007

Gu 2007

Conference participants (government officials, researchers, publishers, etc.); teachers; students

Course providers; teachers; students

Questionnaires; interviews; classroom observations; document analysis; test performance analysis

Questionnaires; interviews; classroom observations; document analysis; tests

Discrepancies exist between IELTS test and participants’ beliefs about academic writing; IELTS preparation courses include a narrower range of academic writing tasks than other unrelated courses; Learners can profit from giving attention to IELTS preparation activities, but the additional benefit is limited. Most CET stakeholders applaud the test, but some problems are mentioned, such as the overuse of MC format, the lack of direct score reports to the teachers, and so on; College English classroom teaching and learning are mainly regular rather than test-oriented, and most teachers use textbooks and other teaching materials; The coaching materials do not have reliability and validity, and test-oriented practice leads to negative effects on teaching and learning; CET writing and speaking tests have exerted positive washback on teaching and learning; CET is an effective means of measurement, but it is just one of the factors that influence college English teaching and learning.

3.2 Empirical Studies on Washback 29

30

3 Theoretical Frameworks for Washback Studies

Washback studies are often carried out on these large-scale high-stakes tests because they tend to exert great impact on language teaching and learning. Qi (2004) investigated the washback effects of NMET, one of the three compulsory tests (the other two are Chinese and mathematics) of the college entrance examinations in China. As a nation-wide high-stakes test, NMET is expected to induce intended washback effects of reforming teaching in secondary schools, shifting its focus from language form to language use. In her study, Qi used interviews, questionnaires and classroom observations to collect data from 8 test constructors, 6 English inspectors, 388 teachers and 986 students. The research findings indicated that NMET failed to achieve its intended washback because of the overwhelming functions it was expected to perform—the function of selection imposed constraints on test design and development, and the function of evaluation boosted test stakes and stimulated intensive test preparation. As a result, Qi came to the conclusion that “tests whose stakes are too high will fail to exert the intended washback effect although high-stakes tests have the potential to do so” (2004: 297). Cheng (2005) examined the washback of the revised HKCEE, which was meant to cause changes in classroom English teaching in secondary schools. She collected data both before and after the revised test was introduced (baseline and follow-up data), so that a comparison could be made. In her study, Cheng developed a three- level working model, which was carried out in three phases, investigating (1) decision making agencies from the top, such as HKEA (Hong Kong Examinations Authority); (2) intervening agencies such as textbook publishers and tertiary institutions; and (3) implementing agencies such as teachers and students in secondary schools. The results show that the revision of a test will have washback on the teaching content, but teachers’ attitudes and behaviors will not change much; therefore the change in the test can hardly fulfill the goal of educational innovation and what actually gets taught is far more complicated than what a test can determine. Wall (2005) explored the washback effects of a new O-level English exam in Sri- Lanka, which, together with a new textbook series, was introduced to bring about innovations in English classrooms. Both baseline and follow-up data were collected through interviews and classroom observations. After careful comparison, Wall came to the conclusion that what teachers teach will be affected more than how teachers teach. In his book, Hawkey (2006) reported two washback studies, one on IELTS and the other on PL2000. The IELTS impact research is part of the IELTS validation systems, which will contribute to the monitoring and test development process of IELTS. Interviews, questionnaires and classroom observations were conducted among candidates, teachers and receiving institution administrators, which led to generally favorable conclusions: IELTS is perceived as a reasonable direct communicative performance test, using authentic and appropriate content; IELTS is fair, but hard, especially Reading and Writing module; IELTS is a high-stakes test, causing motivation and moderate anxiety. The other washback study was conducted on PL2000, a language education reform project in Italy, which aimed to provide foreign language education in state schools to meet the communication and certification needs of students. One of its key policies was to encourage the external

3.2 Empirical Studies on Washback

31

certification of learners’ language proficiency through the examinations of providers such as Cambridge ESOL. Thus, Cambridge ESOL was encouraged to carry out a PL2000 impact study, collecting data from students, teachers, parents, and so on, to hear their perception and evaluation of this project. The research findings are mainly positive: PL2000 appears to have positive effects on motivation and attitudes, communicative teaching, school management, etc., although students’ views, to some extent, differ from teachers’ views on communicative teaching. Green (2007) focused on the washback of IELTS on academic writing. In his study, Green compared IELTS preparation courses and unrelated academic writing courses, through interviews, questionnaires, classroom observations and several tests administered at course entry and exit. The research findings show that discrepancies exist between IELTS writing test and participants’ beliefs about academic writing, and IELTS preparation courses seem to include a narrower range of academic writing tasks than other courses. Besides, although learners can profit from giving attention to IELTS preparation activities, the additional benefit is limited. Gu (2007) discussed the washback effects of CET on college English teaching and learning. A wide range of stakeholders (about 4500 in total), such as administrators, teachers and students, were investigated, through various research methods of interviews, questionnaires and classroom observations. The research findings are mainly positive: most of the stakeholders think highly of the test, believing that the positive washback is much greater than the negative washback, and besides, the negative washback is primarily due to the misuse of test by test users. At the end of the study, various suggestions are made to teachers, students, administrators and test constructors, to promote positive washback of CET.

3.2.2 A Critique of Approaches in Washback Studies From the above studies, it can be concluded that washback studies are mostly large- scale, multi-method, multi-phase and longitudinal. Washback studies are large-scale ones because the tests investigated are often large-scale and high-stakes tests which affect a wide range of stakeholders. Therefore, in terms of participants, all the above studies look into washback on teachers or students, since they are the major stakeholders. Besides, many other stakeholders are investigated, such as test constructors, school administrators, textbook writers and parents. As a high-stakes test, the range of its stakeholders seems to be infinite, so the selection of participants is a vital problem to solve, which is mainly determined by the research purpose and practicality. For instance, if intentionality of washback is to be explored, it is necessary to investigate not only teachers and students, but also test constructors, so that both the intended and unintended washback can be identified. Various methods have been employed in washback studies, among which questionnaires, interviews and classroom observations are the most frequently used. Since washback is a complex phenomenon, it is essential to obtain both quantitative

32

3 Theoretical Frameworks for Washback Studies

and qualitative data, to ensure the validity and reliability of the research. Moreover, document analysis is often conducted, with documents covering related research, syllabuses, textbooks, and so on. Besides, relevant tests can be designed to compare the outcome of test-preparation courses and non-preparation courses, just as what Green (2007) did in the washback study on IELTS. Meanwhile, it should also be noted that the data collection instruments, such as language tests and interview questions, must be validated. When designing the instrument, approaches like brainstorming, expert opinion, reporting and reviewing, piloting and trialing should be carefully planned and meticulously conducted. Moreover, validation techniques can also be employed when processing the data collected, such as factor analysis, multi-trait multi-method validation, triangulation of data collected from different sources or by different instruments, and so on. Washback studies are often multi-phase and longitudinal research because they involve a wide range of stakeholders and employ various research methods. Besides, it takes time for some washback effects to emerge. For instance, in Cheng’s (2005) research, a three-phase study was developed, to investigate policy makers, textbook writers, teachers and students step by step. Moreover, a baseline study had been conducted to collect data before the revised test was introduced, and then more research was done afterwards. Therefore, it is essential to conduct a diachronic study on washback, to explore its causes, working mechanisms and evolving effects. As for research findings, different conclusions have been proposed, with different research foci in each study. In Qi’s (2004) research, it is concluded that high- stakes tests can hardly realize their intended washback because of the overwhelming functions they are expected to perform; while in Gu’s (2007) study, the conclusion is that positive washback of CET far outweigh its negative washback and the negative washback is mainly caused by the misuse of the test. Meanwhile, similar findings have been reported. In many studies (Cheng 2005; Qi 2004; Wall 2005), it is found that a test will have washback effects on the teaching content, but can hardly affect the teaching method. Another common conclusion is that washback is a complex issue which varies from participant to participant and which is mediated by many other factors apart from test design and test quality. To sum up, washback study is comprehensive and systematic, which can be conducted either in the micro context (the school setting) or in the macro context (the educational system or the society), and which involves various factors, such as the test factor (test content, test use, etc.), the prestige factor (test stakes) and the personal factor (characteristics of teachers, students, etc.). Therefore, it is impossible and impractical to develop an inclusive approach which may cover all aspects of washback. In this sense, when planning a washback study, it is important to define the research purpose first, and then relevant participants and methods can be selected. The research findings, whatever they are, may contribute to establishing a washback framework.

3.3 Summary

33

3.3 Summary This chapter has reviewed related literature on washback models and empirical studies, and several key points in washback studies have emerged in the analysis, which are summarized as follows: 1. To have a clear research purpose. A test may affect not only the classroom setting, but also the educational system, even the society as a whole. Therefore, a washback study may involve students, teachers, school administrators, textbook writers, policy makers… It is impractical and impossible to investigate all the stakeholders, so a clear research purpose is essential. Once a research purpose is defined, whether to investigate washback in the micro or macro context, the relevant research participants can be selected, and then the research methods. In this sense, the research will be more emphasized. 2. To investigate different dimensions of washback. Almost all washback studies focus on its direction, such as whether it is positive or negative, or whether its positive washback outweighs its negative washback. However, apart from direction, other dimensions can also be explored, such as whether a test has realized its intended washback, and whether the washback effects will last a long time. Therefore, in a washback study, efforts can be made to probe into various dimensions of washback effects, to obtain a more comprehensive picture of the phenomenon. 3. To draft a practical washback model. Quite a few washback models have been proposed, such as Wall’s Washback Hypotheses and Bailey’s Basic Model. However, it is undecided as to which one can best depict the washback phenomenon and illustrate its working mechanism. It still remains a challenging issue in washback studies, since washback is such a complex phenomenon that it seems impossible to draft a model to cover every aspect of it. Nevertheless, a theoretical model is essential. Therefore, it is necessary to draft a washback model with focused aims to guide the empirical research. 4. To employ appropriate research methods. A multi-method, multi-phase and longitudinal approach will be adopted to investigate the washback effects of TEM. Large-scale questionnaires can be conducted to collect quantitative data from a wide range of stakeholders, and interviews and classroom observations can be conducted as case study to collect qualitative data. Moreover, data collected in the previous phase may assist the development and implementation of research in the coming phases. In this sense, data collected can be triangulated and data collection instruments can be validated. In conclusion, a theoretical framework should be developed to guide the present study, taking the above four problems into consideration.

Chapter 4

Research Framework for TEM Washback Study

4.1 Framework for TEM Washback Study Based on the above theoretical models and empirical studies, a TEM washback model (Fig. 4.1) is drafted, to illustrate its causes, contents and mechanisms. This model, as shown in Fig. 4.1, incorporates Bailey’s basic model and Green’s model, which demonstrates basic factors that may contribute to the generation of washback effects, as well as the working mechanism of TEM washback. The model will be explained in detail in the following part.

4.1.1 Contributing Factors to TEM Washback From TEM washback model, it can be concluded that TEM washback is mediated by the following three factors: 1. The overlap between test construct and test characteristics. The test construct of TEM is stated in its test syllabus, and the test characteristics including test content, method, test result interpretation and use, etc. The overlap between test construct and test characteristics may contribute to the direction of washback— the greater the overlap, the more likely a test will exert positive washback. Conversely, negative washback will be generated, if a test fails to test what is stated in its construct (the part to the left of the “overlap”, e.g. the oral part is missing in some English tests which claim to measure all the four language skills of listening, speaking, reading and writing), or if a test tests something not stated in its construct (the part to the right of the “overlap”, e.g. an English reading exam tests candidates’ knowledge in a specific field, such as law or commerce); 2. Test stakes and test difficulty. These may contribute to the intensity of washback, with important and difficult, but still attainable tests exerting the most intense washback. In this sense, washback of TEM4 and TEM8 may be different, and © Springer Nature Singapore Pte Ltd. 2018 Q. Xu, J. Liu, A Study on the Washback Effects of the Test for English Majors (TEM), https://doi.org/10.1007/978-981-13-1963-1_4

35

36

4 Research Framework for TEM Washback Study

TEM Focal Construct

TEM Test Characteristics Negative

Positive

Negative

stakes

Overlap

difficulty

PARTICIPANTS

Material writers Students

Teachers

Curriculum designers School administrators

Researchers

Experts

Etc.

PROCESSES PRODUCTS

What

What

What

What and

and

and

and

how to

how to

how to

how to

how to

research

develop

learn

teach

design

Learning

Teaching

New materials

Research

New syllabuses

outcome

effect

New curricula

results

New approaches

Etc.

What and

Etc.

Fig. 4.1 TEM washback model

TEM washback may vary from school to school, because of the difference between tests and the gap between schools. 3. Participants’ perceptions and characteristics. Washback effects of TEM vary from participant to participant because of their different perceptions of the test and their unique characteristics. Participants differ in how they perceive test difficulty, test importance, test purpose, test use, and so on. Moreover, participant characteristics and values also differ, such as their previous knowledge and beliefs, available resources, and so on.

4.1.2 TEM Washback Mechanism This model (Fig. 4.1) combines Wall’s Washback Hypotheses with Bailey’s basic model, illustrating both the content and the mechanism of TEM washback.

4.1 Framework for TEM Washback Study

37

TEM will first influence its stakeholders, whose perceptions of English teaching and learning will change accordingly. The stakeholders include teachers, students, school administrators, textbook writers, researchers, and so on. Here, a new category is added—the experts. Experts in TEM refer to TEM test designers, constructors and the Teaching Syllabus designers. It is important to investigate TEM experts because they are familiar with TEM, its implementation, related policies and the intended washback. Therefore, their opinions can provide us with an illuminating insight into TEM and its washback (refer to Sect. 1.1 for more information). Of course, the stakeholders can be more than those listed in the model, if impact in the macro level is to be investigated. Under the influence of TEM, the stakeholders’ learning, teaching, and designing processes will change, yielding different products. Moreover, every category of participants, processes and products may influence each other, as indicated by the arrows. For instance, teachers’ teaching can be affected by the textbooks, the research findings and the new syllabuses. Besides, the participants, processes and products may influence the test, leading to test revision and reform, which is termed “washforward” by Van Lier (1989).

4.1.3 Several Terms Redefined To better apply this model to TEM washback study, it is necessary to clarify a few terms, namely, TEM test construct, TEM test characteristics, overlap, and positive and negative washback. 4.1.3.1 TEM Test Construct Test construct is the trait or traits that a test is intended to measure, which, in the case of a language test, refers to the language ability to be measured (Davies et al. 1999). As for TEM, its construct is stated in its test syllabuses, which include Syllabus for TEM4, Syllabus for TEM8, Syllabus for TEM4-Oral and Syllabus for TEM8-Oral, since TEM is composed of these four separate tests. Moreover, language abilities stated in the Teaching Syllabus should also be included when defining TEM construct, since the test purpose of TEM is to examine whether candidates have met the required levels of English abilities specified in the Teaching Syllabus (Jin and Fan 2011). 4.1.3.2 TEM Test Characteristics Test characteristics refer to the test content, method, marking, as well as the use and interpretation of test results. In this sense, TEM test characteristics include its test contents (listening, reading, writing, speaking, translation, grammar and vocabulary, and so on), test methods (item types used, such as multiple choice, blank

38

4 Research Framework for TEM Washback Study

filling, error correction, and so on), marking (rated by machines or by human beings, using holistic marking or analytic marking, etc.), and use and interpretation of test results (whether TEM results are related to employment, university ranking, and so on). 4.1.3.3 Overlap Overlap means the extent to which the test characteristics can reflect the test construct; that is, whether TEM test can elicit performance which can reflect the English ability stated in the Syllabuses. For example, it is stated in both the Teaching Syllabus and the Test Syllabus that candidates should be able to follow news broadcast from native countries, and in both TEM4 and TEM8, news broadcast is a part of listening comprehension. In this sense, TEM characteristics overlap with TEM construct. However, if test characteristics and test construct do not match, it would be termed either “construct under-representation” or “construct-irrelevant variance” (Messick 1996). Construct under-representation means that some language abilities in the test construct are not measured. For instance, speaking is a language skill listed in the Teaching Syllabus, but only a small fraction of the candidates are eligible to take the oral test, so this skill in TEM construct is under represented in the test. On the contrary, construct-irrelevant variance refers to the problem that test characteristics fail to reflect test construct. For instance, in a reading test, construct irrelevant variance may be caused by background or cultural knowledge. The item which tests candidate’s cultural knowledge is irrelevant to the reading ability to be measured, and will lead to misinterpretation of the test scores. The overlap and mismatch between TEM construct and TEM characteristics would, to some extent, determine whether the test will exert positive or negative washback. 4.1.3.4 Positive and Negative Washback The direction of washback, whether it is positive or negative, is considered the most important issue in washback studies, and promoting beneficial washback is the ultimate goal of most empirical studies on washback. However, what is positive washback and what is negative washback? General criteria have been offered, such as whether the effects are beneficial or damaging to the development of learners’ abilities (Green 2007), or whether it encourages or inhibits the attainment of educational goals held by learners, educators, or both (Bailey 1996). However, no one has developed a set of clear-cut and practical criteria to distinguish positive and negative washback (Qi 2011). The possible difficulty lies in the fact that it is a highly subjective matter—“the distinction between positive and negative could usually be made only by referring to the audience” (Watanabe 2004: 21). What is considered positive washback by the teachers may be considered negative by the experts.

4.1 Framework for TEM Washback Study

39

Therefore, in the current study, two principles will be followed when judging the direction of washback effects. First, participants’ evaluation of washback will be sought separately. Students, teachers, school administrators and experts will all be investigated about their opinions on the washback direction in their respective questionnaires, and in classroom observations, teachers’ and students’ behavior will be analyzed separately, to determine the different washback on them. Moreover, a distinction will be made between positive washback in the micro context and in the macro context. Positive washback in the micro context refers to immediate but short-term benefits, such as improving candidates’ test performance, enhancing efficiency of teaching and learning, and so on; while positive washback in the macro context will last a long time, beneficial to improving candidates’ abilities. For instance, teachers’ moderate use of the mother tongue may help clarify the testing demands more efficiently, so it can be considered positive in the micro context; however, the excessive use of mother tongue, more than 10–15% as suggested by Macaro (2005), may impede the development of learners’ language ability, so it is negative in the macro context.

4.1.4 Research Design As a high-stakes test, TEM will affect not only teachers and students, but also the educational system, and even the society as a whole. The current study will focus on the washback effects of TEM on college education for English majors, so only the stakeholders in the educational system are investigated, while stakeholders like employers and parents are not included. As outlined in the TEM washback model (Fig. 4.1), the two major aspects to be investigated are contributing factors to TEM washback and TEM washback mechanism, and the participants involved are students, teachers, school administrators (material writers, curriculum designers, etc.), researchers and experts. In reference to the TEM washback model, a muti-phase and multi-method research design has been developed to investigate the washback effects of TEM on experts, school administrator, teachers and students, to explore their perception and evaluation of TEM, as well as the impact of TEM on their teaching or learning practice. Moreover, various research methods will be employed, such as document analysis, questionnaire surveys, classroom observations and interviews. As indicated in Table 4.1, the current research can be conducted in the following stages. Firstly, such documents as the Teaching Syllabus, TEM syllabuses, and research findings on washback and TEM will be analyzed. The research findings of TEM studies and washback studies are summarized in Sects. 1.2.4 and 3.2 respectively and in Sect. 1.4 a comparison is made between the Teaching Syllabus and TEM characteristics. Next, in the second stage, large-scale questionnaire surveys are conducted among experts, program administrators, teachers and students, to collect their basic information and to investigate their perception and evaluation of

40

4 Research Framework for TEM Washback Study

Table 4.1 A multi-phase, multi-method research design for TEM washback study Focus TEM-related researches The Syllabus and TEM Syllabus Candidates

Teachers

Administrators Experts TEM lesson observation

Instruments Document analysis Document analysis Questionnaire survey among candidates Classroom observation Interview Questionnaire survey among teachers Classroom observation Interview Questionnaire survey among administrators Interview Questionnaire survey among experts Interview Classroom observation

TEM, as well as the washback of TEM on them. Altogether seven questionnaires are designed, one for experts, and the washback of TEM4 and TEM8 is investigated separately among relevant program administrators, teachers and students. Then, based on the above research findings, some sample classes are selected for observation, to explore the washback of TEM on classroom teaching and learning. Last, supplementary interviews will be conducted among selected participants, to seek answers to problems emerged in the questionnaire surveys and classroom observations.

4.2 Participants Major participants of the current study are experts, program administrators, teachers and students. Experts are specialists in English education who have either participated in the Teaching Syllabus design or TEM development, and some of them are also members of NACFLT, an advisory committee in the Ministry of Education which has been established to guide foreign language education in China. They have a deep understanding of the language education policies in China, the Teaching Syllabus and TEM. Moreover, some of them are also administrators in their respective universities and professors who teach English courses. Therefore, their opinions may provide us with illuminating insights into TEM and its washback, to help us understand this issue from the perspective of supervisors, administrators and teachers.

4.3 Instruments

41

Program administrators are either deans of the English department or course leaders in charge of English teaching in the foundation stage or advanced stage. They are responsible for curriculum design and course planning for English majors in their schools, so they are familiar with both the Teaching Syllabus and the practical situation in their schools. Furthermore, some of them are teachers, and often experienced teachers, so they are familiar with the teaching practice in their own schools as well. In this sense, a survey among them may help us understand how the Teaching Syllabus is implemented and how TEM has affected English program design in different universities throughout China. Teachers are those with TEM related teaching experiences, who have either taught TEM preparation courses or the required courses in the corresponding stage as listed in the Teaching Syllabus (e.g. teachers of fundamental English will be surveyed on the washback of TEM4 since fundamental English is a course recommended for English majors in the foundation stage). As practitioners, they have better understanding of how English is taught in classrooms and how TEM has affected their teaching practice. Students are TEM candidates who have just taken the test. They have prepared for TEM and must have some comments to make on their test preparation experience, such as how the preparation has affected their English learning and how their TEM preparation classes differed from non-preparation classes. An investigation among them will help us understand the washback of TEM on the learning process and products, which is of utmost importance in this research, since the fundamental aim of education is to help students learn.

4.3 Instruments Major research instruments used in the current study are questionnaires, classroom observations and interviews, which will be introduced in detail in the following part.

4.3.1 Research Design for the Questionnaire Surveys Questionnaire is a useful research tool to gather information from a large number of people. Therefore, in the current research, questionnaire surveys are conducted first to collect information from program administrators, teachers and students nationwide, to find out the general situation of TEM washback throughout China.

42

4 Research Framework for TEM Washback Study

4.3.1.1 The Survey Questionnaires Altogether seven questionnaires have been developed to collect information from experts (EQ, questionnaire for experts), TEM4 related administrators (AQ-4, questionnaires to investigate administrators’ opinions on TEM4), TEM4 related teachers (TQ-4, questionnaire to investigate teachers’ opinion on TEM4), TEM4 related students (SQ-4, questionnaire for TEM4 candidates), TEM8 related administrators (AQ-8, questionnaires to investigate administrators’ opinions on TEM8), TEM8 related teachers (TQ-8, questionnaire to investigate teachers’ opinion on TEM8), and TEM8 related students (SQ-8, questionnaire for TEM8 candidates). When designing the seven questionnaires, the researcher has mainly referred to the following resources: 1. The Teaching Syllabus, TEM Syllabuses, TEM test papers, related documents on English education policies; 2. Washback theories, such as Alderson and Wall’s hypotheses, Bailey’s washback model, and so on; 3. Empirical studies on TEM and washback, such as those studies analyzed in Sects. 1.2.4 and 3.2; 4. Communication with experts, administrators, teachers and students; 5. Personal experience as a teacher for undergraduate English majors. When designing and revising the questionnaires, the researcher has frequently consulted the testing experts, discussed with fellow PhD candidates, and communicated with colleagues teaching related courses. Every questionnaire had undergone two rounds of piloting and three rounds of revising before the operational version was finalized. The content and layout of the questionnaires is listed below in Table 4.2. (See Appendix I for the complete version of the questionnaires). Among the seven questionnaires, expert questionnaire employs quite a few short answer questions, 11 altogether, because this is a small scale survey conducted among five experts. Due to some practical reasons, the researcher was not able to interview the experts face to face, so a semi-open questionnaire was designed to investigate their opinions, and the experts all provided detailed answers to the open questions. In the other six questionnaires, most of the items employed are objective questions that are easy to answer, since they are intended for large-scale surveys. The content of the questionnaire surveys mainly covers five aspects. First, the stakeholders’ personal information is collected, because as pointed out in Green’s washback model (2007), different participants’ characteristics may lead to washback variance—stakeholders’ educational background and work experience may contribute to their perception of TEM and its washback, as well as what and how they are affected by TEM. Second, stakeholders’ evaluation of TEM is surveyed, such as whether the test content and method employed in TEM is well selected. This relates to the validity of TEM—whether TEM assesses what it is purported to assess—an essential factor that may determine the direction of TEM washback. Meanwhile, efforts are made to find out how TEM results are interpreted and used in different schools, because the use and interpretation of test results is an indis-

4.3 Instruments

43

Table 4.2 The seven questionnaires Questionnaire EQ

AQ-4

TQ-4

SQ-4

AQ-8

Theme Personal information; Perception and evaluation of the Teaching Syllabus, TEM Syllabuses, TEM, and TEM washback; General comments. Personal information; Perception, views on TEM4 and its washback; Use & interpretation of TEM4 scores; Teaching and test preparation practice; Factors affecting teaching; General comments. Personal information; Perception, views on TEM4 and its washback; Use & interpretation of TEM4 scores; Teaching and test preparation practice; Factors affecting teaching; General comments. Personal information; Perception, views on TEM4 and its washback; Use & interpretation of TEM4 scores; Related learning experience and effects; Factors affecting learning; General comments. Personal information; Perception, views on TEM8 and its washback; Use & interpretation of TEM8 scores; Teaching and test preparation practice; Factors affecting teaching; General comments.

Question type Multiple choice; Blank filling;

Short answer; Ranking. Multiple choice; Five-point Likert scale;

No. of questions 30

54

Ranking; Short answer.

Multiple choice; Blank filling;

52

Five-point Likert scale; Ranking; Short answer. Multiple choice; Blank filling;

55

Five-point Likert scale; Ranking; Short answer. Multiple choice; Five-point Likert scale;

54

Ranking; Short answer.

(continued)

44

4 Research Framework for TEM Washback Study

Table 4.2 (continued) Questionnaire TQ-8

SQ-8

Theme Personal information; Perception, views on TEM8 and its washback; Use & interpretation of TEM8 scores; Teaching and test preparation practice; Factors affecting teaching; General comments. Personal information; Perception, views on TEM8 and its washback; Use & interpretation of TEM8 scores; Related learning experience and effects; Factors affecting learning; General comments.

Question type Multiple choice; Blank filling;

No. of questions 52

Five-point Likert scale; Ranking; Short answer. Multiple choice; Blank filling;

57

Five-point Likert scale; Ranking; Short answer.

pensable part of test validity (Bachman 1990). Third, stakeholders’ perception of TEM washback is investigated, to find out how they think they are affected by TEM, such as their teaching and learning content, methods, etc. Fourth, attempts have been made to explore the TEM preparation practice of teachers and students, to identify the actual washback effects of TEM on English teaching and learning. Last, stakeholders’ opinions are sought as to what are the factors that affect their curriculum design, teaching or learning, and what are the factors that motivate them to learn or lead to the pressure in their teaching. These factors are considered because according to Wall (2000), various factors contribute to the outcome of teaching and learning, so it is essential to weigh these factors before assessing the washback effects of TEM. 4.3.1.2 Data Collection In early 2012, five expert questionnaires were distributed through personal contact, all returned and valid. The questionnaires for TEM8 administrators, teachers and candidates were distributed in March, 2012, together with the test booklets, by special delivery. The administrators’ questionnaire and teachers’ questionnaire were mailed to all the 809 colleges and universities that had students take TEM8, one copy each school for the administrator, and five copies for related teachers. These questionnaires had been distributed and finished before TEM was administered, and returned together with the test paper, but in a separate package. The students’

4.3 Instruments

45

questionnaire was distributed to 250 colleges and universities selected according to university types and regions, to ensure that these institutions included both national key universities and provincial universities in all the provinces in Mainland China. Seventy copies of the students’ questionnaire were mailed to each selected school and were completed in the examination room after the candidates had finished their TEM8 test. The questionnaires were collected soon afterwards and there was no time limit for doing the questionnaire. The questionnaires for TEM4 administrators, teachers and candidates were distributed in a similar way in May, 2012, together with TEM4 test booklets. The administrator’s questionnaire and teacher’s questionnaire were mailed to 869 TEM4 participating schools that year, and the candidate’s questionnaire was mailed to 250 selected schools, with 70 copies for each school. After the questionnaires were returned, they were carefully sorted, to single out invalid copies. The following three types of questionnaires were considered invalid: 1 . More than 20% of the questions were unfinished; 2. The same answer was selected for all items, for instance, selecting “4” for all five-point Likert scale questions; 3. There were some questions with obviously contradictory answers, for instance, when the answer to the question “whether you have used any TEM preparation materials” was “No”, the option of “several TEM preparation books” was selected for the next question “if yes, which TEM preparation book have you used”. The number of questionnaires distributed, returned and valid is listed below in Table 4.3. From Table 4.3, it can be noted that the return rate and valid rate are very high with experts, administrators and teachers. However, for students’ questionnaire, especially TEM8 candidates’ questionnaire, the return rate and valid rate are relatively low, 69.4% and 56.5% respectively, which is probably due to the fact that the candidates had just completed a 195-minite-long test, so some of them gave up answering the questionnaire halfway, leaving more than 20% of the questions unfinished (refer to Sect. 4.4.2.1 for further explanation).

Table 4.3 The number of questionnaires distributed, returned and valid Questionnaire EQ AQ-4 TQ-4 SQ-4 AQ-8 TQ-8 SQ-8

Distributed No. 5 869 4335 17,500 809 4045 17,500

Returned No. (%) 5 (100%) 789 (90.8%) 3758 (86.7%) 15,383 (87.9%) 750 (92.7%) 3479 (86.0%) 12,146 (69.4%)

Valid No. (%) 5 (100%) 786 (99.6%) 3574 (95.1%) 11,033 (71.7%) 724 (96.5%) 3222 (92.6%) 6863 (56.5%)

46

4 Research Framework for TEM Washback Study

4.3.1.3 Data Analysis The valid questionnaires were put into computer by the present researcher with the help of six college students and two graduate students. The answers to multiplechoice and five-point Likert-scale questions were processed by SPSS 17.0 (Statistical Product and Service Solutions 17.0), while answers to short-answer questions were collected, sorted, double-checked, and saved in a Microsoft Word file. Due to its limited number, only descriptive statistical analysis were applied to expert’s questionnaire, reporting such results as frequency and mean, while inferential statistical analysis such as factor analysis and t-test were not conducted. The experts’ perception and evaluation of TEM and its washback will be reported in Sect. 5.1. As for the other six questionnaires, both descriptive and inferential analyses were performed. In this chapter, the results of reliability test and factor analysis are reported, while a further discussion on the views of administrators, teachers and candidates will be provided in Chap. 5. Reliability Test A reliability test was run on the questionnaire data of all the objective items, including both five-point Likert scale items and multiple-choice questions. The questionnaires were analyzed separately, and for each questionnaire, two reliability tests were performed, one on all the objective items and the other on five-point Likert scale items only. The results are listed in the following tables (Table 4.4). From Table 4.4, it can be seen that Cronbach’s Alpha is above 0.8 for all questionnaires, and Cronbach’s Alpha for five-point Likert scale items only is above 0.9, which attests to the satisfactory internal consistency of the questionnaires. Factor Analysis An independent factor analysis was carried out on all the five-point Likert-scale items of each questionnaire to better explain the research findings. First, KMO and Bartlett’s Tests were conducted, the results of which proved satisfactory for further analysis (Table 4.5). In factor analysis, methods of principal components and varimax rotation were employed because they maximize variance and facilitate the interpretation of the constructs deduced (Lu 2006: 477–483). In view of the arbitrary nature of factor extraction, and practicality and meaningful interpretability, the following three criteria were observed in data reduction: (1) the eigenvalue was greater than 1 and there were more than 3 items in one factor; (2) factor loadings lower than 0.4 were deleted and not counted in any factor; (3) when double loadings occurred, decisions were made on meaningful interpretations.

4.3 Instruments

47

Table 4.4 Reliability test for AQ-4, TQ-4, SQ-4, AQ-8, TQ-8 and SQ-8 Cronbach’s Alpha N of Items AQ-4: all objective items .869 116 Five-point Likert scale items .962 58 TQ-4: all objective items .847 130 Five-point Likert scale items .931 71 SQ-4: all objective items .936 200 Five-point Likert scale items .961 118 AQ-8: all objective items .903 116 Five-point Likert scale items .961 58 TQ-8: all objective items .845 106 Five-point Likert scale items .934 49 SQ-8: all objective items .948 203 Five-point Likert scale items .966 121 Note: The number of items in the reliability test for five-point Likert scale items may exceed those in the questionnaires because some multiple-choice questions with five options arranged in degree are included

Based on the above criteria, factor analysis was conducted on the five-point Likert scale items in each questionnaire, and the factors extracted and variance explained are listed in Table 4.6. Based on the three criteria mentioned above, factors were extracted from each questionnaire and named according to the items included. The explanation of the rotated component matrix of administrators’ questionnaires on TEM4 and TEM8 will be grouped together, because they are of similar content and layout, and will be analyzed together in Chap. 5. The same applies to the teachers’ questionnaires and the students’ questionnaires. Program Administrators’ Questionnaires: AQ-4: Seven factors have been extracted and named: Factor 1: washback of TEM4 on English language teaching (16 items); Factor 2: intended use and interpretation of TEM4 scores (11 items); Factor 3: evaluation of TEM4 (8 items); Factor 4: unintended use and interpretation of TEM4 scores (6 items); Factor 5: attitude to unintended uses of TEM4 scores (4 items); Factor 6: familiarity with TEM4 (5 items). Factor 7: washback of the 2005 TEM4 revision on English language teaching (4 items);

48

4 Research Framework for TEM Washback Study

Table 4.5 KMO and Bartlett’s tests AQ-4 Kaiser-Meyer-Olkin Measure of Sampling Adequacy. Bartlett’s Test of Sphericity

TQ-4 Kaiser-Meyer-Olkin Measure of Sampling Adequacy. Bartlett’s Test of Sphericity

SQ-4 Kaiser-Meyer-Olkin Measure of Sampling Adequacy. Bartlett’s Test of Sphericity

AQ-8 Kaiser-Meyer-Olkin Measure of Sampling Adequacy. Bartlett’s Test of Sphericity

TQ-8 Kaiser-Meyer-Olkin Measure of Sampling Adequacy. Bartlett’s Test of Sphericity

SQ-8 Kaiser-Meyer-Olkin Measure of Sampling Adequacy. Bartlett’s Test of Sphericity

.940 Approx. Chi-Square Df Sig.

15334.058 1653 .000 .953

Approx. Chi-Square Df Sig.

58195.807 1176 .000 .948

Approx. Chi-Square Df Sig.

161363.846 2145 .000 .934

Approx. Chi-Square Df Sig.

12479.275 1653 .000 .943

Approx. Chi-Square Df Sig.

45531.756 1176 .000 .947

Approx. Chi-Square Df Sig.

125253.802 2415 .000

Originally there were 10 factors whose eigenvalues exceeded 1. Among them factors 8 and 9 consisted of three items only, and all the items in factor 10 cooccurred in factor 1. So, these three factors were deleted according to the three criteria mentioned above. The items co-occurred in the deleted factors were grouped into the factors named and would be explained in these factors.

.635 .623 .616 .606

Revealing problems in teaching

Promoting teaching reform

Reflecting the Teaching Syllabus implementation

Feedback to course design

.667 .660

Reflecting students’ language proficiency

Predicting future performance

.694 .692

Assessing learning outcome

.511

.603

.563

Washback from TEM4 test method

Washback from TEM4 marking criteria

Reflecting quality of teaching

.482 .573

.613

.605

Washback from TEM4 test content

Washback from TEM4 time allotment

.527

.705 .584

.696

.630

Washback on teaching aims

.684

.689

.705

.661

.688

.676

.775

.710

.743

.785

.856

.786

.871

.751

.764

.755

.781

.787

.821

.756

(continued)

Communality

Washback from TEM4 score allotment

10

.745

9

.771

8

.725

7

Washback on teaching depth

6

Washback on course design

5

.784

4

.772

3

Washback on teaching schedule

2

Washback on teaching attitude

.795

.793

Washback on teaching method

Washback on teaching content

.810

.809

Washback on teaching speed

1

Factor

Washback on teaching ideology

Item

AQ-4: Ten-factor rotated component matrix

Table 4.6 Rotated component matrix for the questionnaires

.639

.591

.778 .757

Washback from TEM4 writing revision

Washback from TEM4 vocabulary revision

.576

.792

.781

.609

.697 .694

Familiarity with the Teaching Syllabus

Familiarity with TEM4 marking criteria

.687 .646

.791 .762

.624

Familiarity with TEM4 Test Syllabus

.751 .707

Attitude to using TEM4 score to confer degrees

Attitude to using TEM4 score to rank schools

.789

.813

.692

.708

Familiarity with TEM4 time allotment

.875 .864

Attitude to using TEM4 score to assess leaders

.679

.570 .476

Affecting student future career

Affecting student further education

Attitude to using TEM4 score to assess teachers

.613

.719 .644

Affecting teacher evaluation

Affecting student ranking

.782 .753

.793 .782

.622

.651

.701

.626

.688

.663

.701

Affecting department ranking

.694 .636

Alignment with the Teaching Syllabus

Convenience of administration

.650 .680

Affecting university ranking

.716 .698

Quality of TEM4 time allotment

Quality of TEM4 test content

.729 .720

Quality of TEM4 marking criteria

Quality of TEM4 test methods

.776

.472

Promoting the Teaching Syllabus implementation

Quality of TEM4 score allotment

.592 .540

Feedback to teaching aims

Feedback to classroom teaching

Table 4.6 (continued)

2

.425

2.57

5

2.36

1.63

8

1.11

2.80

2.13

.706 .676

.750

.742

.694

.605

Washback on teaching content

Washback on teaching depth

Washback on teaching aims

Washback from TEM8 score allotment

.828

.720

.707

.772

.756

Washback on course design

Washback on teaching speed

.734

.772

.721

.785

(continued)

Communality

.717

10

.784

.546

1.92 67.78

.781

9

65.87

Washback on teaching schedule

7

2.34 63.74

Washback on teaching attitude

6

2.59 61.40

.823

4

4.07 56.00 58.80

.820

3

4.43 51.94

Washback on teaching ideology

1

Factor

4.86 47.51

Washback on teaching method

Item

AQ-8: Ten-factor rotated component matrix

6.57

36.08

36.08

Variance explained (%)

Accumulated variance explained (%)

42.65

3.81

20.93

Eigenvalue

1.24

.636 .635 .716

1.36

.798 .748

.649 2.82

.773 .799

Familiarity with the TEM4-Oral test syllabus

.676 .563

Quality of TEM test report

Fairness of administration

.743 .591

Affecting some teachers and students only

.760

1.50

.431

Overall washback on course design

Strictness of administration

.679

.723 .682

.470

Overall washback on students

Overall washback on teachers

.727 .534

Washback from TEM4 listening revision

Washback from TEM4 reading revision

.415 .403

.756

.652

.551

.565

.752

.735 .585

Affecting university ranking

Affecting students’ further education

.626

.769

.537

.775

Affecting department ranking

Strictness of administration

.595

.606 .598

Convenience of administration

Fairness of administration

.675 .598

.650 .647

Quality of TEM8 test content

Alignment with the Teaching Syllabus

.681 .578

.680 .661

Quality of TEM8 test methods

Quality of TEM8 marking criteria

.619 .599

.735 .706

.507

Quality of TEM8 time allotment

.498 .495

Predicting future performance

Quality of TEM8 test report

.637

.716

.713

.743

Quality of TEM8 score allotment

.576 .546

Assessing learning outcome

Promoting teaching reform

.638 .632

Feedback to classroom teaching

Promoting the Teaching Syllabus implementation

.751

.682 .680

Feedback to teaching aims

Feedback to course design

.763 .731

.703 .702

Revealing problems in teaching

.737

Reflecting quality of teaching

.479

.755

.707

Reflecting the Teaching Syllabus implementation

Washback from TEM8 test methods

.556

.828 .722

.564

.578

.567

Washback from TEM8 time allotment

Washback from TEM8 test content

Table 4.6 (continued)

.473

20.93

36.08

36.08

Eigenvalue

Variance explained (%)

Accumulated variance explained (%)

Reflecting students’ language proficiency

Affecting students’ future career

42.57

6.50

3.77

2.90 47.57

4.99

2.30 3.96

52.08 56.05

4.52

2.62

2.00 59.49

3.44

1.65 62.33

2.84

.484

.536

Overall washback on course design

Washback from marking criteria

64.75

2.41

1.40

.706

.744 .725

.444

Overall washback on students

Overall washback on teachers

.565 1.31 67.01

2.27

68.91

1.89

1.10

.544

.618

.646 .535

Familiarity with TEM8 marking criteria

Familiarity with TEM8-Oral Test Syllabus

.720

.663

.782

.810

.834

.747

.582

.653 .633

.762 .722

Familiarity with TEM8 time allotment

Familiarity with the Teaching Syllabus

.585 .764

.849

.716

Attitude to using TEM8 score to rank schools

Familiarity with TEM8 Test Syllabus

.718

.794

.875 .829

Attitude to using TEM8 score to assess leaders

Attitude to using TEM8 score to confer degrees

.301 .812

.893

Attitude to using TEM8 score to assess teachers

.506

.681

.707

Affecting some teachers and students only

Washback from TEM8 listening revision

.664

.712

Washback from inclusion of TEM8 General Knowledge

.759 .643

.770 .756

Washback from TEM8 writing revision

.569

.672 .576

Washback from TEM8 reading revision

.575 .563

Affecting teacher evaluation

Affecting student ranking

(continued)

6

7

8

9

10

11

12

14

15 .799

Communality

.788

.703 .717 .699 .600

.783

.782

.772

.731

.668

.591

Washback on teaching depth

Washback on course design

Washback on teaching aims

Washback on teaching speed

Overall washback on teachers

Overall washback on students

.761

.706

.696

.565

Quality of TEM4 test methods

Quality of TEM4 test content

Convenience of administration

.575

.619

.639

.693

.662

.746

.728

Quality of TEM4 time allotment

Quality of TEM4 marking criteria

.680 .703

.782

.765

Quality of TEM4 time allotment

Alignment with the Teaching Syllabus

.498

.716

.667

.788

.785

Washback on teaching method

.772

.751

Washback on teaching attitude

.804

.801

Washback on teaching ideology

Washback from TEM4 marking criteria

.760

.810

.808

Washback from TEM4 test content

Washback on teaching schedule

.771

.817

813

Washback on teaching content

Washback from TEM4 time allotment

.418

13

.750

5

.813

4

.843

3

.830

2

Washback from TEM4 test methods

1

Factor

Washback from TEM4 score allotment

Item

TQ-4: 15-factor rotated component matrix

Table 4.6 (continued)

.620 .681 .614 .590 .547

Assessing learning outcome

Promoting teaching reform

Predicting future performance

Assessing teaching quality

.692

.440

Reflecting students’ language proficiency

Difference in speaking activities

.710

.592

.699

8

.70

.699

.707

.638

.746 .625

Difference in writing activities

Difference in translation activities

.757 .698

.825 .779

.619

.679

.764

.826

.615

.705

.574

.684

.696

Difference in listening activities

.764 .577

Difference in explaining texts

Difference in other activities

.512 .706

Difference in reading activities

.878 .825

Difference in practicing textbook exercises

.548

Listening activities in TEM preparation classes

Difference in explaining textbook exercisesa

.675 .669

Other activities in TEM4 preparation classes

.703

Writing activities in TEM4 preparation classes

Speaking activities in TEM4 preparation classes

.755

Translation activities in TEM4 preparation classes

.518 .769

Quality of TEM4 test report

Reading activities in TEM4 preparation classes

(continued)

.480

.567

.585

.622 .621 .610

Affecting the organization of routine teaching activities

Affecting teacher evaluation

.727

Affecting student ranking

Affecting students’ further education

.563

Difference in working on test papers

.727

.814

Explaining test papers in TEM4 preparation classes

Affecting students’ future career

.847

Washback from TEM4 reading revision

Working on test papers in TEM4 preparation classes

.511

.586

.690

.661

.663

.769

.794

.495

.715 .695

.725 .645

Washback from TEM4 vocabulary revision

.808 .749

Washback from TEM4 listening revision

.759

Explaining texts in TEM4 preparation classes

.784

.780

.760

Explaining textbook exercises in TEM4 preparation classes

.602 .554

Washback from TEM4 writing revision

.793

.427

Familiarity with TEM4-Oral Test Syllabus

Working on textbook exercises in TEM4 preparation classes

.636

.736

.726 .708

Familiarity with the Teaching Syllabus

Familiarity with TEM4 marking criteria

.479 .733

.824

.471

.638 .605 .820

.428

Familiarity with TEM4 time allotment

.410

Familiarity with TEM4 Test Syllabus

Motivation to learn

Feedback to teaching

Table 4.6 (continued)

9.3

22.1

22.1

Variance explained (%)

Accumulated variance explained (%)

3.3

2.7

6

2.3

7

2.0

1.8

1.5

10

1.3

1.2

1.1

14

1.1

2.9

2.5

2.2

1.9

1.9

1.7

1.4

.737 .771 .817

.780

.753

.751

Washback from TEM8 test methods

Washback from TEM8 score allotment

.819

.817

.773

.763

.786

.776

(continued)

Communality

.754

15

67.8

Washback on teaching ideology

13

1.6 66.4

Washback from TEM8 time allotment

12

1.6 64.8

.764

11

61.4 63.1

.791

59.4

.788

9

57.5

Washback on teaching schedule

8

52.7 55.3

Washback on teaching depth

5

3.2 49.8

.792

4

3.8 46.5

.658

.792

3

4.6

1.1

Washback on teaching method

2

6.4 38.0 42.6

1.3

Washback on teaching aims

.807

.800

Washback on teaching attitude

1

Factor

Washback on course design

Item

TQ-8: 15-factor rotated component matrix

a

Difference between TEM4 preparation and non-preparation classes

31.5

15.7

15.7

Eigenvalue

.726 .717

.661

Attitude to using TEM4 score to confer degrees 4.6

.582 .663 .806 .727

.500

.637

Attitude to using TEM4 score to assess teachers

Difference in explaining test papers

.407

.450

Fairness of administration

Strictness of administration

.423

.404

.761

.760

.759

Speaking activities in TEM8 preparation classes

Working on textbook exercises in TEM8 preparation classes

Reading activities in TEM8 preparation classes

.426

.436

.427

Predicting future performance

Quality of TEM8 test report

Motivation to learn

.547

.533

Strictness of administration

.426

.615

.570

.555

Feedback to teaching

Promoting teaching reform

Assessing learning outcome

.713 .623

.691

.657

Fairness of administration

Convenience of administration

.460

.799

.811

.721

.659

.601

.715

.680

.736

.703

.752 .695

.756

.730

Quality of TEM8 time allotment

Alignment with the Teaching Syllabus

.526

.743 .755

.820

.763

Quality of TEM8 test content

Quality of TEM8 score allotment

.810 .806

.850

.826

Quality of TEM8 marking criteria

Quality of TEM8 test methods

.689 .421 .777

.716

Washback on teaching content

Washback from TEM8 marking criteria .663

.720 .756

.729

.724

Washback on teaching speed

Washback from TEM8 test content

Table 4.6 (continued)

.804

.626

.449

.470

.682 .742

.670 .504

Difference in speaking activities

Difference in explaining test papers

.632 .532

Familiarity with TEM8 marking criteria

Familiarity with TEM8-Oral Test Syllabus

Affecting teacher evaluation

.763 .732

Familiarity with the Teaching Syllabus

Familiarity with TEM8 time allotment

.800

Familiarity with TEM8 Test Syllabus

Difference in working on test papers

.725

.753 .744

.688

.769

.698

.751

.750

.820

.739

.742

.766

.671

.711

Difference in listening activities

.645

Translation activities in TEM8 preparation classes

.447

Difference in reading activities

.649

Other activities in TEM8 preparation classes

.708

.816

.666

Listening activities in TEM8 preparation classes

.764

.773

.691

Explaining texts in TEM8 preparation classes

.752

Difference in translation activitiesb

.706

Writing activities in TEM8 preparation classes

.814

Difference in writing activities

.746

Explaining textbook exercises in TEM8 preparation classes

(continued)

.419

Washback from TEM8 writing revision

.642

.596

.826

Attitude to using TEM8 score to evaluate teachers

Affecting students’ future career

.863

.496

Attitude to using TEM8 score to confer degrees

.533

.495

Assessing teaching quality

.724

Working on test papers in TEM4 preparation classes

Reflecting students’ language proficiency

.752

.514

Explaining test papers in TEM4 preparation classes

Washback from TEM8 listening revision

.679

.730

.742

.807

.637

.758

.709

.796

.583

.728

.754

.729 .659

.716

.455

Washback from inclusion of TEM8 General Knowledge

Difference in explaining texts

.828

.824

Difference in explaining textbook exercises

.517 .843

.885

.414

Washback from TEM8 reading revision

.745 .557

Difference in working on textbook exercises

.787 .511

Affecting student ranking

Affecting the organization of routine teaching activities

Table 4.6 (continued)

Affecting students’ further education

6.9

4.9

6.9

35.8 42.8

9.7

Factor 12

13

14

15

16

17

18

19

20

69.8

1.2

21

71.5

1.7

22

1.1

23

73.1

1.5

24

.69

(continued)

Communality

.60

11

66.2 68.1

1.2 1.7

Clarifying learning requirements

10

1.2 1.8

.63

9

64.3

1.3 1.9

Better arranging .70 learning activities

8

62.2

1.5 2.1

.65

7

60.0

1. 2.1

.767 .708

.74

6

57.4

1.8 2.5

.557 .466

.683 .657

Improving learning efficiency

4

54.7

1.9 2.7

.515

.72

3

51.4

2.3 3.3

.405

.517

Promoting .76 English learning

2

2.7 3.8

.454

.70

1

47.6

4.7

3.3

.462

.77

Improving English proficiency

Item

5

Difference between TEM8 preparation and non-preparation classes

SQ-4: 24-factor rotated component matrix

b

Accumulated variance explained (%) 26.1

18.5

26.1

Eigenvalue

.424

Variance explained (%)

Overall washback on teachers

Overall washback on students

Difference in other activities

.70

Difference in TEM4 grammar practice after class

.42

Enlarging vocabulary by reciting TEM4 words

.74

.67

Identifying weaknesses

Difference in TEM4 reading practice after class

.68

Improving language proficiency by working on test papers

.75

.69

Providing learning aims

Difference in TEM4 cloze practice after class

.69

Spending more time learning

Table 4.6 (continued)

.64

.71

.71

.42

.57

.62

.60

.59

.73

.59

Difference in reading grammar books after class

Washback on course selection

.62

Difference in reciting TEM4 vocabulary after class

.76

.65

Difference in TEM4 note writing practice after class

Washback on learning content

.68

Difference in TEM4 writing practice after class

.78

.68

Difference in TEM4 listening practice after class

Washback on learning schedule

.70

Difference in TEM4 dictation practice after class

.71

.71

.74

.62

.68

.72

.70

.77

.79

(continued)

.72

.72

.69

.65

.61

Preference for translation

Preference for listening

Preference for other activities

Preference for text explanation

.43

Overall washback on students

Preference for writing

.63

Washback on learning aims

.75

.71

Washback on learning depth

Preference for reading

.72

Washback on learning sequence

.75

.72

Washback on learning attitude

Preference for speaking

.73

Washback on learning method

Table 4.6 (continued)

.64

.64

.69

.68

.69

.72

.73

.47

.67

.69

.65

.69

.69

.67

.66

.63

Difference in reading practice after class

Time spent on speaking before TEM4

.41

Preference for doing TEM4 test papers

Difference in free writing practice after class

.53

Preference for TEM4 test paper explanation

.69

.57

Preference for doing textbook exercises

Difference in speaking practice after class

.61

Preference for textbook exercise explanation .40

.41

.57

.63

.65

.64

.60

.64

.66

.70

(continued)

.51

Difference in listening to the radio after class

.66

.65

.64

.63

TEM4 affecting reading proficiency

TEM4 affecting writing proficiency

TEM4 affecting dictation proficiency

.45

TEM4 affecting listening proficiency

.48

.54

TEM4 affecting oral proficiency

Time spent on listening to the radio before TEM4

.61

Time spent on reading books before TEM4

.47

.63

Time spent on free writing before TEM4

Table 4.6 (continued) .44

.65

.71

.71

.72

.53

.60

.63

.63

.70

.73

.60

.59

.53

Time spent on TEM4 dictation before TEM4

Time spent on TEM4 reading before TEM4

Time spent on TEM4 cloze before TEM4

Time spent on TEM4 grammar before TEM4

.57

TEM4 affecting vocabulary proficiency

.76

.58

TEM4 affecting MCQ test taking techniques

Time spent on TEM4 listening before TEM4

.61

TEM4 affecting grammar proficiency

.60

.68

.69

.74

.76

.69

.56

.68

(continued)

.63

.61

Listening proficiency improved in the advanced stage

Grammar proficiency improved in the advanced stage

.68

.65

Vocabulary enlarged in the advanced stage

Reflecting overall language proficiency

.67

Writing proficiency improved in the advanced stage

.68

.68

Speaking proficiency improved in the advanced stage

Reflecting listening proficiency

.70

Reading proficiency improved in the advanced stage

Table 4.6 (continued)

.65

.56

.62

.67

.65

.65

.67

.70

.66

.64

.59

.41

.40

Reflecting reading proficiency

Reflecting writing proficiency

Assessing learning outcome in the fundamental stage

Predicting future performance .78

.74

Reading activities in TEM4 preparation classes

Speaking activities in TEM4 preparation classes

.48

.67

Reflecting mastery of language knowledge

.76

.76

.59

.58

.52

.55

.57

(continued)

.67

.61

.50

Difference in listening activities

Difference in reading activities

Difference in speaking activities

.73

Difference in explaining texts .71

.78

Difference in doing textbook exercises

Difference in speaking activities

.79

.68

Writing activities in TEM4 preparation classes

Difference in explaining textbook exercisesc

.69

Listening activities in TEM4 preparation classes

Table 4.6 (continued)

.68

.68

.70

.73

.68

.74

.76

.69

.70

.71

.62

.74

.53

Explaining TEM4 test papers in TEM4 preparation classes

.45

.77

.54

Affecting student ranking

.59

.75

.56

No washback at all

.63

Time spent on TEM4 note-writing before TEM4

.73

Hindering learning

.67

.77

.75

Taking up the time for skill training

.59

Time spent on TEM4 writing before TEM4

.75

Satisfied with guessing the correct answer to MCQ

(continued)

.74

.69

.57

.52

Familiarity with TEM4-Oral Test Syllabus

Familiarity with TEM4 Test Syllabus

Familiarity with TEM4 marking criteria

.65

Difference in explaining textbook exercises

Familiarity with the Teaching Syllabus

.65

Difference in working on TEM4 test papers .40

.66

Working on TEM4 test papers in TEM4 preparation classes

Table 4.6 (continued)

.44

.66

.61

.65

.71

.72

.68

.57

Difference in translation activities .40

.55

.57

Translation activities in TEM4 preparation classes .45

.64

Difference in other activities

.71

Explaining texts in TEM4 preparation classes .67

.79

Doing textbook exercises in TEM4 preparation classes

Other activities in TEM4 preparation classes

.79

Explaining textbook exercises in TEM4 preparation classes

.71

.68

.72

.69

.64

.77

.78

(continued)

.50

.66

.71

.64

Familiarity with score allotment

Time spent on TEM4 vocabulary before TEM4

.71

.66

.79

.81

.58

.59

.62

.59

Familiarity with time allotment

.83

.46

Time spent on extra-curricular TEM4 training course

Affecting students’ further education

.64

Mock test papers done

.85

.65

Time spent daily on TEM4 preparation after class

Affecting students’ future career

.71

Time spent on TEM4 preparation after class

Table 4.6 (continued)

4.3

3.6

3.0

2.7

2.3

41.7 44.1

2.6

.76

.76

.73

Preference for reading

Preference for speaking

1

Factor

Preference for writing

Item

2

3

4

SQ-8: 24-factor rotated component matrix

c

5

6

7

Difference between TEM4 preparation and non-preparation classes

3.1

27.1 32.3 36.0 39.1

3.6

Accumulated 21.3 variance explained (%)

5.2

5.8

2.4

2.3 2.0

8

9

46.2 48.2

2.0

2.1

10

50.0

1.8

1.9

11

51.6

1.6

1.9

12

53.3

1.6

1.7 1.4

1.6

13

14

54.7 56.2

1.4

1.5

15

57.4

1.2

1.4

1.3 1.1

16

17

58.7 59.8

1.2

1.2

1.2 1.0

18

19

60.9 61.9

1.0

1.1

20

62.9

0.9

1.1

21

63.8

0.9

1.0

22

64.7

0.9

0.8

23

24

65.6 66.5

0.9

1.0

1.0

6.1

6.8

25.1

Variance 21.3 explained (%)

Eigenvalue

.58 .63

.40

.42

Attitude to using TEM4 score to confer degrees

Causing pressure

Time spent on reading grammar books before TEM4

.76

.76

.75

(continued)

Communality

.51

.58

.67

.65

.64

.63

.54

Preference for text explanation

Preference for TEM8 test paper explanation

Preference for doing textbook exercises

Preference for doing TEM8 test papers

.76

.66

Preference for textbook exercise explanation

Difference in reading activities

.68

Preference for other activities

.77

.69

Preference for listening

Difference in writing activitiesd

.71

Preference for translation

Table 4.6 (continued)

.76

.77

.62

.70

.65

.65

.72

.64

.73

.73

.74

.72

.72

.69

.67

.66

.63

.59

.53

Difference in listening activities

Difference in speaking activities

Difference in translation activities

Difference in explaining texts

Difference in explaining textbook exercises

Difference in doing textbook exercises

Difference in other activities

Difference in explaining TEM8 test papers

Difference in doing TEM8 test papers .53

.50

.69

.68

.65

.69

.72

.69

.74

.71

.73

(continued)

.78

.75

.74

.73

.72

.72

.68

.44

Washback on learning schedule

Washback on learning attitude

Washback on course selection

Washback on learning method

Washback on learning content

Washback on learning aims

Washback on learning speed

Overall washback on students

Difference in TEM8 writing practice after classe

.78

Washback on learning depth

Table 4.6 (continued)

.67

.73

.54

.66

.70

.72

.69

.69

.71

.75

.75

.66

.65

.65

.64

.64

.63

.63

Difference in TEM8 reading practice after class

Difference in TEM8 mini-lecture practice after class

Difference in reading culture books after class

Difference in TEM8 general knowledge after class

Difference in TEM8 translation practice after class

Difference in listening to the radio after class

Difference in TEM8 error-correction after class

.40

.72

.67

.74

.68

.62

.72

.70

(continued)

.70

.69

.67

.67

TEM8 affecting reading proficiency

TEM8 affecting writing proficiency

TEM8 affecting grammar proficiency

.53

Difference in speaking practice after class

TEM8 affecting listening proficiency

.58

Difference in TEM8 listening practice after class

.70

.59

Difference in reading practice after class

TEM8 affecting translation proficiency

.61

Difference in free writing practice after class

Table 4.6 (continued)

.41

.69

.66

.71

.69

.68

.73

.70

.62

.73

.73

Improving learning efficiency

.56

TEM8 affecting vocabulary proficiency

.74

.58

TEM8 affecting general knowledge proficiency

Clarifying learning requirements

.59

TEM8 affecting speaking proficiency

.74

.62

TEM8 affecting MCQ test taking techniques

Better arranging learning activities

.63

TEM8 affecting note-taking proficiency

.67

.67

.68

.71

.63

.62

.62

.66

(continued)

.68

.68

Time spent on TEM8 error-correction before TEM8

.52

Improving language proficiency by doing test papers

Time spent on TEM8 mini-lecture before TEM8

.62

Providing learning aims

.69

.65

Identifying weaknesses

Time spent on TEM8 reading before TEM8

.66

Spending more time learning

.71

.66

Improving English proficiency

Time spent on TEM8 listening before TEM8

.70

Promoting English leaning

Table 4.6 (continued)

.70

.72

.68

.72

.55

.63

.58

.64

.64

.70

.80

.80

.79

.77

.76

Writing activities in TEM8 preparation classes

Speaking activities in TEM8 preparation classes

Listening activities in TEM8 preparation classes

Translation activities in TEM8 preparation classes

.68

Reading activities in TEM8 preparation classes

Time spent on TEM8 general knowledge before TEM8

.72

.71

.78

.80

.78

.66

(continued)

.73

.72

.71

.70

.70

.62

.53

.52

Reflecting reading proficiency

Reflecting listening proficiency

Reflecting writing proficiency

Reflecting overall language proficiency

Assessing learning outcome

Reflecting general knowledge proficiency

Predicting future performance

.62

Reflecting translation proficiency

Other activities in TEM8 preparation classes

Table 4.6 (continued)

.63

.47

.63

.68

.63

.62

.64

.67

.59

.70

Listening proficiency improved in the advanced stage

.51

Time spent reading culture books before TEM8

.71

.51

Time spent on TEM8 writing before TEM8

Speaking proficiency improved in the advanced stage

.68

Time spent on listening to the radio before TEM8

.72

.71

Time spent on speaking before TEM8

Reading proficiency improved in the advanced stage

.72

Time spent on reading before TEM8

.42

.72

Time spent on free writing before TEM8

.50

.73

.71

.71

.60

.75

.65

.69

.65

.70

(continued)

.78 .71

.52 .52

Taking up time for skill training

Satisfied with guessing the correct answer for MCQ

No washback at all

Affecting student ranking

.54

General knowledge improved in the advanced stage .79

.61

Translation ability improved in the advanced stage

Hindering language learning

.68

Writing proficiency improved in the advanced stage

Table 4.6 (continued)

.58

.61

.59

.67

.69

.61

.69

.70

Doing textbook exercises in TEM8 preparation classes

.74

.53

Familiarity with TEM8 Test Syllabus

Explaining TEM8 papers in TEM8 preparation classes

.56

Familiarity with TEM8 marking criteria

.75

.70

Familiarity with TEM8-Oral Test Syllabus

Doing TEM8 test papers in TEM8 preparation classes

.73

Familiarity with the Teaching Syllabus

.74

.74

.69

.68

.64

.64

.63

.66

(continued)

.69

.62

The number of mock test papers done

.80

Affecting students’ further education

Time spent on TEM8 preparation after class

.82

.63

Familiarity with TEM8 score allotment

Affecting students’ future career

.65

.65

Explaining texts in TEM8 preparation classes

Familiarity with TEM8 time allotment

.70

Explaining text exercises in TEM8 preparation classes

Table 4.6 (continued)

.63

.59

.74

.76

.67

.68

.65

.75

27.3

8.2

5.6

3.5

3.4

2.5

2.4

2.2

2.1

1.9

1.7

1.7

1.5

1.4

1.3

1.2

1.2

1.2

1.0

1.0

1.0

1.0

2.9

Eigenvalue

.73

.63

4.1

.60

Enlarging vocabulary by reciting TEM8 words

Attitude to using TEM8 score to confer degrees

Causing pressure

.51

Difference in TEM8 vocabulary practice after class

.49

.70

.53

Time spent on TEM8 vocabulary after class

.44

.48

Time spent daily on TEM8 preparation

Time spent on TEM8 translation after class

.52

Time spent on extra-curricular TEM8 training course

.64

.60

.58

.72

.78

.75

.58

.52

(continued)

2.4

43.6 46.0

2.8

Difference between TEM8 preparation and non-preparation classes

40.8

2.9 48.1

2.0 50.1

2.0 51.9

1.8 53.7

1.7

e

Difference in after class learning efforts between TEM8 preparation and non-preparation stages

d

29.4 34.1 37.9

3.8

Accumulated 22.6 variance explained (%)

4.6

6.8

Variance 22.6 explained (%)

Table 4.6 (continued)

55.3

1.5

1.4

56.7 58.1

1.4 59.4

1.2

1.0

60.6 61.7

1.1

1.0

62.7 63.8

1.0 64.8

1.0 65.7

0.9 66.6

0.8

0.8

67.4 68.2

0.8

4.3 Instruments

91

AQ-8: Seven factors have been extracted and named: Factor 1: washback of TEM8 on English language teaching (16 items); Factor 2: intended use and interpretation of TEM8 scores (12 items); Factor 3: evaluation of TEM8 (9 items); Factor 4: unintended use and interpretation of TEM8 scores (6 items); Factor 5: washback of the 2005 TEM8 revision on English language teaching (4 items); Factor 6: attitude to unintended uses of TEM8 scores (4 items); Factor 7: familiarity with TEM8 (5 items). Altogether 10 factors whose eigenvalues exceeded 1 have been extracted. Excluding the items that co-occurred in other factors, the items included in factor 8, 9 and 10 were less than three, so they were deleted. The items co-occurred in the deleted factors were grouped into the factors named and explained accordingly. According to the seven factors extracted from program administrators’ questionnaires, the further discussion in Chap. 5 will be conducted from the following two aspects: 1. evaluation of TEM (Factors 2, 3, 4, 5 and 6 in AQ-4, Factors 2, 3, 4, 6 and 7 in AQ-8); 2. perception of TEM washback (Factors 1 and 7 in AQ-4, Factors 1 and 5 in AQ-8). Teachers’ Questionnaires TQ-4: Eleven factors have been extracted and named: Factor 1: washback of TEM4 on English language teaching (16 items); Factor 2: evaluation of TEM4 (11 items); Factor 3: language skill training activities in TEM4 preparation classes (5 items); Factor 4: difference in textbook activities between TEM preparation and non-preparation classes (4 items); Factor 5: difference in skill training activities between TEM preparation and nonpreparation classes (5 items); Factor 6: intended use and interpretation of TEM4 scores (7 items); Factor 7: familiarity with TEM4 (5 items); Factor 8: washback of the 2005 TEM4 revision on English language teaching (4 items); Factor 9: exercise practice in TEM4 preparation classes (4 items); Factor 10: unintended use and interpretation of TEM8 scores (6 items); Factor 11: attitude to unintended uses of TEM8 scores (2 items); Altogether 15 factors whose eigenvalues exceeded 1 have been extracted. Factors 8, 11, 13 and 14 included no more than three items, so they were deleted. Factor 15 included two items only, but it has been considered a separate factor since the two items in it were on the teachers’ attitude to unintended uses of TEM scores, a factor in program administrators’ questionnaires. In this sense, this factor has been included for further explanation in Chap. 5.

92

4 Research Framework for TEM Washback Study

TQ-8: Nine factors have been extracted and named: Factor 1: washback of TEM8 on English language teaching (15 items); Factor 2: evaluation of TEM8 and intended use and interpretation of TEM8 scores (17 items); Factor 3: textbook and skill training activities in TEM8 preparation classes (9 items); Factor 4: difference in TEM preparation and skill training activities between TEM preparation and non-preparation classes (7 items); Factor 5: familiarity with TEM4 (5 items); Factor 6: unintended use and interpretation of TEM8 scores (3 items); Factor 7: difference in textbook activities between TEM preparation and non-preparation classes (4 items); Factor 8: washback of the 2005 TEM8 revision on English language teaching (4 items); Factor 9: attitude to unintended uses of TEM8 scores (2 items); Altogether 15 factors whose eigenvalues exceeded 1 have been extracted. However, Factors 9, 10, 12, 13, 14 and 15 included no more than three items, so they were not explained separately. Factor 6, unintended use and interpretation of TEM8 scores, was considered a separate factor because it was treated as a separate factor in all other questionnaires. It is the same with Factor 11, the teachers’ attitude to unintended uses of TEM scores. Moreover, Factor 2 consisted of both teachers’ evaluation of TEM8 and the intended use and interpretation of TEM8 scores, which might be attributed to the fact that the quality of TEM8 can be closely related to how the scores are used. Nevertheless, these shall be explained separately in further discussion. Based on the factors extracted from teachers’ questionnaires, the further discussion in Chap. 5 will be conducted from the following aspects: 1. evaluation of TEM (Factors 2, 6, 7, 10 and 11 in TQ-4, Factors 2, 5, 6 and 9 in TQ-8); 2. perception of TEM washback (Factors 1 and 8 in TQ-4, Factors 1 and 8 in TQ-8); 3. TEM washback on English classroom teaching (Factors 3, 4, 5 and 10 in TQ-4, Factors 3, 4 and 7 in TQ-8). Students’ Questionnaires SQ-4: Seventeen factors have been extracted and named: Factor 1: possible positive effects of TEM4 (10 items); Factor 2: difference in after-class TEM related learning activities between TEM preparation and non-preparation period (9 items); Factor 3: washback of TEM4 on English language learning (9 items); Factor 4: preference for classroom activities (11 items); Factor 5: non-TEM related activities in TEM4 preparation classes and their difference between TEM preparation and non-preparation classes (8 items); Factor 6: the effect of TEM4 on the learning effects (8 items); Factor 7: TEM4-related learning activities conducted after class (5 items); Factor 8: language proficiency improved in the foundation stage (6 items);

4.3 Instruments

93

Factor 9: evaluation of TEM4 (7 items); Factor 10: language skill training activities in TEM4 preparation classes (4 items); Factor 11: difference in skill training activities between TEM preparation and nonpreparation classes (4 items); Factor 12: possible negative effects of TEM4 (5 items); Factor 13: TEM related activities in TEM4 preparation classes and their difference between TEM preparation and non-preparation classes (4 items); Factor 14: familiarity with TEM4 documents (4 items); Factor 15: translation and other activities in TEM4 preparation classes and their difference between TEM preparation and non-preparation classes (4 items); Factor 16: TEM preparation activities after class (4 items); Factor 17: attitude to unintended uses of TEM8 scores (1 item); Altogether 24 factors whose eigenvalues exceeded 1 have been extracted. Factors 11, 14, 17, 20, 21, 22 and 23 included no more than three items, so they were deleted. Factor 24 included one item only, but it has been considered a separate factor since the item in it were on the students’ attitude to unintended uses of TEM scores, a factor in program administrators’ and teachers’ questionnaires. In this sense, this factor has been included for further explanation in Chap. 5. SQ-8: Fifteen factors have been extracted and named: Factor 1: preference for classroom activities (11 items); Factor 2: difference in classroom activities between TEM preparation and non-preparation classes (11 items); Factor 3: washback of TEM8 on English language learning (9 items); Factor 4: difference in after-class learning activities between TEM preparation and non-preparation period (12 items); Factor 5: the effect of TEM8 on the learning effects (10 items); Factor 6: possible positive effects of TEM8 (9 items); Factor 7: TEM4-related learning activities conducted after class (5 items); Factor 8: language skill training activities in TEM8 preparation classes (6 items); Factor 9: evaluation of TEM8 (8 items); Factor 10: non-TEM related activities conducted after class (6 items); Factor 11: language proficiency improved in the advanced stage (6 items); Factor 12: possible negative effects of TEM8 (5 items); Factor 13: familiarity with TEM8 documents (4 items); Factor 14: TEM preparation activities after class (4 items); Factor 15: attitude to unintended uses of TEM8 scores (1 item); Altogether 24 factors whose eigenvalues exceeded 1 have been extracted. However, Factors 14, 15, 16, 17, 19, 20, 21, 22 and 24 included no more than three items, so they were not explained separately. Factor 23, the students’ attitude to unintended uses of TEM scores, has been considered a separate factor since it was explained as a separate factor in program administrators’ and teachers’ questionnaires. Based on the factors extracted from students’ questionnaires, the further discussion in Chap. 5 will be conducted from the following aspects:

94

4 Research Framework for TEM Washback Study

1 . evaluation of TEM (Factors 9, 14 and 17 in SQ-4, Factors 9, 13 and 15 in SQ-8); 2. perception of TEM washback (Factors 1, 3 and 12 in SQ-4, Factors 3, 6 and 12 in SQ-8); 3. TEM preparation (Factors 2, 4, 5, 7, 10, 11, 13, 15 and 16 in SQ-4, Factors 1, 2, 4, 7, 8, 10 and 14 in SQ-8); 4. Learning effects (Factors 6 and 8 in SQ-4, Factors 5 and 11 in SQ-8). Generally speaking, the factor structure of the questionnaires complies with the original design, with only a few exceptions. First, in teachers’ questionnaires and students’ questionnaires, the various classroom activities, which had been designed to be a single factor, were divided into several factors, such as TEM related activities, textbook activities and skill training activities. In this sense, some factors (e.g. Factor 14 in SQ-4 and SQ-8) contain two items only. Although they were not listed as a separate factor, they would be explained with other activities in further discussion in Chap. 5. The same practice has been applied to the factors of “difference between TEM preparation classes and non-preparation classes”, “after-class learning activities”, and “difference in after-class learning activities”. Second, in students’ questionnaires, familiarity with TEM was classified as two factors—familiarity with TEM documents and familiarity with TEM practice. There were only two items in familiarity with TEM practice, so it was not considered a separate factor. However, in further discussion, the items of these two factors would be grouped together. Last, in AQ-8, the item of “quality of TEM8 test report” was grouped with other items on “TEM8 quality” designed to elicit participants’ evaluation of TEM8 (Factor 3), but it fell into Factor 2, “intended use and interpretation of TEM8 scores”. This is probably attributed to the fact that a good test report is essential for the intended use and interpretation of test scores, but it would still be explained with other qualities of TEM8. In a word, the results of factor analysis testified the design of the questionnaires and helped simplify explanation, and the further discussion would be conducted according to both the factor analysis results and questionnaire design.

4.3.2 Classroom Observations The advantage of questionnaire surveys is that information can be collected from a large number of people over a short period of time. Therefore, in the current research, questionnaire surveys were conducted first to paint a rough picture of the general situation throughout China. Based on the research findings of the questionnaire surveys, pertinent classroom observation plans were developed, selecting appropriate English classes to be observed and designing relevant classroom observation schemes.

4.3 Instruments

95

4.3.2.1 English Classrooms Observed In the current study, altogether 52 classes have been observed and each class is around 45 min long. The following three principles were followed to select proper English classrooms to be observed.. First, both top universities and less established schools were selected. Questionnaire survey findings indicate that students of various language proficiency levels have been affected differently by TEM, and that teachers from schools of different levels responded variedly to TEM (see Chap. 5). Therefore, in classroom observation, special effort has been made to select English classrooms from both universities and colleges (schools that offer a three-year program for their English majors). However, due to the fact that TEM8 is not required for candidates taking a three-year program, only university classrooms are observed for TEM8 related courses. Second, both TEM-preparation courses and non-preparation courses were selected. As shown in the research findings of the questionnaire surveys (see Chap. 5), some schools offered TEM preparation courses, mainly comprehensive ones teaching all language skills covered in TEM. Moreover, teachers and candidates claimed that TEM preparation courses differed from non-preparation ones. In this sense, both types of courses have been observed to investigate whether they do differ and whether they differ in the same way as claimed by teachers and students. Since TEM is a comprehensive test and the TEM preparation courses observed are often comprehensive ones, equivalent non-preparation courses should also be comprehensive ones which deal with all language skills. Therefore, fundamental English and advanced English courses have been selected. Moreover, it is common practice for some schools to offer TEM preparation in fundamental English and advanced English courses in the semester when TEM4 and TEM8 are held or in the previous semester. Therefore, the TEM preparation classes observed here are of two types, one is in a separate course named TEM Preparation Course, and the other is the TEM preparation classes delivered in fundamental English or advanced English courses. In this sense, whether a class is considered TEM preparation class or nonpreparation class is determined by the teaching content of that class. Last, teachers who teach both TEM preparation and non-preparation courses were selected. It is hoped that the TEM preparation and non-preparation courses can be delivered by the same teacher, so that teacher variance can be limited. 4.3.2.2 Teachers Observed Based on the above principles, six teachers were selected for observation. They are all English teachers with TEM preparation experience. They have taught both TEM preparation and non-preparation classes, but their educational background and teaching experience differ considerably (refer to Sect. 6.1 for detailed information).

96

4 Research Framework for TEM Washback Study

4.3.2.3 The Observation Scheme To better conduct the observation, it is necessary to develop an observation scheme so that attention can be paid to the relevant aspects of the classroom events. The following sources have been exploited to develop an observation scheme for the current research: 1. COLT (Communicative Orientation in Language Teaching), an observation scheme developed by Frohlich et al. (1985) to compare “the effects of instructional differences on the development of L2 proficiency”; 2. Observation schemes used in previous washback studies (Cheng 2005; Gu 2007; Qi 2004; Wall 2005; Watanabe 1997, and so on); 3. Related theories on washback, as well as the washback models; 4. The Teaching Syllabus, TEM Syllabuses, and other documents on English teaching at the tertiary level in China; 5. Extensive consultation with experts, discussion with PhD candidates, and communication with English teachers and students; 6. The researcher’s personal experience as an English teacher. Based on the above sources, an analytic observation scheme was developed, consisting of eight categories: content, organization, materials, classroom language, teacher’s communicative features, student’s communicative features, teacher’s feedback, and atmosphere (see Appendix II). In the first three categories, the time consumed is calculated and recorded in terms of percentage, five-point Likert scale questions are employed in categories four to seven, and in the last category “atmosphere”, the frequency of students’ laughter is counted. A summary of the content and format of the observation scheme is as follows: 1. Content, the teaching and learning content, such as reading practice, listening practice, TEM writing training, and so on; 2. Organization, the way the class is organized, such as teacher lecture, student presentation, pair work, and so on; 3. Materials, e.g. the teaching materials used, such as textbooks, TEM preparation materials, and so on; The items in the above three categories are calculated by time. For example, in a 45-min class, if 10 min is devoted to student presentation, then 20% will be recorded (10/45 = 0.22, which is round down to 20%). 1. Classroom language, the language used by teachers and students, whether it is the target language English or the mother tongue Chinese; 2. Teacher’s communicative features, how the teacher communicates with the students, such as the authenticity of information and the length of questions; 3. Student’s communicative features, how the student communicates with the teacher, such as the authenticity of questions and the length of answers; 4. Teacher’s feedback, how the teacher responds to students’ answers, whether it is positive or negative, and whether the teacher focuses on meaning or on form;

4.3 Instruments

97

Categories four to seven employ five-point Likert scale to measure the frequency of each activity, such as whether the teacher extremely frequently, frequently, sometimes, occasionally or seldom (from 5 to 1) uses English in class. 5. Atmosphere, how often students laugh in class. This is counted by how many times students laugh. The observation scheme had been piloted in three English classes and revised for three times before it was finalized. 4.3.2.4 Data Collection Altogether 52 classes of the six teachers have been observed and it took several semesters (from 2012 to 2014) to observe all the classes, because in some schools, TEM preparation and non-preparation courses were not open in the same semester. To avoid interfering in the classes, the researcher did not observe in the classrooms, but used digital voice recorder to record all the classes, which were analyzed by the researcher afterwards. Of the six teachers, two agreed to have their class video recorded once, so two individual class hours were recorded for each of them, and the recording was transcribed to provide a clearer picture of the characteristics of the English classes (see Appendix II for the transcript). To achieve variety, the classes video recorded are one TEM8 preparation class and one non-preparation class of fundamental English. 4.3.2.5 Data Analysis The analysis was conducted mainly by coding the recorded classes with the help of the observation scheme. The following procedures had been observed when coding each lesson: 1. Listen to the recording once without doing any coding in the observation scheme. Just note down on a blank piece of paper the major classroom events—what was taught, how they were organized, and what kind of teaching materials were used. In this way, the class was divided into several parts; 2. Listen to the recording again, using a stop watch and the time meter in the computer to record the time spent on each activity. After listening, calculate the overall time spent on each item in the content, organization and materials, and fill in the blanks in the first three categories; 3. Listen to the recording once more, focusing on the communicative features of the classroom activities, such as the English used, how the teacher and students responded to each other, and what the teacher focused on when they provided students with feedback information; 4. Listen to the recording for the last time, check the previous information and count how many times students laugh.

98

4 Research Framework for TEM Washback Study

After all the classes had been coded, data were put into computer and processed by SPSS 17.0. Findings of classroom observation will be discussed in Chap. 6.

4.3.3 Interviews Interviews were a supplementary instrument used in this research. They were conducted to facilitate the development of the questionnaires and the classroom observation scheme, to identify possible causes of conflicting research findings, and to verify statistical results in questionnaire surveys. The main participants are teachers (including the six teachers observed), students (including those whose English classes had been observed), PhD candidates (most of whom were also teachers), and school administrators. The interviews were either conducted individually (mainly with teachers and school administrators) or in groups (mainly with students and PhD candidates). Four major rounds of interviews were carried out before and after the questionnaire surveys and the classroom observations. Pre-questionnaire and pre-observation interviews were conducted to facilitate instrument development, so the results were the questionnaires and the observation scheme. The participants were asked about their opinions on the questionnaires and the observation scheme. Therefore, no fixed questions had been designed, and neither had the interviews been recorded. Post-questionnaire and post-observation interviews were meant to solve puzzles, so relevant questions had been designed before the interview (see Appendix III for the interview questions). Moreover, the interview was recorded using a digital voice recorder for further analysis. Key findings of the interview had been collected, sorted, double-checked, and saved in a Microsoft Word file, which shall be discussed in Chaps. 5 and 6.

4.4 Considerations for Validity and Reliability Washback is such a complicated phenomenon that both qualitative and quantitative research methods have been employed to collect data in previous research studies. In the present research, qualitative data are collected through classroom observation transcription, interviews and the open questions in the questionnaires, and quantitative data are collected through objective items in the questionnaires and classroom observation coding schemes. In such a multifarious research, it is necessary to establish reliability and validity of the quantitative data and credibility, transferability, dependability and confirmability of the qualitative data (Watanabe 2004). The 1995 UCLES (University of Cambridge Local Examinations Syndicate) brief and resultant papers on IELTS impact recommend the following key actions and approaches to ensuring validity and reliability of the impact research instrumentation and data (quoted in Hawkey 2006: 36):

4.4 Considerations for Validity and Reliability

99

1. a careful exploration of constructs and advance hypotheses by “insiders” and “outsiders” before the drafting of instruments; 2. checks that every item contributes to the corroboration or refutation of washback and impact hypotheses; 3. the prediction of participant responses to compare with actual responses at piloting and trialing stages, with adjustments to instruments made accordingly; 4. the use of expert and stakeholder judgments on draft instruments; 5. the comparison of draft instruments with other instruments; 6. the use of a combination of methods of data collection, approaching validation from a number of angles with data triangulation used as a check. In the present research, practical measures were taken in each stage to ensure the validity and reliability of the instrumentation and the data collected.

4.4.1 The Development of Research Instruments The following measures have been taken in the development of the research instruments—the questionnaires, classroom observation scheme and interview guiding questions. 4.4.1.1 The Questionnaires Questionnaires are a useful tool to collect information from a large number of people. However, it is difficult to design a valid and reliable questionnaire because the researcher and the participants can’t communicate in person when the questionnaire surveys are conducted. Therefore, it is of vital importance for the researcher to expect all possibilities and try to avoid ambiguity, misunderstanding and missing information. It does not mean that the questionnaire should be perfect, but a great deal of effort should be devoted to make the questionnaires as valid and reliable as possible. In this regard, the current researcher has taken the following steps in the designing, trialing and revision of the seven questionnaires used in the research. First, before drafting the questionnaires, a detailed analysis had been made on the washback theories, such as the definitions, dimensions and models, to explore the construct of washback studies. Second, a careful review was carried out on all the available empirical studies on washback, to gain insight from relevant questionnaire surveys conducted before. Third, the researcher had consulted with experts and school administrators, discussed with PhD candidates and colleagues, and communicated with teachers and students, to obtain a clear picture of the possible TEM washback. Last, two rounds of piloting were performed among school administrators, teachers and students, and the expert’s questionnaire was only piloted once due to the limited number of experts available. Based on the results of the pilot study, a com-

100

4 Research Framework for TEM Washback Study

parison was made between the prediction of participant responses and their actual responses. Meanwhile, every item was checked again to discern their contribution to the corroboration and refutation of the washback hypotheses established. Based on the above findings, relevant revisions were made accordingly and the operational version of the questionnaires was finalized for large scale administration. 4.4.1.2 The Classroom Observation Scheme To explore the washback of TEM, it is necessary to obtain direct information from the language classrooms. However, it remains a difficult problem as to which instrument to use. Numerous classroom observation schemes have been developed to describe classroom teaching and learning practice, such as Communicative Orientation of Language Teaching Observation Scheme (Frohlich et al. 1985) and Target Language Observation Scheme (Ullmann and Geva 1982), but they have been designed to capture the classroom features that the researchers are interested in. Therefore, these observation schemes are hardly applicable to the current research context. In this sense, the present researcher has to develop a new observation scheme to examine the washback effects of TEM on classroom teaching and learning. The following measures have been taken to ensure the validity and reliability of the observation scheme. First, before designing the observation scheme, the researcher had made a detailed analysis of the relevant observation schemes used in previous washback studies (Cheng 2005; Gu 2007; Qi 2004; Wall 2005; Watanabe 1997; etc.), as well as the established observation schemes in language teaching and learning research (Frohlich et al. 1985; Ullmann and Geva 1982; etc.). Second, predictions had been made on the possible washback of TEM, based on the following research findings: (1) the analysis of the Teaching Syllabus, TEM Test Syllabuses, and other relevant documents; (2) the discussion with experts, PhD candidates, school administrators, teachers and students; and (3) empirical findings of previous washback studies. Last, after the observation scheme was drafted, it was piloted in three English classrooms, one TEM4 non-preparation, one TEM8-non-preparation and one TEM4 preparation class. After each pilot lesson, the observation scheme was revised accordingly, based on the research findings and the advice from the observed teachers and the researcher’s PhD classmates. After three rounds of piloting and revision, the operational draft of TEM washback observation scheme was finalized. 4.4.1.3 The Interview Questions In the current research, quite a few interviews have been conducted to seek the opinions, advice, and suggestions from experts, PhD candidates, school administrators, teachers and students. Among all the interviews, the post-questionnaire and postobservation interviews were conducted formally, with guiding questions prepared

4.4 Considerations for Validity and Reliability

101

beforehand. The purpose was to collect qualitative data to confirm research findings from the questionnaire surveys and classroom observations, to seek answers to the contradictory findings, and to obtain supplementary information when further questions need to be asked based on previous research findings. When designing the interview questions, the following steps have been taken to ensure validity and reliability. First, a detailed analysis was made on the research findings of the questionnaire surveys and classroom observations, to locate the problems that need to be tackled. Second, the researcher has consulted the experts, PhD candidates, school administrators, teachers and students for their opinions on the interview questions. Last, the interview questions had been piloted twice among teachers and students, and revised accordingly, before the final version was completed.

4.4.2 Data Collection and Analysis Practical measures have been taken when collecting and analyzing data through questionnaire surveys, classroom observations and interviews, to ensure validity and reliability of the data. 4.4.2.1 The Questionnaire Surveys Altogether seven questionnaire surveys were conducted among experts, TEM4 program administrators, TEM8 program administrators, TEM4 teachers, TEM8 teachers, TEM4 students and TEM8 students. The expert’s questionnaire was distributed in a TEM construction conference, and no time limit was set for the survey. Four experts returned the questionnaires during the conference, and one took it back home to finish and mailed it to the researcher 2 weeks later. All the experts provided their names and E-mail addresses, willing to offer additional information if needed. The other six questionnaires were mailed to the TEM participating schools together with the TEM test papers in 2012, but in a separate package which could be opened before the administration of the test. One copy of program administrator’s questionnaire and five copies of teacher’s questionnaire were sent to each TEM participating school. The personnel in charge of TEM in each school were requested to find the relevant administrators and teachers to complete the questionnaires. No time limit was set and they may return the questionnaire any time before the administration of TEM. The student’s questionnaire was distributed to 250 schools selected from all the TEM participating schools, based on region, school type and school level, to ensure that the research covered a representative sample. The questionnaire survey was conducted in the test room after the TEM test was completed, and no time limit was set. All the six questionnaires were conducted anonymously, to ensure that the participants would feel free to voice their opinions. Moreover, if they were unwilling to complete the questionnaires, they could just refuse or leave the questionnaires unfinished. The blank questionnaires and questionnaires with

102

4 Research Framework for TEM Washback Study

more than 20% of the questions unanswered would be sorted out and marked as invalid questionnaires. This may account for the fact that the valid rate of student’s questionnaire is comparatively low, 71.7% for TEM4 student’s questionnaire and 56.5% for TEM8 student’s questionnaire. Some candidates were unwilling to participate in the survey after taking TEM, so they did not complete the questionnaires. Since it was an anonymous questionnaire, they felt free to hand in a blank or incomplete questionnaire. This practice would, to some extent, uphold the validity of the questionnaire surveys, because the participants were more likely to voice their true beliefs when they were willing to complete the questionnaire. Three criteria were followed while sorting the questionnaires (refer to Sect. 4.3.1.2), and the researcher did all the sorting by herself, because it is believed that the sorting criteria could be better observed if it was done by the same person. This may also contribute to the reliability of the questionnaire surveys. The questionnaire data were put into computer by the present researcher with the help of six college students and two graduate students. Answers to multiple-choice and five-point Likert scale questions were processed by SPSS 17.0, while answers to short answer questions were stored in a Microsoft Word file. The SPSS data files and the Word files were created by the researcher, and the eight research assistants were requested to work in pairs to enter data, one reading aloud the answer and the other inputting data, so that their work was double-checked, which ensured the reliability of data. Moreover, a reliability test was performed on the questionnaires to ensure their inner consistency, and factor analysis was conducted to discern their validity (due to its limited sample, the expert’s questionnaire did not undergo these tests). The test results indicate that the reliability and validity of the questionnaire surveys were satisfactory (refer to Sect. 4.3.1.3 for the test results). 4.4.2.2 The Classroom Observations According to Patton (1990), one major threat to reliability and validity of observation data is the presence of a “stranger” in the classroom—the effect of the observer on what is observed. To reduce this negative effect, the researcher decided to record the classes for further analysis, instead of sitting in the classrooms observing. Each teacher observed was provided a digital voice recorder, and they were free to decide which class to record and how many classes to record. It was suggested that they record classes on different themes, such as writing, translation, reading, and so on, if it was TEM preparation classes, so that the classes observed could cover a wide range. While for non-TEM preparation classes, they were requested to record several consecutive classes because these classes would usually cover one unit, thus focusing on various teaching content, such as vocabulary building, text analysis and exercise explanation. When the teachers felt ready to have their classes recorded, they could just switch on the digital voice recorder on their desk, which would not interfere with their classroom teaching. In this sense, the validity of the observation data collected can be ensured.

4.4 Considerations for Validity and Reliability

103

Table 4.7 Inter-coder reliability (first round) A

B

Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N

A 1 220 .875a .000 220

B .875a .000 220 1 220

Correlation is significant at the 0.01 level (2-tailed)

a

Table 4.8 Inter-coder reliability (second round) A

B

Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N

A 1 220 .967a .000 220

B .967a .000 220 1 220

Correlation is significant at the 0.01 level (2-tailed)

a

Another threat to reliability is inconsistent coding, which has been tackled in the following three ways. First, a four-step coding procedure was developed to standardize the coding process, so that no important information would be missed (refer to Sect. 4.3.2.5 ). Second, a PhD candidate majoring in English pedagogy and testing was invited to recode five classes (10% of the total classes observed), which were randomly selected from all the 52 classes observed. A correlation test showed that the reliability was 0.875, and a further analysis of the data indicated that major differences arose from the following three aspects: (1) the distinction between teacher-fronted interaction and teacher-student interaction; (2) the definition of genuine information; and (3) length of questions and responses (Table 4.7). After extensive discussion and consultation, these three terms were further defined. Teacher-fronted interaction would include only the teacher asking and the whole class responding pattern. Genuine information refers to any information that is unknown to the person who asks the question, so it would be considered a genuine question if the teacher asks, “How do you understand …?”, while “What is the meaning of arbitrary?” would be considered not genuine. As for the length of questions and responses, it is agreed that on the five point Likert scale, (1) means word; (2, phrase; (3), a single sentence; (4), 2–3 sentences; and (5), more than 3 sentences. With the agreed criteria, five more classes were randomly selected for recoding, and the reliability increased to 0.967, which suggested the satisfactory inter-coder consistency (Table 4.8).

104

4 Research Framework for TEM Washback Study

Table 4.9 Intra-coder reliability A1

A2

Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N

A1 1 440 .987a .000 440

A2 .987a .000 440 1 440

Correlation is significant at the 0.01 level (2-tailed)

a

Last, ten classes (20% of the total classes observed) were selected randomly for recoding by the researcher after an interval of 1 month, and the intra-coder reliability was 0.987, which indicated the satisfactory intra-coder consistency (Table 4.9). 4.4.2.3 The Interviews The interviews were conducted either individually or in groups, and both telephone interviews and face to face interviews were done, depending on the situation. To ensure validity and reliability of interview data, the researcher took the following measures. First, a list of questions was prepared beforehand, but in the interview, additional questions might be asked based on the response of the participants. Questions were repeated and explained if the participants had any puzzles. Second, a digital voice recorder was used to record post-questionnaire and postobservation interviews to conduct further analysis. Last, the participants might be interviewed again if any problems arise during data collection and analysis.

4.4.3 Triangulation According to Denzin (1978), triangulation refers to the combination of methodologies in the study of the same phenomenon, which means two or more methods are to be used in a study in order to check results. This is a powerful technique that facilitates validation of data through verification from two or more sources (Bogdan and Biklen 2006). Therefore, in the current research, triangulation is achieved from different approaches, such as data triangulation and methodological triangulation, to enhance validity of the research.

4.5 Summary

105

4.4.3.1 Data Triangulation Data triangulation refers to the use of different sources of information to increase the validity of a research, which is well-suited for the current research since different stakeholder groups had been investigated. Therefore, in the study, responses from the following stakeholders would be compared: (1) experts; (2) program administrators; (3) teachers; and (4) students. Through the comparison and analysis, areas of agreement and divergence can be identified, which may contribute to the validation of the study. 4.4.3.2 Methodological Triangulation Various research methods have been employed in the present research, such as document analysis, questionnaire surveys, classroom observations and interviews. Therefore, data collected by different means can be compared, which are listed as follows: 1 . closed and open-ended questionnaire responses; 2. questionnaire responses and classroom observation findings; 3. questionnaire responses and interview findings; 4. classroom observation and interview findings; 5. current research findings and previous research findings collected through document analysis.

4.5 Summary This chapter began with a washback model employed in the current study, listing the contributing factors and working mechanism of TEM washback. A multi-phase and muti-method research framework was developed accordingy, employing various research methods to investigate the washback effects of TEM on the perceptions and practice of related experts, program administrators, teachers and students. Then, a detailed description was provided of the research participants and instruments—how the participants were selected, and how the research instruments, such as questionnaires and classroom observation scheme, were developed, piloted, revised and finalized. Next, data collection and analysis procedures were explained, with the preliminary analysis results listed in tables, such as the reliability test and factor analysis results. The research data would be further analyzed in Chaps. 5 and 6.

106

4 Research Framework for TEM Washback Study

Last, a detailed account was provided of the measures taken to ensure validity and reliability of the research. Washback is such a complicated phenomenon that it is essential to bear validity and reliability in mind throughout. The research instruments were validated, the data collection procedures were standardized and the research findings from various sources were triangulated, so that the research results are expected to be valid and reliable.

Chapter 5

Stakeholders’ Perceptions of TEM and Its Washback

This chapter presents research findings from the questionnaire surveys. The responses of experts, program administrators, teachers and students will be analyzed one by one, to identify their respective evaluation of TEM and its washback. The sequence in the discussion is as follows: experts, program administrators, teachers and students, according to the three levels presented in the educational hierarchy (Fig. 1.1). The experts are in the top level, responsible for the Teaching Syllabus development and test construction, thus more familiar with the intended washback of TEM; the program administrators are in the middle, who would develop their school syllabuses according to both the Teaching Syllabus and the situation of their schools, so they are familiar with the washback of TEM in the school level; teachers and students are the practitioners of the Teaching Syllabus and TEM Test Syllabuses, so they are familiar with the washback of TEM on classroom practice. In this view, the opinions of experts, program administrators, teachers and students are to be analyzed one by one, and then a comparison would be made to identify areas of agreement and divergence.

5.1 Experts’ Perceptions Five experts participated in the questionnaire survey, and gave detailed answers to the open-ended questions. Due to the limited number of participants, more verbal account of the research findings would be presented than statistical illustrations.

© Springer Nature Singapore Pte Ltd. 2018 Q. Xu, J. Liu, A Study on the Washback Effects of the Test for English Majors (TEM), https://doi.org/10.1007/978-981-13-1963-1_5

107

108

5 Stakeholders’ Perceptions of TEM and Its Washback

5.1.1 Demographic Information The five experts surveyed are all university professors in English with more than 20 years of teaching experience. Two are from comprehensive universities, two from foreign language universities and the fifth from a university of science and technology. Among the five experts, three have participated in the development of both the Teaching Syllabus and the TEM Test Syllabuses, and four have participated in the TEM test construction.

5.1.2 Experts’ Evaluation of TEM As asserted by the experts, the TEM Test Syllabuses are based on the Teaching Syllabus and the TEM tests are constructed according to the TEM Test Syllabuses, so TEM can measure whether students have met the required English proficiency specified in the Teaching Syllabus.

5.1.2.1 TEM Characteristics Generally speaking, TEM was positively evaluated by all the experts, who approved of, or generally approved of the design of TEM, including its content, item types employed, time allotment, score allotment and marking criteria. Moreover, some suggestions have been made on the improvement of TEM, which are summarized as follows. First, the test content of TEM, especially which of TEM8, should include more “professional knowledge for English majors”, such as linguistics, literature and culture of English speaking countries. This may help teachers and students develop a keener awareness of the difference between English majors and non English majors. Second, the item types employed in TEM are mainly traditional, such as multiple choice, translation and writing, so they could be appropriately modified to include more integrative items. Besides, the test content and test method of TEM should be more closely integrated, and appropriate test method can be selected according to different test content. Last, the marking criteria of TEM are well developed and easy to follow, but more importance should be attached to its guiding role in language teaching. For instance, the marking criteria of writing should be able to guide the teaching of writing for English majors.

5.1 Experts’ Perceptions

109

5.1.2.2 Use and Interpretation of TEM Scores Validity involves the quality of test use and interpretation, or the extent to which the inferences or decisions made on the basis of test scores are meaningful, appropriate and useful (AERA, APA & NCME 1985: 9). Therefore, it is essential to investigate the use and interpretation of TEM scores. According to the TEM8 Test Syllabus, the intended use and interpretation of the test scores are: 1 . to check and promote the implementation of the Teaching Syllabus; 2. to measure students’ language proficiency and to evaluate the overall language teaching standard of a university; 3. to provide feedback to language teaching and learning. However, due to the increasing social recognition of TEM, the test results have been used to perform other functions like conferring bachelor’s degree, ranking schools and evaluating teachers. According to the experts, the unintended use of TEM scores should not be encouraged because they will lead to some negative washback. First, to link TEM certificates to the awarding of bachelor’s degree degrees may encourage test-oriented learning, and some test takers may even take the risk of cheating because the certificate means a lot to them. Next, teaching effects may depend on many factors, so it is unjustifiable to use TEM results as the sole criterion to evaluate teaching. Last, to rank schools based on TEM results may lead to undesirable competition between schools, which is harmful to the healthy development of the schools.

5.1.3 Experts’ Perception of TEM Washback All the experts agreed that TEM has exerted positive washback on teaching and learning. TEM can check and promote the implementation of the Teaching Syllabus; TEM provides feedback information to teachers and students, who can adjust their teaching and learning accordingly; and TEM may help improve teaching and promote reform. If TEM is to better play its positive role in teaching and learning, the test reports should be improved, as most experts pointed out. They suggested that more information should be provided in the test reports, to include not only a rank, such as Excellent, Good, Pass and Fail, but also the total score and scores for each section. If possible, verbal description of the language ability for each score band should be provided, so that teachers and candidates can better interpret the test results. Moreover, the number of students in each score band can be provided to give students a rough idea of how they rank in the test population. In a word, the purpose of providing a test report is to facilitate teaching and learning, so the teachers’ and candidates’ needs should be considered. However, it is also likely that TEM may have some negative effects, such as interfering in the teaching plan and promoting test taking technique training. Nevertheless, negative effects can be avoided if TEM test and its scores are properly used.

110

5 Stakeholders’ Perceptions of TEM and Its Washback

5.1.4 Factors Influencing Teaching It is agreed that TEM exerts washback on teaching and learning. However, is it a decisive factor compared with other factors involved in teaching? According to the experts, the top three factors were curriculum design, teacher quality and student efforts, while no one considered TEM a decisive factor in teaching.

5.2 Program Administrators’ Perceptions Program administrators are the stakeholders in the school level, who will set the teaching aims and design the curriculum in their respective schools, based on the Teaching Syllabus and other related factors, such as the language proficiency of the students, the available teaching staff and resources, and perhaps, the TEM Syllabuses. Therefore, it is of significance to survey their opinions on the validity of TEM and TEM washback, to find out the effects of TEM on the English curriculum in different schools.

5.2.1 Demographic Information Altogether 1510 program administrators responded to the questionnaires, including 724 TEM4 program administrators and 786 TEM8 program administrators. Among them, some are deans of the English department at their respective universities, some are directors in charge of English language teaching in departments of foreign languages, and the rest are also leaders, responsible for English language teaching in the foundation or advanced stage (i.e. the first 2 years or the last 2 years of the undergraduate program) (Table 5.1). From Table 5.1, it can be concluded that most program administrators surveyed were well educated, with a master or PhD degree, and they were experienced teachers as well, with the majority having more than 15 years of teaching experience. Therefore, their opinions on TEM and its washback could provide us with illuminating insight into this issue.

5.2.2 Program Administrators’ Evaluation of TEM Program administrators are often responsible for designing school curricula and setting teaching objectives for English majors in their schools, so they should be more concerned about the Teaching Syllabus and TEM. As a result, they are expected to make sensible evaluation of TEM.

5.2 Program Administrators’ Perceptions

111

Table 5.1 Demographic information of program administrators Position:

TEM4 TEM8 Gender:

Dean of English department 27.3% 33.4%

Director in charge of English teaching 51.1% 47.1%

Other 21.6% 19.5%

Male TEM4 45.1% TEM8 49.9% Teaching experience: 0–5 years

Female 54.9% 50.1% 6–10 years

11–15 years

TEM4 3.5% TEM8 2.4% Work experience as a leader: 0–5 years

11.0% 8.1%

17.9% 15.6%

6–10 years

11–15 years

TEM4 TEM8 Degree:

31.2% 34.6%

10.8% 12.0%

More than 15 years 8.2% 8.5%

Master 61.6% 54.4%

PhD 17.2% 22.7%

Other 1.2% 1.0%

University of science and technology 26.0% 27.5%

Language university

Normal university

Other

4.9% 4.2%

15.5% 14.9%

14.9% 12.9%

49.8% 45.4%

Bachelor TEM4 20.1% TEM8 21.9% University type: Comprehensive university TEM4 TEM8

38.8% 40.4%

More than 15 years 67.6% 73.9%

Note: the percentage of each part may add up to 100.1% or 99.9%, because it is round to the nearest one decimal point

5.2.2.1 Familiarity with TEM Before investigating their evaluation of TEM, the current research first surveyed the program administrators on their familiarity with TEM, such as the Test Syllabus, marking criteria, and so on. Most program administrators surveyed were familiar with TEM, as indicated in Table 5.2. Over 90% of the participants were familiar with the Teaching Syllabus and the TEM Test Syllabus. However, the marking criteria were less known, familiar to around 60% of the participants, and the Oral Test Syllabus was the least known, familiar to only around 50% of the participants. Comparatively, TEM4 seems to enjoy a higher familiarity rate, especially in terms of the Oral Test Syllabus and the marking criteria.

112

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.2 Familiarity with TEM

Teaching syllabus Test syllabus Oral test syllabus Time allotment Marking criteria

TEM4 Familiar 96.3% 93.6% 57.8% 90.7% 72.6%

Neutral 3.5% 5.8% 22.3% 6.9% 16.6%

Unfamiliar 0.2% 0.7% 19.9% 2.4% 10.9%

TEM8 Familiar 96.5% 92.7% 51.3% 87.8% 65.4%

Neutral 2.8% 5.9% 21.3% 9.9% 19.9%

Unfamiliar 0.7% 1.4% 27.4% 2.3% 14.7%

Compared with TEM reaching an annual test population of over 200,000, the number of the Oral test takers is much smaller, about 10,000–20,000 every year, which means that many universities do not have students taking the test. So, it naturally follows that only half of the program administrators surveyed were familiar with the oral test syllabus. However, the finding deserves our close attention. As TEM is supposed to assess test takers’ language proficiency specified in the Teaching Syllabus, and speaking is an indispensable part of it, the relatively low familiarity of the TEM-Oral test is not conducive to promoting positive washback. Therefore, it is necessary to popularize TEM-Oral, to make it better known to teachers and students so as to promote a balanced development in students’ language proficiency. Marking criteria usually embody a test’s purpose and focus, and tests, especially criterion-referenced tests like TEM, can hardly achieve expected goals if their marking criteria are unfamiliar to teachers and students. For instance, a scoring scheme combining features of holistic and analytic marking is employed to grade TEM writing scripts. The scheme focuses on language use as well as ideas and arguments. Familiarity with the scoring scheme can help teachers set course objectives and plan their lessons in line with the writing requirements in the Teaching Syllabus, because the scoring scheme incorporates these writing requirements. In this respect, regular communication between test constructors and test users, teachers in particular, is of much importance. Various ways of communication can be employed, such as providing detailed marking criteria with samples, offering video training sessions, so as to foster or intensify positive washback on teaching and learning. 5.2.2.2 Evaluation of TEM Test Quality Another area of inquiry in the questionnaire is program administrators’ evaluation of TEM, including alignment with the Teaching Syllabus, test content, test method and test administration. As is shown in Table 5.3, TEM was positively evaluated by most program administrators, with a satisfaction rate of over 75% for all the items. Among the 10 items in the questionnaire, test content received the highest satisfaction rate whereas test report had the lowest one. Moreover, of all the items, marking criteria received the highest rate of “Not clear”, which is consistent with the previous finding of its unfamiliarity to program administrators. Comparatively, TEM4 had a higher satisfaction rate than TEM8, except for the item of “test report”.

5.2 Program Administrators’ Perceptions

113

Table 5.3 Evaluation of TEM

Alignment with the teaching syllabus Content Item type Time allotment Score allotment Marking criteria Convenient administration Fair administration Strict administration Test report

TEM4 Satisfy 93.9%

TEM8 Not clear Dissatisfy Satisfy 5.2% 0.9% 92.2%

Not clear Dissatisfy 6.4% 1.4%

97.0% 94.7% 91.0% 91.4% 89.0% 89.4% 87.9% 91.7% 78.5%

1.9% 3.7% 6.1% 6.8% 9.6% 7.8% 8.8% 5.1% 11.7%

3.0% 4.0% 8.8% 7.7% 13.7% 10.9% 8.8% 5.3% 9.6%

1.1% 1.5% 2.8% 1.8% 1.3% 2.8% 3.3% 3.2% 9.8%

95.7% 93.8% 87.5% 90.4% 84.9% 80.7% 85.5% 90.2% 80.9%

1.2% 2.2% 3.7% 1.9% 1.4% 8.4% 5.6% 4.5% 9.5%

As is shown in Table 5.3, the overall satisfaction rate is fairly high, with five items above 90%. Among these five items, four are concerned with the test itself: alignment with the Teaching Syllabus, test content, item type, and score allotment; one is related to test administration, strictness of administration. No matter whether these five items refer to internal or external factors, a high satisfaction rate seems to suggest that TEM, as a criterion-referenced test, has satisfactory validity. Feedback from the questionnaire not only provides test developers with achievements of the test, but also helps them identify possible areas of revision. As is mentioned above, convenience of test administration for TEM8 had received a comparatively low satisfaction rate. From participants’ responses to the shortanswer question, it becomes clear that the problem lies with the first task in TEM8 listening section. The first task combines note taking with gap filling. And the invigilators’ manual has stipulated a series of steps to be followed in test administration, which some program administrators said were “quite complicated”. As TEM8 is a large-scale high-stakes test, a minor mistake might cause the listening section to be improperly delivered, which could then result in unfairness. Thus, a review of the test invigilation procedures is essential with a view to improving practicality and thus maintaining validity. Moreover, the satisfaction rate for test report, both TEM4 and TEM8, is low, just around 80%. However, to make better use of the feedback information of TEM, a detailed test report is essential. TEM test report is just a certificate, dividing the test takers into excellent, good and pass (those who fail the test will get no certificates). The test takers can check their overall score in their corresponding schools, but the scores for each section are not provided. In this sense, the information provided in TEM test report is quite limited. As shown in Table 5.4, most program administrators considered it necessary to report not only a composite score, but also section scores and ability description. Moreover, information of students’ ranking in the test population was considered important by around half of the program administrators, and some of them wanted

114

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.4 Information to be provided in the test report

Composite score Section score Ability description Rank in the test population Other information

TEM4 Necessary 78.4% 89.7% 67.9% 57.3% 2.1%

Unnecessary 21.6% 10.3% 32.1% 42.7% 97.9%

TEM8 Necessary 78.9% 85.7% 66.6% 54.5% 2.4%

Unnecessary 21.1% 14.3% 33.4% 45.5% 97.6%

Table 5.5 Use and interpretation of TEM scores

Reflecting syllabus implementation Promoting syllabus implementation Promoting teaching reform Revealing problems &achievements in teaching Reflecting teaching quality Reflecting students’ language proficiency Assessing students’ learning outcome Predicting students’ future performance Feedback to course design Feedback to teaching aim Feedback to classroom teaching

Agree 83.1%

TEM4 Not clear 10.7%

84.4%

11.6%

4.1%

83.3%

12.0%

4.7%

79.5% 80.0%

13.7% 12.2%

5.0% 7.8%

79.0% 77.7%

12.7% 13.3%

8.3% 9.0%

72.4% 80.4%

15.6% 13.6%

11.9% 6.0%

72.0% 77.8%

16.0% 13.6%

12.0% 8.6%

86.3%

9.5%

4.2%

78.7%

12.1%

9.2%

63.5%

24.2%

12.3%

63.6%

20.2%

16.2%

77.0% 83.6% 81.3%

13.7% 11.4% 11.8%

9.3% 5.0% 6.8%

77.7% 78.6% 78.7%

13.3% 11.3% 13.2%

9.0% 10.1% 8.1%

Disagree Agree 6.2% 79.6%

TEM8 Not clear 13.3%

Disagree 7.1%

other information, such as ranking of the schools, strengths and weaknesses reflected in the tests, and so on. Therefore, it is desirable to improve the test report of TEM, to provide more information, so that TEM can play a better role in teaching and learning. Just as an expert pointed out, when drafting a test report, importance should be attached to “the relation between scores and teaching, the feedback a test report can provide for teaching and learning”. 5.2.2.3 Use and Interpretation of TEM Scores As indicated in Table 5.5, around 80% of the program administrators believed that TEM scores can reflect and promote syllabus implementation at the university level, and it can measure students’ language proficiency. Meanwhile, TEM scores can provide feedback to course design and classroom teaching, thus promoting teaching

5.2 Program Administrators’ Perceptions

115

Table 5.6 Influence of TEM scores

Affect university ranking Affect department ranking Affect teacher evaluation Affect student ranking Affect employment Affect further education

Agree 61.1% 58.8% 39.6% 41.8% 79.8% 67.7%

TEM4 Not clear 25.9% 23.0% 27.4% 26.3% 11.6% 18.2%

Disagree 13.0% 18.3% 33.0% 31.9% 8.6% 14.2%

Agree 60.5% 57.5% 35.2% 37.5% 86.2% 72.0%

TEM8 Not clear 25.1% 25.1% 24.9% 24.2% 8.4% 16.0%

Disagree 14.4% 17.4% 39.8% 38.4% 5.4% 12.0%

reform. As to whether TEM scores can reflect teaching quality and students’ future performance, fewer program administrators held a positive view. Therefore, it was doubtful whether TEM results could be used as a criterion for teacher evaluation and job application. Table 5.6 provides information on the impact of TEM scores. As can be seen in the table, TEM scores seem to affect students’ future career and education much more than rankings of various kinds. It can be concluded from the above data that TEM has, to some extent, achieved its intended purpose as is specified in the Test Syllabus. In this sense, TEM scores are used and interpreted in the intended way. Of all the items in Table 5.5, “predicting students’ future performance” received the lowest percentage point (63.5% and 63.6%). This has, from a different perspective, attested to the intended use of TEM, which, as a criterion-referenced test, is designed to evaluate learning outcome according to predetermined criteria, rather than predict what students can do in the future. However, as Table 5.6 indicates, TEM scores have a much greater impact on students’ future career and future education, especially the former. This has aroused concern from TEM test developers, for TEM is not designed to predict students’ performance in their future career. Using TEM scores to make employment decisions may incur problems. Besides its unintended use in job recruitment, TEM scores were believed to affect school ranking and teacher evaluation (Table 5.6). However, were these practices common in the participating universities? According to Table 5.7, the most common practice is to relate TEM4 score to the conferring of bachelor’s degree degree, which was selected by one third of the program administrators. While for the other uses, only about 10% of the program administrators selected “yes”, claiming that TEM scores were used to rank schools, evaluate teachers or confer degrees in their schools, which indicates that these are not common practices. However, program administrators were almost equally divided in their attitudes towards the four practices, as is shown in Table 5.8. Two points came up in Tables 5.7 and 5.8: (1) unintended use of TEM scores in participating universities was not common, which is conducive to maintaining and promoting positive washback; (2) more than half of the program administrators surveyed were not against using TEM scores, especially TEM4 scores, in unintended ways, which poses potential hazards in misuse of test scores.

116

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.7 Unintended use of TEM scores

Conferring degrees Evaluating teachers Evaluating leaders Ranking schools

Yes 33.5% 14.7% 18.6% 14.6%

TEM4 No 65.1% 83.7% 78.4% 59.4%

Not clear 1.4% 1.6% 3.0% 26.0%

Yes 12.4% 8.3% 13.2% 11.8%

TEM8 No 86.9% 91.4% 84.7% 57.0%

Not clear 7.0% 3.0% 2.1% 31.2%

Agree 47.6% 46.9% 42.1% 30.4%

TEM8 Neutral 7.7% 8.8% 14.6% 17.5%

Disagree 44.7% 44.2% 43.3% 52.1%

Table 5.8 Attitudes towards unintended uses of TEM scores

Conferring degrees Evaluating teachers Evaluating leaders Ranking schools

Agree 65.5% 55.4% 52.1% 48.0%

TEM4 Neutral 8.2% 10.2% 17.8% 20.6%

Disagree 26.3% 34.4% 30.0% 41.1%

Therefore, test developers are expected to “anticipate the common uses of certain kinds of tests, and the potential consequences of such use, even if these uses are not explicitly advocated” (Kane 2001: 338). It is advisable that TEM test developers identify possible uses and interpretations of TEM scores and provide guidelines on how decisions about scores are to be made.

5.2.3 Program Administrators’ Perception of TEM Washback It is generally assumed that tests may exert influence on teaching and learning, and the higher the stakes, the more intense its washback. The existing research on washback effects indicate that washback effects do exist, and they may take different forms in different settings or among different stakeholders. Moreover, a test is more likely to influence teaching content than teaching method (Cheng 2005; Gu 2007; Wall 1996). Table 5.9 provides program administrators’ perception of TEM washback. Generally speaking, their views on washback were positive, though their opinions varied on some items. From Table 5.9, it can be inferred that most program administrators held a positive attitude towards TEM, especially its overall washback on program design and (classroom) teaching. With regard to other items, TEM was perceived to have exerted more positive influence on setting teaching aims (85.2% and 83.5%) than scheduling courses (75.0% and 71.4%); and teaching method (27.6% and 31.0%)

5.2 Program Administrators’ Perceptions

117

Table 5.9 Views on TEM washback

Overall washback on program design Overall washback on teaching Teaching aims Course design Teaching schedule Teaching content Teaching method Teaching ideology Teaching attitude Teaching depth Teaching speed

Positive 86.8%

TEM4 No washback 11.6%

Negative 1.5%

Positive 84.5%

TEM8 No washback 14.6%

Negative 0.9%

86.7%

11.5%

1.8%

83.7%

15.1%

1.1%

85.2% 78.3% 75.0% 78.2% 68.6% 68.4% 72.9% 75.6% 66.3%

13.6% 20.1% 22.3% 18.9% 27.6% 27.1% 25.6% 21.5% 30.0%

1.1% 1.7% 2.7% 2.8% 3.8% 4.6% 1.6% 2.9% 3.7%

83.5% 75.7% 71.4% 76.1% 67.2% 70.6% 73.0% 75.8% 61.2%

15.7% 23.2% 26.5% 22.2% 31.0% 26.4% 25.7% 22.1% 35.3%

0.8% 1.1% 2.1% 1.7% 1.8% 2.9% 1.3% 2.1% 3.5%

No washback 16.3% 20.1% 33.7% 37.9% 32.9%

Negative 0.9% 1.5% 1.5% 1.6% 1.6%

Table 5.10 Washback exerted by different aspects of TEM

Test content Test method Score allotment Time allotment Marking criteria

TEM4 Positive 83.4% 81.7% 71.0% 65.6% 71.5%

No washback 15.3% 16.8% 27.6% 32.7% 26.8%

Negative 1.3% 1.5% 1.5% 1.7% 1.7%

TEM8 Positive 82.7% 78.3% 64.7% 60.5% 65.4%

seemed to be influenced less than teaching content (18.9% and 22.2%), which complies with previous research findings (Cheng 2005; Gu 2007; Wall 2005). In addition, TEM also seemed to have impact on teaching ideology and attitude in a positive direction. On the whole, the statistics have shown that TEM has had positive effects on teaching, but the intensity of its effect varied across different aspects of the teaching context. A further investigation was made to find out which aspect of TEM exerted more washback effects (Table 5.10). According to most program administrators, various aspects of TEM exerted positive washback on course design, and the influence of test content and method surpassed that of the other aspects. However, marking criteria, an aspect that should play a guiding role, were not considered important, with around 30% of the program administrators selecting “no washback”. Possible reason is that some program administrators were not familiar with the marking criteria (see Table 5.2).

118

5 Stakeholders’ Perceptions of TEM and Its Washback

5.2.4 Washback on Course Design Table 5.9 also indicates that around 75% of the program administrators held a positive view on TEM8 washback on course design, which could further be manifested in two aspects: introduction of TEM related courses and TEM preparation courses, as the following tables show. TEM4 consists of six parts—dictation, listening, grammar and vocabulary, cloze, reading and writing, and TEM8 also consists of six parts—listening, reading, general knowledge, proofreading, translation and writing. According to the Teaching Syllabus, the recommended courses for students in the fundamental and advanced stage are: 1. Fundamental stage: fundamental English, phonetics, listening, speaking, reading, writing, grammar, and culture of English speaking countries; 2. Advanced stage: advanced English, writing, translation, oral interpretation, academic writing, linguistics and literature. As is indicated in Table 5.11, TEM related courses seem to be more often introduced than non-related courses, especially for TEM8 non-related courses such as academic writing and interpretation, which may be viewed as a sign of TEM washback on teaching as well. Moreover, in the fundamental stage, skill training courses (listening, reading, speaking and writing) were more often introduced than courses on language knowledge, such as grammar and phonetics, which may indicate that the focus of college English teaching is language use rather than language knowledge. TEM underwent some substantial revisions in 2005. In TEM4, the requirement on listening, writing and vocabulary was raised, and the item of fast reading was deleted; while in TEM8, the requirement on listening, reading and writing was raised and a new component, General Knowledge, was introduced to assess students’ knowledge of linguistics, literature and culture. The current research thus makes an initial investigation into the influence of TEM revisions on course design, as shown in Table 5.12. As for TEM4 revision, it seems that the raised requirement exerted more influence on course design than the absence of a particular test component. A possible reason is that fast reading is only a kind of reading skill, so the deletion of fast reading would not influence the introduction of the reading course. While for TEM8 revision, the newly introduced component, General Knowledge, was expected to exert more influence on course design. However, our research findings indicate that listening and writing revisions seemed to have affected course design to a greater extent. This phenomenon was explained by some program administrators in the short-answer question. Special courses on listening and writing had been introduced in some institutions for juniors and seniors because these skills were generally supposed to be more difficult for students. For example, many institutions offered advanced listening courses, even though listening is not a compulsory subject for the advanced stage in the Teaching Syllabus. Thus, higher test requirements

Advanced English 4.1% 95.9%

Fundamental English 3.0% 97.0%

4.5% 95.5%

Writing

10.2% 89.8%

Phonetics

12.7% 87.3%

Translation

2.7% 97.3%

Listening

16.9% 83.1%

Interpretation

3.4% 96.6%

Speaking

24.1% 75.9%

Academic writing

5.9% 94.1%

Reading

8.3% 91.7%

Linguistics

4.1% 95.9%

Writing

5.4% 94.6%

Literature

12.5% 87.5%

Grammar

7.7% 92.3 %

Culture

Note: Culture (of English Speaking Countries) is included in TEM8 because it is the content domain of 3–4 questions in the General Knowledge Section of TEM8, despite the fact that it is a compulsory course for the sophomore year

Not introduced Introduced

Not introduced Introduced TEM8

TEM4

Table 5.11 Courses introduced by TEM participating universities

5.2 Program Administrators’ Perceptions 119

120

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.12 Washback from TEM revisions TEM4 Not influenced Not clear Influenced TEM8 Not influenced Not clear Influenced

Listening 6.2% 9.0% 84.9%

Fast reading 17.9% 14.4% 67.7%

Writing 4.7% 8.6% 86.7%

Vocabulary 6.3% 8.1% 85.6%

Listening 8.5% 11.8% 79.8%

Reading 17.4% 13.4% 69.2%

Writing 10.1% 9.4% 80.4%

General Knowledge 12.6% 11.0% 76.4%

prompted universities to continue to focus on skill training, which is an encouraging sign of TEM8 washback on course design. As Table 5.12 shows, participants’ perception of the General Knowledge component varied. Responses to the short-answer question indicate that some admitted there had been a revision to their school curriculum accordingly to introduce related courses; some believed what was tested was too easy and not worth preparing; still others claimed that linguistics, culture and literature had long been part of their curriculum, and thus there was no need to revise the existing curriculum. In a word, TEM has exerted some influence on course design, but different schools responded in different ways. As for TEM preparation courses, they were offered by some schools, but mostly in the form of lectures, as is indicated by Table 5.13. It seems that TEM4 preparation courses were more often introduced than TEM8 preparation courses, and the length of preparation is longer. However, for both TEM4 and TEM8, when special training courses were offered, more than half lasted less than 20 h. So, they covered no more than half a semester. Thus, in terms of test preparation, TEM has had some mild effect on course planning.

5.2.5 Individual Differences As proposed by many researchers, the washback effects of a test may vary from participant to participant, due to their different characteristics, such as educational background and work experience (Alderson and Wall 1993; Green 2007). In this sense, program administrators’ perception of TEM washback is further analyzed to identify individual differences. Since program administrators are often responsible for course planning, the washback effects on course design were further analyzed, and it turned out that the washback effects perceived by TEM4 program administrators varied significantly—the overall washback on course design differed among program administrators with different educational background and teaching experience.

5.2 Program Administrators’ Perceptions

121

Table 5.13 TEM preparation courses TEM4 Preparation courses No 18.1% Several lectures 36.3% Overall training 33.5% Skill training 16.0% Grammar & Vocabulary 4.3% TEM8 Preparation courses No 27.1% Several lectures 40.1% Overall training 25.2% Skill training 15.4% 8.1% General knowledge training

Less than 10 h 10–20 h

More 20–30 h 30–40 h than 40 h

40.9% 38.9% 46.8%

13.4% 11.6% 8.2%

26.3% 33.7% 31.2%

12.1% 8.5% 7.4%

7.3% 7.3% 6.3%

Less than 10 h 10–20 h

More 20–30 h 30–40 h than 40 h

51.4% 55.2% 69.3%

11.7% 7.6% 5.4%

23.6% 26.0% 17.1%

8.6% 6.3% 2.9%

4.7% 4.8% 5.4%

Note: The percentage may exceed 100%, because participants could select more than one answer Table 5.14 TEM washback perceived by different program administrators Overall washback perceived by program administrators with different teaching experience (X2 = 8.394; p = .039) 0–5 years 5–10 years 10–15 years >15 years Positive 96.2% 94.0% 89.0% 84.7% Neutral 0 4.8% 9.5% 13.7% Negative 3.8% 1.2% 1.5% 1.6% Overall washback perceived by program administrators with different educational background (X2 = 9.010; p = .029) Bachelor Master Doctor Others Positive 88.4% 87.0% 84.0% 77.8% Neutral 10.9% 11.7% 13.0% 11.1% Negative 0.6% 1.3% 3.0% 11.1%

As shown in Table 5.14, program administrators with a longer teaching experience and higher education took a more neutral perspective of the washback effects of TEM4 on course design. Perhaps they were more confident and had clearer ideas as to how teaching should be conducted for English majors, so they were affected less by TEM4. However, no significant difference was identified among TEM8 program administrators, which is possibly due to the fact that TEM8 is considered to play a less important role in college English teaching (see Table 5.15), so the washback it exerts is less intense, and so is the difference of the washback perceived by the program administrators.

TEM4 TEM8

Teacher quality 70.9% 64.3%

Student effort 80.8% 61.0%

Student quality 58.0% 69.0%

Table 5.15 Factors affecting English language teaching Leader attention 61.9% 40.1%

Course design 36.7% 20.8%

Teaching facility 42.8% 9.6%

Teaching material 28.9% 4.3%

Teacher income 22.4% 22.4%

Research 20.8% 8.3%

TEM 22.0% 2.5%

122 5 Stakeholders’ Perceptions of TEM and Its Washback

5.3 Teachers’ Perceptions

123

5.2.6 Factors Influencing Teaching The above discussion shows that TEM has more positive washback effects on English language teaching than negative effects, and it seems to have affected those courses recommended in the Teaching Syllabus that are more related to test content domain. Moreover, TEM has had some impact on the continued delivery of skill training courses in some universities, even though the skill/competency is not a compulsory subject for the advanced stage. However, is TEM a decisive factor compared with other factors involved in language teaching? From Table 5.15, it can be noted that program administrators consider student quality, teacher quality, student effort and leader attention the most important four factors. In contrast, the influence of TEM was limited. Comparatively, TEM4 played a more important role in college English teaching, acknowledged by 22.0% of the program administrators, while TEM8 was considered important by only 2.5% of the program administrators surveyed. In other words, although TEM has exerted some washback on English language teaching, its effect is limited, as compared with other factors. This finding is desirable because the relationship between teaching and testing should be “that of partnership”—a good test should be “supportive of good teaching” and “exert a corrective influence on bad teaching” (Hughes 2003: 2). In this view, the role TEM plays in English teaching should be facilitative rather than decisive, so it is natural that TEM was not considered an important factor by most program administrators. In addition, this is also in line with the experts’ opinion, who believed that the top three factors affecting English teaching and learning effects should be curriculum design, teacher quality and students efforts, while no one has selected TEM as an important factor.

5.3 Teachers’ Perceptions Teachers are the practitioners of the Teaching Syllabus and their school curricula. Meanwhile, their teaching activities are influenced by TEM. Therefore, questionnaire surveys were conducted among TEM4 and TEM8 teachers, to investigate their evaluation of TEM, perception of TEM washback, classroom teaching practice, and perception of the factors influencing teaching for English majors.

5.3.1 Demographic Information Altogether 6796 teachers participated in the questionnaire surveys, including 3574 TEM4 teachers who teach students in the foundation stage and 3222 TEM8 teachers who teach students in the advanced stage. They may not have taught TEM preparation courses, but the students they have taught would take TEM4 or TEM8 (Table 5.16).

124

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.16 Demographic information of teachers Gender: Male 25.0% 28.3%

Female 75.0% 71.7%

Professor

Associate professor

Lecturer

TEM4 3.1% TEM8 4.3% Teaching experience: 0–5 years

22.3% 26.6%

58.7% 56.0%

6–10 years

11–15 years

TEM4 TEM8 Degree:

28.0% 25.0%

34.2% 32.0%

18.9% 19.6%

More than 15 years 19.0% 23.4%

Bachelor TEM4 15.8% TEM8 14.7% University type: Comprehensive university TEM4 43.9% TEM8 44.3%

Master 77.7% 76.1%

PhD 5.9% 8.7%

Other 0.6% 0.4%

University of science and technology 20.3% 23.6%

Language university 6.4% 5.7%

Normal university 15.9% 14.9%

TEM4 TEM8 Title:

Teaching assistant 14.7% 12.3%

Other 1.2% 0.8%

Other 13.5% 11.6%

From Table 5.16, it can be concluded that the teachers surveyed constitute a representative sample, covering teachers from various types of universities, with different educational background and teaching experience. The academic title of the teachers ranged from teaching assistant to professor, but with lecturers being the majority, and the academic degree of the teachers ranged from Bachelor to Doctor, with Master being the most common. There are new teachers with no more than 5 years of teaching experience, and also experienced teachers with more than 15 years of experience. In terms of academic title and degree, TEM8 teachers seemed to be slightly higher than TEM4 teachers, and TEM8 teachers had a longer teaching experience. The number of female teachers far outweighs that of male teachers, both for TEM4 and TEM8. The wide coverage of the sample ensures the representativeness of the research findings, and meanwhile, a comparison can be made between teachers with different characteristics.

5.3.2 Teachers’ Evaluation of TEM Teachers are the practitioners of the Teaching Syllabus and their school curricula, and TEM is one of the methods used to measure their teaching effects. Therefore, they may have developed their unique perceptions of TEM and its washback. In this

5.3 Teachers’ Perceptions

125

Table 5.17 Familiarity with TEM

Teaching syllabus Test syllabus Oral test syllabus Time allotment Marking criteria

TEM4 Familiar 79.9% 81.8% 39.6% 81.5% 63.5%

Neutral 16.5% 14.6% 29.1% 14.6% 22.9%

Unfamiliar 3.7% 3.7% 31.3% 3.9% 13.5%

TEM8 Familiar 79.0% 72.9% 33.6% 73.2% 49.3%

Neutral 17.7% 20.4% 27.9% 19.9% 29.0%

Unfamiliar 3.4% 5.6% 38.5% 6.9% 21.7%

sense, teachers were surveyed on their familiarity with TEM, evaluation of TEM, and their use of TEM results. 5.3.2.1 Familiarity with TEM Before investigating their evaluation of TEM, the current research first surveyed the teachers on their familiarity with TEM, such as the Teaching Syllabus, the Test Syllabus, and so on. Whether teachers are familiar with TEM, to some extent, is determined by whether they are teaching TEM related courses, whether they are preparing students for TEM, and whether they are concerned about the test. The majority of the teachers surveyed were familiar with the Teaching Syllabus, the TEM Test Syllabus and TEM time allotment, as indicated in Table 5.17. Around 80% of the participants claimed to be familiar with the Teaching Syllabus and the TEM Test Syllabus. However, the marking criteria were less known, familiar to 63.5% of the TEM4 teachers and 49.3% of the TEM8 teachers, and the Oral Test Syllabus was the least known, familiar to only one third of the participants (39.5% and 33.6%). Both TEM4 and TEM8 teachers were familiar with the Teaching Syllabus, but TEM4 seemed to enjoy a higher familiarity rate, especially in terms of the Test Syllabus and the marking criteria. Teachers’ questionnaire findings coincide with that of the program administrators’ questionnaires, and the possible causes are the same: the small test population of TEM Oral Tests and the lack of communication between test developers and test users. 5.3.2.2 Evaluation of TEM Test Quality Since most teachers were familiar with TEM, it is likely for them to make sensible judgment on the quality of TEM; hence, another area of inquiry in the questionnaire is teachers’ evaluation of TEM, including alignment with the Teaching Syllabus, test content, test method and test administration. As is shown in Table 5.18, TEM was positively evaluated by most teachers, with a satisfaction rate of over 75% for all the items. Among the 10 items in the questionnaire, test content received the highest satisfaction rate whereas test report had the lowest one. Moreover, of all the items, marking criteria received the highest rate of “Not clear”, which is consistent

126

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.18 Evaluation of TEM

Alignment with the teaching syllabus Content Item type Time allotment Score allotment Marking criteria Convenient administration Fair administration Strict administration Test report

TEM4 Satisfy 90.6%

Not clear 7.5%

Dissatisfy 1.9%

TEM8 Satisfy 89.5%

Not clear 8.9%

Dissatisfy 1.7%

91.7% 90.6% 86.9% 88.6% 85.3% 86.1% 84.0% 90.1% 74.3%

5.5% 6.6% 8.4% 8.7% 12.2% 9.9% 10.9% 6.9% 14.9%

2.7% 2.9% 4.7% 2.7% 2.6% 3.9% 5.0% 3.0% 10.8%

92.0% 90.7% 83.7% 87.0% 81.4% 75.4% 79.0% 86.6% 75.5%

5.8% 6.8% 10.7% 10.5% 16.6% 14.6% 12.7% 8.9% 17.0%

2.3% 2.5% 5.6% 2.6% 2.0% 10.0% 8.2% 4.5% 7.6%

Table 5.19 Information to be provided in the test report

Composite score Section score Ability description Rank in the test population Other information

TEM4 Necessary 65.5% 81.4% 64.0% 46.6% 1.8%

Unnecessary 34.5% 18.6% 36.0% 53.4% 98.2%

TEM8 Necessary 62.6% 78.5% 62.1% 43.2% 0.9%

Unnecessary 37.4% 11.5% 37.9% 56.8% 99.1%

with the previous finding of its unfamiliarity to teachers. Comparatively, TEM4 had a higher satisfaction rate than TEM8, except for the item of “test report”. As shown Table 5.18, the items of test report and convenient administration for TEM8 received the lowest satisfaction rate, which complies with the findings of the program administrators’ questionnaires. Possible reasons are analyzed in Sect. 5.2.2.2. Moreover, teachers’ opinions were sought as to what should be included in the test report. As shown in Table 5.19, what teachers wanted most is the section score, which can reflect students’ strengths and weaknesses, such as whether they are better at reading or listening. In this sense, learning activities can be organized to cater to the needs of the students. 5.3.2.3 Use and Interpretation of TEM Scores Test characteristics include not only test content and method, but also the use and interpretation of test results. In this view, the current research attempted to investigate teachers’ perception of how TEM scores should be used, as well as how the score were actually used in their respective schools. As shown in Table 5.20, more

5.3 Teachers’ Perceptions

127

Table 5.20 Use and interpretation of TEM scores

Feedback to teaching Assessing teaching quality Promoting teaching reform Reflecting students’ language proficiency Assessing students’ learning outcome Predicting students’ future performance Motivating students to learn

Agree 80.0% 68.5% 73.0% 77.7%

TEM4 Not clear 13.1% 16.3% 15.6% 14.1%

Disagree 6.8% 15.1% 11.4% 8.1%

Agree 75.3% 64.5% 67.3% 74.8%

TEM8 Not clear 15.7% 19.1% 19.3% 15.9%

82.7%

11.6%

5.7%

79.0%

13.0%

7.9%

57.7%

25.4%

16.8%

58.1%

25.6%

16.3%

80.0%

12.6%

6.5%

76.4%

14.7%

8.9%

Disagree 9.0% 16.5% 13.5% 9.4%

Table 5.21 Influence of TEM scores

Affect teacher evaluation Affect student ranking Affect employment Affect further education

Agree 33.4% 35.0% 71.4% 55.8%

TEM4 Not clear 25.7% 34.5% 17.8% 26.0%

Disagree 40.8% 30.4% 10.8% 18.2%

Agree 24.8% 32.7% 80.0% 58.7%

TEM8 Not clear 25.3% 32.0% 12.8% 24.4%

Disagree 49.9% 35.3% 7.2% 17.0%

Yes 9.2% 9.7%

TEM8 No 83.2% 80.7%

Not clear 7.5% 9.6%

Table 5.22 Unintended uses of TEM scores

Conferring degrees Evaluating teachers

Yes 34.6% 15.7%

TEM4 No 53.5% 70.1%

Not clear 11.9% 14.2%

teachers agreed on the use of TEM score to assess students learning outcome (82.7% and 79.0%) and provide feedback to teaching (80.0% and 75.3%) than to predict students future performance (57.7% and 58.1%). It suggests that TEM has been used in the intended way—to measure students’ language proficiency and to provide feedback information to facilitate English teaching. As to whether TEM results can be used in job application, “predict students’ future performance”, most teachers were doubtful, which is the same as the program administrators. Table 5.21 provides information on the impact of TEM scores. As can be seen in the table, TEM scores seem to affect students’ future career and further education much more than teacher evaluation and student ranking. Further investigation was made into the actual practice in different universities— how TEM scores were used and the teachers’ attitudes towards such uses. From Tables 5.22 and 5.23, it can be noted that TEM scores were seldom used to evaluate teachers, but one third of the schools linked TEM4 score to the bachelor’s degree degree. However, teachers’ attitudes towards these uses deserve our attention: more teachers agreed to these uses than not.

128

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.23 Attitudes towards unintended uses of TEM scores

Conferring degrees Evaluating teachers

Agree 62.9% 45.5%

TEM4 Neutral 13.0% 19.8%

Disagree 24.1% 34.6%

Agree 53.1% 42.7%

TEM8 Neutral 11.3% 14.8%

Disagree 35.6% 42.5%

TEM8 No washback 14.4%

Negative 2.2%

Table 5.24 Views on TEM washback

Overall washback on teaching Overall washback on learning Teaching aims Course design Teaching schedule Teaching content Teaching method Teaching ideology Teaching attitude Teaching depth Teaching speed

Positive 84.8%

TEM4 No washback 11.0%

Negative 4.2%

Positive 83.3%

92.5%

3.4%

4.2%

93.5%

3.4%

3.2%

84.6% 75.1% 73.7% 76.6% 68.8% 66.0% 71.8% 71.9% 63.3%

11.8% 21.4% 19.8% 16.9% 22.5% 24.5% 24.0% 19.8% 24.4%

3.6% 3.5% 6.5% 6.6% 8.7% 9.5% 4.1% 8.3% 12.4%

82.9% 73.2% 70.1% 76.2% 67.6% 67.0% 71.5% 76.1% 62.3%

15.3% 24.9% 26.8% 20.5% 28.0% 28.2% 26.2% 20.0% 32.0%

1.7% 1.9% 3.1% 3.2% 4.4% 4.9% 2.2% 3.9% 5.7%

Therefore, it is advisable that TEM test developers identify possible uses and interpretations of TEM scores and provide guidelines on how decisions about scores are to be made.

5.3.3 Teachers’ Perception of TEM Washback Table 5.24 is about teachers’ perception of TEM washback. Generally speaking, their views on washback were positive, though their opinions varied on some items. From Table 5.24, it can be concluded that most teachers held a positive attitude towards TEM, especially its overall washback on learning, with more than 90% of the teachers claiming that TEM exerted positive washback on learning. As for teaching, most teachers agreed that TEM exerted positive washback on the overall teaching and teaching aims, while TEM seemed to affect teaching method, ideology, attitude and speed less, as indicated by the number of teachers selecting “no washback”. This finding complies with the previous research findings (Cheng 2005; Gu 2007; Wall 2005) and the program administrators’ perception. Teachers’ perception of TEM washback was further explored to find out which aspect of TEM exerted more washback effects (Table 5.25).

5.3 Teachers’ Perceptions

129

Table 5.25 Washback exerted by different aspects of TEM

Test content Test method Score allotment Time allotment Marking criteria

TEM4 Positive 82.2% 78.2% 71.7% 68.5% 69.9%

No washback 14.4% 17.8% 25.6% 28.3% 26.7%

Negative 3.3% 4.1% 2.7% 3.2% 3.3%

TEM8 Positive 80.4% 77.3% 67.7% 65.1% 66.3%

No washback 17.9% 20.8% 30.5% 32.9% 31.2%

Negative 1.6% 1.9% 1.9% 2.0% 2.6%

From Table 5.25, it can be inferred that various aspects of TEM exerted more positive than negative effects on teaching. Test content and method seem to have exerted more influence than score allotment, time allotment and marking criteria, which is similar to the perception of program administrators. However, it deserves our attention that the marking criteria failed to achieve the expected effects. Marking criteria should have played a guiding role in English teaching, but it failed to do so since some teachers were not familiar with them (see Table 5.17).

5.3.4 Washback on Classroom Teaching According to Alderson and Wall (1993), a test will influence teaching, such as its content and method, rate and sequence, degree and depth, as well as teachers’ attitude towards teaching. Therefore, in the present study, teachers were surveyed on the washback of TEM on their classroom teaching practice. 5.3.4.1 TEM Preparation Classes One possible washback of TEM on teaching is the introduction of TEM preparation courses, which may differ from non-preparation courses in both the teaching content and the teaching method. In this sense, teachers were surveyed on whether they have prepared students for TEM, and if yes, how these courses were delivered, and whether they were different from non-preparation courses. Table 5.26 summarizes whether teachers have prepared students for TEM and how long they have spent on TEM preparation. From Table 5.26, it can be inferred that TEM4 preparation was more often introduced than TEM8 preparation, since around half (45.0%) of the teachers claimed that they had never prepared students for TEM8, while only 25% of the teachers selected “no” for TEM4 preparation. For those who had prepared students for TEM, around one fourth (27.0% and 27.4%) employed the format of lectures, while for those who prepared students for TEM in their classrooms, most of them spent no more than 10 h on TEM preparation, which indicates that the washback of TEM on course design is not intense.

130

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.26 TEM preparation experience TEM4

No Several lectures Overall training Skill training Grammar & Vocabulary TEM8

No Several lectures Overall training Skill training General knowledge training

TEM preparation 25.3% 27.0% 26.3% 26.5% 11.4%

Less than 10 h

10–20 h

20–30 h

30–40 h

More than 40 h

53.8% 50.6% 56.4%

24.9% 29.8% 25.2%

11.6% 11.7% 10.0%

5.2% 4.3% 5.2%

4.5% 3.6% 3.2%

Preparation courses 45.0% 27.4% 14.8% 15.9% 5.5%

Less than 10 h

10–20 h

20–30 h

30–40 h

More than 40 h

60.8% 60.2% 76.3%

20.2% 24.1% 14.5%

10.2% 9.5% 6.1%

5.3% 3.4% 1.8%

3.5% 2.7% 1.4%

Note: The percentage may exceed 100%, because participants could select more than one answer

Furthermore, questions were asked to find out how teachers taught in TEM preparation classes, the major activities conducted in classes. Activities conducted in TEM preparation classes were varied, because in most schools TEM preparation was carried out in fundamental English or advanced English classes, rather than in a separate course. From Table 5.27, it can be inferred that TEM preparation, such as having students work on the test paper and explain the exercises in the test paper, is often conducted in classes, especially for TEM4. However, textbooks were still used and skill training activities were often carried out. It seems that TEM8 preparation was much less intense than TEM4 preparation, with only one third of the teachers claiming that test preparation activities were frequently conducted in class. From Table 5.28, it can be inferred that TEM would have some washback on the teaching content. More time would be devoted to working on the test paper and explaining it. Also, TEM related skills would be practiced more, such as listening, reading and writing, while non-related skills would be practiced less, such as translation for TEM4 and speaking. It seems that most activities had been intensified in TEM preparation classes, which, according to findings from the interviews, may have been attributed to the tight schedule, for teachers had to finish the textbook and prepare students for TEM at the same time. As a result, more activities would be conducted in the classes.

5.3 Teachers’ Perceptions

131

Table 5.27 TEM preparation classes Frequent Work on test papers 55.1% Explain test papers 52.9% Explain texts 62.9% Work on textbook exercises 59.4% Explain textbook exercises 57.2% Listening activity 61.0% Speaking activity 53.7% Reading activity 64.0% Writing activity 57.8% Translation activity 48.2% Other activity 42.9%

TEM4 Sometimes 26.6% 29.3% 20.7% 21.2% 22.8% 20.4% 24.9% 21.2% 25.4% 27.1% 32.5%

Seldom 19.3% 17.9% 16.4% 19.4% 20.0% 18.6% 21.4% 14.9% 16.8% 24.7% 24.5%

Frequent 37.2% 35.1% 65.3% 57.6% 52.3% 54.6% 46.5% 63.3% 52.7% 55.8% 34.2%

TEM8 Sometimes 34.7% 34.8% 19.6% 24.5% 27.5% 26.5% 30.2% 23.2% 30.9% 29.6% 33.9%

Seldom 28.0% 30.2% 15.1% 18.0% 20.2% 18.8% 23.3% 13.5% 16.5% 14.7% 31.9%

Table 5.28 Difference between TEM preparation and non-preparation classes

Work on test papers Explain test papers Explain texts Work on textbook exercises Explain textbook exercises Listening activity Speaking activity Reading activity Writing activity Translation activity Other activity

More 81.3% 80.4% 23.7% 34.5% 30.3% 65.7% 40.5% 61.4% 65.8% 45.3% 31.1%

TEM4 Same 15.9% 15.8% 35.9% 34.7% 35.6% 26.3% 38.2% 30.0% 26.5% 37.9% 41.9%

Less 2.9% 3.8% 30.4% 30.7% 34.0% 7.9% 21.3% 8.6% 7.7% 16.8% 27.0%

More 73.9% 68.3% 27.1% 37.2% 32.5% 60.3% 39.1% 58.6% 62.9% 62.3% 34.8%

TEM8 Same 20.8% 25.3% 46.1% 37.3% 38.7% 30.1% 41.2% 32.4% 30.2% 30.2% 37.1%

Less 5.2% 6.4% 26.9% 25.5% 28.8% 9.6% 19.7% 9.0% 6.9% 7.5% 28.2%

5.3.4.2 Washback from TEM Revision TEM underwent some substantial revisions in 2005. Since many teachers have more than 10 years of teaching experience, several questions were designed to find out how TEM revision has affected them in classroom teaching. (See Sect. 5.2.4 for the content of revision). It can be noted from Table 5.29 that as for TEM4 revision, it seems that the raised requirement (listening, writing and vocabulary) exerted more influence on course design than the deleted item (fast reading), which is the same as program administrators’ opinion. While for TEM8, the newly added item, General Knowledge, seems to have exerted more influence on English teaching. Moreover, listening and writing revision seemed to influence English teaching more than the revision in reading. As a

132

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.29 Washback from TEM revisions TEM4 Not influenced Not clear Influenced TEM8

Not influenced Not clear Influenced

Listening 4.8% 11.1% 84.0%

Fast reading 18.7% 17.5% 63.8%

Writing 4.1% 10.3% 85.6%

Vocabulary 5.1% 9.5% 85.4%

Listening 5.2% 13.2% 81.6%

Reading 15.5% 17.9% 66.6%

Writing 5.0% 12.3% 82.7%

General Knowledge 3.9% 10.3% 85.8%

new item, it is natural that General Knowledge will affect teachers more, because they had to familiarize themselves with both its content and method, so that they can better prepare students for the test. According to the findings from the interviews and the open questions in the questionnaires, the reason for listening and writing to exert more influence is that these two parts were very difficult, so teachers were more concerned about their revision.

5.3.5 Individual Differences The current research covers teachers from various universities with different background, so an investigation was made to find out whether their perceptions of TEM washback differ (Table 5.30). Several Samples Tests were done and difference among teachers with different educational and teaching background was found to be significant, while there seemed to be no significant difference among teachers from various universities. As shown in Table 5.30, those affected less by TEM (selecting “no washback”) were teachers with higher titles, longer teaching experience, or higher degrees. Moreover, they seemed to take a more critical attitude towards TEM (selecting “negative”). Possible reasons are: (1) better-educated and more experienced teachers tend to be more confident about their teaching, so it is natural that they would be affected less by TEM; (2) the longer one teaches, the more problems they may identify in the educational system, including TEM, so they are more likely to hold a negative attitude towards TEM; and (3) more teachers with bachelor’s degree degrees were found to be negatively influenced by TEM8, which is probably due to the high difficulty level of the test.

5.3 Teachers’ Perceptions

133

Table 5.30 TEM washback on teaching perceived by different teachers TEM4

TEM8 No No Positive washback Negative Positive washback Negative Professor 83.4% 11.0% 5.5% 74.2% 21.3% 4.4% Associate Prof. 80.6% 14.8% 4.6% 79.0% 17.8% 3.1% Lecturer 85.4% 10.1% 4.5% 85.1% 13.1% 1.8% Teaching Assist. 88.5% 9.3% 2.1% 87.7% 11.0% 1.3% Other 90.0% 7.5% 2.5% 75.0% 16.7% 8.4% TEM washback on teaching perceived by teachers with different titles (TEM4: X2 = 23.417, p = .000; TEM8: X2 = 40.568, p = .000) 0–5 years 88.4% 8.8% 2.8% 88.1% 10.8% 1.0% 6–10 years 85.8% 10.0% 4.1% 85.2% 13.1% 1.7% 11–15 years 83.9% 11.0% 5.2% 82.3% 15.3% 2.5% >15 years 78.3% 16.2% 5.5% 76.9% 19.1% 4.0% TEM washback on teaching perceived by teachers with different teaching experience (TEM4: X2 = 31.797, p = .000; TEM8: X2 = 51.992, p = .000) Bachelor 85.0% 11.2% 3.9% 81.2% 14.8% 4.0% Master 85.3% 10.8% 4.0% 84.8% 13.5% 1.7% PhD 78.2% 14.6% 7.3% 72.7% 23.6% 3.7% Other 73.7% 15.8% 10.6% 73.2% 23.1% 0.0% TEM washback on teaching perceived by teachers with different degrees (TEM4: X2 = 13.096, p = .004; TEM8: X2 = 20.038, p = .000)

5.3.6 Factors Influencing Teaching Generally speaking, most teachers agreed that TEM had exerted some positive effects on English teaching. However, is TEM an important factor that may determine their teaching effects? From Table 5.31, it can be noted that TEM4 and TEM8 teachers’ responses were quite similar, considering student quality, student effort and teacher income the three most important factors. In contrast, the influence of TEM is limited, considered important by no more than 10% of the teachers (8.1% and 6.3%). In this view, teachers’ view coincided with that of the program administrators, and it is the desirable role of TEM, to be facilitative rather than decisive in college English teaching. As a result, TEM was not considered a major source of pressure by most teachers, as indicated in Table 5.32. As indicated in Table 5.32, teachers’ pressure mainly came from their sense of responsibility and their work load, while TEM preparation and being evaluated based on TEM results would not lead to pressure for most teachers.

134

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.31 Factors affecting English teaching Student Student Teacher Leader Course Teacher Class quality effort income attention design quality size TEM4 62.0% TEM8 61.2%

57.5% 51.1%

49.8% 42.0%

44.1% 41.8%

42.5% 21,3%

37.9% 36.7%

Teaching facility Research TEM

40.2% 29.4% 34.2% 25.8%

26.7% 23.6%

8.1% 6.3%

Table 5.32 Sources of pressure for teachers

TEM4 TEM8

Teacher’s responsibility 64.2% 65.4%

Too many classes 41.9% 39.2%

Students’ non-cooperation 13.6% 12.1%

TEM preparation 10.7% 10.3%

Evaluation based on TEM 6.8% 6.6%

Other 5.7% 4.9%

5.4 Students’ Perceptions Altogether 17,896 TEM candidates participated in the questionnaire surveys, including 11,033 TEM4 candidates and 6863 TEM8 candidates. The following part will discuss the students’ evaluation of TEM, their perception of TEM washback, and their TEM preparation practice.

5.4.1 Demographic Information TEM4 candidates were mainly second-year English majors and TEM8 candidates were English majors in their fourth year. However, there were also test takers majoring in other fields, such as law, economics, and so on, but they often had English as their minor. Moreover, there were candidates majoring in other foreign languages, such as Japanese, French, and so on. The candidates were from universities of all types, such as comprehensive universities, universities of science and technology, language universities, and so on (Table 5.33). Based on the figures listed in Table 5.33, the following findings can be summarized. First, the number of female test takers far surpasses that of male test takers, which might be attributed to the common phenomenon that there are more girls majoring in English. Second, TEM4 is mainly taken by second-year students, but quite a few thirdyear students take the test, because for those who have failed their TEM4 in the second year, they can take TEM4 again in their third year. Likewise, for those who have failed TEM8 in the fourth year, they can take TEM8 the next year, perhaps after their graduation. However, the number of TEM8 test takers in their fifth year is much smaller than that of TEM4 test takers in their third year. This is not because TEM8 has a much higher pass rate, but that some candidates who fail TEM8 for the first time do not take the test again after their graduation.

5.4 Students’ Perceptions

135

Table 5.33 Demographic information of students Gender TEM4 TEM8 Grade TEM4 TEM8 Major

Male 15.2% 16.4%

Female 84.8% 83.6%

First year 1.6% 0.1%

Second year 76.3% 0.2%

Third year 21.8% 2.9%

English major

English minor 17.3% 6.7%

Other languages 2.8% 0.8%

University of science and technology 15.5% 17.1%

Language university 17.2% 14.9%

TEM4 79.8% TEM8 92.5% University type Comprehensive university TEM4 38.6% TEM8 36.4%

Fourth year 0.3% 96.5%

Fifth year 0 0.3%

Normal university 22.4% 25.8%

Other 6.3% 5.9%

Last, more candidates of other majors take TEM4 than TEM8, which is possibly due to the fact that TEM8 is not a required test for non-English majors in most schools.

5.4.2 Students’ Evaluation of TEM Test takers’ opinions are seldom sought as to whether a test is valid or not (Cheng 2008, 2011; Cohen 2006; Hamp-Lyons 2000a, b). However, research in language assessment has demonstrated strong evidence of test validation from multiple stakeholder perspectives (Cheng 2011; Moss et al. 2006). Therefore, it is essential to investigate test takers’ perception in the validation of TEM. In this view, test takers’ evaluation of TEM is explored in the questionnaire surveys, and the following part will report their familiarity with TEM and their evaluation of TEM. 5.4.2.1 Familiarity with TEM Before investigating test takers’ evaluation of TEM, it is necessary to find out how familiar they are with TEM, such as TEM Test Syllabus, TEM test administration, and so on. As indicated in Table 5.34, TEM candidates were more familiar with the administration of the test, such as its time allotment and score allotment. Meanwhile, they

136

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.34 Students’ familiarity with TEM

Teaching syllabus TEM test syllabus TEM-oral test syllabus TEM marking criteria TEM time allotment TEM score allotment

TEM4 Familiar 35.7% 54.8% 32.5% 47.9% 69.6% 66.5%

Neutral 35.1% 29.9% 31.7% 32.2% 20.4% 20.3%

Unfamiliar 29.2% 15.4% 35.8% 19.9% 10.0% 13.2%

TEM8 Familiar 36.1% 57.0% 32.4% 39.3% 69.1% 64.8%

Neutral 37.2% 30.4% 32.2% 37.3% 22.2% 20.8%

Unfamiliar 26.8% 12.6% 35.4% 23.5% 8.8% 14.5%

were more concerned about the TEM Test Syllabus than the Teaching Syllabus. This conforms to our expectation because candidates care more about the test content (as listed in the Test Syllabus), time allotment and score allotment, so that they can better perform in the test; while the Teaching Syllabus is none of their concern since it is meant to guide teaching for English majors. It should also be noted that the candidates were not so familiar with TEM marking criteria, especially TEM8 marking criteria. 47.9% of the TEM4 candidates claimed to be familiar with TEM4 marking criteria and 39.3% of the TEM8 candidates were familiar with TEM8 marking criteria. This might be attributed to the fact that more subjective items are employed in TEM8, such as translation and writing, which account for 40% of the total score, while there is no translation in TEM4 and the writing part only accounts for 25% of the total. As proved by the interview findings, most candidates surveyed claimed that they were not so concerned about the marking criteria, especially that of writing and translation, because they either “had no access to them” or “had difficulty understanding them”. Moreover, candidates seemed to believe that the marking of writing and translation was “teachers’ business”, while they “had no say” in this matter. This may also account for the fact that translation and writing were less practiced in TEM preparation after class (see Table 5.45). Candidates may feel their efforts in vain because they can’t get prompt feedback on their performance in translation and writing. As shown in Table 5.34, TEM-Oral Test Syllabus was the least familiar to test candidates, 32.5% for TEM4 candidates and 32.4% for TEM8 candidates. Possible explanation is that the test taking population of TEM oral tests (TEM4-Oral and TEM8-Oral) is much smaller than TEM written tests (TEM4 and TEM8). No more than 10% of the written test takers would take the oral tests—this makes speaking an untested ability in TEM for most English majors. As a result, speaking proficiency might be neglected in teaching and learning. 5.4.2.2 Evaluation of TEM Candidates’ opinions are sought as to whether TEM can assess their language abilities validly, both general language ability and individual language skills.

5.4 Students’ Perceptions

137

Table 5.35 Students’ evaluation of TEM TEM4 Satisfy Overall language proficiency 59.2% Learning achievement 57.4% Predict future performance 34.6% Listening 72.2% Reading 69.9% Writing 63.3% Grammar&vocabulary 71.1% (TEM4) Translation (TEM8) General knowledge (TEM8)

Not clear 22.6% 21.0% 26.2% 13.8% 17.9% 18.8% 15.9%

Dissatisfy 18.2% 21.6% 39.2% 14.1% 12.3% 17.9% 12.9%

TEM8 Satisfy 53.8% 53.5% 38.1% 65.6% 68.2% 67.2%

Not clear 27.4% 23.8% 26.5% 17.3% 17.9% 19.4%

Dissatisfy 18.8% 22.7% 35.5% 17.1% 13.8% 13.4%

68.6% 66.0%

19.6% 18.1%

11.9% 15.9%

Table 5.35 shows that most TEM candidates agreed that the individual sections in TEM were able to assess their language ability, such as the listening part can validly assess their listening ability. However, they seemed to be less satisfied with the role of TEM in assessing their overall language ability and learning achievement. There are two possible reasons. First, the overall language proficiency may include more than what is assessed in TEM. For instance, most TEM candidates would not take TEM oral test, so oral proficiency, an indispensable part of overall language ability, is not assessed in TEM. Second, learning achievement in college involves much more than language ability, so it can hardly be assessed by a single test. As to whether TEM can predict future performance, opinions of candidates vary. Around one third of the candidates believed that it could predict their future performance, one third were not sure, and the rest believed that it could not serve such a function. As stated in the test syllabus, the purpose of TEM is to measure whether candidates have met the required language proficiency specified in the Teaching Syllabus. Therefore, TEM is not designed to predict candidates’ future performance, such as whether they are competent for a certain job or whether they are qualified for a bachelor’s degree. In this question, more TEM8 candidates (38.1%) agreed on the predictive role of TEM than TEM4 candidates (34.6%), which might be attributed to the fact that TEM8 is taken at the end of the fourth year when candidates are about to graduate, so TEM8 results are more likely to be used to predict candidates performance in their future jobs. Whether a test can achieve its intended washback depends, to a large degree, on the feedback information it provides, and test report is a basic means of providing information. Therefore, candidates’ opinions are sought as to what information should be provided in the TEM test report. From Table 5.36, it can be inferred that TEM candidates wanted more information than what is provided in the present TEM test report. The scores of each section are considered necessary by most candidates, even more important than the composite score. One possible reason is that candidates would be able to identify their

138

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.36 Information to be provided in TEM test report

Composite score Section score Ability description Rank in the test population Other information

TEM4 Necessary 56.5% 61.0% 47.2% 28.0% 1.0%

Unnecessary 43.5% 39.0% 52.8% 72.0% 99.0%

TEM8 Necessary 61.5% 63.0% 43.5% 23.5% 0.5%

Unnecessary 38.5% 37.0% 56.5% 76.5% 99.5%

Table 5.37 Use and interpretation of TEM results

TEM4 TEM8

Affect ranking Affect employment Affect further education

Whether TEM score is related to degree Yes Not clear No 59.5% 15.5% 25.0% 16.0% 11.9% 72.1% TEM4 Agree Not clear Disagree 34.4% 32.8% 32.9% 59.1% 21.9% 19.0% 53.7% 24.7% 21.6%

Attitudes to relating TEM score to degree Agree Neutral Disagree 44.4% 19.3% 36.3% 44.9% 21.6% 33.6% TEM8 Agree Not clear Disagree 26.9% 29.0% 44.1% 68.1% 17.4% 15.4% 54.2% 22.5% 23.4%

strengths and weaknesses based on the section scores. Moreover, some candidates believed it necessary to describe the language proficiency for each score band, so that they would know what their marks truly mean. Only one fourth of the test takers believed it necessary to provide their rank in the test population, which suggests that candidates are against the competition entailed in TEM. 5.4.2.3 Use and Interpretation of TEM Results Whether test scores are used in the intended way may, to some extent, affect the validity of a test. However, in test use and interpretation, test takers often play a passive role, powerless as to how test scores should be interpreted and used. Nevertheless, it is still important to investigate how they believe the test scores are used and how they are affected by it, so that a comparison can be made between the beliefs of various stakeholders (Table 5.37). According to the candidates, TEM4 scores (59.5%) were more often related to conferring bachelor’s degree degrees than TEM8 scores (16.0%). However, TEM4 and TEM8 candidates held similar attitudes towards relating TEM scores to conferring bachelor’s degree degrees, half agree and half disagree. Besides conferring bachelor’s degrees, TEM results may also affect candidates’ school rank, future employment and further education. The candidates’ responses suggest that TEM results have affected their future employment more than school rank, especially for TEM8 candidates who were to graduate soon.

5.4 Students’ Perceptions

139

Table 5.38 Views on TEM washback

Overall washback Learning aims Course selection Learning plan Learning content Learning method Learning attitude Learning depth Learning speed

Positive 82.4% 76.8% 63.8% 68.9% 70.8% 68.1% 72.3% 69.5% 68.0%

TEM4 Neutral 8.5% 15.2% 26.3% 18.8% 18.1% 20.6% 17.5% 18.5% 20.4%

Negative 9.0% 8.0% 9.9% 12.4% 11.1% 11.3% 10.1% 12.0% 11.6%

Positive 82.1% 76.1% 62.8% 69.7% 70.9% 66.0% 71.5% 71.4% 68.1%

TEM8 Neutral 10.5% 16.8% 29.5% 20.1% 20.7% 24.0% 20.0% 19.8% 22.5%

Negative 7.4% 7.1% 7.9% 10.2% 8.4% 10.1% 8.5% 8.8% 9.4%

5.4.3 Students’ Perception of TEM Washback According to Alderson and Wall (1993), a test will affect what learners learn and how learners learn, such as the learning content, method, rate and sequence, and degree and depth. Therefore, the candidates were first surveyed on whether they were positively or negatively affected by TEM. As indicated in Table 5.38, most TEM candidates believed that TEM has exerted more positive washback on their English learning, especially the overall washback of TEM. Moreover, it seems that TEM has helped candidates set their learning aims, so most of them were quite positive about the washback of TEM on their learning aims. Besides, TEM seems to affect the learning content more than the learning method, with more candidates selecting neutral to the question of the washback of TEM on the learning method. What is least affected by TEM is course selection, with around 30% of the candidates choosing “Neutral”, which is possibly due to the fact that most TEM related courses are compulsory courses, such as fundamental English and advanced English. A further investigation is made on the candidates’ attitudes towards the possible positive and negative washback of TEM—whether they agreed that TEM would generate washback effects, and the findings are listed below. From Table 5.39, it can be concluded that a lot more candidates agreed on the positive washback than the negative washback. More than 70% of the candidates agreed that TEM would provide them with learning aims, so that they would spend more time learning English. Moreover, preparing for TEM could help them clarify the learning requirements and identify their weaknesses, so that they could better arrange their learning tasks, allocating time for each skill training. Besides, most candidates believed that their vocabulary had been enlarged and their learning efficiency improved. As a result, 78.9% of the TEM4 candidates and 72.7% of the TEM8 candidates agreed that TEM had promoted their language learning.

140

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.39 Positive and negative washback of TEM

Increase pressure Provide learning aims Promote learning Improve learning efficiency Spend more time learning Clarify learning requirement Help better arrange skill training Help identify weaknesses Improve English proficiency Improve English through mock tests Enlarge vocabulary Interfere with regular learning Interfere with skill training Satisfy with identifying the correct answer for MCQ

Agree 81.1% 75.3% 78.9% 68.4% 82.2% 64.5% 64.1% 79.1% 74.4% 66.7%

TEM4 Neutral 9.8% 13.9% 11.9% 17.6% 9.9% 20.3% 19.3% 12.8% 14.7% 18.1%

Disagree 9.1% 10.8% 9.1% 13.9% 7.9% 15.3% 16.6% 8.1% 10.9% 15.3%

Agree 82.0% 71.6% 72.7% 61.7% 76.3% 59.9% 62.7% 74.4% 72.1% 65.4%

TEM8 Neutral 10.1% 16.0% 15.5% 22.1% 13.0% 21.8% 19.6% 14.5% 16.5% 19.9%

Disagree 7.9% 12.4% 11.8% 16.2% 10.7% 18.3% 17.7% 11.1% 11.5% 14.6%

85.4% 40.3% 51.6% 38.6%

8.4% 20.5% 18.6% 19.6%

6.3% 39.2% 29.8% 41.8%

75.5% 34.8% 43.2% 40.6%

14.5% 20.9% 20.6% 21.8%

10.0% 44.3% 36.2% 37.5%

However, it is also possible that TEM will exert some negative washback on candidates. For example, preparing for TEM may interfere with their regular learning practice; they would focus more on test paper work than language skill training; and they may lower their requirement, satisfied with identifying the correct answers to the multiple choice questions, instead of seeking the reasons. As indicated in Table 5.39, such effects did exist, with around 40% of the candidates claiming to be affected in a negative way.

5.4.4 TEM Preparation According to Hughes (1993), a test will influence the participants, the processes and the products. TEM may first affect the perceptions and attitudes of the students, and then, their learning activities will differ, so will their learning products. In this sense, a further investigation was made into students’ learning processes and learning effects, to find out how their in-class and after-class learning activities are affected by TEM and what kind of learning effects have been achieved. 5.4.4.1 In-Class Learning TEM preparation courses are provided by some schools and TEM preparation activities are conducted in some classes, so students’ learning in classes may differ from that of the non-preparation stage. In this sense, the students were surveyed on their in-class learning experience, to find out possible differences.

5.4 Students’ Perceptions

141

Table 5.40 TEM preparation courses TEM4

No Several lectures Overall training Skill training Grammar & Vocabulary TEM8

No Several lectures Overall training Skill training General knowledge training

TEM preparation 28.4% 19.1% 32.4% 25.2% 14.6%

Preparation courses 45.9% 18.0% 23.5% 18.2% 8.8%

Less than 10 h

10–20 h

20–30 h

30–40 h

More than 40 h

43.9% 41.9% 48.0%

27.9% 29.0% 25.3%

14.3% 15.4% 13.3%

6.9% 7.0% 7.4%

6.9% 6.7% 6.0%

Less than 10 h

10–20 h

20–30 h

30–40 h

More than 40 h

52.3% 48.9% 60.2%

26.3% 27.7% 19.8%

10.9% 12.7% 10.5%

5.9% 5.6% 5.1%

4.6% 5.0% 4.5%

Note: The percentage may exceed 100%, because participants could select more than one answer Table 5.41 Activities conducted in TEM preparation classes Frequent Work on test papers 57.5% Explain test papers 52.2% Explain texts 57.7% Work on textbook exercises 55.6% Explain textbook exercises 53.8% Listening activity 68.3% Speaking activity 47.5% Reading activity 52.7% Writing activity 61.0% Translation activity 33.5% Other activity 30.2%

TEM4 Sometimes 20.6% 22.6% 20.3% 21.4% 21.3% 19.8% 26.2% 26.2% 23.9% 24.2% 30.9%

Seldom 22.0% 25.2% 22.0% 23.0% 25.0% 11.9% 26.3% 21.1% 15.1% 42.4% 37.8%

Frequent 45.6% 36.8% 46.4% 45.3% 38.8% 57.1% 43.7% 51.5% 49.2% 52.0% 30.1%

TEM8 Sometimes 24.8% 24.6% 22.8% 25.2% 26.1% 25.1% 28.8% 28.4% 29.4% 28.9% 31.2%

Seldom 29.6% 38.6% 30.8% 29.6% 35.1% 17.7% 27.5% 20.1% 21.4% 19.1% 38.8%

From Table 5.40, it can be noticed that TEM4 preparation was more often introduced than TEM8 preparation, since around half (45.0%) of the TEM8 students selected “no preparation”, while only 28.4% of the students selected “no” for TEM4 preparation. When TEM preparation courses were offered, most of them would last no more than 20 h. This roughly coincides with program administrators’ and teachers’ responses. Furthermore, questions were asked to find out the major activities conducted in classrooms (Table 5.41).

142

5 Stakeholders’ Perceptions of TEM and Its Washback

Table 5.42 Difference between TEM preparation and non-preparation classes

Work on test papers Explain test papers Explain texts Work on textbook exercises Explain textbook exercises Listening activity Speaking activity Reading activity Writing activity Translation activity Other activity

More 78.5% 69.8% 35.5% 41.9% 38.0% 65.9% 42.9% 51.9% 62.3% 34.9% 36.2%

TEM4 Same 15.2% 21.2% 30.8% 26.6% 29.5% 24.3% 37.7% 33.2% 26.5% 39.9% 37.8%

Less 6.3% 9.0% 33.8% 31.5% 32.4% 9.8% 19.3% 14.8% 11.2% 25.1% 25.9%

More 63.4% 52.1% 35.8% 42.3% 37.7% 53.8% 41.2% 50.3% 50.6% 52.8% 35.1%

TEM8 Same 26.6% 34.0% 38.5% 34.0% 36.4% 32.6% 38.5% 34.7% 35.4% 32.9% 38.0%

Less 10.0% 14.0% 25.6% 23.7% 25.9% 13.6% 20.1% 15.0% 14.0% 14.4% 26.9%

According to the students, various learning activities were conducted in TEM preparation classes, including test preparation, textbook learning, and language skill training. However, activities not directly related to TEM seemed to receive less attention, such as translation (33.5%) in TEM4 preparation classes and speaking (43.7%) in TEM8 preparation classes. Meanwhile, TEM8 preparation seemed to be less intense than TEM4 preparation, both in test paper practice (45.6% for TEM8 and 57.5% for TEM4) and test paper explanation (36.8% for TEM8 and 52.2% for TEM4). From Table 5.42, it can be inferred that TEM would have some washback on the learning content. Students would spend more time working on the test papers and listening to teachers’ explanation. Meanwhile, TEM related skills would be practiced more, such as listening, reading and writing, while non-related skills would be practiced less, such as translation for TEM4 and speaking. It seems that TEM preparation classes were more tightly organized, with more activities conducted, which is in line with the teachers’ response. Moreover, students’ preference was also investigated. It seems that more than half of the students preferred all kinds of learning activities, and skill training activities enjoyed a slightly higher preference rate. The learning activity with the highest preference rate was listening and the lowest, working on textbook exercises. Generally speaking, students’ preference was in line with the frequency of activities conducted in class (Table 5.43). 5.4.4.2 After-Class Learning It is common practice that students would spend some time preparing for the test before the exam, so an investigation was made into students’ after-class learning activities before TEM, to seek possible differences between preparation and nonpreparation stages (Table 5.44).

5.4 Students’ Perceptions

143

Table 5.43 Students’ preference for classroom activities

Work on test papers Explain test papers Explain texts Work on textbook exercises Explain textbook exercises Listening activity Speaking activity Reading activity Writing activity Translation activity Other activity

Like 55.4% 63.9% 53.2% 44.8% 51.6% 71.1% 66.0% 65.2% 63.4% 57.9% 54.4%

TEM4 Neutral 24.3% 21.5% 28.0% 29.0% 27.4% 17.7% 21.5% 22.9% 21.1% 23.8% 30.6%

Dislike 20.2% 14.7% 18.7% 26.2% 21.1% 11.2% 12.5% 11.8% 15.5% 18.2% 15.0%

Like 52.4% 59.9% 52.5% 45.4% 52.5% 71.8% 67.8% 66.3% 62.5% 67.6% 57.4%

TEM8 Neutral 25.0% 23.1% 27.7% 29.3% 27.0% 17.5% 20.2% 22.4% 20.9% 18.7% 29.4%

Dislike 22.5% 16.9% 19.8% 25.3% 20.6% 10.8% 11.9% 11.3% 16.6% 13.5% 13.1%

Table 5.44 After-class preparation efforts Length of TEM preparation No preparation 1 month TEM4 7.5% 43.2% TEM8 7.6% 61.1% TEM preparation time every day No preparation

A Study on the Washback Effects of the Test for English Majors (TEM)

Recommend Stories

Idea Transcript

Helpful Links

Smile Life

Get in touch