The impact of student satisfaction surveys on staff in HE and FE institutions

Congress Motions passed or remitted - allocated to Education Committee, as approved by NEC 19 June 2010.

Motion 2: Measuring the quality of education

Congress believes that students should have the best possible experience of post-compulsory education. However Congress is concerned that the criteria and methods used to capture the quality of students' experience is narrow, short-term and utilitarian which results in measuring cost effectiveness rather than the real value of educational experience.

Congress further believes that crude ranking of colleges and universities based on these measures results in superficial, inadequate and inaccurate conclusions being drawn about the potential experience of students whilst studying.

Congress identifies that, as well as students, staff should also have a role in measuring the quality of an institution and that dignity and respect for all should be inherent in the process.

Congress calls on the NEC, together with input from regions and branches/LAs, to develop a national campaign to expose this aspect of commodification of the education experience.

Background

The 'culture of complaint' on campus and its consumerist corollary, 'student satisfaction' surveys in HE and FE, did not stem directly from issues intrinsic to universities and colleges but from external developments elaborated by the Conservative government of the late 1980s and early 1990s launch of 'consumer charters': the Patient's Charter, the Citizen's Charter and the Parent's Charter.

The 'Charter Initiative' promoted customer complaints, senior management designed complaint-management systems and 'customer satisfaction surveys' of patients, citizens and parents as a means of encouraging the efficient delivery of public sector services, 'complaints' conceived of as crucial quality improvement management information.¹

The remit of the Parents' Charter ranged over children's services and, crucially, primary and secondary schooling, installing the 'parent-customer' as the key driver of school improvement, simultaneously opening up a window to map the 'Charter Initiative' onto further and higher education, an opportunity enthusiastically taken up by the National Union of Students when it launched its NUS HE and FE Student Charters in December 1992.² In one way or another, NUS' re-invention as an HE and FE consumerist lobby group has been and remains central to the serial cultural transformation and institutional development of the university and FE student as a 'customer'.

The marketisation of public services and the role of customer satisfaction processes, procedures and practices begun under Conservative prime minister John Major grew ever more elaborate within the New Labour administration, ranging across all public services and refreshed by New Labour's importation of 'public value' theories from the Clinton administration, a means to embed and evaluate private sector efficiency measures, 'best value' business practice and an increasing measure of competition into public service provision.

This has been especially prevalent in FE and HE, both nationally and internationally, in the process elaborating whole new interlinked sets of top down HE and FE management theories that have moved from processing student complaints into management quality improvement strategies to methodologies that supposedly capture 'the student experience', in the process becoming the orthodox official doctrine on the 'quality' of HE and FE institutions or individual departments within them.

Introduced in 2006, the HE National Student Survey (NSS) supposedly measures student satisfaction but is equally used to measure, rank and audit the quality of HE institutions, explicitly used by HEFCE and the QAA (who jointly 'own' the NSS) as a bureaucratic auditing tool to hold universities publicly to account for the 'experience' they provide to students.

Unsurprisingly, there have been high profile major disputes about the validity of the NSS process. The most serious broke out in April 2008 between Paul Ramsden, the then CE of the HEA whose Australian student experience surveys were a forerunner of the NSS and Professor Lee Harvey, then director of Research and Evaluation at the HEA and equally an international expert in quality assurance and assessment. Lee Harvey's 'private capacity' letter to the 'THE'³ describing the NSS as a "hopelessly inadequate improvement tool" led to his suspension, publicly condemned by many academics as a breach of academic freedom by the HEA.

Embedding student complaint and satisfaction processes in FECs' quality improvement procedures followed a similar course but were inflected by FE's different funding and inspection regime. In HE, 'Student Charters' were voluntary: in FE, charters were an OfSTED inspected, audited requirement. From the late 1990s onwards these were leveraged up by NUS' 'FE Student Voice' campaign into the LSC 'Framework for Excellence' quality improvement machinery matched in to OfSTED inspection. This required colleges to develop, operate and evaluate 'student engagement strategies' sited within colleges internal quality improvement machinery. A hybrid system was developed by the QAA, with OfSTED input, to measure HE in FE student satisfaction, a mix of student survey and staff peer review.

In both HE and FE national 'student voice' bodies were set up to review the impact of student satisfaction measures in both sectors. Following intense NUS lobbying the 2005 'Foster Review of FE' recommended the establishment of an FE 'National Learner Panel' in 2006, followed a year later by an HE 'National Learner Forum', NUS further valorising 'student voice' in HE by gaining equal status student membership of QAA Institutional Audit Review Teams in 2007.

The Consumer Model

Student: 'I want you to re-mark my work'.

Lecturer: 'Why is that?'

Student: 'I was expecting a much higher mark'.

Lecturer: 'Have you read the feedback sheet I provided?'

Student: 'No, it's the mark I'm not happy with'.⁴

The imposition of a consumer model of education:

• Short-circuits the educational process. A dissatisfied student 'customer' becomes 'proof' of institutional failure, positive student satisfaction results 'proof' of institutional success.
• Hollows out what it purports to assess as students are often unable to distinguish between inadequate and quality education: the ability to discriminate and assess the quality of an academic experience is primarily produced through experience gathered over time, predominantly by practitioners' professional reflection on their pedagogy.
• Excludes academic and academic-related staff from a meaningful role in measuring the quality of their institutions. As a result, both the NSS and the Framework for Excellence template in the FE sector produce data on how well student customers' expectations have been managed rather than the quality of academic experience in institutions.
• Assumes a conflict of interest between the 'student customer' and the education 'service provider' which erodes the trust relationship between teacher and student that necessarily defines both academic and vocational education.
• Pushes students into regarding their education as a commodity that must fundamentally represent 'value for money', channelling students into predominantly complaining about dissatisfaction with marks for essays, examinations, projects, grades awarded and degree classifications.⁵

Student feedback was and is a major focus of the NSS and the National Learner Panel for FE. Led by NUS, the effect has been to accentuate student concern about the quality and timing of lecturer feedback, a concern continually pumped up by NUS' campaigns on feedback to students. This alleged 'quality deficit' - although the NSS itself shows that satisfaction with student feedback improves annually - has nonetheless led both HEIs and FECs to review defensively their institutional arrangements for both feedback methods and assessment.

Complaints about feedback rarely express concern about the absence of genuine dialogue and educational exchange with a teacher - precisely what feedback from a professional educator is for - but are frequently veiled dissatisfaction with marks awarded for projects or examinations, along the lines of 'We were not told what to expect or what we should be doing'.

Academic activity, in colleges and HEIs, does not often 'give customers what they want', neither does academic and vocational education dialogue with and instruction of students through feedback provide the student 'customer' with a clearly defined product, but it does provide students with what they need, the means to access, reflect on and make their own the knowledge, skills, understanding and technique necessary to mastery of their subject.

HEIs and FECs have become very keen for their institutions to score well in student satisfaction rating league tables to avoid complaints and the potential risk of litigation and disputes with 'customers',⁶ producing an institutional form of 'defensive education'⁷ that seeks to minimise occasions for dispute that could lead to complaint, a poor NSS rating, FE student feedback to OfSTED or litigation.

'Defensive education' can institutionally discourage individual educators from the proper exercise of their professional judgement when offering feedback, responding to disputed marks for essays, projects or degree grades or even grades across a whole department as in the recent case of Professor Paul Buckland.⁸ Professor Buckland won an employment tribunal case for constructive dismissal after he failed a quarter of his environmental archaeology students at Bournemouth University whose exam scripts were subsequently arbitrarily re-marked and graded up on the instructions of the chair of the board of examiners and the dean of the school of conservation.

Institutions adopting 'defensive education' strategies will often modify courses to make them more customer-friendly, especially ones that do not gain high ratings in student satisfaction surveys⁹; use 'progressive' marking that validates student efforts rather than highlighting weaknesses in presentation, argument or technique that students need to improve; and discourages tutors in both academic and vocational education from dealing effectively with plagiarism.

Because professional educators in all sectors face complex and unpredictable situations they need a specialised body of knowledge of their subject discipline and an appropriate pedagogy to teach it effectively. To apply that knowledge and practice responsibly and accountably they equally need the autonomy to make their own judgements and the time and space, professionally and intellectually, to reflect on their practice. The consumer model lodged in student satisfaction surveys, the 'complaints culture' and 'student engagement strategies' clearly undermines that autonomy and the professionalism which underwrites a worthwhile education.

Current developments

Prior to the general election David Willetts assiduously courted NUS over the terrain of 'student engagement' and 'student satisfaction', lending a spurious substance to press moral panics concerned with 'dumbing down education', 'Mickey Mouse' degrees and grade inflation. He floated the incongruous notion of either using networked social media or setting up a 'Facebook' style website for students to report on the quality of their student experience despite the wholly adverse reaction from education professionals in all sectors towards websites such as 'rate myprofessor.com'.

The Browne Review of HE¹⁰ argues in chapter 4, 'Enhancing the role of student choice', that student choice 'will drive up the quality of higher education' primarily through the market mechanism but with improved information informing 'customer choice'. Proposals include:

• Improved school-based careers guidance (no mention is made of FE providing over 40% of Level 3 HE entry qualifications!)
• Using HEFCE research on student satisfaction¹¹ (Student evaluation, including standard of teaching and feedback on assessment; Course information, including weekly contact time and the proportion of students employed in a full time professional or managerial job one year after completion and average salary after one year¹²; and finance information) to develop a single web portal integrating the Universities and Colleges Admission Service (UCAS) and the Student Loans Company (SLC)
• 'UCAS will work with institutions to gather the information identified [in the HEFCE research] so that it is available to students when they are applying to higher education'. (Browne, page 31)
• A further recommendation that 'institutions and students work together to produce Student Charters that provide detailed information about specific courses'.
• When developing charters, 'Institutions may want to include commitments to students on the minimum contact time with teachers...and promise timely individual feedback on assignments...[and]...students may decide to include commitments on attending a minimum number of classes or completing a minimum number of assessments per term'.
• There is no mention whatsoever of academic and academic-related staff involvement in the proposed HE charters.
• Disputes about charter commitments will be dealt with by the institution and, failing internal resolution, by the proposed HE Council (a proposed merger of HEFCE, the QAA, OFFA and the OIA).

Recommendations from the Browne Review to thoroughly marketise 'student choice' can only exacerbate the problems already identified with 'student satisfaction' surveys, especially item 4.3 in Browne, which proposes that:

'The UCAS portal (above) will allow students to compare courses on the proportion of students in employment after one year of completing the course; and average salary after one year.'

'Where a key selling point of a course is that it provides improved employability, its charge will become an indicator of its ability to deliver - students will only pay higher charges if there is a proven path to higher earnings...[which] will help students make a better choice about what to study. Courses that deliver improved employability will prosper; those that make false promises will disappear.'

This proposal seems to imply that HEIs and their staff can somehow be held accountable for the occupation and salary that students access a year after graduation. Such proposals:

• proved difficult to action in the USA
• would require unambiguous linkages between education and labour markets which are difficult to theorise (except demand for some professional roles, e.g. teachers and nurses) and even more difficult to effectively operationalise
• do not fit the nature of national labour markets which are always changeable and currently volatile, increasingly subject to ever-changing global labour market pressures where the trend is to push down the price of highly skilled intellectual labour
• would in no sense explain why the fastest growing sub-category within current NEET statistics is for 18 to 24 year olds with 3 A Levels and/or a degree
• fail to take into consideration the equality dimension of graduate (from HE or FE) 'employability'

Recommendations

• Except for some useful information in the regions and individual 'fire fighting' incidents, UCU has no direct body of working knowledge on the impact of student surveys, student engagement strategies and complaint systems on UCU members in HE and FE.
• To support campaigning on this issue and in consultation with members of the Education Committee, HEC, FEC and regional staff, develop a questionnaire to get a measure of the impact of student surveys, etc, on members.
• Use the data and results from the questionnaire to continue work on UCU 2010 Congress Motion 2 where it states: 'Congress identifies that, as well as students, staff should also have a role in measuring the quality of an institution'.
• Similarly, to analyse and support Branches and LAs in responding to any development of the HE charter proposals made in the Browne Review.
• As there are wholly legitimate issues that students raise - reduction in teaching hours, timetabling in general, large seminar groups, essays marked by undergraduates, etc - establishing further contact with the NUS (beyond contact made on fees and funding) with a view to closer collaboration on 'student experience' issues.
• Monitor any private sector involvement in the management of 'learner voice', as there are both established USA-based private sector 'learner voice management' companies and UK private research and facilities management companies¹³ - e.g. 'SHM'¹⁴, who managed the FE National Learner Panel for the LSC - who could exploit the proposals made in the Browne Review on 'student choice'.

Notes

1. A February 1993 Conservative Cabinet seminar defined public service customer complaints as "jewels to be cherished" by public sector managers.
2. The NUS President in 1992, Lorna Fitzsimons, was widely praised for her statement that "students as consumers have a right to quality education, equivalence and choice".
3. Academics criticise HEA for flouting principles of intellectual freedom, THE
4. Posted by a lecturer on the 'THE' website,5 June 2009, in response to Frank Furedi's article 'Now is the age of the discontented', 'THE', 4 June2009
5. In 2008, there were only 900 student complaints - predominantly concerned with marks and grades - to the Office of the Independent Adjudicator (OIA),with a mere 63 upheld, which did not stem an inevitable slew of newspaper headlines about rising complaints and 'the shocking state of Britain's universities'.
6. One of the 1994 Group's three key policy areas as an HE mission group is 'The Student Experience'.
7. A critical category developed by Professor Frank Furedi, analogous to 'defensive driving', in his June 2009 'THE' article.
8. Bournemouth University slammed over remarking of exams
9. Better feedback is an obvious safe issue for NUS to run with but more problematic from a UCU point of view. Highlighting good practice may be fine but UCU members are generally sceptical about the value of the NSS and in some cases fear that it may be used as a stick to beat lecturers in low scoring departments.
10. The Independent Review of Higher Education Funding and Student Finance was launched on the 9th November 2009.
11. Browne, page 30.
12. This practice, with the addition of the earnings rates expected over a 5 year span, is commonplace in many US community colleges.
13. StudentVoice, LLC
14. SHM's projects for the LSC