27 January 2005
The problem with NHS star ratings
Since 2001, the star rating system of NHS acute hospital trusts have given the public the ability to compare one hospital from another using a standard evaluation process. What are the consequences of the results, and is this enhancing patients’ experience in NHS hospitals? Dr Russell Mannion and colleagues at the Centre for Health Economics, University of York present new research in the current issue of the Journal of Health Services Research and Policy, published by the Royal Society of Medicine Press.
The ‘importance’ of
star ratings
Although the star ratings system is not the only process
in place to measure the performance of NHS acute hospital
trusts, the ratings system and its results gain broad media
exposure and is therefore scrutinised by the public. They
are also viewed as a benchmark for internal processes. ‘In
addition to facilitating accountability to patients and
the public, they also serve as an important tool for concentrating
management attention on key strategic priorities and national
targets,’ Dr Mannion writes.
Are
star ratings a ‘balanced scorecard?’
Staff members at some of the hospitals surveyed thought
that star ratings ‘did not represent a rounded or balanced
scorecard of their own organisation’s performance.’ One
of the inconsistencies of the NHS star ratings system are
the sometimes conflicting results of similar, commercially-funded
surveys, the author says. These differences are ‘assumed
to be due to the increased weight accorded to clinical
indicators in the methodology used’ by other organisations,
the article states. Other complaints from staff included
that they did not take into consideration ‘local
contingencies’ and specific ‘mitigating factors
that might help explain variations in the measure performance
of hospitals.’ Many of these unique conditions are
beyond the hospital’s control and it is therefore
viewed as unfair that this would not be taken into consideration
when the rating is calculated.
Inaccurate
data is ‘not in the patients’ best
interest’
This study also highlighted the ‘widespread belief
that the data used to calculate the star rating were often
incomplete and inaccurate,’ Dr Mannion says, to the
extent that the staff at one highly-rated hospital admitted
to ‘purposefully manipulating and misrepresenting data
in order to improve their rating.’ Some of the tactics
used included scheduling cancellations at a time to avoid
unfavourable data and re-classifying trolleys as beds. One
hospital with a low rating thought this was a result of the
accuracy of their data, and another expressed concern that
this practice was ‘not in the patients’ best
interests.’
Responses
to star ratings
In addition to the data and methodology of the star ratings
system being evaluated, the internal effects is also relevant
in determining its integrity. There were many positive
hospital responses to the star ratings, including the alignment
of ‘internal performance management and reporting
systems with key national targets.’ The communication
between central government and hospitals was improved and ‘dysfunctional
senior management’ was identified that otherwise
may have stayed hidden within the hospital, Dr Mannion
asserts. The negative responses included ‘evidence
of tunnel vision and a distortion of clinical priorities,’ reduced
staff morale and public trust, and ‘bullying and
intimidation.’
‘Gaps in knowledge’ and ‘dysfunctional
consequences’
The authors conclude that ‘this study highlights some
important gaps in knowledge and failings in current policy
and practice. That star ratings may be used for good or ill
makes it imperative that the Healthcare Commission monitors
the impact of these data, not only in terms of the degree
to which they lever beneficial change, but also in terms
of any unintended and dysfunctional side-effects they generate
for patients and staff.’
Full article [PDF 105k]
[ends]
