Peer Assessment

Peer Assessment

Article Information
Category:

Description

Peer Assessment involves the observation, by an experienced operator, of the operator carrying out the exercise.

Discussion/References

Peer Assessment techniques are essentially a craft skill, and are rarely discussed in the published literature.

  • EEC Report # 164 (David and Noonan, 1983) discusses the pitfalls of peer assessment. In particular, ATC working methods, which reflect operators’ experience of widely varied sectors, are extremely variable. This presents a danger that the peer assessor will either grade the operator in terms of how difficult he thinks he would himself find the traffic, or of the extent to which the operator uses the same techniques as he does.
  • Dubey (1998) in a social anthropologist’s study of real-time simulation discusses the emotional links that develop in operator and pilot teams. These may well affect peer assessments.
  • ANON, RE ASSESSMENT OF THE CAPACITY OF THE CONTROL SECTORS OF MAASTRICHT UAC., Feb 1984, pp 6 + 21 tables. presents an example of peer assessment for capacity assessment.
  • Oppenheim A.N., Questionnaire Design, Interviewing and Attitude Measurement, 1992 (1st ed. 1996), Cassel: London (ISBN 1 85567 044 5) gives a general (non-ATC specific) introduction to this type of measurement.
Categories
Human Factors Methods
Generics
Type of method Subjective
Target of method Strain, Stress
Peer Assessments are generally used to assess workload, but in terms of how it affects the operator. It is therefore inclined to be ambiguous about what it actually measures.
Time Scale of method Hours
Peer Assessments usually require at least an hour of continuous assessment, although the assessor may record his opinion at five minute intervals, it usually requires a complete simulation to provide a definite assessment.
Portability of method Yes
In principle, Peer Assessments, since they need no equipment, can be held wherever and whenever convenient - during a working shift in an operational centre, for example.
Observer Effect Yes
The operator will be aware that his actions are being assessed by another operator, and may modify his behaviour, either by being over-precise, or by trying to appear relaxed and ‘on top of the job’.
Context of studies
Laboratory studies Use
 
Simulation studies Avoid
 
Field studies Caution
 
Potential problems with the method
Failure risk Moderate
Some assessors find that they cannot make assessments, particularly where a novel system is being introduced.
Bias risk High
The assessor, unless carefully briefed, may either start to assess the traffic rather than the operator, or may mentally impose his own ‘style’ of control as a standard.
Ethical problems Privacy
Operators carrying out assessments must not discuss their observations of specific operators in public.
Costs of the method
Staff Cost High
Peer Assessments must always be carried out by skilled experienced operators, capable of understanding the operators’ actions and judging their relevance and effectiveness. One assessor is needed per operator measured.
Set-up Cost High
Considerable effort must be invested in training and standardising the assessors.
Running Cost High
The cost of the staff required is high.
Analysis Cost Low
Peer Assessments are summarised by the assessor at the end of each exercise. Formal analysis is neither required nor needed.
Analysis data
Analysis Speed Fast
Conclusions are available immediately.
Data Automation No
Not needed. Assessments could be input and handled in questionnaire form if required.
Analysis Automation No
 
Status Established
Routinely used (in National ATC Centres).

SKYbrary Partners:

Safety knowledge contributed by: