|
Evaluation of
Data Collection Development Project
Project
Objective
To conduct the enumeration of the population of Northern
Ireland by delivering census forms to every household and securely
collecting them for processing.
Background
Census Data Collection was designed to contribute to the
strategic aims of the Census as laid out in the 2001 Census of
Population White Paper (1997)
including “to ensure all aspects of the census data collection
operation…..are acceptable to the public and comply with Data
Protection law”. To this end the primary goals included:-
 |
To
attempt to achieve completeness of coverage of the population
by delivering appropriate census forms to every household and
communal establishment with an efficient use of resources; |
|
To design, produce and distribute
materials that are fit for purpose to the field force; |
|
To maximise the overall level of
quality of incoming data |
|
To enable respondents to post
back their census forms or have them collected for processing
in a secure manner |
|
To provide assistance and help to
the public where and when needed |
|
To ensure the field force was
instructed, trained and managed to meet project objectives; |
|
To maintain confidentiality and
security of information collected from the public |
|
To manage the census operation in
a cost effective and efficient manner to achieve Census
enumeration on 29th April
2001. |
This data collection evaluation summary
is based on current information and will be updated as further
information from other sources such as results from the data
processing exercise becomes available.
Methodology
A key change in 2001 was the opportunity for members of
the public to return their forms by post. The introduction of
postback was designed to help reduce unit costs and enable resources
to be targeted on difficult to enumerate areas.
The Census was carried out by some
2,980 field staff which included a 30% reduction in the number of
Enumerators compared to 1991. The field staff was comprised of 6
Census Area Managers (CAMs), 97 Census District Managers (CDMs), 281
Census Team Leaders (CTLs) and around 2,590 Enumerators (CEs). CTLs
had not been used in Northern Ireland in the 1991 Census and their
introduction was designed to strengthen the quality management role
of the supervisory field staff grade and to allow greater control of
the supply and collection of forms. In the 1991 Census the
equivalent management tier to that of CDM was Census Officer (CO).
There was close collaboration between
the Census Offices in the methodology applied to ensure printing,
logistics, supply of materials, training and response to public
enquiries and liaison with service providers were successfully
achieved, while taking into account necessary variations in the
design of Northern Ireland forms and materials.
Enumeration
Enumerators were given a computer generated map and a
pre-printed list of addresses (within the Enumeration Record Book
– ERB) as well as their instructional material. Unique to Northern
Ireland an information leaflet on the Census was posted to all
households just prior to Enumerators commencing their visits.
Enumerators delivered forms to all households in two phases
during the three weeks before Census day (Sunday 29th
April). The public was asked to complete the form and post it back
in a distinctive yellow pre-addressed envelope.
If additional census forms were required they could be
requested via a Census Helpline.
Local census management teams received and checked the forms.
A period after Census day was allowed
for the forms to reach the local field management teams and for them
to record and check the returns. Flexibility was allowed as to when
enumerators could begin the Follow-up phase to collect non-returns
and resolve queries, and managers were able to re-assign enumerators
between areas. Follow-up (9th – 18th May)
consisted of at least two visits by Enumerators, and a Mop-up phase
carried out by Census Team Leaders and Census District Managers. A
Non-compliance routine was then followed where necessary. Completed
census forms were collected under secure conditions from 281 Census
Team Leaders (CDMs in England) and transported to the processing
centre.
Staff in Census Office in Northern
Ireland made local arrangements with representative organisations to
ensure the enumeration of Armed Forces, prisoners, students in halls
of residence, shipping personnel, persons sleeping rough and Irish
Travellers.
An additional initiative was undertaken
in Northern Ireland whereby postal reminders, census forms and
pre-paid envelopes were issued to householders from whom a Census
return had not been received at the end of the fieldwork period.
Field
checks
New or streamlined checks were introduced:
|
For reconciling the numbers and
types of forms received; |
|
For a focused geography check of
districts prior to enumeration which helped identify potential
problems during the enumeration and assisted with the
allocation and support for enumerators; |
|
For coverage and quality. |
Field
staff learning
As in Great Britain, a greater emphasis was placed on
enabling field staff to work as a team, recognizing the skills of
the workforce and the need to be flexible to target more difficult
to enumerate areas. The most senior field staff (Census Area
Managers – CAMs) were directly trained by Census Office.
Because of the relatively smaller scale of operations some
direct training by Census Office was provided to CDMs in Northern
Ireland and a series of training sessions were held in different
regional locations. Field
managers then trained their own managers who in turn trained their
own teams. The detailed instruction manuals, guides, videos, special
pocket guides and workbooks for Enumerators used throughout the UK
were adapted to reflect any Northern Ireland variations.
Technology
New applications were:
|
The use of software to design
census forms, and the latest print and barcoding technology
for printing of forms; |
|
The design process that took
account of the use of scanning technology for processing
whilst recognising the important aspect of public
acceptability of the form; |
|
Email
and a Field Management Information System (FMIS) for
communications between the Census Area Managers and Census HQ
and Touchphone Data Entry (TDE) for Census District Managers
to report progress with the field operations; |
|
Call Centre technology for the
public Census helplines; |
|
On-line stock control information
for logistics management purposes. |
Outsourcing
Northern Ireland closely followed Great Britain in the
degree of outsourcing used following recommendations made after the
1991 Census. After
assessment of the appropriateness of this route in the Census Test
(1997) and Rehearsal (1999) outsourcing included handling postback;
telephone helplines; designing and printing forms, distribution and
collection of forms, printing of other documents and other materials
and their subsequent disposal. Planning and producing field staff
videos were also outsourced.
Foot
and Mouth Disease Outbreak
Modified enumeration procedures were quickly developed to
conduct the Census during the Foot and Mouth Disease outbreak. Close
liaison was maintained throughout with other agencies such as the
Department of Agriculture and Rural Development to ensure that the
Census would not compromise the work being done to manage the
outbreak. Special methods for delivery and collection were arranged
to ensure that as complete an enumeration as possible was achieved.
Assessment
and Lessons Learned
Initial
Assessment
The Census Office has estimated that around 98% of forms
delivered to dwellings from which a response was expected have been
returned. An exact
figure for the number of people counted in the 2001 Census will be
available in late summer 2002 when all the information, including
the results of an independent Census Coverage Survey to assess the
extent of any under-enumeration, has been analysed. An examination
of the Enumerator Record Books conducted after fieldwork estimated
that of those forms returned just over 92 per cent were returned by
post. It is estimated that approximately 19% of forms were posted
before Census day. Over 4 per cent were collected by field staff in
the follow-up exercise and the remainder as a result of a postal
reminder exercise that was conducted by Census Office.
Enumeration
A survey of CDMs indicated that the use of pre-printed
address lists were regarded as generally accurate and well ordered
and the maps of Enumeration Districts provided were welcomed by
field staff, though
the pre enumeration field check was still regarded as
worthwhile. Virtually all CDMs felt the two phase delivery
procedures were good though the number of calls to the Helpline
requesting forms pre census day suggesting more publicity explaining
that delivery continued up to Census day. The pre census day
delivery was successfully accomplished but inevitably there were
households that did not receive their forms before Census day and
approximately x thousand calls were received on or in the week after
Census day to this effect.
While the postal response rate was
high, some CDM’s reported that the resulting flow of envelopes was
uneven and that they experienced some missorted envelopes in their
returns. This feedback combined with the FMIS information led to a
decision to delay the start of follow-up until 11th May
and to extend the collection by an equivalent period. As the
collection phase continued there were also reports from fieldstaff
that some members of the public were claiming that they had already
posted their form back. The postal flow issues and the effect on
follow-up also resulted in many calls to the Helpline. The postal
methodology allowed greater targeting of non-returns and
strengthening the monitoring of postal flows would further enhance
future operations of this type.
Current information derived from a
sample of ERBs indicate that checks on data quality were undertaken
and about 5% of cases are estimated to have failed the prescribed
checks and resulted in further information being obtained.
Field
Staff
In general CDMs thought the Census 2001 was well planned
and were for the most part positive about Census Office’s role.
The majority of CDMs surveyed responded very favourably to the
training with particular reference to the quality of the videos.
Virtually all the CDMs felt the exercise books were well structured
and about the same thought the role playing exercises worthwhile.
Some CDMs and CTLs felt that they could have been given more detail
earlier in the process. Others remarked that the number of forms was
too high leaving an impression of bureaucracy and providing too much
information to absorb. A consensus emerged that CTLs played an
important role in being able to manage closely and quality assure
the work of their teams. Some CAMs felt that the CDM:CTL ratio could
have been increased slightly.
About 4% of Enumerators had to double
up as a result of either a shortfall of candidates or resignations.
As in Great Britain, some CDMs and CTLs also had to cope with
heavier than anticipated workloads particularly when responding to
revised enumeration procedures as a result of Foot and Mouth and the
high postal response. The
standardized procedures are felt to have worked well in conveying
the same message to everybody. It was felt that the Census Office
training to CDMs helped reduce the communication gaps and that in
any future operation such direct training should cover the CTLs to a
greater extent. The instructions and training programme were
considered successful, though the opportunity to further simplify
some of the materials (e.g. payroll forms) should be examined.
Regional
Management
The relatively small scale of field operations in
Northern Ireland facilitated good communication between CAMs, CDMs
and Census Office. This enabled generally fast response times and
Census Office and CAMS worked collectively to issue numerous written
communications to CDMs to respond to issues in a dynamic fieldwork
environment. The Field Management Information System generally
worked well in Northern Ireland although several malfunctions were
experienced and there was an overhead in following up field staff
who were unable to deliver reports on time. CDMs generally welcomed
the FMIS, though there were mixed views on the number of reports
required. Most welcomed the information it brought, though some
thought the reporting time window too narrow.
Foot
and Mouth
Liaison with DARD and the farming agencies was very good.
The special arrangements involving the delivery and collection of
supplies and completed census forms seems to have worked well. There
is no evidence to date that the response was seriously affected by
the outbreak. There were a few complaints from members of the
farming community about Enumerators transgressing farm boundaries.
Not all of these proved justified on examination and written
instructions had been issued to CDMs in good time.
Public
Enquiries
The Helpline had been planned on the basis that there
would be somewhere in the region of 6,000 to 8,000 calls for
Northern Ireland. However, in effect, some 47,655 calls were
received with 4,866 of these on the busiest day (23rd
April). During 1 April to 30 June 2001, 56% or 26,687 calls were
dealt with by the Integrated Voice Response (IVR) system and of
these 99% or 26,420 callers were successful in getting through. This
remained the usual daily success rate, but at the period of peak
demand (23-24th April) this reduced to 91%. After this,
the number of call centre operators was increased and a direct line
to Census Office in Northern Ireland was opened and advertised. Many
of the calls before Census Day were requests for Census forms: some
of which might have been avoided if there had been more information
indicating that form delivery continued up to Census Day. About,
3,500 calls during the whole census period were requests for field
staff assistance which were directed to CDMs. Overall, 1.4% or 381
calls could be regarded as a complaint and most of these related to
the Census in general and census field staff. There were 3434
queries about individual questions and of these (177 queries)
referred to the question on qualifications.
Response time to the surge in demand
approaching Census Day was good. However, an
examination of the call profile leading up to and immediately
after Census Day should assist in the planning of future Census
Helpline operations and resource requirements. The preference of
members of the public to speak to an adviser is noted, though the
cost implications need to be taken into account.
Forms
Design and Logistics
In terms of the design and production of public forms
NISRA considers these to have been largely successful but some
people did find the Relationship Matrix to be slightly confusing.
The quality of the public forms in terms of their design and print
were suitable for the automatic processing systems. The organization
of supply and distribution programme enabled supplies to reach the
field staff efficiently. Timing problems were encountered at times
and reprints needed. All completed census forms and materials were
collected from around 380 locations and transferred safely to the
processing centre or for disposal. Suppliers were felt to be
reasonably flexible in delivery times and the supplies delivered
matched requests. Most CDMs felt that the system for delivery of
ad-hoc supplies worked well. Some redistribution of supplies between
CDMs was anticipated and this was for the most part managed
successfully at the local level.
NISRA received few reports of
difficulties with storage facilities and this was felt to justify
the decision to locate forms storage with CTLs rather than with CDMs
as in England and Wales.
Outsourcing
and Contract Management
Northern Ireland benefited from the economies of scale
through working with ONS and GROS colleagues on delivering service
through externally contracted providers. NISRA was also required to
build and manage partnerships with service providers at the local
level to ensure the ability to respond to the dynamics of the census
operation and in general this worked well. NISRA
also had to deal with the same contract management issues
that applied to the larger organizations and while this is much less
than if the service was developed in house it nevertheless made
considerable demands. In general, relationships with service
providers on Data Collection Development were constructive. NISRA
concurs with the ONS view that the procurement approach used for
this project – i.e. contracting services rather than systems –
was appropriate and welcomed by service providers.
Conclusion
Conducting a Census is a complex and challenging
logistical exercise. The social and technical issues that the 2001
Census has had to deal with, the large amount of change taken on
board, amid an outbreak of Foot and Mouth disease and the prospect
of a General Election, made Data Collection more difficult than
anticipated.
Preliminary information indicates that
98% of households from which a response could be expected was
achieved and this can be regarded as a positive outcome to the Data
Collection exercise. More accurate information on the degree of
coverage achieved will be known in late Summer 2002 and the extent
of any differential under-enumeration.
The flexibility to delay the
commencement of follow-up was important and such contingency needs
to be built into planning. The difficulty in communicating a
standardized message via a number of management levels to the
fieldforce should not be underestimated. Predicting the number of
calls to the Helpline was a difficult task. While additional
publicity might have helped to alleviate calls about
non-availability of forms, the awareness and willingness of the
public to be involved was reflected in the number of calls.
The Foot and Mouth outbreak presented
significant opportunity costs to the overall management of the
Census and stretched existing resources. The Republic of Ireland
took the decision to suspend their Census and careful co-ordination
was required to maintain the effectiveness of the operation in
Northern Ireland without posing a risk to the rural community.
The degree of co-operation experienced
between the Census Offices and outsourcing was of benefit in
enabling a relatively small Census Office conduct an operation of
this scale. There nevertheless remain considerable overheads
associated with managing such a diverse range of projects. Detailed
planning is essential with careful attention given to
interdependencies between projects and even then events differ from
census to census.
The 2001 Census in Northern Ireland
received widespread community support and also managed to avoid any
damaging controversy which might have impacted on the overall
response rate.
|
