Northern Ireland Statistics and Research Agency - Census - Census Evaluation

On Sunday 29 April 2001, a Census of Population was held in Northern Ireland and Great Britain. Carrying out the Census is a huge undertaking involving every household in the land. Work on the 2001 Census began almost as soon as the last census in 1991 had been taken. Each stage of development involved much detailed planning and testing. A key component in this programme was that of Census Data Collection.

Planning and development of the Census Data Collection project had to take account of changes in the way people live, substantial advances in technology, the growth in single person households, changing work patterns and a potentially less compliant society. To this end, Census Data Collection featured as one of the four broad strategic aims of the Census as laid out in the 2001 Census of Population White Paper (Cm 4253) published in March 1999. These were:

Objectives

The objectives of the Data Collection Development Project were:

	To attempt to achieve completeness of coverage of the population by delivering appropriate census forms to every household and communal establishment with an efficient use of resources;
	To design, produce and distribute materials that are fit for purpose to the field force;
	To maximise the overall level of quality of incoming data;
	To enable respondents to post back their census forms or have them collected for processing in a secure manner;
	To provide assistance and help to the public where and when needed;
	To ensure the field force was instructed, trained and managed to meet project objectives;
	To maintain confidentiality and security of information collected from the public; and
	To manage the census operation in a cost effective and efficient manner to achieve Census enumeration on 29^th April 2001.

Methodology

Due to changes in society, family structures, work patterns and advances in technology, the business scope of Data collection for the 2001 Census had to be considerably broadened. Early on in the development of the 2001 Census Programme it was decided that the Data Collection Project should be divided into Data Collection Development and Data Collection Support. The Census Data Collection Development Project involved:

The Census Data Collection Support Project, details of which are included under its own Project Evaluation report, involved:

Close links between the Geography and Data Collection Support and Data Collection Development projects enabled the effective and expeditious planning of enumerator workloads, facilitating the supplies of field maps and other materials and assisting the conduct of pre-Census geography checks.

A key change in 2001 was the opportunity for members of the public to return their forms by post. Introduction of postback was designed to help reduce unit costs and enable resources to be targeted on difficult to enumerate areas.

This evaluation describes how the Data Collection Development project was developed and implemented, the methodology used, the outcome and the lessons learned. However, it should be explained that this is based on current information and will be updated as further details from other sources such as results from the data processing exercise become available.

A detailed examination of the various tasks to be performed by each level of field staff was undertaken and modifications introduced in light of experiences in the Census Rehearsal. In Northern Ireland, the Census was carried out by some 2,970 field staff reflecting a 30% reduction in the number of Enumerators compared to 1991. Field staff composition was 6 Census Area Managers (CAMs), 97 Census District Managers (CDMs), 281 Census Team Leaders (CTLs) and around 2,590 Enumerators (CEs). CTLs had not been used in Northern Ireland in the 1991 Census and their introduction was designed to strengthen the management of the field staff grade and to provide greater control of the supply and collection of forms. In the 1991 Census the equivalent management tier to that of CDM was Census Officer (CO).

There was close collaboration between the three UK Census Offices in strategic planning, contracting out of various services, printing, logistics, supply of materials, training and response to public enquiries and liaison with service providers. However, account was taken of the necessary variations in the design of Northern Ireland forms and materials.

Enumerators were given a computer generated map and a pre-printed list of addresses (as part of the Enumeration Record Book – ERB) as well as their instructional material. Unique to Northern Ireland, during 2^nd to 7^th April 2001 the Royal Mail using door-to-door services delivered an Advance Information Leaflet on the Census to residential addresses just prior to Enumerators commencing their visits. The purpose of the Advance Information leaflet was to alert householders to the forthcoming Census and their requirement to complete a census form. This was followed by the Delivery Round, the purpose of which was to ensure that a Census form was delivered to every household in NI before Census Day (29^th April). The Delivery Round was carried out in two phases. The first phase from 9^th to 20^th April consisted of a single visit. If contact was made, a census form was left, if not a ‘No contact’ leaflet was left at the address. During the second phase from 21^st to 27^th April, Enumerators returned to deliver census forms to addresses where they had not previously made contact during Phase One. If there was still no contact at single household addresses, Enumerators left a census form addressed ‘To the occupier’ and a pre-addressed return envelope. At addresses where it was thought there was more than one household, a further attempt to make contact was initiated. If no contact was established, sufficient census forms, based on a best estimate of the number of households, together with pre-addressed envelopes were left and addressed ‘To the Occupier’. The two phased Delivery approach was aimed at ensuring that enumerators didn’t delay the start of their work until too near Census Day (29^th April). The public were instructed to post back their Census forms in the distinctive yellow pre-addressed envelope on, or as soon after Census Day as possible. If additional census forms were required they could be requested via a Census Helpline.

NI Census Office staff made local arrangements with contacts in representative organisations to ensure the enumeration of Armed Forces, prisoners, students in halls of residence, shipping personnel, persons sleeping rough and Irish Travellers. These arrangements included members of homeless organisations accompanying Census Office staff to identify anybody who was absolutely homeless on Census Day (habitually sleeping rough) and for the enumeration of hostels for the homeless. Census Office took responsibility for enumeration of any communal establishment with an expected 100 or more usual residents and treated these as special enumeration districts.

As stated earlier, a key change in enumeration procedures for the 2001 Census was the opportunity for the public to return their census forms by post. The aim of postback was to remove the necessity for Enumerators to call back at every household to collect census forms, thereby enabling field resources to be focused on difficult to enumerate areas and groups of the population. A secondary effect was to reduce the total number of Enumerators required thus alleviating some of the recruitment difficulties. The main postback period was from 30^th April to 29 May. In Northern Ireland, CDMs began collecting posted back census forms from their local Royal Mail Delivery Office from 30^th April and distributed them to their teams of CTLs and Enumerators for checking and recording receipt in the Enumeration Record Books (ERBs). New and streamlined completeness checks were introduced to focus attention on areas where the potential for error was greatest. The checks were:

In addition, CTLs checked the responses to four key questions for each person on the Census form thereby enabling them to determine the non-returns and incomplete forms for the enumerators to follow-up. The CTL role was critical in ensuring that checks were carried out to monitor the progress and effectiveness of enumerators in the field. These checks also assisted CDMs/CTLs in identifying areas where enumerators needed assistance. CDMs in turn were also able to check that enumeration procedures were being followed and that every address listed had been visited.

Follow-up took place from 9th to 18^th May and consisted of at least two visits by Enumerators to households to collect outstanding forms. If contact was not made on the first occasion, Enumerators were asked to leave reminder leaflets to try to prompt a postback. If unsuccessful, then they were to make another visit. If no contact was made on the second visit a second reminder leaflet was left and a note of the date and time made in the ERB. During this period, CDMs continued to call at their local Royal Mail Delivery Office to collect postal returns. Some flexibility was allowed as to when enumerators could begin the follow-up phase to collect non-returns and resolve queries, and managers were able to re-assign enumerators between areas. In addition to the follow-up visits by Enumerators, a Mop-up phase was carried out during the same period by CTLs and CDMs. The Mop-up exercise was aimed at giving CTLs a final opportunity to retrieve as many outstanding census forms as possible, or if necessary, commence non-compliance action. A non-compliance procedure was then followed where necessary. In NI, completed census forms were collected under secure conditions from 281 CTLs and transported to the processing centre.

An additional initiative was undertaken in Northern Ireland from Census Office whereby postal reminders, census forms and pre-paid envelopes were issued to householders from whom a Census return had not been received at the end of the fieldwork period. This resulted in forms being returned and was regarded as a more valuable initiative.

As in Great Britain, a greater emphasis was placed on enabling field staff to work as a team, recognizing the skills of the workforce and the need to be flexible to target more difficult to enumerate areas. Training began with training courses for CAMs, the first tier of field staff appointed. Census Office staff were responsible for imparting to the CAMs the important messages and instructions of Census fieldwork. In NI, because of the relatively smaller scale of operations Census Office provided CDMs with direct training in addition to that delivered by CAMs. CDMs in turn trained their own teams of CTLs and Enumerators.

Training of field staff was carried out on a ‘cascade’ basis timed to coincide with, and immediately precede, key events in the field. The objective was to ensure that the training was consistent across the whole of the country. Detailed field training packages were produced and designed to enable them to be used by people with minimum experience of delivering training.

The detailed instruction manuals, guides, videos, special pocket guides and workbooks for Enumerators used throughout the UK were adapted to reflect any Northern Ireland variations. There were also guides on health and safety in the field. The field staff instructions were supplemented as necessary to meet issues arising such as the access restrictions to farms in light of the Foot and Mouth disease outbreak that occurred prior to Census Day.

Following recommendations made after the 1991 Census, an important change in the undertaking of the 2001 Census was the decision to outsource a substantial part of the Census operation. Northern Ireland closely followed Great Britain in the degree of outsourcing involved. After assessment of the appropriateness of this route in the Census Test in 1997 and the Census Rehearsal 1999 outsourcing included handling postback; telephone helplines; designing and printing forms, distribution and collection of forms, production and distribution of of other documents and materials and their subsequent disposal. Planning and producing field staff videos were also outsourced.

The greater use of outsourcing of the 2001 Census activities introduced a range of external expertise and knowledge and necessitated effective management of risk transfer from the Census Offices to the external service providers. Allied to this was the acquisition by the staff of the three Census Offices of contract management skills necessary to ensure attainment of the required standards and quality of services required.

The Public Enquiry Unit (PEU) was set up to deal with enquiries from the public during the enumeration of the 2001 Census. Census publicity material and the NI Census forms carried the local telephone rate Helpline number. The public could telephone this number to access a number of services. A front-end Interactive Voice Response (IVR) system operated that allowed calls to be routed quickly and thereby reduce the burden on the Helpline Advisors. In addition to providing a cost effective filter the IVR was also a contingency tool for handling upsurges in call volume. On telephoning the Helpline, the caller would be given information about the services available and given the opportunity to get further information on some Frequently Asked Questions (FAQs). After this the caller could opt to connect directly with an Advisor, to request assistance with completing the Census form, or ask for more detailed questions; request Census Fact Sheets, additional/replacement Census forms, envelopes; or indicate a desire to communicate in Irish or Ulster Scots. In addition, blind callers could request Braille information sheets and /or audio tapes.

The PEU Helpline was in operation from 1st April until 30^th June 2001. It was estimated that during this time the Helpline would deal with 8,000 calls and that the peak week for calls would be week beginning 30 April 2001. In fact there was a total of 47,655 calls to the helpline (i.e 6 times the number anticipated) and the peak week was week commencing 23 April 2001. The peak day for calls was 23 April 2001.

Of the 47,655 calls made to the Helpline, 43,149 (90.5%) resulted in access to the IVR. Some calls involved multiple enquiries and accordingly the 18,925 calls handled by advisors were logged against 21,925 enquiries. The type of enquiries received fall largely into six categories and included – complaints, requests for assistance, question and answer topics, and ‘other’. The most frequently asked questions of the 3,434 categorised as ‘question and answer’ topic enquiries were:

The ‘General Census queries’ of which there were 1,205, were not specific enough to enable them to be categorised into precise question type enquiries but 136 of them could be classed as ‘Census Form’ questions and most of these involved the enquirer having misunderstood when the Census form had to be returned by and was simply seeking clarification, other enquiries were from concerned householders asking whether they would be fined if the Census form was not returned by Census Day (29^th April). Near the end of April a notice was place in the main newspapers advising that those households who had not received a Census Form could phone Census HQ direct and request one. In all there were some 381 enquiries categorised as complaints of which 338 were dealt with by the PEU advisors, 9 were directed to Census Office and 34 of the former were subsequently re-directed to Census Office. Some of these were about field staff operations during the Foot and Mouth outbreak while others were from householders expressing dissatisfaction with the content of the Census form, especially the inclusion of questions on Irish, Religion and Qualifications. Some callers were complaining, prematurely, that they had not received their form. The request was passed on to the relevant Census District Manager to ensure a Census form was delivered. Of the 18,925 calls to an advisor only 364 (1.9%) needed redirecting to Census HQ.

Table 1 below provides details of the total number of calls received from the public about the NI Census:

Table 1: Calls to the Helpline (1 April 2001 - 30 June 2001)
Week Beginning	No of calls
1 April	57
9 April	2,333
16 April	5,150
23 April	17,234
30 April	8,562
7 May	2,272
14 May	3,604
21 May	1,854
28 May	948
4 June	620
11 June	298
18 June	154
25 June	63
Total	43,149

Modified enumeration procedures were quickly developed to conduct the Census during the Foot and Mouth Disease outbreak. Detailed advertisements in the local and farming press were provided to advise householders in rural areas how their forms would be delivered. Close liaison was maintained throughout with other agencies and primarily with the Department of Agriculture and Rural Development to ensure that the Census would not compromise the work being done to manage the outbreak. Special methods for delivery and collection were arranged involving post out and post back to ensure that as complete an enumeration as possible was achieved.

In the White Paper on the 2001 Census of Population (Cm4253) the Government stated that the Census Offices would undertake a review or reviews of the confidentiality and security arrangements for the 2001 Census. Security consultants were employed to ensure that effective security measures were in place to cover every stage of the distribution, collection and processing of census forms. The ‘Independent Security Review’ of Census security and confidentiality carried out on behalf of the three Census Offices ensured that key security and confidentiality requirements applicable to the Data Collection phase were reinforced through the training of all levels of field staff. Instructions and training programmes included appropriate references to the importance of security and confidentiality. Every person employed by the Census Offices and contractors involved in providing services to them, either in the field or elsewhere, were required to sign a Census Confidentiality Undertaking, which indicated they understood their responsibilities and were aware of the penalties involved.

The relatively small scale of field operations in Northern Ireland facilitated good communication between CAMs, CDMs and Census Office. This enabled better response times to issues raised by the field staff. Census Office and CAMS worked collectively to issue numerous written communications to CDMs to respond to issues in a dynamic fieldwork environment. A Field Management Information System (FMIS) was designed to assist the flow of information from the field to Managers. The system worked on the basis of information flowing from Enumerators to CTLs to CDMs. The latter would then make their report to their CAMs using Touchphone Data Entry (TDE). Each grade of field staff was given a report manual showing the information to be relayed and the date it was due.

Assessment and Lessons Learned

The Census Office has estimated that around 98% of forms delivered to dwellings from which a response was expected have been returned. An exact figure for the number of people counted in the 2001 Census will be available in late summer 2002 when all the information, including the results of an independent Census Coverage Survey to assess the extent of any under-enumeration, has been analysed. An examination of the Enumerator Record Books conducted after fieldwork estimated that of those forms returned just over 92 per cent were returned by post. It is estimated that approximately 19% of forms were posted before Census day. Over 4 per cent were collected by field staff in the follow-up exercise and the remainder as a result of a postal reminder exercise that was conducted by Census Office.

A survey of CDMs indicated that the use of pre-printed address lists were regarded as generally accurate and well ordered and the maps of Enumeration Districts provided were welcomed by field staff, though the pre enumeration field check was still regarded as worthwhile. Virtually all CDMs felt the two phase delivery procedures were good though the number of calls to the Helpline requesting forms pre census day suggests that more publicity explaining that delivery continued up to Census day might have helped. The pre census day delivery was successfully accomplished but inevitably there were households that did not receive their forms before Census day and over 3,000 calls were received in Census Office on or in the week after Census day to this effect.

While the postal response rate was high, some CDM’s reported that the resulting flow of envelopes was uneven and that they experienced some wrongly sorted envelopes in their returns. This feedback combined with the FMIS information led to a decision to delay the start of follow-up until 11^th May and to extend the collection by an equivalent period. As the collection phase continued there were also reports from fieldstaff that some members of the public were claiming that they had already posted their form back. The postal flow issues and the effect on follow-up also resulted in many calls to the Helpline. The postal methodology allowed greater targeting of non-returns and strengthening the monitoring of postal flows would further enhance future operations of this type.

Current information derived from a small sample of ERBs indicate that checks on data quality were undertaken by enumerators and about 4-5% of cases were estimated to have failed the prescribed checks and resulted in further information being obtained.

In general CDMs thought the Census 2001 was well planned and were for the most part positive about Census Office’s role. The majority of CDMs surveyed responded very favourably to the training with particular reference to the quality of the videos. Virtually all the CDMs felt the exercise books were well structured and about the same thought the role-playing exercises worthwhile. Some CDMs and CTLs felt that they could have been given more detail earlier in the process. Others remarked that the number of forms was too high leaving an impression of bureaucracy and providing too much information to absorb. A consensus emerged that CTLs played an important role in being able to manage closely and quality assure the work of their teams. Some CAMs felt that the CDM:CTL ratio could have been increased slightly.

About 4% of Enumerators had to double up as a result of either a shortfall of candidates or resignations. As in Great Britain, some CDMs and CTLs also had to cope with heavier than anticipated workloads particularly when responding to revised enumeration procedures as a result of Foot and Mouth and the high postal response. The standardized procedures are felt to have worked well in conveying the same message to everybody. It was felt that the Census Office direct training to CDMs helped reduce the communication gaps and that in any future operation such direct training should cover the CTLs to a greater extent. The instructions and training programme were considered successful, though the opportunity to further simplify some of the materials (e.g. payroll forms) should be examined.

Communications between the various levels of census field staff generally worked well in Northern Ireland. However, the FMIS posed problems whenever malfunctions were experienced and there was an overhead in following up field staff who were unable to deliver reports on time. CDMs generally welcomed the FMIS, though there were mixed views on the number of reports required. Most welcomed the information it brought, though some thought the reporting time window was too narrow.

Liaison with DARD and the farming agencies was very good. The special arrangements involving the delivery and collection of supplies and completed census forms seems to have worked well. There is no evidence to date that the response was seriously affected by the outbreak. There were a few complaints from members of the farming community about Enumerators transgressing farm boundaries. Not all of these proved justified on examination and written instructions had been issued to CDMs in good time.

The Census Helpline was planned on the basis that there would be somewhere in the region of 6,000 to 8,000 calls for Northern Ireland. These figures were based on calls received during the 1991 Census and included an uplift factor. However, increased telephone ownership and availability and repeated calls by callers who experienced confusion when connected to the Integrated Voice Response (IVR) technology, are factors to be considered when preparing future Census Public Helplines installations. The higher than expected number of calls to the Helpline also included misunderstandings by some members of the public. These mainly related to concerns about the delays in delivery of census forms, and the fear of penalties being imposed if forms were not returned returned by 29th April (Census Day). However, taking account of these factors, some 47,655 calls were received with 4,866 of these on the busiest day (23^rd April). During 1 April to 30 June 2001, 90.5% or 43,149 calls were dealt with by the IVR system and of these 45% or 18,925 callers subsequently spoke to an advisor. This remained the usual daily success rate, but at the period of peak demand (23-24^th April) this reduced to 55%. After this, the number of call centre advisors was increased and a direct line to Census Office in Northern Ireland was opened and publicised in the NI press as quickly as possible for those persons wanting to request a Census form. Many of the calls before Census Day were requests for Census forms: some of which might have been avoided if there had been more information indicating that form delivery continued up to Census Day. About 3,500 calls during the whole census period were requests for field staff assistance that were directed to CDMs and can be regarded as successfully actioned. Overall 1.4% or 381 calls could be regarded as a complaint and most of these related to the Census in general and census field staff. There were 3,434 queries about individual questions and of these (177 queries) referred to the question on qualifications.

An examination of the call profile leading up to and immediately after Census Day should assist in the planning of future Census Helpline operations and resource requirements. The preference of members of the public to speak to an adviser is noted, though the cost implications need to be taken into account.

In terms of the design and production of public forms NISRA considers these to have been largely successful. The quality of the public forms in terms of their design and print were suitable for the automatic processing systems. The organization of supply and distribution programme enabled supplies to reach the field staff efficiently. Timing problems were encountered at times and some reprints needed. All completed census forms and materials were collected from around 380 locations and transferred safely to the processing centre or for disposal. Suppliers were felt to be reasonably flexible in delivery times and the supplies delivered matched requests. Most CDMs felt that the system for delivery of ad-hoc supplies worked well. Some redistribution of supplies between CDMs was anticipated and this was for the most part managed successfully at the local level.

NISRA received few reports of difficulties with storage facilities and this was felt to justify the decision to locate forms storage with CTLs rather than with CDMs as in England and Wales.

Northern Ireland benefited from the economies of scale through working with ONS and GROS colleagues on delivering service through externally contracted providers. NISRA was also required to build and manage partnerships with service providers at the local level to ensure the ability to respond to the dynamics of the census operation and in general this worked well. NISRA also had to deal with the same contract management issues that applied to the larger organizations and while this is much less than if the service was developed in house it nevertheless made considerable demands. In general relationships with service providers on Data Collection Development were constructive. NISRA concurs with the ONS view that the procurement approach used for this project – i.e. contracting services rather than systems – was appropriate and welcomed by service providers.

Conclusion

Conducting a Census is a complex and challenging logistical exercise. The social and technical issues that the 2001 Census has had to deal with, the large amount of change taken on board, amid an outbreak of Foot and Mouth disease and the prospect of a General Election, made Data Collection more difficult than anticipated.

Preliminary information indicates that 98% of households from which a response could be expected, was achieved and this can be regarded as a positive outcome to the Data Collection exercise. More accurate information on the degree of coverage achieved will be known in late Summer 2002 and the extent of any differential under-enumeration.

The flexibility to delay the commencement of follow-up was important and such contingency needs to be built into planning. The difficulty in communicating a standardized message via a number of management levels to the fieldforce should not be underestimated. Predicting the number of calls to the Census Helpline was a difficult task that should have taken greater regard of the wider availability of telephone technology and greater public ownership. Notwithstanding this though, confusion experienced by some of the public when connected with IVR technology and about the timing of the delivery of Census forms, generated repeat and additional calls which only exacerbated the situation. Additional appropriately timed publicity might have alleviated the position.

The Foot and Mouth outbreak presented significant opportunity costs to the overall management of the Census and stretched existing resources. The Republic of Ireland took the decision to suspend their Census and careful co-ordination was required to maintain the effectiveness of the operation in Northern Ireland without posing a risk to the rural community.

The degree of co-operation experienced between the Census Offices and outsourcing was of benefit in enabling a relatively small Census Office conduct an operation of this scale. There nevertheless remain considerable overheads associated with managing such a diverse range of projects. Detailed planning is essential with careful attention given to interdependencies between projects.

	Back to Top
	Last updated 12 November 2004
	© 2000-2003 - The Northern Ireland Statistics and Research Agency (NISRA)
	Enquiries : info.nisra@dfpni.gov.uk \| Webmaster : info.nisra@dfpni.gov.uk \| Contact Information
	An Agency within the Department of Finance and Personnel

Data Collection Development Evaluation Report

Contents

Executive Summary

Evaluation of The Northern Ireland Census Data Collection Development Project

Introduction

Objectives

Methodology

Assessment and Lessons Learned

Conclusion