NISRA Home

Census Home

Census Review and
Evaluation

CCS Evaluation Report

Contents

Executive Summary

Overview

	Introduction
	Aim of the survey
	Nature of the survey
	Sample design
	Data collected by the survey
	CCS Management structure
	Timing
	Progress reporting

Evaluation

	Scope
	Project Sections
	Progress Reporting
	Emergencies and changed procedures
	Recruitment
	Field methodology
	Maps
	Geography and workload planning
	Training
	Logistics
	Public forms
	Public helpline
	Overall survey performance

Evaluation of the Northern Ireland Census Coverage Survey

Overview

Introduction
This report provides a high level overview and evaluation of the practical aspects of the Northern Ireland Census Coverage Survey (CCS).

The purpose of this report is to highlight procedures that are considered to have worked well in running the survey, to make note of those that didn't work so well, and to document the lessons learned from the experience.

The Aim of the Survey
The main aim of the CCS was to collect the information required to estimate the coverage of the 2001 Census. The coverage itself will be estimated by comparing the information collected in the CCS with the Census returns from the Survey areas.  The results will be used to calculate the adjustments required in the “One Number Census” (ONC) outputs.

A further aim of the CCS was that it should also yield some information about the quality of the 2001 Census data.

The Nature of the Survey
The Survey took the form of an independent, intensive re-enumeration of a sample of postcodes involving a short doorstep interview (as opposed to using a self-completion questionnaire like that used in the Census) with each household that could be contacted in the sampled postcodes.

The Survey Questionnaire used for the interviews was designed to be as concise and manageable in the field as possible.  A matrix format was used to enable the Interviewers to see the response spaces for as many members of the household as possible on one page.  The questionnaire, like the Census form itself, was designed for capture by scanning and recognition.

The Interviewers firstly checked if the people they contacted were resident at the address in question on Census Day.  They then carefully checked on who was in the household to ensure that no households or residents were missed.  They asked probing questions (helped by a prompt card and/or prompts on the interview form) about those groups likely to be under-reported, such as babies, students and young people, and the elderly.  The Interviewers also enquired about visitors as an aid in identifying the household correctly but the information about visitors was not recorded.

Sample Design
The Northern Ireland Census count in 1991 was about 1.6 million in 3,729 EDs and the population was projected to be about 1.7 million by 2001, suggesting the use of 3 Estimation Areas for 2001 CCS purposes. The country is split up into 26 Local Government Districts (LGDs) of which Belfast is easily the largest in population terms, with a population of about 300,000. The LGD boundary defines the urban area very tightly, and many areas that might be considered as suburbs of the city are in neighbouring LGDs such as Castlereagh and Newtownabbey. A consequence of this is that the population of the LGD area is declining as people move to the suburbs. However, it is still important and likely to differ in terms of underenumeration from the rest of Northern Ireland. Therefore Belfast was considered as a design group on its own. The rest of the LGDs were grouped into the standard East and West classification as shown in Table 1.

This classification means that, using 1991 Census figures, the population of the Belfast design group was 

279,215, the population of the East design group was 700,364, and the population of the West design group was 598,111.

Enumeration District Type (EDT) Index for Northern Ireland
In England and Wales, EDs have been classified using a Hard to Count Index which incorporates local indicators, such as multi-occupancy housing and ethnic minority populations, which research has demonstrated to be related to underenumeration. For Northern Ireland no similar research is available, and at least some of the indicators have very low incidence levels.

The stratification of 1991 EDs in Northern Ireland was based on observed response rates to Northern Ireland’s (voluntary) 1997 Census Test, which was designed as a fractional replicate of a 2x2x3 experiment where EDs were the sampling units, classified by predominant religious background (3 levels), urban/rural and deprived/non-deprived. [Further details on the classification methods used in the 1997 Census Test and the observed response rates can be found in NISRA Occasional Paper Number 13 (1999)].

The religious background classification was reduced from 3 levels to 2 on the basis of similar response rates, giving eight initial strata for the EDs. While it was desirable to spread the sample over all the eight categories it was not possible to estimate independently in all eight. Therefore, estimation used a three level categorisation that combined the categories. Levels 1 to 5 formed an ‘easy to count’ group containing about 33 per cent of the population, levels 6 and 7 formed a middle group containing about 50 per cent of the population, while level 8 formed a ‘hard to count’ group.

Allocating the Sample at Stage One
The approach used in England and Wales forms the basis of the allocation, with some specific differences. The number of Northern Ireland EDs sampled was specified using the same sampling fraction as for England and Wales (approximately 3.6 per cent). For Northern Ireland this implied a first-stage sample of 134 EDs. An initial allocation to the design group by collapsed EDT index was made proportional to the population sizes within the groups. The use of population at this stage rather than number of EDs reflected the fact that the East design group was the largest in terms of population but had less EDs than the West design group. If any allocation was less than eight EDs this was forced to equal eight and the proportional allocation was repeated for the remaining groups. This was to guarantee sufficient sample size within each collapsed EDT index category for estimation. The specified sample of EDs was then proportionally allocated (by number of EDs) to the full EDT index. This ensured that although the sample was designed for estimation using the specified collapsed categories the sample was spread across all eight categories and allowed estimation within a different set of collapsed categories.

The ONS design assumed a second level of stratification below EDT index based on population size. The problem was choosing which age-sex ED counts to use as the size variable. In England and Wales this was solved by constructing a design variable based on the first three principal components derived from six age-sex groups (males 0-4, females 0-4, males 20-24, males 25-29, males 30-34, females 85+) that suffered high underenumeration in 1991. The within index stratum boundaries were then defined using minimum variance cluster analysis on the three principal components. Optimal allocation based on the design variable was used to allocate a pre-specified within index sample to the size strata such that the relative standard error (RSE) for the estimate of the design variable total was minimised. (RSE is the standard error of the estimated total expressed as a percentage of the total and is also called the ‘coefficient of variation’.)

The same approach was adopted in Northern Ireland. However, proportional allocation, rather than optimal allocation, was used to allocate the specified sample to the within EDT index size strata such that the relative standard error (RSE) for the estimate of the design variable total was minimised across the whole design. This approach was not necessarily as ‘efficient’ as the ‘optimal’ allocation approach but it had one advantage, it spread the sample evenly across all the strata. This is important for two reasons; firstly it is robust when you have little information about the expected pattern of underenumeration, secondly it looks fair to the user if they perceive there to be little information about the expected pattern of underenumeration.

Allocating the Sample at Stage Two
In England and Wales, simulation work  suggested that a random selection of five postcodes from each sampled ED (or less in situations where the ED does not contain five postcodes) was a good compromise between clustering for cost efficiency and spreading the sample of postcodes for statistical efficiency. The expected household sample was approximately 75 households but this was subject to considerable variation.

Northern Ireland’s experience in the 1999 rehearsal was that the distribution of postcodes (in terms of population size) was very skewed with many postcodes having very small numbers of households. Accordingly, in Northern Ireland, postcodes were successively drawn at random until the number of households in the sample set reached at least 70, when sampling stopped. In practice, most of the sampled postcodes contained approximately 75 households.

The Data Collected by the Survey
The CCS interviewers collected the following information:

	the address including the postcode;
	basic information about the house;
	information on each person; name; relationship within household; date of birth; sex; marital status;
	whether a student and if so whether at term-time address;
	religion;
	whether the person had a different address one year ago; and
	economic status

The Survey used pre-coded classifications similar to those used in the Census. No write-in responses were required for the “Other” category in the religion question.

A postcard informing residents about the survey was left at each address during the initial property listing round of each area.  In addition, Interviewers had an official letter from NISRA to give to residents seeking further reassurance of the official nature of the survey.  Unlike the Census, the 2001 Census Coverage Survey was voluntary.

At the end of the fieldwork a self-completion questionnaire was left at each address where no contact has been made, in a final attempt to secure a response.  In an attempt to secure as many interviews as possible, CCS interviewers were not informed in advance that they would be completing their fieldwork in this way.  A stamped and addressed envelope was provided for sending the questionnaire back to NISRA.

The CCS Management Structure

Headquarters Staff
The survey was directed by the CCS Project Manager who was located in Demography and Methodology Branch, NISRA.

There were 3 NISRA HQ based CCS Team Managers – one for each of the Estimation Areas. The Team Managers were trained by the GRO Scotland CCS Project Team with regard to fieldwork management procedures, interviewers training and management, fieldwork progress reporting and logistics.

Field Staff - Interviewers
There were 100 Central Survey Unit interviewers used for the CCS.

Field Staff Pay
Team Managers, as members of the Central Survey Unit, received their normal rates of pay and working conditions. Interviewers likewise were paid by the CSU in accordance with their 2001 hourly rate.

Field Staff Training
The Team Managers were trained by the GRO Scotland CCS Project Team. The training was delivered in two parts; the first part covered the background to the Survey and Team Managers’ duties; the second part covered the training programme for the Interviewers.

The Team Managers trained the interviewers in accordance with their own training and survey field instructions.

Timing
The survey was carried out as soon as practicable after the Census in order to minimise the effect of migration and the problems faced by respondents in recalling the position on Census Day.  Interviewer field duties (geography checks and property listing round) began on 17 May; actual interviewing began on 21 May. 

Interviewing duties ceased on 27 June.

Progress Reporting
Northern Ireland employed a “low tech” approach to reporting weekly progress of the interviewers on the CCS. The interviewers were supplied with paper pro-formas which they completed weekly on both an actual and cumulative basis. They were then posted into the Survey HQ weekly for collation by the Team Managers and reporting to senior management.

Evaluation

Scope
This report is limited in scope to comment on the practical aspects of the survey.  It makes no attempt to evaluate the statistical impact of:

	How well the survey picked up people missing from the Census - this information is not yet available;
	How well the One Number Census procedures have worked.

There is very little in the way of quantitative assessment. Full analyses of response rates and other measures can only be done once the CCS and Census processing is complete.  As a result the report concentrates mainly on the qualitative aspects.

Project Sections 

Field Management Structure
The hierarchical structure enabled the survey to be closely managed at all times for all areas of the country despite the  Headquarters Team consisting of only 3 Team Managers and 1 Project Manager. The reduced number of levels in the management structure actually facilitated the speed of communication through the hierarchy.

The original survey design allowed for three Headquarters based Field Managers to manage 6 Team Managers working in the field. Resource constraints imposed by the Central Survey Unit however meant that this was not possible. The fact that there were no Team Managers working in the field necessitated a change to the progress reporting arrangements.

Lessons learned
Formal contract arrangements need to be employed at all times (even within different parts of the same organisation) in order to make sure that the level of staffing originally promised is provided.

Progress Reporting
Rapid communication to and from the field is essential so that problems encountered can be dealt with as soon as possible.  The absence of Team Managers in the Field meant that weekly progress reporting meetings between Interviewers and Team Managers were not possible. Instead, Team Managers relied on Interviewers posting their weekly update sheets in to the Survey Headquarters, at which point the information provided was keyed onto a spreadsheet.

The main problem with this method of reporting is that the onus is on the Interviewer to prepare a hard copy report and take it to a point of posting. There was no requirement for them to have their reports ready for specific meeting dates and therefore the possibility of slippage existed and was exhibited. This, together with delays in the postal system itself, meant that some weekly progress reports could be up to a week late. In several cases the Team Managers even had to telephone the Interviewers concerned  to prompt their returns.

Lessons learned
The method finally employed, as a result of insufficient management resource in the field, left a lot to be desired.  The Interviewers need to have reporting deadlines imposed on them or face the consequences (whatever these are deemed to be) if these are not met. The postal system is too slow and irregular for Headquarters management to be guaranteed comprehensive reports on a regular weekly basis. The employment of some sort of electronic reporting mechanism, as was the case in Great Britain, would have been more efficient.

Emergencies and Changed Procedures
The Foot and Mouth outbreak in the UK was an unforeseen problem that coincided with the timing of the CCS.  Emergency procedures had to be compiled and disseminated to field staff quickly to ensure the adverse impact of the disease was kept to a minimum.

At all times Headquarters staff were able to devise alternative interviewer plans within a very short timeframe and to disseminate them into the field.  The reactive capability of the Headquarters team was excellent.

The field staff were highly receptive to the necessary alterations to survey procedures.  They provided useful feedback on the state of the situation at all times.

Lessons learned
Some allowance must be made in the resource allocation of the survey project for the chance that something unexpected will occur during the survey period.  This includes consideration of funding as well as staff time available.  A degree of contingency time for Headquarters staff should be built into the survey plans.

Recruitment
As NISRA employed its own Central Survey Unit Interviewers to carry out the CCS interviews, recruitment was not an issue.

Lessons learned
Although the CSU Interviewers are highly proficient in their jobs the CCS Project Manager’s own findings in this respect were that the experienced Interviewers were too set in their ways with regard to their survey/interviewing techniques, that they were not receptive enough to following the CCS instructions to the letter. It is the Project Manager’s view that ‘fresh recruits’ would have been more predisposed to specific CCS instructions.

Field Methodology
Most of the field methodology used was developed as a direct result of the findings of the 1999 CCS Rehearsal and other pilot exercises.  All the methodology was geared towards achieving a high response rate to a voluntary survey and therefore finding as many people as possible who had been missed by the Census.

There is real evidence that the interviewers found whole properties that were missing from other address lists during their on-the-ground property listing phase.

Having a public interface in the form of the interviewer meant that any questions members of the public had about why the survey was being run could be answered straight away.   Persuasion could be used to keep compliance rates up, and the form-filling burden for the public was non-existent.

The recommended calling strategy (suggesting that interviewers vary their calling times in order to increase the likelihood of finding the householder at home at some point in the survey period) limited the number of wasted visits

The postback "last chance" option involving a self-completion questionnaire for all households not contacted by the last day of the survey resulted in a further 2 per cent response.

There is anecdotal evidence that the probe questions on the survey form worked well, with people who would have otherwise been missed being found in households.

Lessons learned
The field methodology implemented is sound - the final response rates to the survey are testament to this. 

Maps
Interviewers were provided with street maps for their postcode areas.  The maps highlighted a rough boundary for the selected postcodes but this was only to be used as a guide.  On-the-ground checks by the interviewers were used to identify households within the selected postcodes rather than identification from the maps or other means.  This meant that if any of the maps were out of date in any way, the property listing round managed to account for them.

Geography and Workload Planning
This was a complex area that involved manually grouping selected postcodes together into interviewer workloads based on a number of criteria (e.g. expected household count, distance apart).  There was a real need to keep them as equitable as possible to avoid conflict between interviewers. In general however this exercise  was successful. The interviewers considered the size of their workloads (in terms of number of households) to be about right. The whole workload planning was kept entirely independent from the Census operation and the sample treated with strict confidentiality at all times.

Training
Training was delivered by GRO Scotland in a two part residential course to Team Managers and passed on by the Team Managers to the Interviewers.  All training material was developed by GRO Scotland and the Northern Ireland CCS Headquarters team.  The training covered all aspects of the survey including the history of the Census and Census Validation Surveys, conducting an interview, geography, documentation, logistics etc. The Northern Ireland Team Managers who attended the courses in Scotland found it to be very informative and beneficial in getting across the importance of the CCS. It was also particularly useful for explaining all of the paperwork involved in the survey.

Logistics
A survey operation the size of the CCS required some careful planning of logistics, on both the delivery and the pick-up of supplies in the field.  It required close liaison with ONS and GROS on forms design and delivery. All  completed survey forms were delivered to the processing centre in time for processing to commence.

TNT, the delivery contractor, was efficient and flexible. Any problems were dealt with swiftly and efficiently.

Lessons learned
Careful monitoring of the delivery of all items to and from the field is absolutely essential.  Headquarters need to know immediately of any problems arising.  The amount of resource required to perform this function should not be underestimated.

Public Forms
The survey questionnaires were developed and tested during the rehearsal and other pilot exercises carried out by ONS on behalf of the three UK Census Offices.  Although the forms were designed to be completed by the interviewers there was still a requirement for clarity in order that  information recording mistakes did not occur.  The choice of questions, particularly the inclusion of probing questions, were designed to find people within households who would otherwise have been missed.  Therefore as well as finding missing households (using the property listing field methodology) the nature of the questioning meant that it should also possible to find missing people within households. In addition to this the employment of clear and simple showcards kept the interview time short - multiple choice responses could be displayed to the householder rather than asked. The late decision to merge individual country forms into a single common UK form for contractual reasons did not present any difficulties for the experienced interviewers.

The colour of the form however did present some problems in that a number of people got it confused with their Census form and wondered why they were being asked to complete a second Census form.

Public Helpline
A helpline number was provided that allowed members of the public to phone in if they had any questions regarding the purpose or legitimacy of the CCS.  Unlike the Census helpline number the CCS one was not widely publicised as it was relevant to only the sampled 1.6 per cent of households.  Instead it was  printed on official explanatory letters and given out by interviewers on request. Of the 40 calls received, all were from people who had been uncontactable and had therefore received their CCS form in the post.  About three quarters of these were from people who had got the CCS form confused with their Census form, the remainder were complaints, predominantly from people asking why they had to answer the CCS questions when they had already given their answers to these questions on their Census forms.

Overall Survey Performance
It is worth recording the overall performance of the CCS because this is a good indicator of the balance between things that worked well and things that did not go entirely to plan.  The number of people found who had been missed by the Census is the key measure of its success, and although this will not be known until processing is complete evidence gathered so far indicates that the survey will have met its objectives. Of particular note is the response rate (number of successful interviews achieved as percentage of properties found) of 92 per cent.