NISRA Home

Census Home

Census Review and
Evaluation

Data Needs Evaluation Report

Contents

	Executive Summary
	Objectives
	Background
	Methodology
	User consultation
	Research and question testing
	Consultancy
	Implementation
	User's requirements for information
	Testing
	Northern Ireland specific questions
	Form design
	Agreed content of the census questionnaire
	Lessons learned
	Conclusion
	Notes

Objectives
The main objective of the Data Needs Project was to ensure that users’ requirements for 2001 Census information were met as far as possible through the inclusion of relevant questions and the design of acceptable Census forms. This paper outlines the process of selecting topics, developing appropriate questions and suitable Census forms and broadly describes the user consultation and testing programme involved.

As another aim of the 2001 Census was to ensure a high level of comparability across the UK in terms of the questions asked and the forms design, the three UK Census Offices employed a ‘joined-up’ approach to the data needs research and question development.

Background
The tasks undertaken by the Data Needs Project were part of the complex programme of statistical work which constituted an integral part of the planning for the 2001 Census. While some activities were based on tried and tested practices, others required new approaches and working methods. In particular, significant changes were made to accommodate the scanning of forms using optical character and optical mark recognition, and to work effectively with external contractors and sub-contractors.

In many respects the direction and focus of the work was determined by the findings of a comprehensive review of the 1991 Census which underlined the importance of consultation as well as the need to demonstrate that the census was the best and most effective means of meeting users’ requirements for information.  At the same time detailed evaluation highlighted a number of problems with individual questions and the layout of the form which in some cases had an adverse effect on the quality of the data. These findings underlined the need to develop new ideas for the overall look and detailed design of the census forms as well as the importance of examining the format and wording of individual questions.

The overall scope of the work undertaken by the project reflected these key priorities. Detailed responsibilities included:

	all aspects of user consultation on 2001 Census content;
	co-ordination of the small-scale testing programme;
	research and evaluation of possible questions and alternative designs for the census forms;
	assessment of alternative sources;
	proposals for the content of the 2001 Census and the design of the public forms drawing on consultation with users, assessment of business cases and evaluation of results from the testing programme.

While these core tasks constituted the majority of the work undertaken by the Data Needs Project, activities associated with the development of appropriate definitions and classifications including the provision of comprehensive meta-data will be considered as part of the work on census output.

A full and final evaluation of the work undertaken by the Data Needs Project can only be accomplished when work on processing and analysing data from the Census forms has been completed and the quality of the data assessed.

Methodology
The NISRA Data Needs Project Team worked in conjunction with staff from the Census Division of ONS and representatives from the General Register Office for Scotland. This joint approach contributed to the overall efficiency, co-ordination and effectiveness of the work.

The nature and diversity of the work determined the approaches that were adopted. Thus an extensive programme of consultation was complemented by qualitative and quantitative research and question testing as well as advice from specialist consultants. An outline of each of these approaches enables the nature and complexity of the key tasks to be understood.

User Consultation
The Census offers a unique opportunity to obtain a wealth of information on households and individuals on a consistent basis at national and local levels. A key strategic aim was to ensure that the content of the census meets the requirements of users.

The importance of consultation was recognised from the outset of the programme for the planning and development of the 2001 Census. A number of working groups and sub-groups were set up across the UK in addition to the established Census Advisory Groups. These groups were made up of census users and other experts drawn from central and local government, the health service, academic and business communities, who were able to provide detailed advice and guidance on topics including ethnicity, religion, qualifications, disability, migration, and the labour market. A wide range of papers were prepared to provide regular feedback on work in progress, methodological developments and test results. Regular meetings of these core groups were complemented by bilateral discussions, special events and presentations as well as extensive correspondence.

Research and Question Testing
An extensive programme of around 40 tests was conducted. Although this work was carried out primarily by the ONS Question Testing Unit, members of the Data Needs Projects in each of the Census Offices were responsible for a small number of tests. Work began in April 1995 and was completed in December 1999.

A variety of techniques, including focus groups, cognitive interviewing, small-scale and large-scale tests as well as qualitative follow-up surveys, were used to assess the overall acceptability of individual questions, the understanding and interpretation of specific terms. These detailed investigations were complemented by two comprehensive field tests - the 1997 Census Test and the Census Rehearsal in 1999.

Consultancy
On a number of occasions specialist advice was obtained from external experts. At an early stage in the question testing programme, graphic design consultants were invited to put forward new ideas and approaches for the overall look and design of the census form, to produce illustrative designs in both matrix and page-per-person formats and to provide advice on specific graphic features including routing. These requirements were reinforced by the need to obtain independent advice on the suitability of specific designs for scanning and storage purposes. In the final stages of preparation for the printing of the Census forms the Census Offices worked with specialist print experts to ensure that each document complied with the detailed specifications required to meet scanning requirements, contractual obligations and individual country sensitivities.

Implementation
This section focuses on the processes associated with identifying information requirements, selecting topics, developing appropriate questions and suitable forms for the 2001 Census. This work relied on extensive consultation with users, members of the Census Offices, contractors and sub-contractors as well as a comprehensive programme of research, question testing and evaluation. In practice this work was conducted in parallel, with feedback from users, findings from the testing programme and advice from consultants contributing to the on-going process of refining requirements, improving questions and informing decisions on the layout of the census forms. However, for the purposes of this review individual aspects of work will be examined in turn.

Users’ Requirements for Information
The importance of ensuring that the content of the Census met the requirements of users was a key strategic aim of the 2001 Census. Consultation began in 1995 and continued until final decisions about the content of the Census were agreed by the respective legislatures for Northern Ireland, England & Wales and Scotland. The process of consultation can be divided into three phases:

	identification of user requirements for information;
	the preparation of detailed business cases;
	clarifying outstanding issues with a view to finalising proposals in time for the preparation of the Census legislation.

User groups were asked to provide advice and guidance on changing requirements for information including priorities for the development of new questions. Early consultation focused on amending and updating a provisional list of topics, drawn up by the Census Offices, based primarily on needs identified for the 1991 Census.

Detailed comment and review enabled this list to be amended and updated to take account of changing information needs. As a result the Census Offices were able to refine requirements and produce an amended list of 62 questions for which there was prima facie justification. The number of questions was reduced after further analysis and assessment, either because there was not sufficient evidence to substantiate the need for the information, or because the data could be obtained more effectively from other sources, or were not suitable for a Census. As a result of this iterative process questions including private education, private pensions, date of most recent marriage or divorce and age of property were excluded from further consideration.

The second stage of the consultation involved the production of business cases for each possible question. During this phase special attention was given to the interaction between questions and the value of combinations of questions rather than single topics. The business cases were prepared by Census users from central and local government, the business sector, professional organisations and a number of special interest groups. All cases were based on detailed guidance provided by the Census Offices regarding overall format and content. It was agreed that all questions required a detailed business case, apart from counts of households and counts of people by age and sex which were considered to be an integral part of any Census. The main criteria used by the Census Offices to determine the selection of questions included:

	the information is needed by a substantial number of users;
	the information is not available from any other source in the detail or at the area level required;
	it is practicable to collect good enough data to meet users’ needs; and
	the topic and the form of the question are acceptable to the general public.

Each business case needed to provide clear evidence of the way in which information would be used, what users would do if the data were not available and the risks associated with the loss of information. These key points were to be supplemented by information on the availability of alternative sources of information and comments on why the Census was essential, including details of what would happen in the absence of the required data. Written submissions were complemented by a table showing the interdependencies between questions and uses with ratings of 'essential’, 'valuable’ or ‘relevant’ attached to the variables associated with each use to indicate overall importance. These tables provided a basis for computing an overall score for each possible question, taking into account the importance of each question for each use as identified by the user.

The results of this consultation were used in conjunction with research findings and test evidence to identify possible questions for inclusion in the 2001 Census. In addition, special arrangements were made to keep users informed of developments associated with decisions on the inclusion of new questions such as income.

Testing
Decisions about the format and wording of questions for the 2001 Census were based on a comprehensive programme of question testing. This was designed to deal with a number of issues including:

	users’ requirements for new questions in the 2001 Census;
	the need to overcome problems with existing Census questions;
	the importance of updating questions to take account of underlying changes in society;
	the development of new classifications most notably the National Statistics Socio-economic Classification.

The testing programme may be divided into two key stages:

	in phase one the Census Offices focused on developing questions for a major field test in June 1997; and
	in phase two efforts were concentrated on dealing with any outstanding work and rectifying problems with existing questions.

During phase one the overall scope of the small-scale testing programme was determined as priorities and limitations were set. Decisions took account of comments from users including members of the Working Group on Content, Question Testing and Classifications and Census Advisory Groups. Special attention was given to any major new topics and thus much early work focused on developing questions on income, disability and caring. At the same time a considerable amount of time and effort was spent on reviewing, updating and improving topics that had been included in the 1991 Census. For example, detailed work was undertaken on extending the relationship question to collect information on all relationships within the household including concealed families and stepchildren. Other work resulted in the development of a comprehensive tick-box question covering all formal qualifications as well as minor amendments to the format and wording of questions on type of accommodation, tenure, number of rooms, number of cars and marital status. These revisions informed decisions on the questions to be included in a major field test in June 1997.

One of the primary objectives of the 1997 Census Test was to assess public reaction to the new questions included on the form. More specifically, attention focused on assessing the acceptability of including a question on income.

The 1997 Census Test used both quantitative and qualitative methods. This test involved 9,000 households drawn from a number of (1991 Census) enumeration areas chosen to be representative of the socio-economic circumstances of Northern Ireland in terms of its urban/rural split, its level of relative disadvantage and its community background. A follow-up survey, based on interviews with 780 respondents and 213 non-respondents, carried out by the NISRA Central Survey Unit provided information on public reaction.

The 1997 Census Test was statistically designed to allow comparison of response rates for households that received forms with and without an income question. Overall, although the response rate for forms with an income question was 2.4% lower than the response rate for forms without an income question, which was statistically significant at the 5 per cent level, there was no evidence to suggest that the inclusion of the income question made any difference on a practical level. There was some evidence however to show that the inclusion of an income question appeared to effect the response rate to some of the individual questions on employment and, as a result of this and qualitative information which conveyed a general disike for the inclusion of such a question, the decision was taken to leave income off the Census form.

In addition to this specific investigation, details of completion rates enabled the overall performance of individual questions to be assessed. In general results were positive, but respondents had difficulties with a number of questions, particularly those on economic activity, number of employees and qualifications. These problems were compounded by difficulties with filter questions as well as perception problems which affected understanding and interpretation. These findings were used to determine priorities for the final stage of the testing programme.

During the last major phase of testing, work on untested questions or new topics was combined with efforts to resolve problems with existing questions. Thus a series of small-scale tests were conducted using the ONS Omnibus Survey to investigate the feasibility of developing new closed questions on occupation and industry. This work was carried out in conjunction with investigations designed to assess the merits of individual relationship questions, improve the employment questions, notably those designed to be compatible with International Labour Office definitions, and overcome specific problems identified in the 1997 Test including the shortcomings of the question on qualifications.

Much of this work was brought together in two Whole Form Tests carried out in June and September 1998. The results of this work contributed to a number of decisions which had direct implications for the questions included in the Census Rehearsal including endorsement of an amended version of the relationship matrix. Special attention was also given to who should answer the series of questions on employment. After careful consideration a progressive filtering strategy, incorporating an age based cut-off for questions on qualifications and employment, was agreed. This was designed to take account of users' requirements for information, the quality of responses, overall acceptability, burden on the public and processing costs.

The Census Rehearsal took place in April 1999 and involved a total sample of 147,000 households across the United Kingdom.

Northern Ireland Specific Questions
In addition to the work being carried out across the UK with regard to the ‘core’ questions, NISRA contracted their Central Survey Unit (CSU) to test certain Northern Ireland specific questions using their Omnibus Survey. This was carried out in September 1999. A short self-completion (Census style) questionnaire was issued by the Omnibus Survey interviewers while they were preparing for the main part of their computer assisted interview. This only took a few minutes after which the interviewers asked their respondents to comment on the questions. The questions asked referred to the location of rooms, knowledge of Irish and qualifications. The choice of religion question for Northern Ireland was influenced by earlier Omnibus Survey results which showed that the format of the question finally chosen worked well and was acceptable to the public.

Form Design
Detailed decisions about the design and layout of the Census forms were based on successive test results, detailed discussion with Census Project Managers, design consultants and print experts as well as thorough quality assurance and proofing procedures. Special attention was given to a combination of factors including:

	public acceptability;
	technological considerations;
	burden on the public; and
	financial constraints.

Early work drew on published material about how respondents to self-completion questionnaires were affected by layout and presentation as well as research in other Census organisations. Initially, both the matrix format, used in previous Censuses, and a new page-per-person layout were tested. External design consultants were then appointed to redesign both documents. The results of this exercise provided excellent working models for subsequent developments, most notably the 1997 Census Test.

The statistical design of the 1997 Census Test allowed comparisons between reactions and responses to the matrix and pages-per person layouts. Completion rates for both household and person questions, overall discrepancy rates, levels of objection and refusal as well as concerns about privacy showed that the pages-per-person form outperformed the matrix style document.

These findings were substantiated by an evaluation of technical and operational issues including data capture, scanning and recognition which highlighted a number of problems with the matrix form. As a result it was concluded that the pages-per-person format would be more suitable for the 2001 processing systems.

A series of recommendations took account of results from the 1997 Census Test, research on estimated costs of printing and postage, acceptability, burden on the public as well as the preferences of Census users and managers. Most proposals were adopted, including the preferred pages-per-person design and overall size of 24 pages in Northern Ireland allowing the collection of information on six persons per form.

The primary purpose of the Census Rehearsal was to check methodological and operational procedures rather than the design of the form. However, results from data processing and preliminary analysis of returns highlighted a number of problems with form completion, colour and design. Supplementary information from the Census Quality Survey demonstrated that the public’s reactions to the form were favourable with most respondents (76.8%) finding it simple or easy to understand. A few people (7.2%) found the form too cluttered or complex and there were some problems with routing and the differentiation of questions for individual household members.

Although the basic design and layout of the form were not changed a number of minor revisions were implemented to overcome the shortcomings identified in the Census Rehearsal and the Census Quality Survey. Experts recruited by the Census Offices and the processing contractor recommended further minor adjustments in order to resolve specific printing or technical issues. These changes were accomplished without compromising public acceptability or burden. Copies of the amended documents were used as the basis for printing the public forms.

Agreed content of the Census questionnaire
Decisions on the selection of questions and the design and layout of forms for the 2001 Census were based on a careful assessment of information requirements, detailed discussions with users and a comprehensive programme of question testing. All aspects of the work have been governed by a combination of ethical and practical considerations including equity, public acceptability, the limitations imposed by a self-completion questionnaire, processing and cost constraints. Furthermore, recommendations took account of the quality of the expected data, the burden on the public and statutory restrictions.

The Government set out proposals for the 2001 Census in a White Paper published in March 1999. Decisions about the content of the Census, including details of the questions, were set out in the Census Order approved by Parliament in May 2000. Copies of the Census forms became available when the Census Regulations came into force on 26th June 2000. Table 1 provides a list of the questions included in the Census.

Table 1: Questions Included in the 2001 Census in Northern Ireland

Demographic Topics

Name (first name and surname) Sex Date of Birth Marital Status Relationships within the Household Usual Address One Year Ago Term-time Address for Schoolchild or Student in Full-time Education

Cultural Topics

Country of Birth Ethnic Group Religion

Health Related Topics

General Health Provision of Unpaid Care Limiting Long-term Illness

Education Topics

Whether Schoolchild or Student in Full-time Education Academic and Vocational Qualifications

Labour Market Topics

Economic Activity (including whether working, unemployed or retired) Whether or Not Ever Worked (including year last worked) Employment Status Size of Workplace Occupation (job title and description) Supervisory Status Nature of Business (Industry) Name of Employer (Industry) Hours Worked

Transport Topics

Number of Cars and Vans Address of Place of Work Method of Travel to Work

Household Accommodation Topics

Type of Accommodation Self-contained Accommodation Number of Rooms Amenities (availability of bath/shower and toilet) Lowest Floor Level of Accommodation Central Heating Tenure Type of Landlord (for rented property only)

Despite our best efforts it was not possible to meet all requirements and as a result a number of topics were excluded from the Census. In some cases this was because there was insufficient need for the information, in others because of concerns about public acceptability, problems with the accuracy or reliability of responses or difficulties in devising suitable questions. In particular, the case for including a question on income was rejected by the Government despite strong support from users because of concerns about public acceptability, the implications for response and coverage. Details of the questions not selected are presented in Table 2.

Table 2: Questions Considered but Not Proposed for Inclusion in the 2001 Census in Northern Ireland

Demographic Topics

Extended Family outside the Household Date of Most Recent Marriage or Divorce Usual Address Five Years before the Census Term-time Address of Students One and Five Years before the Census Duration of Residence at Current Address

Cultural Topics

Duration of Residence in Northern Ireland and Reason for Coming Taught Languages Nationality

Health Related Topics

Health/Lifestyle Disabilities Receipt of Unpaid Care Private Health Insurance

Education Topics

Private Education Qualifications Write-in

Labour Market Topics

Number of Jobs Closed Industry and Occupation

Transport Topics

Individual Access to Private Transport Commuting Address Size of Car and Number of Miles Travelled per Year Travel to Place of Study Use of Public Transport

Income

Amount of Social Security Benefits Received Source of Income Gross Individual/Household Income Private Pensions

Household Accommodation Topics

Age of Property Value of Home Availability of Garden or Yard Furnished/Unfurnished Accommodation (for rented accommodation only) Tenure Write-in Landlord Write-in

Lessons Learned
In view of the long lead time associated with the planning of the Census and the complexity of the issues affecting the selection of questions, the work was completed successfully, providing the input needed for the content of the census, the design and printing of the census forms.

Work on the 2001 Census has highlighted the importance of the following:

	co-operative working arrangements and good communication including the use of timely and well-informed publicity;
	advice and support from Census users. The Census Offices need to build on the strengths of the Census Advisory Groups, Census Working Groups and Sub-Groups;
	the invaluable advice provided by members of the ONS Question Testing Unit;
	retaining independent print and processing experts to advise and support Census staff;
	better co-ordination of the legislative and census timetables;
	improved awareness of the possible implications of political developments; and
	contingency arrangements to deal with unexpected problems with questions and changes in Government policy.

Conclusion
The importance of consultation, research and question testing were recognised from the outset of the programme for the planning and development of the 2001 Census. Staff from the Census Offices have worked co-operatively with users to meet their needs, develop effective questions and design suitable questionnaires. Every effort was made to maintain regular contact with users, report progress and provide information on successive stages of the work.

Notes:
The Omnibus Survey is a multi-purpose survey which provides a fast effective and reliable way of obtaining information from a representative sample of adults.