Washington Group on Disability Statistics (WG)

Development of an International Comparable Disability Measure for Censuses

A new set of questions on disability for use on national Censuses has been developed, tested, and adopted by the Washington Group on Disability Statistics (WG). The WG is a United Nations (UN) sponsored City Group commissioned to improve the quality and international comparability of disability measures. The approach taken by the WG represents a break with methods used in the past. The questions reflect advances in the conceptualization of disability and use the World Health Organization’s International Classification of Functioning, Disability, and Health (ICF) as a conceptual framework. The focus is on functioning in basic actions in contrast to approaches that are based on impairments or bodily functions. The UN Principles and Recommendations for Population and Housing Censuses incorporates the approach taken by the WG. (See: Section VI-8: Disability Characteristics pages 178-183, and Tabulations on Disability Characteristics [PDF – 2.5 MB] pages 292-294. A more detailed discussion of the conceptual framework and data collection objectives of the WG can be found in the Washington Group Position Paper: Proposed Purpose of an Internationally Comparable General Disability Measure (WG3.6).

Background

In June 2001, the United Nations International Seminar on the Measurement of Disability identified the need for comparable population-based measures of disability for individual country use and for international comparisons. This determination was based on the scarcity and general poor quality of data on disability, especially in developing countries, and the lack of internationally comparable measures, even among developed countries. The Washington Group on Disability Statistics (WG) was formed to address this urgent need.

The main purpose of the WG is to promote and co-ordinate international co-operation in the area of health statistics focusing on disability measures suitable for censuses and national surveys. The major objective is to develop tools to collect the basic data necessary to provide information on disability that is comparable throughout the world. The first priority of the WG was to guide the development of a short set of disability measures suitable for use in censuses, sample-based national surveys, or other statistical formats, for the primary purpose of informing policy on equalization of opportunities for the population with disabilities. A second priority is to recommend one or more extended sets of survey items that elaborate the measurement of the multiple concepts associated with disability and can be used as components of population surveys, as supplements to surveys or as the core of a disability survey. There will be a connection between these extended sets of survey items and the short set of disability measures. The disability measures recommended by the group will be accompanied by descriptions of their technical properties, and methodological guidance will be provided for their implementation and their applicability to all population subgroups.

I. Recommended Short Set of Questions on Disability for Censuses

At the 6th Annual Meeting of the WG in Kampala, Uganda, in 2006, test results were reported and the short set of questions on disability was endorsed by the 23 countries and 5 international agencies in attendance. Some minor wording modifications were suggested based on pre-test results presented at the meeting (see Section V below). The set comprises questions on six core functional domains: seeing, hearing, walking, cognition, self care, and communication. In countries where resources do not permit inclusion of six questions on a census, the first four domains are recommended for inclusion (seeing, hearing, walking, and cognition); however the WG strongly endorses the use of the six questions.

The final short question set is:

The next questions ask about difficulties you may have doing certain activities because of a HEALTH PROBLEM.

Do you have difficulty seeing, even if wearing glasses?
Do you have difficulty hearing, even if using a hearing aid?
Do you have difficulty walking or climbing steps?
Do you have difficulty remembering or concentrating?
Do you have difficulty (with self-care such as) washing all over or dressing?
Using your usual (customary) language, do you have difficulty communicating, (for example, understanding or being understood by others)?

Each question has four response categories: (1) No, no difficulty, (2) Yes, some difficulty, (3) Yes, a lot of difficulty, and (4) Cannot do it at all. The severity scale is used in the response categories in order to capture the full spectrum of functioning from mild to severe.

II. Rationale for Choice of Questions

Disability represents a complex process and is not a single, static state. It refers to the outcome of the interaction of a person and his/her environment (physical, social, cultural, or legislative) and represents a measure of the negative impact of environmental factors on one’s ability to function. The complexity of the concept has resulted in the proliferation of statistics on disability that are neither comparable nor easy to interpret. Furthermore, disability data are collected for different purposes such as to estimate the prevalence of physical impairments or to plan for the provision of services. Each purpose elicits a different statistic and even when the intention is to measure the same concept, the actual questions used differ in ways that severely limit comparability. The conclusion is not that some estimates are right and others are wrong, but that they are measuring different things. The WG chose to develop questions that would address the issue of whether persons with disability participate to the same extent as persons without disabilities in activities such as education, employment, or family/civic life. A major reason for this choice is the pivotal importance of the issue of social participation and equal rights from a policy perspective as illustrated by the recently ratified UN Convention on the Rights of Persons with Disabilities. In addition, there was agreement that it would be possible to develop a question set to meet this objective, that could be administered using Census methodology and that could produce internationally comparable data.

One approach to measuring social engagement is to ask directly if a disability has impacted participation. An example of such a question is “Are you limited in the kind or amount of activities that you can do because of on-going difficulties due to long term physical, mental, or health problems?” Such questions are difficult to ask in a way that produces comparable data. An alternative approach is to obtain information on difficulty in functioning in basic actions (c.f. the WG short set of six questions above) since these actions form the building blocks for more complex activities and, when restricted by the environment, can result in disparities in participation. The task is then to determine whether persons with difficulties or limitations in basic actions have participation rates equal to those without these limitations.

The WG questions were designed to provide comparable data cross-nationally for populations living in a variety of cultures with varying economic resources. While the ideal would be to collect information on all aspects of the disablement process and to identify every person with a disability within every community, this would not be possible given the limited number of questions that can be asked on a National Census. The basic actions represented in this question set are those that are most often found to limit an individual and result in participation restrictions. Domains were selected using the criteria of simplicity, brevity, universality, and comparability. It is expected that the information that results from the use of these questions will, a) represent the majority of, but not all, persons with limitation in basic actions, b) represent the most commonly occurring limitations in basic actions, and c) be able to capture persons with similar problems across countries.

III. How to Use the Data

As mentioned above, this short set of questions will identify the majority of persons with a disability—but not all; and they will identify most limitations—but not all. Psychological limitations, for example, are not covered by the short set of questions. Neither were the WG questions developed specifically for use with children. Obtaining information on disability in children, especially in young children, using a limited number of questions is very difficult. These limitations would preclude the use of the short set of questions as screeners for disability in a population.

However, the recommended set of questions identifies the majority of the population with difficulties in functioning in basic actions that have the potential to limit independent participation in society. The intended use of these data is to compare levels of participation in, for example, employment, education, or family life for those with disability versus those without disability and thereby to assess equitable access to opportunities as mandated by the UN Convention. In addition, the data can be used to monitor prevalence trends for persons with limitations in the specific basic action domains.

The WG recognizes that the short set of questions for censuses may not meet all the needs for disability statistics, nor will it replicate a survey of the population that can collect information across a wider range of disability domains. While a census can provide valuable information on disability, especially for local areas, other data collection mechanisms are necessary to obtain a more complete understanding of disability nationally and internationally. The WG is currently developing extended question sets for use on surveys.

IV. Context for the Work of the WG

The finalization of the short question set will facilitate the inclusion of the questions in the 2010 census round. The questions were developed according to the Fundamental Principles of Official Statisticsand are consistent with the International Classification of Functioning, Disability, and Health.Most importantly, however, the endorsed questions support the UN Convention on the Rights of Persons with Disabilities. The short set addresses equalization of opportunities for persons with disabilities, which is one of the General Principles listed in Article 3 and is the focus of Article 5. It is also particularly relevant to the collection of data for policy purposes outlined in Article 31 and will facilitate the monitoring of participation in cultural life, leisure, recreation, work, and employment that is called for in Articles 27 and 30 of the Convention.

V. Question Testing Protocol

Pre-test process

In addition to developing the short question set, the WG also developed a plan for both cognitive and field testing of the questions.

Standardized testing was undertaken in 15 countries, including 13 that were funded through the World Bank to assess the validity of the questions and to better understand how they operate in a variety of regions, cultures, and languages. Cognitive tests were designed to provide insight about how respondents comprehend, retrieve, judge, and respond to questions. Cognitive tests were also intended to identify response errors related to question design and to reveal how and why these errors may have occurred. These tests involved in-depth, face to face interviews with a small sample of respondents representing the group of interest. For the purpose of the WG test, the interviews were semi-structured and the analysis of results was largely qualitative. Field tests were conducted to determine whether the questions were being interpreted as intended by the developers in that they captured the most relevant key aspects of the functional domains selected (i.e. whether the single question per domain captured a reasonable proportion of those with functioning difficulties in that domain). Field tests were intended to simulate the conditions of an actual survey. The objectives of the WG field test were to determine whether the single question per domain was representative of that domain, whether the questions produce comparable data across countries, and how the WG questions work as a set in comparison with other questions used by the country.

Summary of results

Based on combined test results from all countries, the WG questions were well understood and interpreted consistently across countries. Only a few cases of inconsistent response patterns for each of the six WG questions were observed. Most notable were ‘literal’ translation difficulties from English into local languages and it was suggested that better conceptual translation was needed since certain phrasing is not easily interpretable in some languages. Countries are instructed to translate these phrases in a way that is culturally appropriate to capture the concepts in the question. In earlier versions, the communication question (Question 6) had an introductory text that was considered somewhat cumbersome. In order to reduce some erroneous responses and to improve precision, this particular text was removed since it repeated the introduction to the entire set of questions. Furthermore, some respondents to this question reported a second language problem (i.e. they had communication problems because the primary language in the country was not their native language), therefore, a reference to communicating in one’s normal language was added in the preface to this question.

Correspondence between each WG question and the extended questions for each domain was generally good. In some instances, respondents answered affirmatively to detailed questions about a particular domain of functioning, but did not respond affirmatively to the WG question for that domain, and visa versa. For instance, a respondent may have reported some difficulty with near or far vision, but did not report difficulty seeing in general as captured by the WG question on vision. Misidentification can occur for a number of reasons and it is important to determine whether false positives or negatives are occurring systematically (true errors) or randomly. For cases identified as true errors, it is furthermore important to determine if these are associated with gender, country, age, disability, or health status. True error was identified for the vision question but the error was related to the glasses clause rather than the result of misunderstanding of the question. There is a potential for false positives in extended questions on cognition as a high rate of inconsistencies was found; however, unlike the vision question, the inconsistencies are more likely to be a result of interpretation issues and not blatant misunderstanding.

There was discussion about whether the WG questions should be more specific, for instance, by adding a clear reference to distance for the question on difficulty walking (Question 3). However, the overall level of error with the WG questions was small and bias in responses due to contextual differences across countries did not appear to be a problem. It was agreed that increasing the specificity of the questions would not be beneficial, particularly in a census format and given the potential for cultural and contextual differences across countries.

In certain instances the WG question represents only a portion of the entire domain of functioning. In terms of cognition (Question 4), for example, memory and concentration do not represent all aspects of mental functioning. In addition to memory and concentration, the extended questions included aspects of learning new tasks and finding solutions to problems. Field testing revealed good correspondence between the WG question and the extended questions that captured similar aspects of functioning (memory and concentration) but poor correspondence between the WG question and the extended questions that captured different aspects of mental functioning (learning and finding solutions). This was interpreted as a limitation in terms of the aspects or scope of functioning in the domain captured by the WG question, but not a problem with the question itself.

It was anticipated that the question on self-care (Question 5) would be somewhat problematic since it was oriented toward more complex activities than the other WG questions (and therefore potentially more culturally influenced); however, this did not seem to pose a problem when data were compared across countries.

Overall, the inconsistencies identified in the field and cognitive tests were not considered to be significant problems; the number of false negatives and false positives were few and responses to the WG questions appeared to reflect respondents’ overall abilities in that domain. The testing suggested that potential errors could be minimized by addressing issues of inconsistency at the country level, ensuring that questions are interpreted and translated according to the customary language of the country and providing country-specific cognitive testing in terms of question development. A single question per domain was deemed appropriate to capture those with functioning difficulties in that domain. The questions also captured the intended aspects of the functional domains selected, although not all aspects of those domains. It should be reiterated, however, that not all functional domains are covered by the six WG questions; for example, no information is collected that deals with psychological limitations or difficulties.

In further analyses using data from existing national surveys from developed countries where questions similar to the WG questions were included as well as questions on many other aspects of functioning, the domains captured by the WG questions represented the majority (but not all) of respondents with self-reported limitations in any aspect of functioning.

These tests, as well as studies in other countries employing the WG approach, show an improvement over use of more traditional impairment focused census questions on disability. For example, according to the 2000 Census in Zambia , disability referred to individuals who were limited in the kind or amount of activities they could do because of on-going difficulties due to long term physical, mental, or emotional health problems. The actual questions used to capture disability in that Census, however, were: “Are you disabled in any way?” (Yes/No), and “What is your disability?” (Response categories included: blind, partially sighted, deaf/dumb, hard of hearing, mentally ill, ex-mental, mentally retarded, and physical handicapped.) This approach yielded a disability prevalence rate in Zambia of 2.7% and represented, in fact, a trebling of the 1990 prevalence rate of 0.9%, which used the same approach but included only 4 impairment categories: blind, deaf/dumb, mentally retarded, and crippled.^, When the short set of WG questions was included in the 2006 Living Conditions Survey in Zambia, 13.3% of respondents reported a lot of difficulty or unable to do any one activity or some difficulty in two or more of the six domains (minimum criteria set for presence of disability by the Zambian research team). Also, there was a low rate of misclassification resulting from the WG questions when compared to information obtained from subsequent detailed interviews with respondents.

VI. The Washington Group on Disability Statistics:

The Washington Group on Disability Statistics (WG) was organized in 2001 following the United Nations International Seminar on Measurement of Disability to address the need for statistical and methodological initiatives at an international level to facilitate the measurement of disability and the comparison of data on disability cross-nationally. To date, the WG has met seven times, in: Washington DC, USA (2002); Ottawa, Canada (2003); Brussels, Belgium (2004); Bangkok, Thailand (2004); Rio de Janeiro, Brazil (2005); Kampala, Uganda (2006); and most recently in Dublin, Ireland (2007). All National Statistical Offices are eligible for membership in the WG. Currently, 77 National Statistical Offices are represented, as well as 7 international organizations, 6 organizations that represent persons with disabilities (DPOs), the UNSD, and 3 other UN affiliates. The Secretariat for the WG is located at the National Center for Health Statistics (NCHS), USA. The main objective of the WG is the promotion and coordination of international cooperation in the area of health statistics by focusing on disability measures suitable for censuses and national surveys.

Details of the WG organization, history and accomplishments Internet site provides access to lists of participants, proceedings from the meetings (presentations and papers), reports to the UN Statistical Commission and information on upcoming meetings.

Building Capacity for Disability Data Collection in Developing Countries:

Several government statisticians from developing countries have been trained on disability measurement methodology through the WG efforts. Regional training meetings held in Kenya (June, 2005) and Brazil (September, 2005) were an integral part of this effort. Presently, countries that received training are working internally to improve their overall approaches to dealing with the issue of disability measurement through ongoing data collection activities.

Fostering International Cooperation:

The WG has cooperated with the United Nations Statistics Division (UNSD), the World Health Organization (WHO), the Statistical Office of the European Communities (Eurostat), the United Nations Economic and Social Commission for Asia and the Pacific (UNESCAP), the United Nations Economic and Social Commission for Western Asia (UNESCWA), the International Labor Organization (ILO), the Budapest Initiative group, the World Bank, and others to promote a unified approach to disability measurement. Several World Bank data instruments have been heavily influenced by the work of the WG, and related disability questions are currently being tested as part of the Living Standards Measurement Study (LSMS).

Future Work:

The WG remains active and will continue to work on the development and testing of extended question sets for surveys and survey modules and the production of technical reports on methodological issues such as dealing with special populations (e.g., children and institutionalized persons). If resources allow, it also plans to continue to offer technical assistance to countries to build capacity for disability measurement and analysis.

References

The inclusion of assistive devices was considered for two domains only, seeing and hearing, as limitations in these domains can often be overcome with the use of glasses or hearing aids, 2001.)
United Nations Enable: Rights and Dignity of Persons with Disability.
See Statistical Commission, Report on the Special Session (11-15 April 1994), Economic and Social Council, Official Records, 1994, Supplement No. 9, Series No. E/CN. 3/1994/18, United Nations, New York, 1994, para.59.
International Classification of Functioning, Disability and Health (World Health Organization, Geneva, 2001.)
Funding secured through the World Bank Development Grant Facility.
Extended questions: Do you have difficulty (i) concentrating on doing something for 10 minutes? (ii) learning a new task, for example, learning how to get to a new place? (iii) finding solutions to problems in day to day life? (iv) remembering the names of people or places? (v) remembering appointments? (vi) remembering how to get to familiar places? (vii) remembering important tasks, like taking medications or paying bills?
CSO, 2000 Census of Population and Housing. See Chapter 9: Disability.
CSO, 1990 Census of Population, Housing and Agriculture.
Eide AH, Loeb ME (eds.) (2006) Living Conditions among people with activity limitations in Zambia: A national representative study. Report No. A262, SINTEF Health Research, Oslo.