It is important to consider who is going to collect the data and build the indicators. Indeed, the nature and reputation of the people and/or organization(s) involved in the measurement exercise can have an enormous impact on the use, or lack thereof, of the data and indicators produced. Factors to consider include:
Who collects the information will affect the ownership, sustainability, integrity and quality of the assessment.
Who collects the information can have a double positive or negative impact: on people, when deciding to report events, and on stakeholders and policymakers, when considering whether to use the data produced for policymaking. Notions of trust, integrity and independence come into play and, if not handled properly, can lead to situations of under- or over-reporting.
Depending on the nature of the organization, some individuals may either not respond to questions at all or respond untruthfully because of fear or intimidation. This is particularly true with sensitive questions relating to democratic governance and human rights.
Subsequent use of the data and indicators by policymakers depends on whether the data and indicators are considered to be the result of objective, impartial and professional work. Thus, as noted above, the overall reputation of the organization(s) involved, along with the quality of the information, are extremely important.
National statistical offices (NSOs), non-government organizations (NGOs), commissions on human rights (CHRs), governmental bodies and agencies, and academics may be involved at different stages and to different degrees. A multi-disciplinary approach should be used whenever possible, to enhance the comprehensiveness, usefulness and reliability of the exercise.
Data Collection Actors
A range of actors will be involved in data collection activities. The following examines a selection of these actors.
Locally based academics have conducted governance assessments in several countries. For instance, in Tanzania researchers from the University of Dar es Salaam have been conducting democracy assessments for more than 10 years with the REDET project, funded by DANIDA (hyperlink). The quality of academics varies for assessments relying on survey data, especially in terms of indicator and questionnaire development and sampling. For archival research, it is always a good practice to look for someone who has successfully published a book or published in a peer-reviewed journal. One advantage of contracting nationally based academics is that these researchers often have talented and dedicated graduate students as research assistants. The fees charged by local academics vary considerably. Academics also may need to adjust their fees downward from "donor levels" for nationally based assessment to be financially sustainable.
Locally based research companies may lack certain survey skills, but generally have a good feel for the cultural context and other local or indigenous issues. In some cases, they also may lack sophisticated data analysis skills required to move beyond just reporting aggregate results. The quality of these organizations varies widely from country to country; established, high-quality companies are costly. When considering a local research firm, principals in the firm should have advanced degrees in a social science (such as sociology or political science), and at least one staff member should have some statistical training. Contacting past customers is a good way to make sure you hire the right company. For nationally based assessments to be feasible in the long run, local researchers will need to be realistic in the prices they charge. Most countries will simply not be able to afford to pay "donor prices."
Locally based NGOs, including local chapters of international NGOs, are active in collecting governance data in many countries. Local chapters of Transparency International are a good example of NGOs that actively assess, measure and monitor corruption- related issues and that conduct surveys and qualitative research on the problem. NGO expertise in governance data collection will vary enormously: Some NGOs will have a great degree of technical and professional experience in collecting information in ways that will make the data widely trusted. Other NGOs might have fewer resources and limited capacities to collect data, using techniques that are not technically robust or taking short cuts. It is important, wherever possible, to strengthen NGOs’ data collection capacities, and care should be taken in the selection of NGOs to partner with in this regard. The selection of NGOs is especially important in facilitating culturally specific data collection and data relating to marginalized groups. In many cases, local NGOs that advocate on behalf of vulnerable groups need support to enhance their own monitoring capacities with regard to the groups whose interests they represent.
National statistical offices (NSOs) have the capacity and experience to conduct high-quality governance assessments. Recently, interest has increased from these organizations to conduct nationally based governance assessments. Experience and capacity are clearly positives. However, in some countries NSOs may suffer from being perceived as not politically independent enough. However, NSOs are a viable option if the process has wide stakeholder participation and transparency, especially in countries where there exists little or no capacity in civil society to conduct these studies. The marginal cost of adding a set of governance questions to an existing household survey is quite small compared to a stand-alone survey of typical citizens.
External experts are commonly used in desk studies where little or no original data are collected. These experts often do not reside in the country they are assessing and thus have very little in-country ownership of the assessment process. External experts are often quite expensive.
International teams who conduct survey-based assessments often have superior survey research skills, but lack the inside knowledge needed to get good-quality data and draw representative samples, such as where census data are out of date and other challenges are present. Usually there exists no ownership of the process by the government or local stakeholders. International teams are expensive and not really suited for nationally based governance assessments.
DIAL 1,2,3 surveys.
Développement et Insertion Internationale (DIAL), a French development organization, works with national statistical offices (NSOs) to measure governance and democracy amongst the general public, using surveys. Data are collected through the specific survey modules on democracy, governance and subjective poverty, which are attached to questionnaires of regular household surveys conducted by NSOs. Results obtained include objective indicators such as absenteeism of public functionaries in different public services, incidence of corruption in various administrations, participation in previous elections, and reasons for non-participation. It also includes subjective perceptions and opinions in areas such as the functioning, trustworthiness and shortcomings of government institutions and policies, as well as the most important problems facing the country.
One of the main strengths of this approach is the high level of disaggregation available. Direct policy implications are possible, for instance, when a particular institution is identified as especially prone to corruption. Because the modules are attached to surveys that provide rich socioeconomic information, all governance-related phenomena can be disaggregated in relation to the incidence in poor/non-poor households, households with/without higher education, and households with a woman as household head, etc. This allows for a clearer picture of the vulnerability of different social classes. By adding a module to an existing survey, this method also is cost-efficient.