Methodology Guidelines for LEAD RESEARCHERS

This document provides instructions to researchers for the Global Integrity Report 2010. It compliments Indaba Platform Overview and Indaba Platform Tip Sheet for LEAD RESEARCHERS. Related documents are at the Indaba Help Desk.

Overview

Global Integrity is an international NGO that tracks governance and corruption trends around the world. Our core product each year is the Global Integrity Report. The Global Integrity Report comprises individual Country Reports generated by in-country experts assessing governance, anti-corruption mechanisms, and government accountability at a national level.

Each Country Report has two major components:

• A scorecard component in which a lead in-country researcher scores approximately 320 Integrity Indicators. The Integrity Indicators break down a country’s anti-corruption mechanisms into six broad dimensions of governance and from there into 320 individual indicators.
• A journalistic component in which a leading in-country journalist produces a 1,250-word “Reporter’s Notebook” describing how corruption manifests itself in day-to-day life in the country; in other words, how it looks, tastes, feels, and smells for the average citizen.

Additionally, three to five Peer Reviewers for each country (who are neither the lead researcher nor the lead journalist for the country) independently verify both the draft Reporter’s Notebook as well as the draft Integrity Indicators scorecard. They provide feedback and suggestions to Global Integrity staff, who then make final changes to both the reporting and the data, when necessary, to ensure consistency and accuracy. The role of Peer Reviewers is to verify the factual content of and provide third-party perspective to both the Reporter’s Notebook and the Integrity Indicators scorecards.

The Global Integrity staff also prepares basic Country Facts and a Corruption Timeline (reviewing major governance- or corruption-related events in the country during the past decade) for each country report. Those two pieces accompany the final Country Report.

In sum, each country report contains:

• Reporter’s Notebook: prepared by the lead in-country journalist; reviewed by Peer Reviewers
• Integrity Indicators scorecard: prepared by the lead in-country researcher; reviewed by Peer Reviewers
• Country Facts: prepared by Global Integrity staff
• Corruption Timeline: prepared by Global Integrity staff

Maintaining the Independence of Roles: There is to be no formal interaction between the lead journalist, the lead researcher, and/or peer reviewers. The Reporter’s Notebook and the Integrity Indicators scorecard are to be prepared independently and without collaboration from other country team members (if they happen to be known to one another). Peer reviewers should not be involved with the preparation of either the Reporter’s Notebook or the Integrity Indicators; their function is to review and comment on both products after the lead reporter and lead researcher have drafted them and submitted them to Global Integrity.

Part of the strength of Global Integrity’s approach in assessing governance and anti-corruption is the unique mix of journalism and social science inherent in our Country Reports; maintaining the separation between the two is therefore critical to the value of our final product.

Review Process

The Integrity Indicators submitted to Global Integrity will be integrated into the overall Country Report. After you submit your work, we will review your work and contact you for clarification on a few indicators. This communication, and your subsequent changes (if necessary) to any of your original scores will occur through Indaba. You will receive an email indicating that Global Integrity staff has reviewed your draft indicators and has a few questions about them; you will then be asked to log in again to Indaba to review those indicators and provide, when needed, additional explanation or comment.  The companion instructions Indaba Platform Tip Sheet for LEAD RESEARCHERS provides details about how these exchanges will work in practice.

In our experience, even very good work benefits from this review. Your prompt reply to this request is important to avoid delays in the process.

As mentioned above, the draft Integrity Indicators that you submit (as well as the draft Reporter’s Notebooks submitted by the lead reporter) will then be sent to the country Peer Reviewers for review and comment prior to publication. Global Integrity staff will review all Peer Review comments and identify any indicators that require additional input or explanation from you before determining a final score. You will then be asked (via email) to log in to Indaba again to review those specific indicators in question and make any necessary changes. The Peer Reviewers’ comments will be published transparently alongside the final Integrity Indicators scorecard and final Reporter’s Notebook.

Global Integrity staff may contact lead researchers during the peer review and/or editing process should we come across questions or issues that require additional input from you or that could result in substantial changes to your original scoring. We ask that you remain engaged in this dialogue until final publication of your work.

The Integrity Indicators Scorecard

The Integrity Indicators scorecards that comprise part of each country assessment of the Global Integrity Report consist of two types of indicators: “In Law” (de jure) and “In Practice” (de facto). The format of the indicators typically includes the description of the indicator (or question), the possible responses (“Yes” and “No,” or scores ranging from 0 to 100), scoring criteria to guide the selection of the score for the indicator, and space for references and comments.

To properly respond to an indicator, the researcher must select the response related to the scoring criteria that best describes the research findings for the aspect(s) of anti-corruption/accountability that is assessed in that indicator. The researcher must also substantiate each response by providing adequate sources and narrative comments.

For an indicator’s sources and comments, Global Integrity expects certain minimum quality standards, as outlined below.

Please be advised that failing to meet those standards will force Global Integrity’s staff to return the indicators in question to the researcher for improvement until adequate sources and narrative comments are provided.

“In Law” Indicators

These indicators refer to the existence, or not, of specific laws, regulations, institutions, or governmental mechanisms. The responses are usually limited to “Yes” or “No.” In both cases, the researcher must provide all relevant information that helps explain why one or the other response was chosen:

a) Full name of the law(s) or regulations(s), number and year that govern the issue addressed in the particular indicator.

Examples:

• “The Philippines Anti-Graft and Corrupt Practices Act, Republic Act No. 3019, August 17, 1960.”
• “Ley del Servicio Publico de Defensa Penal de la Republica de Guatemala, Decreto numero 129-97, 5 de Diciembre de 1997.”

b) Location of the specific provision within the law or regulation (title, article, section or subsection, etc.) that most closely related to the indicator question itself.

Example:

• “Paragraph (2) of subsection (a) of Article 1.”

c) Transcript or explanation about the specific provision, if appropriate.

Example:

• “Article 200 – Final Provisions: The Constitution is the supreme law of the State. Any law which is contrary to this Constitution is null and void.”

d) If there is not a law, regulation or institutions that explicitly regulates the given aspect of anti-corruption/accountability/transparency assessed by the indicator, the researcher is expected to conduct further research on whether there are any other legal sources, such as jurisprudence (case law or decisions of the higher court), decrees, administrative regulations, orders, or customary law that would apply.

e) For citations of cases, sentences or resolutions, since legal citation formats vary greatly across national contexts, researchers may use the format most appropriate for that country. However, the citation should include the following information: 1) case number, 2) name of the court and year of decision, 3) published source where the case can be found (for example: The Bluebook, p. 6) and a web link if the information is available online.

f) For citation of reports, papers, Internet texts, journals, magazines, newspapers and other sources see point “l” at the end of this document.

g) If after thorough research (including step “e”) the researcher concludes that there is no existing law or regulation for the particular dimension of anti-corruption/accountability assessed by an indicator, the researcher must still include a supporting reference to substantiate the “No” score, such as an interview with an expert or systematic desk research of legal sources. Responses that reference a “researcher’s own knowledge/expertise” or “not applicable” or “no such bodies exist” will not suffice. Instead, “No such law/mechanism exists” is the preferred response.

h) Consistency is essential throughout all the indicators. Some indicators are related but spread throughout different categories or sub-categories, and the researcher should make sure that there are no contradictions or discrepancies in the different responses. In some cases, responses may seem inconsistent even though they are not, and the researcher should make sure the proper clarifications are always included to avoid confusion. This is an example of a useful type of comment to avoid confusion and ensure consistency: “While there are indeed regulations governing gifts and hospitality offered to most public officials, including ministers, these regulations do not apply to members of Congress.”

i) Global Integrity expects the researcher to use the scoring criteria to guide the decision about which score to choose, but the scoring criteria should never be simply copied into the comments box. The researcher’s comments must be original, context-specific and well-sourced.

j) Global Integrity strongly encourages researchers to review their submitted comments and references to avoid accuracy and spelling mistakes.  Common errors include providing different names for the same law or the same institution or omitting the last name or the position of a source.

“In Practice” Indicators

These indicators refer to the way in which anti-corruption and accountability regulations, laws, institutions, or mechanisms are implemented and enforced in practice, and the responses are usually limited to scores on an ordinal scale from 0 to 100 (0, 25, 50, 75, or 100), with 100 being the best possible score that a particular “in practice” indicator can earn. In all cases, the researcher must provide robust information with supporting references to help explain why the particular response was chosen:

a) As the name suggests, an “in practice” indicator examines the actual effectiveness or ineffectiveness of laws, regulations, mechanisms, or governmental institutions in practice. Therefore, the responses must always directly refer to what happens in the day-to-day practice of the specific aspect of anti-corruption/accountability assessed in the indicator, and should never be based on what the regulation or procedure says should happen (that theoretical instance should be captured in the related “in law” indicator). Here is an example of an inappropriate response to an in practice indicator:

Indicator: “In practice, budget expenditure records are audited or evaluated according to international standards.”

Inappropriate response:

• “There is an internal audit department subject to the General Auditor’s standards.”

This response is insufficient because it assumes that because the internal audit department theoretically adopts the General Auditor’s standards, it actually follows such standards in practice. The intent of an in practice indicator is to go beyond that type of assumption and assess if, in reality, the internal audit really adheres to those standards. That answer is often best substantiated by interviewing officials and practitioners familiar with the law or mechanism’s function in practice.

b) As a general rule, responses to each indicator must include all sources of information that have been used to support the choice of the score, as well as the substance of the information used to substantiate the score. These might include quotes from interviews with experts and practitioners related to the issue assessed in the indicator, studies and reports (by media, academia, government, and international organizations), statistical data, public records and documents, and/or conclusions drawn from direct observations made by the researcher.

c) In exceptional cases, the response to an “in practice” indicator can be based on a single source, for instance, when there is a recent credible investigative report based on numerous sources that offer solid conclusions specifically related to the issue assessed by the indicator. In most cases, however, Global Integrity expects the researcher to use multiple sources to substantiate the response chosen for each indicator.  As a general rule, two different sources are required for in practice indicators.

d) Under no circumstances should a single interview, media report, or document be used as the supporting reference for all or most of the indicators within a particular category or sub-category.  Such indicators will be returned to the researcher for additional research and referencing.

e) Under no circumstances should the score for an “in practice” indicator be substantiated by a reference to a legal statute/article or regulation. Rather, the sources used should be based on interviews, media reports, and other documents. Legal references may be included as supporting or explanatory information, however.

f) In some cases, the researcher will discover contradictory views amongst various interviewees. To decide which score is most appropriate, the researcher must weigh all the information that is available and, using his/her knowledge about anti-corruption in the context of his/her country, arrive at an informed conclusion about which score best reflects the situation and why. For example, it is insufficient to select a 50 score simply to “balance out” two different views without more rigorous reasoning.

g) A careful selection of who to interview is essential in guaranteeing the credibility of the “in practice” responses. Global Integrity trusts that lead researchers will exercise good judgment in making that selection. We recommend that one of the first steps in your research process is to create a list of desired interviewees that will have insights to share about each sub-category of indicators and to schedule those interviews well in advance of your deadline. Bear in mind that not all “experts” or interviewees are independent or are able to offer a reasonably honest and unbiased account of what happens “in practice.” For example, the head of the public information office of a government institution may have a clear interest in saying that the office works efficiently regardless of what the reality actually is. Your final choice of score for that indicator should bear this in mind.

h) While interviewees are free to express their views and even personal opinions on the issues, the researcher is expected to make his/her best effort to extract specific details that support the sources’ views. Vague or uninformed responses are usually of limited value and credibility, as well as responses based on perceptions, assumptions, and suppositions. For example, the following response would not meet Global Integrity’s standards: “Regulations governing gifts and hospitality to public officials are usually respected, but there are always exceptions.” Something more specific is needed to defend the score. What are those exceptions, how often do they occur, and are there specific examples that the interviewee can cite?

i) Global Integrity expects the researcher to use the scoring criteria to guide the decision about which response to choose, but the scoring criteria should never be simply copied into the comments box. The researcher’s comments must be original, context-specific and well-sourced.

j) The researcher must keep in mind that the indicators will be read by both local and global audiences. Therefore, all comments and references provided must be clear and accessible enough for a wide range of audiences to understand.

Global Integrity strongly encourages researchers to review the material to avoid accuracy and spelling mistakes.  Common errors include providing different names for the same law or the same institution or omitting the last name or the position of an interviewee.

k) Reports by media, academics, auditors, and even lawsuits and legal records can be good sources of information. Global Integrity trusts the researcher will exercise good judgment in selecting: 1) recent reports, and 2) reports authored by credible sources and organizations.

l) All sources must be identified by their full name, position and, when necessary, an explanation of their background to demonstrate why they are knowledgeable about the issue being assessed in the indicator. All other data, such as public information and third-party reports, should include standardized reference information (see points “m” and “i” for details).

m) All mention of data that can be found online must include the direct web link, when available, and the date the link was accessed by the researcher. (For more details on proper formatting of sources, please see “i” below.)

l) To quote interviewees, reports, and other sources, please use the following formats based on guidelines in the Chicago Manual of Style:

Books - Single author

• Michael Pollan, The Omnivore’s Dilemma: A Natural History of Four Meals (New York: Penguin, 2006), 99-100. [If citing specific page numbers.]
• Pollan, Michael. The Omnivore’s Dilemma: A Natural History of Four Meals. New York: Penguin, 2006. [For general bibliographic reference.]

Books – Two or more authors

• Geoffrey C. Ward and Ken Burns, The War: An Intimate History, 1941-1945 (New York: Knopf, 2007), 52. [If citing specific page numbers.]
• Ward, Geoffrey C., and Ken Burns. The War: An Intimate History, 1941–1945. New York: Knopf, 2007. [For general bibliographic reference.]

Chapter in an edited book

• John D. Kelly, “Seeing Red: Mao Fetishism, Pax Americana, and the Moral Economy of War,” in Anthropology and Global Counterinsurgency, ed. John D. Kelly et al. (Chicago: University of Chicago Press, 2010), 77. [If citing specific page numbers.]
• Kelly, John D. “Seeing Red: Mao Fetishism, Pax Americana, and the Moral Economy of War.” In Anthropology and Global Counterinsurgency, edited by John D. Kelly, Beatrice Jauregui, Sean T. Mitchell, and Jeremy Walton, 67–83. Chicago: University of Chicago Press, 2010. [For general bibliographic reference.]

Journal articles

• Joshua I. Weinstein, “The Market in Plato’s Republic,” Classical Philology 104 (2009): 440. [If citing specific page numbers.]
• Weinstein, Joshua I. “The Market in Plato’s Republic.” Classical Philology 104 (2009): 439–58. [For general bibliographic reference.]

News stories in newspaper or popular magazines

• Daniel Mendelsohn, “But Enough about Me,” New Yorker, January 25, 2010, 68.
• Sheryl Gay Stolberg and Robert Pear, “Wary Centrists Posing Challenge in Health Care Vote,” New York Times, February 27, 2010, accessed February 28, 2010, http://www.nytimes.com/2010/02/28/us/politics/28health.html. [If citing specific page numbers.]
• Mendelsohn, Daniel. “But Enough about Me.” New Yorker, January 25, 2010.
• Stolberg, Sheryl Gay, and Robert Pear. “Wary Centrists Posing Challenge in Health Care Vote.” New York Times, February 27, 2010. Accessed February 28, 2010. http://www.nytimes.com/2010/02/28/us/politics/28health.html [For general bibliographic reference.]

Quotes from interviews - Direct interview

• Jenifer Harrison, interview by author, July 11, 2004, Little Rock, AR, digital recording (or other type of recording). “It is essential that they strengthen the fight against corruption- All these stories about irregularities and corruption are damaging for public support for our presence in Afghanistan.”

Quotes from interviews – Published interview

• Benjamin Spock, interview by Milton J. E. Senn, November 20, 1974, interview 67A, transcript, Senn Oral History Collection, National Library of Medicine, Bethesda, MD.

Quotes from interviews – Anonymous interviewee

Anonymous interviews can be cited in whatever format is appropriate for the interviewer’s field.

Example:

• Interview with government official, 3 March 2010.

Website and online texts

• Google. “Google Privacy Policy.” Last modified March 11, 2009. http://www.google.com/intl/en/privacypolicy.html.

Legal documents (sentences, resolutions, audit reports, etc.)

• Varies by country. Use standard practice for citing these documents in your location.

Thesis/dissertations

• Mihwa Choi, “Contesting Imaginaires in Death Rituals during the Northern Song Dynasty” (PhD diss., University of Chicago, 2008). [For general bibliographic reference.]

Conference papers

• Adelman, Rachel. “‘Such Stuff as Dreams Are Made On’: God’s Footstool in the Aramaic Targumim and Midrashic Tradition.” Paper presented at the annual meeting for the Society of Biblical Literature, New Orleans, Louisiana, November 21–24, 2009. [For general bibliographic reference.]

Ensuring Consistency of Scoring: No/Zero Relationships

For some sub-categories of indicators, you will observe an initial “in law” question (or series of “in law” questions) followed by a series of “in practice” indicators that attempt to assess the law’s, mechanism’s, or institution’s implementation or enforcement. If you score “NO” for an “in law” indicator, its accompanying “in practice” indicators should normally also be scored 0 since the mechanism does not exist and therefore cannot be implemented or enforced in practice.

However, one important exception to this rule is when there is an informal or alternative mechanism in place that, on at least some occasions, performs the role of the missing law, mechanism, or institution. In that case, the “in law” indicator may still be scored “NO” while the subsequent “in practice” indicators can receive more positive, non-zero scores. Please make a special effort in these cases to explain the situation to the reader through your Comments.

Example:

In some countries that lack an access to information/freedom of information law, government records are still sometimes accessible to the public through an informal request to government officials. Although these requests may not always be met, and officials are under no legal obligation to disclose the requested information, it would be appropriate to score “NO” for the “in law” indicator asking about the existence of a freedom of information law/mechanism but 25 or 50 for “in practice” indicators that assess the time and cost involved for accessing such government records (since the records can occasionally be accessed, though such access is not legally guaranteed).

Potentially Confusing Double-Negative Indicators

Because of the way the Integrity Indicators are normally written, there are several potentially confusing indicators where we ask whether key anti-corruption stakeholders such as journalists, CSO activists, or judges have been imprisoned, assaulted, or killed because of their work on corruption issues. The key to understanding these indicators is that a “Yes” score is always a positive assessment.

If you read the scoring criteria carefully for these indicators, a “Yes” score should be your choice when no one has been imprisoned, assaulted, or killed even though it may appear from a quick reading that “Yes” implies that something “bad” has happened. You will notice that in the “Yes” scoring criteria for these indicators we have reminded you that “Yes is a positive score.” In other words, always choose “Yes” when the situation is good and “No” when it is bad.

Hints

For many indicators, we have provided you with “Hints” to help guide your choice of scores.  In Indaba, simply hover your mouse over the “Hint” button to display the guidance.

Using the Indicators Spreadsheet

To help get you started with your research, we have provided you with a hard copy version of the Indicators and all related scoring criteria and “Hints” in an Excel spreadsheet.  You should feel free to use that spreadsheet to organize your research as you see fit, but ultimately all scores, comments, and references will need to be inputted into Indaba.

Related documents are at the Indaba Help Desk.