Methodology Guidelines for PEER REVIEWERS

Back to the Indaba Help Desk.

This document provides instructions to peer reviewers for the Global Integrity Report 2010. It compliments Indaba Platform Overview and Indaba Platform Tip Sheet for PEER REVIEWERS.

Global Integrity is an international NGO that tracks governance and corruption trends around the world. Our core product each year is the Global Integrity Report. The Global Integrity Report comprises individual Country Reports generated by in-country experts assessing governance, anti-corruption mechanisms, and government accountability at a national level.

Each Country Report has two major components:

• A scorecard component in which a lead in-country researcher scores approximately 320 Integrity Indicators. The Integrity Indicators break down a country’s anti-corruption mechanisms into six broad dimensions of governance and from there into 320 individual indicators.
• A journalistic component in which a leading in-country journalist produces a 1,250-word “Reporter’s Notebook” describing how corruption manifests itself in day-to-day life in the country; in other words, how it looks, tastes, feels, and smells for the average citizen.

Additionally, three to five Peer Reviewers for each country (who are neither the lead researcher nor the lead journalist for the country) independently verify both the draft Reporter’s Notebook as well as the draft Integrity Indicators scorecard. They provide feedback and suggestions to Global Integrity staff, who then make final changes to both the reporting and the data, when necessary, to ensure consistency and accuracy. The role of Peer Reviewers is to verify the factual content of and provide third-party perspective to both the Reporter’s Notebook and the Integrity Indicators scorecards.

The Global Integrity staff also prepares a Corruption Timeline (reviewing major governance– or corruption-related events in the country during the past decade) for each country report.

In sum, each country report contains:

• Reporter’s Notebook: prepared by the lead in-country journalist; reviewed by Peer Reviewers
• Integrity Indicators scorecard: prepared by the lead in-country researcher; reviewed by Peer Reviewers
• Corruption Timeline: prepared by Global Integrity staff

Maintaining the Independence of Roles: There is to be no formal interaction between the lead journalist, the lead researcher, and/or peer reviewers. The Reporter’s Notebook and the Integrity Indicators scorecard are to be prepared independently and without collaboration from other country team members (if they happen to be known to one another). Peer reviewers should not be involved with the preparation of either the Reporter’s Notebook or the Integrity Indicators; their function is to review and comment on both products after the lead reporter and lead researcher have drafted them and submitted them to Global Integrity.

Part of the strength of Global Integrity’s approach in assessing governance and anti-corruption is the unique mix of journalism and social science inherent in our Country Reports; maintaining the separation between the two is therefore critical to the value of our final product.

As a Peer Reviewer, we ask you to use your expertise to comment on each of the two main components of the Country Reports: the Reporter’s Notebook and the Integrity Indicators scorecard. We require critical and constructive feedback on the raw field research – in the form of your comments – to tell us whether the material matches your own knowledge of the situation in the particular country. Your review of both the Reporter’s Notebook and the Integrity Indicators scorecard is “blind”: you will not be told who prepared the draft Reporter’s Notebook or draft Integrity Indicators scorecard. In addition, your specific comments on both products will be published anonymously to avoid you feeling constrained in providing us with a frank and honest assessment. While your name will be listed as a member of the Country Team, your specific comments on both the Reporter’s Notebook and the Integrity Indicators scorecard will not be directly attributed to you individually (instead they will read: “Comment 1: XXX” as opposed to “Nathaniel Heller’s Comment 1: XXX”).

Comments on the Reporter’s Notebook

The Reporter’s Notebook is intended to be a short 1,250-word country-specific investigative overview/essay on corruption and government accountability. Reporters are asked to prepare a narrative essay on corruption and government accountability in their respective countries with a particular focus on events during the past year. The Reporter’s Notebook is NOT a review of the most important corruption scandals. Instead, the Reporter’s Notebook is intended to be an exploration of important aspects of corruption in that country. A Notebook should illustrate the state of corruption and governance in the country through specific human stories. It is a compelling, rather than comprehensive, view of corruption.

The Reporter’s Notebook is to be written as a stand-alone “news analysis” column for a newspaper or magazine.

Your Peer Reviewer comments on the Reporter’s Notebooks will be in the form of narrative, paragraph-style comments (as opposed to the more targeted comments for each of the Integrity Indicators; see details below). You should aim to limit your comments on the Reporter’s Notebook to around 2 to 4 paragraphs. Specific issues to focus on in your comments include:

* Is the Reporter’s Notebook factually accurate?
* Is the Reporter’s Notebook fair? Is anything misrepresented or unclear?
* Within the theme or story chosen by the author, is it a complete view of the events discussed? Are there important elements of this story that have been left out?
* What additional context would a reader benefit from knowing in order to more fully understand the situation?

Comments on the Integrity Indicators

The Integrity Indicators have been developed over a number of years by the Global Integrity team with informal collaboration by a number of outside experts in the governance and anti-corruption fields. They are designed to quantitatively assess, at a national level in any country:

• 1. the existence of laws, regulations, and institutions designed to curb or deter corruption;
• 2. the extent to which those mechanisms are implemented or enforced;
• 3. the access that average citizens have to those mechanisms.

The Integrity Indicators assess the strengths or weaknesses of a national anti-corruption/national integrity framework; they do NOT attempt to measure corruption itself. The Integrity Indicators seek to assess the de jure existence of anti-corruption mechanisms as well as their de facto implementation and enforcement.

The Integrity Indicators are grouped into four levels:

• Categories (aggregated scores)
• Sub-categories (aggregated scores)
• Indicators (some scored by lead researcher)
• Sub-indicators (all scored by lead researcher)

The lead researchers are tasked with scoring the Indicators and Sub-indicators. Sub-indicators are averaged to produce “composite” Indicators – in these cases, the lead researcher will only score the component Sub-indicators. In other cases, a particular Indicator will have no Sub-indicators; the researcher will score those Indicators directly.

Sub-category and Category scores are generated by calculating the average of their components (Categories are based on Sub-categories; Sub-categories are based on Indicators). They are not scored directly.

You will be asked to comment on the individual Indicators and Sub-indicators. When you see an indicator, you will be given four options with which to respond:

I agree with the score and have no comments to add.

I agree with the score but wish to add a comment, clarification, or suggest another reference. If you select this choice a comments box will appear in which you can type your comment.

No, I do not agree with the score. If you disagree with a score and select this third choice, you will be required to provide an explanation and recommended scoring change. Your explanation for why you disagree with the score is essential for helping us understand why the score needs to be changed. A “disagree” answer without an explanation to support that position will be ignored.

I am not qualified to judge this indicator.

In reviewing a particular Indicator or Sub-indicator, you should consider the following criteria:

• Is the particular Indicator or Sub-indicator factually accurate?
• Are there any significant events or developments that were not addressed?
• Does the Indicator or Sub-indicator offer a fair and balanced view of the anti-corruption environment?
• Is the scoring consistent within the entire set of Integrity Indicators?
• Is the scoring controversial or widely accepted? Is controversial scoring sufficiently sourced?
• Are the sources used reliable and reputable?

Following your review of the data scorecard for the country, Global Integrity staff will review all Peer Review comments for the country from each Peer Reviewer. We will pay special attention to indicators flagged by any Peer Reviewer as “disagree” (the third choice) and will then carefully review the specific indicator in question, including all information provided by the lead researcher as well as by other Peer Reviewers, to determine if a scoring change is indeed necessary. It is essential that you mark an indicator as “disagree” if you believe that the score should be changed; Global Integrity staff may not review indicators for potential scoring changes if they are not marked by at least one peer reviewer as “disagree.”

We typically process and review approximately 10,000 individual Peer Review comments. As such, the standardized method described above for sending us your feedback is essential if we are to publish the Global Integrity Report: 2010 on time. Please contact us via the Indaba Help Desk (http://getindaba.org/help-desk/) if you require additional clarification or assistance.

As mentioned above, Peer Reviewers’ comments will be published transparently alongside the final Integrity Indicators scorecard and final Reporter’s Notebook for each country. In the event of any major factual discrepancies that arise during the peer review process, Global Integrity staff will discuss the issue in question with all country team members and decide on appropriate changes, if necessary. If this should be necessary, we ask that you remain engaged in this dialogue until the final publication of the report.

Timely Submission

You will have approximately two weeks each to review and provide any comments on the Reporter’s Notebooks and Integrity Indicators scorecard. It is essential that all members of the country teams meet their deadlines in order to ensure timely publication of the final Country Report. As described in separate instructions, you will use the Indaba platform (http://indabaplatform.com) to access the Reporter’s Notebook and Integrity Indicators scorecard in order to provide your comments. Indaba will notify you via email when both products are ready for your review.