Introducing the Indaba Fieldwork Platform

Why We Built It and What It Can Do for You

The Indaba fieldwork platform helps organizations collect data and reporting and publish it. This document describes the need for Indaba, our thought process when building it, and what it does. See related documents here:

• The Feature List (a short version of this paper)
• Indaba Adoption Scenarios
• Video demo and discussion

Indaba enables groups of people to design projects, collect data, write reports, edit documents, clean datasets, conduct quality control and peer review, and then publish or export the results, all from a web browser. Indaba’s workflow manager allows geographically distributed teams to collaborate online with minimal oversight, allowing projects to scale up.Indaba also has tools to manage relationships, control access to common content, and promote accountability for a group of contributors working on a data gathering or publishing project.

Designed by a coalition of non-governmental organizations (NGOs) convened by Global Integrity, Indaba aims to dramatically lower the time and cost of measuring and tracking fuzzy social issues (transparency, service delivery, human rights, etc.), while promoting and sharing best practices and methodologies across the social sector. By using a common, open platform, data from across the social sector can be compared and reused, encouraging the use of best practices and open, interoperable data formats. Indaba is a Web-based software-as-service requiring no local installation with field staff or home office.

Why We Built Indaba: A Brief History of NGO Data Gathering

At the turn of the century, voices within the aid donor community were calling for more systematic approaches to aid, particularly around issues of corruption and governance. Aid, it was argued, should be concentrated in countries with good performance on anti-corruption. Corruption rankings such as the World Bank’s Worldwide Governance Indicators or Corruption Perceptions Index were suggested as triggers for aid flows.

This was a good idea, but with a serious flaw: the metrics being used to rank countries on issues crucial to development were, in the view of the field’s best thinkers, entirely inappropriate for this use. [1, 2, 3]

Governance indices typically took three forms: mashups of third-party survey data, statistical analysis of newswire headlines, or a handful of hyper-educated Westerners ranking every country in the world based on things they had read or heard from friends on the ground. These rankings were frequently of limited value for suggesting specific policy reforms and tended to track very closely to the consensus views of Western elites.

The development sector desperately needed better data, especially data reflecting local realities. Several organizations waded into this tiny crisis with a better vision: break complex issues (“corruption”) into hundreds of discrete, measurable concepts (“Are jobs in ministry XYZ given out through a competitive process?”). These assessments focused only on things we could actually change. Rather than use a single number to assess an entire country, these second-generation datasets produced actionable, disaggregated insights, often with narrative explanations for why a score was given.

Answering these questions required better primary sources, which led to the creation of the distributed NGO, a new media model where a loose network of reporters, activists, or researchers feeds information to a central organizer. These organizations represented a radical shift away from Washington/London/New York/Brussels as monolithic voices in development data and towards the simple but powerful idea that people living in developing countries have crucial insights into what is and isn’t working in development.

The challenge in executing this new approach was that collaborating with hundreds of people in dozens of countries is a lot harder than the old five-people-in-one-room-in-New-York approach. None of the groups working on these issues had a budget to open dozens of field offices or fly everyone to one place. Using the Internet was the only way it could ever work. Indaba is the latest iteration in eight years of technology development at Global Integrity aimed at building tools to make distributed NGOs work over the Internet.

Why Existing Technology Wasn’t Good Enough

In the summer of 2010, the Indaba team went on a listening tour with projects at seven distributed NGOs. What we found was troubling. The majority of institutions we talked to use Microsoft Word documents or Adobe PDFs as their primary database for quantitative datasets. Typically, data are collected in Word documents or PDFs, then emailed back and forth during quality control processes. Finally data are transferred, usually by hand, to an Excel spreadsheet as the final data repository. In one case, an organization emailed Word files to and from the home office until the files became so huge that the files inevitably became corrupted in transit. They would then print the documents onto paper and retype the data into a spreadsheet by hand. The project managers did this for more than 10,000 data points.

Suffice it to say that there are a number of disadvantages to this approach. The more interesting question is why smart people at these organizations continued using Word and PDF documents as their solution.

One answer is that there aren’t a lot of tools off the shelf that address the issue of managing workflows in organizations, and fewer that integrate content and relationship tools. Content management systems (like the very good Drupal CMS) are good at storing and publishing content, but don’t reflect the internal workflow of organizations. Project management systems, on the other hand, typically are not good content databases.

A second answer can be found by focusing on the end users, examining the constraints facing staff in distributed NGOs and what software would be realistic in improving their situation. These are hyper-connected people with multiple jobs, working from Internet cafes and university computer centers. Pushing new tools on these overworked people is not a good use of project managers’ limited influence, so organizations use the ubiquitous tool of our time — Microsoft Word — and do whatever it takes back at the home office to make it work. This is, in fact, good user-focused design.

But it’s awfully hard on those project managers. A sad consequence of this is that NGO data tends to be published in difficult-to-use formats, without metadata, common IDs with other datasets, or even good internal structure. NGOs avoid a highly effective methodological approach of matching quantitative data with deep narrative description (getting beyond “what” and to the “why”) simply because the technology to manage these datasets is unavailable to them. Much of this is compounded by fatigue, as exhausted project managers simply want to get something out the door before time or money runs out.

In some cases, organizations focused on beautiful printed materials treat online access as an afterthought. In one particularly strange technology decision, an NGO aimed to replicate the print experience so completely online that the high bandwidth website played the sound of a crisp paper page turning to online visitors. It did not, however, allow readers to copy and paste text.

Consistently applying best practices for publishing would be an improvement. But there’s potential for much more. In the late 1990s, blogging software radically transformed individual publishing by making a simple thing — putting text and images onto a website — much easier. In the decade since, an explosion of new, unexpected expression has occurred for individual authors. Today, organizations that publish data and reporting are in that “pre-blog” state, where doing simple things, like emailing a Word document, processing it for quality control and publishing it to the Web, print, or datasets are simple but tedious, difficult to automate, and stressful.

If we can take that stress away, the potential growth of new data gathering and publishing is huge. If we lower the costs of publishing for distributed NGOs the same way that blogs did, we can transform not only established organizations but also empower start-up, grassroots, and unexpected sources of structured analysis and data to inform public debate. That is our ultimate goal with Indaba: the democratization of voice and official knowledge to include every group of people with something to say.

Indaba: What It Does

The Indaba fieldwork platform is a collection of tools that organizations can use to run projects that collect and publish text, data, or any kind of file (such as images or video). It is well suited to large teams of contributors working remotely on projects which collect structured information such as scorecards, comparative analysis, or other publishing efforts where content is created in discrete chunks and repeated at large scale.

Indaba is highly flexible, allowing each project to define workflows, user roles, content types, and other key variables to match the preferred working style of that project. Once created, these project definitions become useful starting places for future projects, encouraging incremental improvement and the transmission of best practices across the social sector. When a project is complete, the data is stored in a well-structured database ready to send content to web, print, social media platforms, mobile devices, or file exports. Indaba consists of a large database with several software applications that move information in and out of that database.

There are currently three core applications involved in an Indaba project: Designer, Builder and Publisher. Additional input or output applications can be added as technology matures, such as mobile device or tablet input.

Designer

Designer is a web interface used by project managers to define a new project. For new organizations, this is a consultative process of making a project design explicit and repeatable.

Most of this effort happens offline with a pencil and paper, guided by a representative from the Indaba team. The most important aspect of this is the creation of a workflow for each type of content to be published. The workflow defines each goal that must be completed before a document can be published. Even on paper, making a workflow visible tends to improve how projects run. With Indaba, the workflow becomes a foundation upon which to build your project. At all steps, existing Indaba projects can serve as a template to modify from or adopt entirely. Once created, your project definition becomes a valuable starting place for future projects at your organization and others. A project definition involves deciding the following:

• what products you want to publish,
• what format (text, questions, files) those products will take,
• what kind of roles your staff will have,
• privacy and peer-to-peer visibility rules for each role,
• permissions and access to features for each role,
• setting a workflow for each product,
• creating rules for each workflow assignment such as deadlines or completion requirements,
• how will users will be given assignments (in advance, self-claimed from a queue, or assigned on the fly), and
• creating automated and custom notifications, such as deadline reminders or thank you notes.

Designer’s workflow manager is one of the few unique aspects of the platform. Workflows are defined with a high degree of flexibility, meaning that workflows can branch, circle and recombine in whatever way the organization requires. An example workflow might look like this:

• Field staff signs contact
• Field staff submits data
• Office staff reviews data and discusses revisions with field staff
• Editors, libel reviewers and fact checkers each claim this document out of a pending work queue, and engage with the content simultaneously.
• Peer reviewers gain access to the document after all three of the previous staff have completed their work. Peer review lasts until a deadline, or until 80% of peer reviewers have completed a review.
• Office staff reviews peer review feedback and discusses results with field staff.
• A manager makes a final review of the document before approving it for publication.

Each step in the workflow has unique rules that control deadlines, notifications, and other options that are configured in the Designer application. Designer is a hosted service using the following technology stack: PHP, Tomcat, MySQL, Linux, Amazon cloud hosting.

Builder

Builder is a core web interface used by all contributors to a project, including authors, editors, managers and reviewers. Builder is the centerpiece of the Indaba platform, the tool used to automate and manage the workflows defined in the Designer tool. For most users Builder is the only part of Indaba they will see.

In Builder, users can do several things…

Complete assignments.

The concept of assignments is the core of Builder. Each workflow goal (defined in Designer) is broken into one or more specific assignments that go to individual users, who are then accountable for finishing them. When any user logs in, they are presented a list of their pending assignments. From that list, each assignment links to a specific piece of content, viewed with a specific tool. Each tool is a unique interface devoted to doing a specific kind of work, such as editing a text document, or reviewing and commenting on a dataset or scorecard. Existing tools allow for several forms of text editing, data input and cleaning, comment and peer review, or simply logging that an offline event (such as hiring or paying a researcher) has occurred. New tools can be created with relatively simple Web development, making the system extensible to new kinds of work.

Monitor work in progress.

There are a number of ways to see into the progress of work in flight. Depending on a user’s permissions, this can range from only seeing the relative progress of that document compared to others generating the same content, up to detailed status reports and the ability to export or edit the document. In particular, Indaba focuses on two key metrics: identifying areas where work has stalled, and making projections about when workflows will finish.

Use the case manager to fix problems.

Automated workflows are great until something unexpected happens. When it does, humans need to get involved, collect information and make decisions. Indaba has an integrated case manager for addressing fieldwork problems on or off the system. The case manager serves as a unified place to collect information on a problem, share that with relevant users (users and content can be “attached” to a case), and rank the severity of the problem. The case manager is integrated with the document workflow: if needed, the case manager can either freeze progress of a document or prevent a document from publishing until the problem is resolved.

Manage assignments via queues.

As goals are completed, documents will pile up waiting for review or editing. Indaba uses queues to allocate these assignments when a group of people are sharing responsibility. This allows for very efficient allocation of assignments to people with capacity to complete the work, and allows people to self-assign into assignments of personal interest. Queues can also be owned by a manager who assigns the work as it comes in. Once a user claims an assignment, they are accountable for finishing it, but if needed a manager can shift assignments in progress to new people.

Send messages to individuals, or groups.

Builder has a built in sitemail messaging system that works like a very light version of a webmail system. This exists to solve two problems. One, sitemail enables an email-like communication with people working anonymously or in areas where email censorship is a barrier to reliable communication. A more common application is to quickly find and send messages to your field staff. Indaba’s sitemail has your entire team loaded into the “to” box. For users with appropriate permissions, Indaba also enables messages to preset teams (arbitrary groups of users), to specific roles (for example, all field researchers), or to the entire project. This is also possible over email, but in practice, we believe having a place for instant lookup of up-to-date field staff contact information leads to better communication between the field team and managers. Sitemail is forwarded to users’ real-life email inboxes by default. Builder is a hosted service optimized for reliability and stability under heavy load. Builder uses the following technology stack: Java, Tomcat, MySQL, Linux, Amazon cloud hosting. Builder is actually two independent Java applications, one to serve pages, and another to manage background events like email alerts.

Publisher

Publisher is a web interface that allows project managers to export and publish content stored on the Indaba platform. As with all Indaba tools, this is permission controlled; users can access only eligible content. Publishing takes three forms:

• Bulk export: text, datasets, and files can be exported in a variety of file formats including the most common, as well as open standards. With a bulk export, content can be transferred to the content manager of your choice.
• Widgets: widgets allow organizations to quickly and easily serve content created in Indaba over the Web, similar to embedding a YouTube video. Indaba widgets are open source (using an Affero GPL license), and we encourage customization and sharing across our community. Have you finished compiling a bunch of scorecards and want to quickly publish them on the web? Indaba’s scorecard widget can get your content online, with professional polish and navigation, in minutes.
• API interface: for greater control over information served via Publisher, an API allows custom built applications to display content with a high degree of flexibility and customization.

The Indaba Design Concept

The previous section explained what Indaba does. Now we turn to explaining why Indaba is designed the way it is. Let’s go back to our project managers using Microsoft Word-over-email. The decision to use Word is very important: a technical solution must reflect the working style of all participants, and these people clearly prefer using simple solutions and ubiquitous tools over more complicated approaches. This has guided our thoughts towards the following design goals.

Integrate project management, content management, and relationship management.

Indaba is not a universal solution that will manage all of your organization’s projects, manage all of your content, or store all of your contact information. But in the context of a group of people working together on a large publishing project, it does each of those in a clean, quick, integrated way.

Access from any Internet link.

Users should be able to perform all system functions from the computer they already use, regardless of its condition, and without installing software or plugins. Many of our users work from Internet cafes or university computer labs. Using our entirely browser-based approach, users can write, edit, or manage data from any Internet-connected machine on the planet. In Global Integrity’s eight years of fieldwork in more than 100 countries, we’ve never had a contributor that didn’t already value their Internet connection. They already do whatever it takes to stay online, a process that requires no additional encouragement from central managers.

Today, all Indaba functions can be accessed from any computer with access to the Internet and either Internet Explorer (version 7 or better) or the Firefox browser. Indaba can also be used in smartphone Web browsers (iPhone, Android). Additional browser compatibility, as well as more purpose-built mobile integration, will be added over time, driven by dialogue with our user community. Based on community feedback so far, we are exploring offline data entry via a desktop application. We are also looking at ways to enable paper-based data collection (perhaps via bar coded forms printed from Indaba and returned to the Indaba database via a smartphone camera) in challenging, remote circumstances.

A personalized, workflow-aware experience.

While technology can be a barrier to adoption, a more powerful barrier is the difficulty of training remote, part-time users. Our solution is to mimic the existing working style of our community: present field staff with a very simple, highly structured experience. Indaba’s workflow manager allows each user to see exactly what that person needs to be working on, and can limit visibility into non-essential content and features. As workflows progress, access to documents shifts in response. Peer-to-peer visibility can be limited to working groups of manageable size (i.e. a country team, or a technical specialty), or turned off entirely, as configured by project managers. Privacy can be configured on a user-to-user basis, allowing double-blind peer reviews, for example. Instructions and help are provided on-screen throughout the system, customized to the specific assignment the user is working on. Instructions are unique to each project and can be improved on the fly as field staff run into problems.

Managers can quickly see problems, sort by importance and solve them.

While we want to limit the visibility of field staff to their task at hand, the opposite is true of managers. These people need real-time visibility into every aspect of the project, distilled into useful dashboards and status icons. This goes beyond access to in-progress content to indicators of what’s happening inside of each workflow, with escalating alerts that notify managers of potential problems. Deadline awareness is built into every assignment. Email alerts can be fired to remind staff of an approaching or missed deadline, escalate to emailing managers, or start a new ticket in the case manager; these rules and messages are configurable for each assignment. As a result, managers can avoid having to constantly check field staff deadlines until someone actually misses them. User statistics provide useful insights into field staff engagement. For example, if a manager is investigating an overdue assignment, Indaba can indicate if someone hasn’t logged in for weeks (big problem), or if the field staff is regularly completing other work, but hasn’t gotten to that assignment yet (smaller problem). This keeps managers focused on solving the truly critical issues. Indaba allows managers to shift their perspective from a narrow view (a list of assignments they must complete) to a universal view (real-time status of every assignment, document, or case) as they like.

Grassroots and security ready.

Using an entirely web-based publishing platform has two important advantages. First is convenience. The total technical know-how needed for an organization to use Indaba is… none. There’s nothing to install at the home office, no databases to back up, no hosting to buy. An organization using Indaba doesn’t need a website at all, and in the case of start-up projects, they probably won’t have one until information is ready to publish. While this is merely convenient to traditional institutions, this may be essential to grassroots groups or non-traditional projects that lack dedicated technology staff.

Second, hosting everything in an online database limits risk for field staff in difficult security environments. A reporter can walk into an Internet cafe, contribute to a project and walk out again, carrying nothing incriminating with them. There’s nothing on a laptop to confiscate, and it is difficult to connect a visit to indabaplatform.com to a specific project. Indaba stores very little user data that are not voluntarily submitted (for instance, we do not log IP addresses), allowing anonymous contributors to work in relative safety.

Current status of Indaba

Please see our running status reports for up to date information on progress and availability. Your feedback is warmly welcomed below. Or join our email list here.