Governance and Stakeholders Survey

Overview of Survey

Of the 12 proposed sub-indicators within the Governance & Stakeholders category, the majority involve some amount of subjectivity reflecting stakeholders’ perceptions. While it is desirable to use objective, empirical data, such as counting the number of multi-stakeholder meetings a river basin organization holds in a year, such data are imperfect proxies for the actual principles or processes of interest. When measuring governance, perception data (what an individual believes to be occurring) is particularly valuable, because decision makers base their actions on their perceptions, and there is also frequently a divergence between de facto and de jure governance (Kaufman et al. 2010). Put another way, in a perception-based survey, individuals are asked to apply a subjective rating scale rather than answer only objective questions (yes/no or numeric responses). To develop comprehensive, systematic and comparable data for our Governance & Stakeholder indicators, we recommend deploying a survey to a cross section of stakeholders and then subsequently repeated for the next round of assessments.

Implementing the survey

A survey instrument has been developed to be administered to a group of regional experts familiar with water management issues in the basin. The instrument has been designed to correspond to 11 of the Governance and Stakeholder sub-indicators – one “module” per sub-indicator, containing between 3-7 questions each (see Appendix B). The master file is in English but has also been translated into Chinese, Spanish and Portuguese. The original version has been screened and approved by Conservation International’s Institutional Review Committee, but before administering the survey, it is advisable to ensure that it is also compliant with your own institution’s research ethics policy as well as any other ethical policies that might be in place in the jurisdictions where you wish to survey. It might be also helpful to have a small group of stakeholders to review the survey to adapt the terminologies and examples to the local context to ensure correct interpretation of the questions.

The survey was designed to take approximately one hour for respondents to complete and can be administered in person or remotely (e.g., mailed or sent electronically), in one-to-one settings or in groups. An in-person group meeting may be most efficient for the surveyor(s) and can be built into the agenda of a broader meeting. If administered in person, the surveyor(s) must take care to not suggest answers or otherwise bias respondents; information should be limited to the survey instructions and clarifying terms that may be unclear.

In principle, any stakeholder in the basin can take the survey, but participants should only complete the questions that they feel qualified to answer. Many questions require familiarity with details about water resource management and related topics, and so we recommend using a non-probability sampling technique referred to as expert sampling. This is not meant to be a representative sample of the population, which requires probability sampling and hundreds or thousands of respondents to infer that results represent the perception of the general population in the basin. Instead, respondents should be invited based on their experience with water governance issues in the basin. We recommend referring to your stakeholder mapping first and identifying stakeholder groups with high levels of interest and engagement in the basin, as they should be the most familiar with the current governance dynamics. With perception-based surveys, there are no “wrong” answers, but respondents should be able to explain their responses and to provide insight into areas of poor (or strong) performance as well as areas of disagreement among respondents.

Your sampling should cover all tiers of government, along with industry representatives, non-profit organizations and academic researchers. There is no minimum number of respondents per stakeholder group, and in some cases, it may be desirable for stakeholder groups to be given an opportunity to formulate an “official” response that incorporates multiple inputs and allows them time for consultation before answering. Survey responses should be kept anonymous, but we recommend recording respondents’ sector affiliation (e.g., provincial government) as well as their location within the basin, which can be as simple as “upstream,” “mid-stream,” and “down-stream.” With this information, and with enough respondents, it is possible to analyze the data for differences or commonalities. A larger number of respondents reduces the influence any one individual respondent has on results and can even offer some statistical significance to analyzing sectoral or geographical differences. It should be made clear to respondents and in subsequent communications about the results that the results reflect the perceptions of an expert panel, and the number of respondents should be noted.

It may also be necessary to allow respondents to reconsider and adjust their responses after learning about the mean values for the group. Respondents would receive the averaged results (and standard deviations) calculated for the group and then be asked to reconsider their initial responses that diverge from the mean. However, the means do not represent a “true value” since the topics are subjective, and so respondents can have legitimate reasons for deviating substantially from the mean. For this reason, standard deviations should be calculated and recorded as a measure of the uncertainty of the final indicator values.

While the priority of the survey is to elicit information that translates into quantitative sub-indicators, it will likely be necessary and useful to follow up and elicit qualitative information, particularly in areas of poor performance. The timing and amount of effort on this will relate to expectations for the narrative report that accompanies the indicators. This is one advantage of administering the survey on a one-to-one basis, though further interviewing should take place once the survey has been completed. If qualitative information is collected through a group discussion format, there is no guarantee that all perspectives will be heard.

Importing responses in FHI Toolbox

FHI Toolbox can import the responses from a csv template. The template requires responses to be formatted as follows: