Unique Methods in Advocacy Evaluation

This AEA session discussed common advocacy evaluation methods:

• Stakeholder surveys or interviews - Print, telephone, or online questioning that gathers advocacy stakeholder perspectives or feedback.
  
• Case studies - Detailed descriptions and analyses (often qualitative) of individual advocacy strategies and results.
  
• Focus groups - Facilitated discussions with advocacy stakeholders (usually about 8-10 per group) to obtain their reactions, opinions, or ideas.
  
• Media tracking - Counts of an issue's coverage in the print, broadcast, or electronic media.
  
• Media content or framing analysis - Qualitative analysis of how the media write about and frame issues of interest.
  
• Participant observation - Evaluator participation in advocacy meeting or events to gain firsthand experience and data.
  
• Policy tracking - Monitoring of an issue or bill's progress in the policy processes.
  
• Public polling - Interviews (usually by telephone) with a random sample of advocacy stakeholders to gather data on their knowledge, attitudes, or behaviors.
  

And highlighted four new methods that have been developed specifically to address advocacy evaluation's unique challenges:

• Bellwether methodology - Interviews conducted with "bellwethers" or influential people in public/private sectors whose positions require that they track a broad range of policy issues. Part of sample is not connected to issue of interest and sample does not have advance knowledge of interview topic. Used to assess political will as outcome, forecast likelihood of future policy proposals/changes, assess extent that advocacy messages have "broken through", and to gauge whether an issue is on federal/state/local policy agenda and how it is positioned.
  
• Policymaker ratings - Advocates (or other informed stakeholders) rate policymakers of interest on scales that assess policymakers' support for, and influence on, the issue. Used to assess extent to which a policymaker supports an issue and whether that support is changing over time.
  
• Intense period debriefs - Advocates are engaged in evaluative inquiry shortly after a policy window or intense period of action occurs. Used when advocacy efforts are experiencing high intensity levels of activity and advocates have little time to pause for data collection.
  
• System mapping - A system is visually mapped, identifying the parts and relationships in that system that are expected to change and how they will change, and then identifying ways of measuring or capturing whether those changes have occurred. Used to try to achieve systems change.
  

Please note that the above notes are credited to the "Unique Methods in Advocacy Evaluation" by Julia Coffman and Ehren Reed.

See posts about other sessions I attended at this year's AEA: "American Evaluation Conference Summary Post"