Evaluating Academic-Community Partnerships

Several resources are available for evaluating academic-community partnerships and for planning community-engaged research. Researchers at the South Carolina Clinical and Translational Research (SCTR) Institute at the Medical University of South Carolina created a toolkit for academic-community partnerships that are preparing for CBPR.16-17 The toolkit defines CBPR partnership readiness “the degree to which academic/community partners fit and have the capacity and operations necessary to plan, implement, evaluate, and disseminate CBPR projects that were informed by a partnership readiness model that will facilitate mutual growth of the partnership and positively influence targeted social and health needs in the community.”16 In addition to using a trained facilitator to guide the process, the authors offered several suggestions for using the toolkit (Box 16.1).

The National Institute of Environmental Health Sciences published the Partnerships for Environmental Public Health Evaluation Metrics Manual, which provides examples of metrics that Partnerships for Environmental Public Health grantees and program staff can use for program planning, implementation, and evaluation.18 Evaluation tools are included for grantees and program staff to measure the effectiveness of partnerships and the impact of research on public health

Formation of CAB

Clarifying purpose, functions, and roles

Potential purpose, function, and roles may include:

  • 1. community perceptions, preferences, and priorities in the development of a research agenda and research processes
  • 2. advising on study protocol design and implementation, facilitating community consent, evaluating and communicating the risks and benefits of research, helping provide resources, evaluating education materials, disseminating information, and using research findings to advocate for policy change
  • 3. partner or advisor; clarification of role is essential

Determining membership composition and recruitment strategies

To select appropriate board members:

  • 1. specific inclusion criteria should be established that reflect the goals of the research and the intended functions and purpose of the CAB
  • 2. assess community and capacity to guide identification of potential partners
  • 3. create a “potential member matrix” that includes the types of organizations to be considered; their reputation, activities, and achievements in the community; their capability to contribute resources; their self-interests; and their potential conflicts
  • 4. screen through telephone and personal interviews
  • 5. invite through personal invitations and follow-up with letter to organization
  • 6. review the potential member’s intended role and clarify expectations, including and defining mechanisms of communication to help ensure a shared understanding of the requirements of the board member position
  • 7. signed letter of commitment provides documentation of the agreement and helps to minimize potential misunderstandings

Operations

Establish operating procedures and principles

Operating procedures guide how the team works together to complete tasks and include:

  • 1. setting the agenda and documenting minutes
  • 2. considering group dynamics and accepted social norms, listening to all, letting members agree to disagree
  • 3. having all members participate in board meetings and activities, which start and end on time.

Members periodically reassess and revise the procedures, on the basis of process evaluations, to maintain an equitable balance of power Defining community values or principles that guide research is another initial task of a CAB, and this provides the opportunity to integrate the local context, develop trust, and build relationships among board members

Operations

Establishing leadership, balancing power, and making decisions

Fair and appropriate distribution of power and leadership may be

addressed with community and academic cochairs

Effective leadership and balancing of power supports members’

satisfaction, participation, and overall effectiveness by using

democratic and consensus-based decision-making

Establish process for decision-making and consensus decisions but

increase group solidarity

Maintenance

Evaluating

partnership

processes

A multimethod approach to collecting evaluation data increases

the likelihood of well-rounded assessment of CAB structure and

processes and may include:

  • 1. key informant interviews, meeting observations, focus groups, documents such as activity logs, and member surveys to provide different perspectives of the partnership and enhance the comprehensiveness and credibility of evaluation
  • 2. qualitative methods, such as key informant interviews, provide a platform for CAB partners to address frustrations and concerns
  • 3. quantitative methods, such as surveys, provide a standardized measure of partnership processes that allows a baseline measure to be established and reevaluated over time to gauge continued effectiveness
  • 4. measures of process evaluation incorporate items to assess group dynamics within a CAB partnership framework, including shared leadership, open communication, mechanisms for resolving conflicts, and trust and cohesion
  • 5. evaluation of CAB leadership considers whether leaders provide praise and recognition, seek out members’ opinions, and approach members for help with specific tasks
  • 6. process evaluation also includes assessment of more pragmatic issues such as turnover rate of board members, success in recruiting members with specific skills or connections to influential leaders, members’ perceptions of the benefits

and costs of participation, and the degree to which members perceive the partnership to be effective and sustainable over time

7. evaluations that address partnership priorities increase the likelihood that partnership collaboration continues, thus promoting sustainability

Maintenance

Sustainability

Develop plan for sustainability early in the life of the CAB Recognition of CAB members’ contributions of time, resources, and expertise, through some type of compensation, promotes continued engagement in partnership

Continuing relationships informally during gaps in funding or activities helps to maintain communication between partners and provides opportunity for brainstorming about next steps for partnership

When sustainability is not possible, dear communication between researchers, CAB, and community members will leave door open for future collaborations

Box 16.1 Recommendations for Using the “Are We Ready? Toolkit for Academic-Community Partnerships for Community-Based Participatory Research,” South Carolina Clinical and Translational Research Institute

  • 1. Schedule a retreat in which all partners can participate (1-2 days): or
  • 2. Plan five to six 2-hour sessions in which each section can be reviewed and discussed by all partners.
  • 3. Each partner should have his/her own workbook.
  • 4. Before the retreat or scheduled sessions, each partner should review and complete the reading materials for the planned session.
  • 5. When the partners meet to discuss the assigned sections, individual responses are shared, discussed as a team, and tentative action plans are made. A trained facilitator will help guide this process. At the end of all of the sessions, the team will derive a final comprehensive action plan to guide next steps.
  • 6. Prior to starting the first session, the partnership may want to decide on principles or ground rules to guide this process. For example, honest and transparent communications are vital for the team to make accurate assessments of the partnership readiness and growth opportunities to leverage the partnership for future success.16

at local, regional, and national levels.18 The manual explains how to use a systematic, strategic analysis of program activities (actions that use available resources), outputs (direct products of activities), and impacts (benefits or changes resulting from activities and outputs) to identify meaningful metrics that can be used to document program achievements. For example, under “Output 3: Translation of scientific findings among partners,” several potential metrics are listed in the manual (Box 16.2).

Szilagyi et al.19 proposed a framework for evaluating the community engagement activities of academic medical centers. The framework includes broad goals and specific activities within each goal; goals and activities are evaluated using a health services research framework consisting of structure, process, and outcomes. In the framework, “structure represents the administrative arrangements and committees that are developed, the new organizations established to enhance community engagement goals, any new facilities or space, and financial as well as non-financial arrangements regarding community engagement.”19 Szilagyi et al.19 recommended both qualitative and quantitative process assessments of activities (for example, What services were delivered? Were they appropriate and necessary? Were they delivered with fidelity and rigor, and in a timely, patient- centered, and culturally sensitive manner?).19 The framework is likely to be helpful for evaluating the effectiveness and return on investment of academic medical center community engagement activities.

Box 16.2 Example Metrics for Output 3: Translation of Scientific

Findings Among Partners, Partnerships for Environmental Public Health Evaluation Metrics Manual

  • • Number and description of materials that translate findings.
  • • Lists of coauthorship on materials that demonstrate a mix of partners.
  • • Description of subsequent funding for translation efforts.
  • • Description of support provided by target audience for translation efforts.
  • • Number of publications that report on translation activities.
  • • Description and counts of how partners are using findings in other settings.
  • • Number of requests for translated information by partners.
  • • Description of requests for materials by others.
  • • Anecdotal evidence indicating successful translation of scientific findings to new audiences.
  • • Number and description of materials or products produced by partners that include research finding.18

To lay the groundwork for evaluating community-academic contributions to research conducted by CTSAs, Eder et al.12 proposed a typology consisting of three relationship types: engagement, collaboration, and shared leadership. The authors defined engagement as “an intention to exchange information and possibly resources (including money) through an individual event or a short-term series of events (including clinical trials).”12 They defined collaboration as “an intention for members of a partnership to cooperate over time for the purpose of achieving specified goals,” which may or may not be shared.12 In addition, they defined shared leadership as “an intention for the partnership to achieve shared goals.”12 Rather than focusing on specific community-academic activities, the typology encourages CTSAs to measure the strength of relationships using synergy and trust as core metrics.12

Models that have been proposed for evaluating translational research,5-20-23 may also be useful for evaluating community-engaged research. After considering prominent models of translational research and identifying their commonalities, Trochim et al.23 proposed a process marker model for evaluating translational research. The model identifies key operational and measureable markers along a generalized process pathway from research to practice.23 According to the model, translational research is a continuous process that moves from basic biomedical research through clinical and practice-based research, and ultimately to health policies, outcomes, and impacts.23 The model assumes that this process may be bidirectional, variable, and complex and that there are many potential markers along this process.23 The process marker model proposed by Trochim et al.,23 which is grounded in process modeling research,19-24 provides a useful framework for evaluating interventions aimed at improving translational research (for example, interventions designed to reduce the relatively long time from discovery to routine use and health impact).

 
Source
< Prev   CONTENTS   Source   Next >