The evaluation design was based on a review of the literature and consideration of coalition goals and objectives and served as the basis for decisions related to evaluation methods and measures.  Potential instruments and measures were discussed within the evaluation group and examined by individual sites. Factors for consideration included:  cultural relevance; potential for translation; feasibility of data collection requirements; fit with data collection timelines; appropriate literacy levels; types (quantitative/qualitative) and quality of data generated.

The evaluation instruments used for the Allies cross-site evaluation are available for use by others.

Design components:

(a) Contextual factors related to the community where the coalition operates
As part of the Allies cross-site evaluation, coalition leadership were interviewed using the context survey at several time points throughout the planning and implementation phases to document contextual factors. 

(b) Coalition processes and structures
Coalition processes and structures were measured both quantitatively and qualitatively in the Allies evaluation. The Coalition Self-Assessment Survey (CSAS) was administered annually to the coalition membership, and key informant interviews were conducted with coalition members, staff and leadership, and community members outside of the coalition.

(c) Planning and planning products
The Allies cross-site evaluation assessed the quality of each coalition's action plans using criteria that included how representative the plan is of the coalition's mission, how it engages diverse stakeholders, how feasible the timeline is, the adequacy of resources allocated, and the dissemination of written plans within the coalition. 

(d) Implementation actions, activities and collaborations
Information about coalition activities was gathered throughout the implementation period using Program Reach, a computerized web-based tracking system.

return to top

(e) Anticipated intermediate outcomes
In the Allies evaluation, individual level changes were defined broadly and included exposure to asthma-related activities, parent management of asthma, and changes in quality of life and asthma related symptoms. To measure these changes, each site administered the Asthma Core Caregiver Survey to a cohort of children affected by pediatric asthma that were expected to participate in the coalition's primary interventions, as well as a comparison group. Both groups were followed for one year between measurements.

Community-level changes include improvements to system level function and changes in system capacity. Examples include standardization of asthma action plans, protocols and referral processes; tracking systems and registries; policy changes; quality improvement; and system integration and linkage. These changes reflect both health care system and provider level changes. Such activities are tracked through the Program Reach web-based system. Results of such efforts, including both system capacity changes and coalition impact in the community, have been measured qualitatively through key informant interviews with coalition members, staff and leadership, as well as key community leaders outside of the coalition, as described in the following section.

(f) Expected asthma related health outcomes
Allies is working in partnership with the Center for Medicare and Medicaid Services (CMS) and the Agency for Healthcare Research and Quality/Healthcare Cost and Utilization Project (AHRQ/HCUP) to obtain and analyze health care utilization data for the Allies sites and comparison communities for the periods before, during and after RWJF Allies funding.  CMS is providing annual zip code level data related to Medicaid-supported hospitalizations, ED visits and medication use for the period 1999 to 2007. AHRQ is providing hospitalization data supported by multiple payers for this same time period.

return to top

Evaluation

		Introduction
		Design & Instruments
		Findings