Director: Abdolmehdi Baghaei, MD
Deputy director: Maryam Boshtam
Members in alphabetic order:
Shahram Oveis Qaran, MD, Afshan Akhavan, MD, Mansoureh Boshtam, MD, Shidokht Hosseini, Niloofar Shabani, D, Masood Mahvash
Program evaluation policy
Population interventions aimed at promoting health in the society call for studying the existing relations between risk factors and diseases. These studies attempt to employ the most efficient way of using the present knowledge to overcome health problems. So these programs should be evaluated continuously for their efficiency.
Evaluation is one of the most important elements of every interventional program in the society. Program evaluations assess the degree to which predetermined objectives have been achieved and contribute to the trend of intervention by providing data. Designing, performing and evaluating each program calls for continuous data collection. Evaluation in Isfahan Healthy Heart Program is described in detail in book 2.
Types and levels of evaluation:
1- Formative evaluation: It aims at the improving the program during its implementation. This kind if evaluation assesses the relation between program objectives and planned interventions.
2- Process evaluation: This kind of evaluation evaluates the process whereby each program is implemented.
Is the program being implemented as it was designed?
Has the intervention reached the intended target populations?
3- Impact evaluation: The emphasis of this evaluation is on assessing the short-term health outcomes of an intervention. These outcomes are divided into 2 categories: behavioral and non-behavioral
· Used services
· Environment and Policies
4- Outcome evaluation: To evaluate the degree of achieving ultimate program objectives.
When and at what level the evaluation should take place?
An intervention program takes at least 10 years to bring about health changes in the society (Crow et al. 1996). Before such changes occur, mortality, morbidity and intermediate indices (impact) should change. These indices are knowledge, performance, attitude and risk factors (RF).
The first requirement for the success of every intervention program is "effective intervention". To be assured of effective intervention, informative evaluation is helpful. The second step in succeeding to achieve program objectives is good implementation.
Process evaluation is recommended at this stage. After this stage, the efficiency of the program in affording early results is assessed by impact evaluation.
Finally, evaluation of results is necessary at the end of the program.
Possibility of evaluation
Before implementation, each program should be planned in such a way that it can be evaluated. To achieve this, each program should be able to answer the following questions:
Are the program objectives written well enough to be:
Executable within the intended timeframe
Do the selected indices for the program comply with the following requirements?
2-1) providing a clear definition of standards or a way for comparing results
2-2) be measurable and quantitative to the extent that is possible
2-3) be readily measurable
- The relation between objectives, indices and program interventions should be outlined clearly.
- Present the details of program execution
- Evaluation check list should be added to the program
- Means of measuring the indices and the way of measuring should be defined
Evaluation committee was established simultaneously with the primary design. The evaluation activities of the committee take place within its 4 units (figure 2).
Monitoring: it does the annual assessment of knowledge, attitude, behavior and skills among adults, children, adolescents, health workers and patients (BASK).
Risk assessment unit: in this unit all over-35 individuals whose data were collected in the phase 1 of the program are followed up, data related to their cardiovascular accidents are collected and analyzed with the early data related to the distribution of CVD risk factors in the intended society.
Event registry unit: functions based on protocols of Monica's study on population intervention, registry and follow-up of cardiovascular accidents and strokes in control and intervention populations.
Data analysis unit: all of the processes of collecting, surveillance and qualitative control of data are followed by data analysis.
Evaluation committee undertakes early evaluation of the intended population in the demonstration phase and also conducts outcome evaluation. Process evaluation is done with the supervision of this committee and related units. After being analyzed the results are provided to projects managers to be used in efficient design of future intervention programs.
According to evaluation planning mentioned in previous chapters, we designed Isfahan Healthy Heart Program Evaluation format as below. Because of our restrictions in human and funding resources, we allowed some alterations in planning of evaluation system.
In the future sentences, we demonstrate the evaluation system planning, step by step. We discuss about types and levels of evaluations, organization chart of evaluation committee in IHHP, IHHP indicators and the methods of data collecting and analysis. The surveillance system will be discussed in detail. The evaluation report of demonstration phase of IHHP will be presented in the 3rd volume of these book series.
First Step: Choosing Evaluation Design for IHHP
IHHP evaluation is designed as a quasiexperimental (community trial) with reference community. For this reason, Isfahan and Najaf-abad (rural and urban areas) were selected as interventional community and Arak was selected as reference community. These areas are located in the central of IRAN have the same climate. They have similar situation about sex and age distribution and somehow have the similar culture in many aspects. But they differ from each other about industrial context and medical care providing, specially in third level of health cares (medical specialty and subspecialty). The acceptance of Arak medical university and health services for cooperation in this trial was the main reason for selecting this community as reference.
Because of these dissimilarity (mentioned above), we compare the trend of changes in both communities in our analysis. Also we have the basement characteristics and background variables of both communities for adjusting the indicators and stratified analysis (look at 3rd volume of these book series).
Second Step: Determination of Program Objectives and Goals
According to general aim of IHHP, in the second step we determined the goal and objectives of IHH. They were mentioned in volume 1 of these book series.
Third step: Determination of Evaluation System in IHHP
Evaluation in IHHP is done in 4 level and types.
1st level (preimplementation): Before any activity by projects, they must write their intervention in a formatted proposal (evaluable format) that is constructed by evaluation committee. Also, they must complete formative evaluation worksheet. This is 30 items questionnaire prepared for IHHP.
In this level of evaluation, we want to test agreement between projects’ objectives and each activity. Another reason for this kind of evaluation is to estimate the feasibility and accountability of each activity, before beginning.
2nd level (implementation): After beginning of any activity the process of that activity is evaluated. This kind of evaluation tests the agreement between actual performances with planned objectives. The community satisfaction, community motivation and barrier finding are the other purposes of this form of evaluation.
3rd level (Impact evaluation): In this level of evaluation, we measure the impact indicators in communities (interventional and reference) annually. The method of sampling, data collection and analysis is discussed in previous chapters.
4th level (Outcome evaluation): Outcome indicators are assessed in the 3rd phase of IHHP. The aim of this type of evaluation is to test the achievability of main objectives of the program.
IHHP evaluation in the above mentioned levels are conducted in 3 phases.
1st phase (Community diagnosis): In this phase, all background variables in the community plus outcome indicators were measured. Socioeconomic state, sex and age distribution, educational level and any other variables that may have a confounding effect on the results of intervention were assessed.
2nd phase (monitoring): In this phase, impact indicators are measured as described in surveillance method (WHO stepwise approach). The method of sampling and data collection were mentioned in previous chapters.
In this phase, 6300 adults will be followed for determining the risk chart.
3rd phase (outcome evaluation): At the end of interventions, all outcome indicators will be assessed again as the same as in the first phase of evaluation.
Fourth Step: Team Making and Organizing Duties
Evaluation needs a multidisciplinary team (Epidemiologist, Statistician, Scientist, Methodologists). So, we organized our team as the following chart with different specialty.
· Monitoring Unit: Impact evaluation and Process evaluation annually.
· Risk assessment Unit: Follow up study for risk chart drawing
· Event Registry Unit: This unit was discussed in detail before.
· Data analysis Unit: Data qualification, Analysis and Report
Fifth Step: Implementation of Evaluation in IHHP
1. Determination of Indicators:
· Review the literature
· Scientific Advises (focus groups, nominal groups)
After the above steps, we determined the IHHP indicators in three levels and two types (individual versus environmental). Indicators were categorized in impact, outcome and process levels.
2. Determination of Data Collection Methods and Materials:
We used validated and reliable questionnaires for each kind of data about BASKs.
The methods of data collection were discussed in the methods of IHHP