Before designing an evaluation, you must assess the feasibility of answering the evaluation questions identified in Stage 1, Planning.

Reality Testing Implementation Questions

If any of your "top priority" questions pertain to program operations and implementation (concerning program goals, underlying assumptions, program inputs, program activities, program outputs, and/or participants' immediate outcomes) then discuss the following with your evaluator:

Design considerations:

  • Can the evaluator obtain program records, such as the grant application and reports to funders, as well as administrative records and intake/registration information?
  • Can the evaluator have class instructors fill out a short form after each session, rating such things as whether they provided all program content, whether participants appeared confused or disinterested in the material, and whether there were any circumstances that may have affected the fidelity of program implementation?
  • Can the evaluator observe a class session?
     

Reality Testing Immediate Outcomes Questions

Data considerations:

  • What data elements are needed to capture immediate outcomes targeted by the program (e.g., knowledge, skills, attitudes, behavior)? Can that necessary data be collected?
  • Will the evaluator do pre/post or post-only data collection to assess immediate outcomes? If your evaluator uses a pre/post survey (and/or any follow-up data collection), discuss how s/he will develop and use a unique identifier to link respondents' answer over time and across data collection instruments.

Time and budget considerations:

  • How long will it take, and how much will it cost, to collect and analyze the range of quantitative and qualitative data needed to assess program implementation and immediate outcomes? Are resources available to survey all program participants, or will a representative subsample need to be surveyed, due to cost?
  • Is there time and money to develop and test new outcome measures if no valid and reliable measures exist for particular outcomes of interest?
  • Will evaluation findings be ready if there is a particular deadline required by the funder or other stakeholders?
     

Reality Testing Impact Questions

To reality test impact questions, you need an experimental or strong quasi-experimental design in which you have obtained data from a randomly assigned control group or a reasonably similar comparison group. Without a control or strong comparison group, you will not be able to attribute outcomes to the program.

Design considerations:

  • What is the political, ethical and logistical feasibility of random assignment, or of finding, recruiting, and collecting data from a similar enough comparison group?
  • Can you afford to conduct a random assignment or quasi-experimental evaluation?
     

Data considerations:

If any of your impact evaluation questions pertain to subsequent outcomes, then you will also need data on subsequent outcomes from both the program and control/comparison groups.

  • Is it feasible and within budget limits to collect follow-up data (e.g., participant surveys)? If so, for how long should participants be tracked, and by what method?
  • If the budget cannot support both a follow-up survey and collecting outcome data from both program and comparison/control groups, which is more important for addressing your top priority evaluation questions?

Remember, no matter how long participants are followed over time, or how many respondents you are able to track and keep in the study over time, you cannot attribute outcomes to the program without a control or strong comparison group!

Other Resources

  • Marriage And Relationship Educators