Communities of Practice Evaluation Guide

Primary tabs

    Prepared by Michael Lambur, eXtension Evaluation and Research Leader
    Email: mike.lambur@extension.org

     

    Contents

    Introduction

    Much of the work in documenting the processes and outcomes of eXtension will be come from Communities of Practice (CoP) assessing the use and results of the content they provide on the eXtension public website. This evaluation guide is intended to assist CoP’s in carrying out evaluation of their content on the eXtension public website.

    Evaluation resources

    This guide is not the end all to program evaluation, but rather a set of guidelines for CoP’s to use as they approach evaluating their work. Other information and resources will be included where appropriate throughout the guide. Two particularly good ones are the Extension evaluation websites at the:

    Also, check out the Just in Time Parenting CoP Evaluation Guide

    Another good evaluation resource is Financial Education Evaluation Toolkit

    A good reference for online surveys is:

    • Dillman, D.A. (2007). Mail and internet surveys: The tailored design method. Hoboken, NJ: John Wiley & Sons, Inc.

    If you know of other useful evaluation resources, please let me know or feel free to post them here to share with others.

    I consider this guide to be a work in in progress, in that I will continue to add information and resources and refine it as we work together through the process of conducting eXtension evaluations. I am also available and willing to work with you on any and all aspects of your evaluation.

    Steps to Evaluating CoP Content

    Step 1: Establish an Evaluation Sub-Committee/Contact Person(s)

    While establishing an evaluation sub-committee isn’t required, I think it is a good step because it spreads the work around and it allows CoP members who have an interest in evaluation to focus on this important task. As you’ll see below, to be effective and contribute to the success of your CoP work, as well as eXtension, evaluation is best planned for as you develop your content. However, if you already have content on the public website, you can still conduct effective and useful evaluation. Either way, an evaluation sub-committee or at the least, an evaluation contact person(s), can spend the necessary time to make evaluation a productive part of your CoP experience.

    You can see who the evaluation contacts are for each of the CoP's at: CoP Evaluation Contacts

    Please do join the eXtension evaluation contacts CoP at: people.extension.org

    If you don’t have an eXtension ID, you can get one at this site. After getting in, please click on Browse, join, and create communities and join our CoP. This is important because I will use this as our email list--and you can too, to communicate with all eXtension CoP evaluation contacts.

    Step 2: Review Your Content and Develop an Evaluation Plan

    What do you want to accomplish?

    The definition of evaluation I have used in my Extension work for the past 22 years is:

    A process to bring information to bear on decisions about programs.

    Decisions can be driven by process oriented questions, called formative evaluation. Formative evaluation questions include, Is the program working effectively and as planned? Are people accessing the content information? To what extent is the content appropriate for the target audience?

    Decisions can also be driven by results oriented questions, called results or impact evaluation. Results evaluation questions include, Did the program achieve its intended outcomes? Did the program make a difference/impact? Should we continue doing the program?

    These questions help you focus your evaluation. The CoP, under the direction of the evaluation sub-committee, should discuss and develop a list of these types of questions they want to address through evaluation.

    Will your content achieve your intended accomplishments?

    OK, now before you go too far, you need to take a look at your content and make sure that it will do the things you want it to do. This is especially important for questions and expected accomplishments associated with results evaluation. So for example, if you want people who use your eXtension content to make a change in their behavior, will they be able to do this after interacting with your information and educational materials on the public website? This will be especially challenging for eXtension because you will have no direct control over when and how people interact with your content. The key here is to know and incorporate appropriate web-based instructional design principles into your content, depending on your expected accomplishments.

    Experience to date with eXtension has shown that individuals who access the public site, find it via searches, spend a short time on the site, and then exit. Given this, connecting with site users for evaluation is a tremendous challenge. Consequently, I have been encouraging CoP's to develop ways to engage their users in a more educational way so that they will have a greater likelihood of achieving and measuring an outcome (knowledge gain/behavior change). Examples include webinars, chats, and use of Web 2.0 applications like Facebook, Myspace, blogs, twitter, etc.

    A good way to check your content against your expectations is to develop a model of your program, using a logic model. A logic model is a graphic representation of the key elements in your content that let you see and check how they are related, from the situation you are addressing, to the content you are posting, and finally to the accomplishments or results you expect. Two resources for logic modeling are the:

    Develop your evaluation plan

    Once you have articulated your questions and have determined that your content can be reasonably expected to achieve them, then you need to develop your evaluation plan. Your evaluation plan should include at a minimum:

    • Your evaluation questions.
    • The specific measures, or indicators, you want to use to provide the data to answer your questions.
    • When and from whom you will collect evaluation data (evaluation design).
    • How you will collect the evaluation data (see Step 3 below).
    • How you will report the evaluation data (see Step 4 below).

    Measures/indicators. Developing measures or indicators means getting real specific on the data that you will be collecting to provide evidence to answer your evaluation questions. So for example, let’s say that you wanted to know if people interacting with your content learned more about financial security. What specific knowledge are you targeting—credit ratings, APR, specific investments? Or let’s say that you wanted to know if people interacting with your content adopted a new behavior related to horse management. What specific behavior would you target and expect, given your content and web design? Knowing your target audience and your content are key in selecting the most appropriate measures/indicators for your evaluation needs.

    In addition to developing your own measures, there may be existing evaluation instruments and questions that others have developed that you could use. These are great sources, especially if they have been tested and standardized. The key is to make sure that these measures are appropriate for your subject matter and evaluation purpose.

    Evaluation design. Evaluation design involves articulating when and from whom you will collect evaluation data. In traditional face-to-face Extension programs, this could include collecting data from program participants either after the program only (post only design), or before and after the program (pre and post design). In addition, it might include comparing program participants with non-program participants (comparative design). Evaluation design addresses the issue of the extent to which the program participants' made changes that could be attributed to the Extension program, reasonably ruling out other factors.

    Well, we are limited here for the most part to a post only design because of our delivery mechanism—the web. However, if you are using engagement strategies like webinars and chats, you may be able to collect before and after data.

    Knowledge acquisition or learning can be reasonably documented immediately after an individual reads your content. This can be accomplished using a post only design in a one or two question format to document this in a one time online survey.

    Changing behaviors typically requires a time period between acquiring information (learning) and then putting that learning into action. So these questions would typically be asked after an appropriate amount of time had transpired after users had read your content. To achieve behavior change, your content must be specifically targeted and appropriately developed on eXtension for this level of outcome. This can also be accomplished using a one question or two question format to document behavior change in a one time online survey.

    As you develop your evaluation plan, don’t be surprised if you need to go back and review and adjust your questions and perhaps even your content. The key is to make sure that all these elements are in sync. I would be happy to help you develop your plan. Once you have developed your evaluation plan, please share it with me.

    Step 3: Collect Data

    There are currently two ways to collect data to assist CoP's in their evaluation efforts: Web Analytics and Online Surveys.

    Web Analytics

    eXtension is using Google Analytics (GA) to capture user interactions with the public website. Some metrics we are reporting on a regular basis for the public website overall, and in our state and institutional reports are:

    • Number of page views
    • Number of unique visitors
    • Number of new and return visitors
    • Number of pages viewed per visit
    • Number of visits by state and counties/cities per state

    CoPs can also access GA data for their content.  if you are interested in looking at GA for your CoP, please click here for access information.

    There are several informative videos on GA that we have done.  So do check them out.

    Check out our Overall eXtension Metrics Report and eXtension State and Institutional Reports to see the information we report on a regular basis.

    Online Surveys

    Online surveys will be one of the ways that you can collect both formative and results evaluation data directly from users of the website to determine their reactions, use, knowledge, skill, and behavior changes as a result of interacting with your content on the public website. The product we are using is InstantSurvey, a web based survey application. In InstantSurvey, you have access to an eXtension CoP Online Survey Template that has a set of questions that can be used to collect information on:

    • How users discovered the website
    • How often they access the website
    • Usefulness of the information
    • Other sources of information used
    • How they rate the ease of use
    • Self-reported knowledge changes in a one or two-question format
    • Self-reported behavior changes in a one or two-question format
    • Demographics, including state and county pull-down responses

    CoP’s can modify these questions and add others to meet their particular needs.

    These online surveys will likely be administered in an after/post program evaluation design (i.e., the user would receive the online survey after they had come in contact with the CoP’s content). The self-reported knowledge and behavior change questions in the CoP Online Survey Template were prepared for this evaluation design.

    In addition, CoP’s can share their survey work in InstantSurvey with other CoP’s, thus avoiding duplication of effort.

    To use InstantSurvey, I must establish a user account for any CoP members who wish to use this application, so please send me the names and email addresses of CoP members who will be working in InstantSurvey.

    To begin using InstantSurvey, review the Getting Started Guide at:

    InstantSurvey also provides an excellent comprehensive User Manual. You can access the User Manual at:

    You may want to save this manual to your PC. It is also availaible in the InstantSurvey application.

    We have also conducted three 30 minute professional development sessions on working in InstatSurvey that you might like to view to get started.

    I am also available to consult with you on InstantSurvey in your evaluation work. So don't hesitate to get in touch with me.

    Pilot testing surveys. It is always a good idea to pilot test a survey before it is administered. Pilot testing involves administering the survey to a small number of users, or people similar to users, before it is finally released and getting feedback on the survey. This increases the validity of the survey, i.e., getting the correct information that you want. Some questions that you might ask pilot test respondents after completing the survey include:

    • How long it took them to take the survey.
    • Overall, what did they think of the survey.
    • Would they take the survey if they received it by email.
    • Were the directions clear.
    • Were the questions understandable.
    • Were the questions appropriate given the purpose of the survey.
    • Are there other questions on the topic that should be included.

    CoP's can engage users to participate in online surveys in two ways

    • Imbedded URL in content. A URL to InstantSurvey can be imbedded in a particular page of the CoP's content inviting users to provide evaluative feedback. In this way the CoP can target specific content areas/pages for evaluation purposes. Please see IRB Protocol below for additional information.
    • Surveys to engaged users. CoP’s can send surveys to engaged audiences (webinars, chats, etc.) both before and after these educational events. Please see below for IRB Protocol below for additional information.

    Institutional Review Board (IRB) Protocol

    Any online survey work we do with users of the public website, must be vetted through an IRB process to protect our users. There are three categories of users we must consider with respect to IRB in our online evaluation survey work: 1) adults 18 years old and older (19 in Nebraska and Alabama), 2) youth 14-17 years old (18 in Nebraska and Alabama), and 3) children 13 years old or younger. Following are the IRB requirements for these three groups.

    • Adults 18 years old and older (19 in Nebraska and Alabama)

    As you may know, we initially secured blanket Institutional Review Board (IRB) approval from the University of Nebraska at Lincoln (UNL) for survey work done for CoP evaluations. The approval was for surveys that were similar to the CoP Survey Template in InstantSurvey. Increasingly, CoP’s are developing surveys that are appropriate but different from the CoP Survey Template.

    In discussions with UNL IRB, we feel it would be more appropriate for all CoP surveys to be reviewed by UNL. In that way, we will not take the risk of sending out an inappropriate survey. Once you have completed a draft of your survey, please send me electronically:

    1. How you will identify/recruit possible participants.
    2. A test distribution url for the survey from InstantSurvey.
    3. The name, email address, and phone number of a contact person.

    Please allow a minimum of two weeks for the review.

    Also, in some cases, your institution’s IRB may also want to review the survey. That is fine. You may want to see if your IRB and UNL IRB can establish a collaborative agreement so that you only need to go through one of them. The UNL IRB contact person is:

    Becky Freeman
    UNL IRB
    IRB and Research Compliance Manager
    312 N. 14th St, Ste 209
    Lincoln NE 68588-0408
    Email: bfreeman2@unl.edu
    402-472-8127

    Please refer to:

    IRB Number: 20110811872 EX
    Project ID: 11872
    Project Title: eXtension – Building Cooperative Extension's 21st Century Network

    One important element in developing surveys for this age group is the introduction to the survey that must contain the information in the example cover letters provided below.

    The following links with IRB information for this age group are provided for your use

    • Youth 13-17 years old (18 in Nebraska and Alabama)

    Survey evaluation work with this age group needs to be treated carefully. Signed parental consent and youth assent forms may need to be used, depending on the content of the survey. In addition, the introduction to the survey will need to be carefully constructed to be appropriate for this audience (an example introduction/cover letter is provided below).

    According to UNL IRB:

    If the participant is 13-18 years of age, a Youth Assent Form is required. The Youth Assent Form is based on the adult consent form, but should be revised to meet the cognitive and educational level of an average youth. The assent form must contain simple language written at the appropriate educational level of the youngest prospective participant in the youth age range. In some research projects, it may be necessary to utilize two assent forms written to accommodate participants at either end of the age range. The Youth Assent Form must contain all of the required elements of consent previously outlined in the IRB Guidelines except instructions about emergency care and rights of research participants, and should follow the general format of the adult consent form.

    If you plan on collecting evaluation data from this age group, please contact me to discuss.

    • Children 12 years old or younger

    Because of legal precedent, we will need parental consent before we can register these individuals on the public website. We are investigating various protocols for accomplishing this. Should CoP’s have a need to interact with this age group, then they need to provide justification of the value added in working programmatically with this targeted audience.

    According to UNL IRB:

    If the participant is under the age of 7 years, only a Parental/Guardian Consent Form is required. However, verbal assent should be obtained as appropriate.

    If the participant is 7 through 12 years of age, a Child Assent Form is required. The Child Assent Form must be brief, without subheadings, and contain extremely simple language arranged in brief paragraphs. The assent form must contain the following elements: title of the research study; opportunity to ask questions; basis for participant selection; purpose of the study; explanation of procedures; potential risks/discomforts; potential benefits; statement concerning consultation with parents; freedom to withdraw; and confidentiality statement.

    With respect to evaluation, any online surveys with this age group would need to be approved by UNL IRB and perhaps other institutional IRB’s with respect to language appropriateness, as well as whether or not signed parental consent and/or signed children assent forms are needed or can be waived. Please contact me to discuss if you have questions.

    Step 4: Report Results

    Reporting the results of your evaluation efforts to appropriate groups will help tell the story of the use and impact of your CoP content in eXtension. And these results should also inform your CoP as to what is working well and where adjustments need to be made.

    Evaluation results can be shared:

    • With your CoP members.
    • In your CoP Profiles.
    • With members in your institutions (administrators, specialists, and agents/educators).
    • With any other appropriate stakeholders.
    • In Journal articles.
    • In your annual faculty and Extension reports.

    One of the features in InstantSurvey, for your online survey work, is the ability to generate reports by state and county (if you use the state and county questions and pull down responses provided). When your online survey is finished, InstantSurvey can generate a report url that anyone can access and generate a report of frequencies of response to all questions in the survey for a specific state and/or county. While you will need to think through this option, it does provide a way to generate individual state and county level reports.

    As always, don't hesitate to contact me to discuss your reporting needs for your evaluation work.

    Category: 
    Evaluation
    Tags: 
    Evaluation