eXtension Evaluation CoP

Primary tabs

    Contents

    Purpose

    The purpose of this Wiki page is to capture ideas and promote an ongoing dialogue about forming an eXtension Community of Practice (CoP). Please feel free to add your ideas, comments, and questions and to comment on other's ideas.

    How to contribute

    To be able to contribute to this page, you will need to have an eXtension ID. To get one, go to How to get an eXtension IDOnce you have your eXtension ID, it's real simple to add your ideas, comments, and questions. All you have to do is:

    • Access this page.
    • Log in (upper right) using your eXtension ID and return to this page.
    • Click on the blue edit preceeding the Ideas, comments, and questions section.
    • Type in your input. Please identify yourself in bold before making your contribution (see format below). To bold, highlight the text and then click on the bold icon (B) at the top of the page.
    • Scroll down to the bottom of the page and click Save page.

    Voila, you are now a Wiki expert! If you don't feel comfortable in using the Wiki, send me your ideas at mike.lambur@extension.org and I will post them here.

    Ideas, comments, and questions

    From Lisa Townson, New Hampshire

    Seems to be a couple of ideas -- to provide professional development to Extension staff AND to provide a resource repository -- I am very excited about both. I know Ben S. is transitioning his webinar series to be promoted through eXtension, and I just spoke with one of my tech colleagues who indicated there is a need for more "non-tech" professional development offerings through eXtension, so any thing we can provide would be greatly appreciated. It would be nice to continue with Ben's series -- with a goal of one every month on a different program development or evaluation topic. Even basic logic model webinars would be appreciated by some of our newer staff.

    I can't wait to see Ellen's train the trainer manual!


    From Ellen Taylor-Powell, WI

    CoI: Do we already have a CoI or do we need to define that? Is it one CoI - e.g., all CES faculty and staff (assuming that all have to do/engage in evaluation at some level); or do we segment the audience into different groups: e.g., those interested in developing knowledge and skill in evaluation for their own program improvement and accountability needs; those training/mentoring others in evaluation; leaders/supervisors/administrators who need evaluation results; etc...or some other way to cut the pie?

    What is our purpose? What is our goal? What will be better/different as a result of this CoP?

    Possible products: As I've mentioned, we are nearly finished creating a comprehensive resource - a 'train the trainer' type manual - with activities, handouts, powerpoint, etc. - for community-based educators and Extension staff who are teaching/mentoring others in evaluation, particularly outcome measurement. I would think this resource would be useful to others.

    One area I'm interested in/we don't have resources on is: self-assessment tool/process for extension education evaluation. How do members self-assess to assess their level of knowledge/skills; identify areas for work; monitor progress; and continue to self-assess along their journey of gaining evaluation competence?

    I see this CoP as a place for sharing resources, networking, and building capacity within ourselves that extends to evaluation across the Extension system at individual, group and organizational levels (state and national).

    Sorrel Brown, Iowa State, Ag

    It's been a rollicking couple of years as I have learned the ropes of evaluation, with special thanks to the folks at WI and Nancy Ellen Kiernan at Penn State for their help.

    Ellen is on target w/ asking what we think/want this CoP to be. Will it be an 'internal' group that we can use as a resource as we develop tools for our constituents, as Ellen suggests, or will it eventually be a place the public can go to for information on how we evaluate ourselves? Would we want that? What would be the pros/cons of having a public eye on our efforts to evaluate our programs? Would it serve the greater good or just draw fire?

    Heather Boyd, Virginia

    What if we had a few different goals and then prioritized them. For example, goal one would be to gather all the best Extension education eval resources into one place to ease of use by people who use eXtension. Goal two could be the self-assessment that Ellen mentions regarding each of our own evaluation competencies. Just an idea.

    Becky White

    Sorry I missed the recent meeting. I have just listened to the meetng from the link Mike sent out. I think a great product or 'resource' for our COI or COP to eventually produce for our Extension system professionals would be online courses in general evaluation and perhaps particular evaluation topics. Similar in format to the CECP online courses, Extension professionals could sign up, take Evaluation 101 (or maybe 1001 is more appropriate), an interactive, guided educational course complete with assessment at the end. Eventually we might have several advanced courses, and perhaps specialized courses. For example, we could have online courses in Using Focus Groups in Program Evaluation, or Program Evaluations Using Surveys. I had a former colleague who developed a web-based course for our nutrition educators. It was used as a professional development technique for our EFNEP educators and also as a way to assign pay raises for them. It has been a success in our state. I see potential for Extension professionals to significantly enhance their professional skill in the area of program evaluation when the product is structured in this way. Just an idea!

    Notes from the initial organizational meeting, May 6, 2008

    Prepared by Mike Lambur

    To view and hear the recording, go to

    May 6 Meeting Recording

    Present: Heather Boyd (VA), Sorrell Brown (IA), Liza Guion (NC), Michael Lambur (VA), Larry Lippke (TX), Ben Silliman (NC)

    The agenda included:

    • Introductions
    • Purpose of the meeting
    • What is our purpose/goal?
    • Who is/are our target audience(s)?
    • What are our products?
    • Who will commit to work on this effort?
    • What are our next steps?

    Purpose of the meeting

    The purpose of the meeting was to begin an initial discussion of organizing an eXtension Evaluation Community of Practice (CoP). Larry Lippke, also provided some background information on the National Extension Professional Development Community and internally focused CoP’s. See National Extension Professional Development Communityfor more details and background.

    What is our purpose/goal?

    There was much discussion on this agenda item. We need to be more than just a place to archive evaluation information. Here is my best stab at what we eventually came up with as some initial purposes. We need to spend more time on this topic.

    • Enhance the use of best practice evaluation resources.
      • Pull together current resources on evaluation from around the country (and possibly review and endorse them).
      • Identify gaps in current resources and develop materials to fill those gaps.
    • Be a national evaluation resource for Cooperative Extension

    Who is/are our target audience(s)?

    In priority order:

    • 1. Agents/educators (especially in states with no evaluation specialist)
    • 2. Specialists
    • 3. Administrators

    An audience analysis/needs assessment was also suggested. This could be done with evaluation specialists and/or a national survey of agents/educators (perhaps a phase 1 and 2 approach). This might also help us in shaping our purpose and work.

    What are our products?

    Possibilities:

    • Courses/modules (Moodle)
    • Frequently Asked Questions
    • Ask an Expert
    • Links to other resources

    What are our next steps?

    We agreed that this was a valuable first meeting and discussion. We need to continue meeting to discuss further, especially the purpose. We need to involve as many people in these meetings as possible. We will plan on meeting again in late July or August. Mike Lambur will coordinate this meeting.

    Notes from the second organizational meeting, August 12, 2008

    Prepared by Heather Boyd

    To view and hear the recording go to:

    August 12 Meeting Recording

    Present: Michael Lambur, Becky White, Ben Silliman, Heather Boyd, Jay Jayaratne, Larry Lippke, Lisa Guion, Michael Newman, Sorrel Brown, Suzanne LeMenestrel, Ellen Taylor-Powell

    Some of the themes shared in the conversation included:

    • The CoP should go beyond having a web site to house information.
    • Our first audience might be agents and educators, second audience specialists, third audience administrators, perhaps.
    • Evaluating for Impact work from 4-H has been a major effort. CYFERNet is also a great source.
    • AEA EEE-TIG has been a great source of professional development opportunities. And, the Southern Region meeting has been a good source.

    Winter Institute in Florida, Minnesota Summer School, and Wisconsin workshop for administrators and other state/regional initiatives, such as the FCS evaluation group, the different tracks NAE4-HA offers and CYFAR offers sessions. There had been more leadership in evaluation from the feds when they had their program development unit.

    A CoP has the potential for helping evaluation specialists improve helping people learn and practice evaluation, especially through a web site. Agreement that this CoP would be a national resource for Cooperative Extension, a national vehicle for professional development and resources around evaluation.

    Learning (resources), encouraging (resources and FAQs), doing (group projects), innovating (developing approaches unique to our Extension niches – a LEDI Academy of sorts.

    Next steps and volunteers:

    • Examine successful communities of practice – Michael Lambur
    • Catalog and review what resources and professional development opportunities are available – Suzanne LeMenestrel will send the cataloging done for *Evaluating for Impact – Sorrell Brown and Michael Newman will work together on summary of what is available
    • One page summary of LEDI – Ben Silliman will use the wiki
    • Lead or co-lead or support leadership – Heather Boyd and Ben Silliman
    • Share current draft of CoP application—Larry Lippke
    • Talking with Michael and Larry about the CoP application – Heather Boyd
    • Brief summary of meeting – Heather Boyd
    • Post audio – Michael Lambur
    • Doodle dates for next meeting – Heather Boyd
    • Added CoP progress as an agenda item for EEE-TIG business meeting – Heather Boyd

    LEDI: eXtension Evaluation Community of Practice Strategies

    Ben Silliman, Assoc. Professor and Extension Specialist Department of 4-H Youth Development and Family and Consumer Sciences, North Carolina State University August 12, 2008

    The eXtension Evaluation Community of Practice will target four levels of engagement to serve the professional development and programming needs of Extension professionals. This model could be expanded for use with state and county Communities of Practice or with collaborating organizations outside Extension.

    Learning

    • Objective: To foster professional development in evaluation knowledge and practice for Extension staff at all levels of competence (e.g., novice, advanced beginner, practitioner, mentor, expert) and assignment (federal, state, county; administrator, specialist, agent, program assistant, volunteer)
    • Measurable Outcomes: Competency tests, attribution and aspiration surveys with site users
      • Documenting key evaluation standards and strategies in user-friendly terms, including:
        • Best practices (optimal conditions and procedures) and necessary practices (minimal conditions needed for a valid study)
        • Federal reporting guidelines
        • Professional standards (practices and ethics)
        • Linking to web sites providing reliable information and instruction in all phases of program development and evaluation (stand-alone, structured topics, search engine)
        • Self-assessments and recommendations for professional development•
        • Evaluability assessment protocols to guide program evaluation
        • Specific methods and measurement tools (e.g., fact sheets, videos, online chapters)
        • Training curricula like 4-H Evaluating for Impact, SAMMIE, etc.
        • Online graduate programs
        • Facilitating learning opportunities for specialists and agents through
        • Existing professional conferences (e.g., face-to-face connections at AEA and Extension conferences)
        • Web conferences on specialized topics

    Site Recommendations

    NC 4-H Greenlight http://www.nc4h.org/professionals/greenlight/programming-knowledge-base.html

    Southern Region Program and Staff Development http://www.ca.uky.edu/agpsd/soregion.htm

    CYFERNet Evaluation http://cyfernet.ces.ncsu.edu/cyfres/browse_3.php?cat_id=608&category_name=National+CYFAR+Evaluation+Studies&search=Evaluation&subcat=Evaluating+Technology&search_type=browse

    SAMMIE http://www.sammie.osu.edu/

    Cornell Extension Evaluation:

    -William Trochim’s Social Research Methods http://www.socialresearchmethods.net/

    -Michael Duttweiler’s Program & Professional Dev. http://www.people.cornell.edu/pages/mwd1/

    Encouraging

    • Objective: To foster professional development in evaluation knowledge, practice, self-efficacy, and initiative for Extension staff at all levels of competence and assignment
    • Measurable Outcomes: Self-efficacy and social support assessments, tracking of cognitive and emotional components of interactive tools
      • Fostering dialogue (blogs) and iterative learning (wikis) about evaluation among Extension professionals…and possibly among 4-H youth leaders
      • Tracking continuing needs and capabilities via online surveys and quizzes
      • Recognizing exemplary efforts and providing insights on less-than-exemplary projects as an iterative learning tool (e.g., agents and specialists describe their efforts and an expert group selects, comments, and posts some as samples)

    Doing

    • Objective: To foster professional development in evaluation knowledge, practice, self-efficacy, and initiative for Extension staff at all levels of competence and assignment
    • Measurable Outcomes: Authentic assessment of program evaluation quality and outcomes, competency tests, attribution and aspiration surveys with site users, self-efficacy and social support assessments
      • Initiating multi-state evaluation projects through collaborating with existing program-based groups and organizing new groups to investigate specific program or outcome issues

    Innovating

    • Objective: To foster professional development in evaluation knowledge, practice, self-efficacy, and initiative for Extension staff at all levels of competence and assignment
    • Measurable Outcomes: Authentic assessment of innovation impact on individual competence and program evaluation quality
      • Developing new tools and methods for evaluation of Extension programs and outcomes
      • Developing new learning opportunities for Extension staff, volunteers, and clients at all levels

    Notes from the third organizational meeting, September 30, 2008

    Present: Melissa Cater, Michael Lambur, Suzanne LeMenestral, Larry Lippke, Allison Nichols, Ben Silliman, Becky White Assistance from Dustin Hiatt

    To view and hear the recording go to:

    September 30 Meeting Recording

    Review topics addressed in previous conference call

    • Examine successful communities of practice – Michael Lambur
      • Michael--CoPs to complete profiles every six months; He will post CoP goals and report summaries; Keys include communication, sharing work, face-to-face when possible, tech assistance invaluable
      • Allison--Process evaluation with Family CoP indicates how role with eXtension evolved and provided benefit to participants (JOE article); Intentional recruiting/survey key in recruiting motivated participants; “Workers” only a small percentage of participants; Making time in schedule and staying on task key; “Cheat Sheets” for wiki also helpful
    • Catalog and review what resources and professional development opportunities are available (Michael Newman not attending)
      • Suzanne—summarized Evaluating for Impact project; asked about sub-groups within
      • Michael—initial group goals of Eval CoP were to 1) enhance use of best practice resources (review) and gaps; 2) evaluation resource to agents, specialists, administrators (possible needs assessments, Moodle, ask the expert widget; FAQ)
      • Larry—Moodle provides a good way to organize new or existing materials
      • Allison—remember that there is a review process for CoP materials
      • Becky—Can we request input for this at AEA?

    Review LEDI document and target initial objectives

    • Michael--Organize resources that are out there into one place is primary; adding value via Moodle and FAQ
    • Larry—developing resources based on FAQ
    • Allison—Learning activities important beyond facts; role of academic credit for contributions (work group now at eXtension)
    • Becky—reference possible products noted at May 6 meetings; FAQs are often relevant points-of-departure; face-to-face
    • Allison—ditto; even brainstorming topics
    • Becky—4-H evaluation for impact provides
    • Larry—FAQ ready-to-go—identify tags
    • Becky—NEI categories could serve as tags
    • Allison—specialized topics could also complement
    • Group—focus on best ways to organize initiatives;
      • Allison—Analyzing and Interpreting Data
      • Ben—Methods and Tools
      • Becky—Program Planning
      • Melissa—Collecting and Handling Data
    • Becky—Similar group identified key questions and provided answers as a resource
    • Melissa—Q on paragraph
    • Allison—paragraph description should follow NEI categories, focus on details, application questions, gaps not addressed by
    • Needed areas—statistical analysis
    • Becky—Does everyone know how to use eXtension via asking a question and providing a correct answer; Guide from “Better Kid Care America” may help process

    Review Evaluation Community of Practice application and target completion

    • Larry—application outline may guide draft proposal for the group
      • Gain Moodle space
      • Recognition for activity, accomplishments; link to wiki
      • Multi-institutional involvement; leadership and evidence for sustainability
      • Is an Evaluation CoP a possible project/framework of the AEA-TIG? (possibly as a motion at Business Meeting)
      • Michael and Heather will examine application process

    Keys to Effective CoP's

    From Michael Lambur assignment. More to come...

    These items were garnered from CoP Profiles posted on the CoP Wiki in 2007-08.

    • Regular meetings via Connect keep people informed and on track.
    • Develop short and long term goals.
    • Create sub-groups to create content.
    • Create marketing materials.
    • Develop external partnerships with other groups and organizations.
    • Relationship building at first is important.
    • The CoP needs a task master to keep things on track. This person needs to do a lot of leading, shepherding, and direct leadership.
    • Break down large tasks into smaller ones for people to do.
    • Specific training is needed on working in the Wiki.
    • Start with Frequently Asked Questions (FAQ’s).
    • Think beyond the way things have always been done and how to engage clientele in a different way.
    • Much progress is made in face to face meetings. It helps to maintain camaraderie.
    • CoP members with strong Wiki and technology skills can serve as coaches for other CoP members.
    • Working in this virtual environment has strengthened the capacity of CoP members to focus their work in a more structured, time efficient manner.

    CoP Application

    DRAFT of Letter for COMMUNITY OF PRACTICE APPLICATION FORM for Feb. 4, 2009 discussion


    Dr. Dan Cotton, Director eXtension University of Nebraska - Lincoln 6 Ag Communication Building PO Box 830918 Lincoln, NE 68583-0918

    Dear Dr. Cotton:

    Please review the attached proposal to establish an eXtension Evaluation Community of Practice (CoP). A leadership group has been working with Dr. Michael Lambur in your office since May 2008 to develop a plan of work and partnerships with educational organizations and professional groups. The attached document describes the foundations of our work in professional standards and training resources. The Evaluation CoP would collect and extend these resources through a web-based clearinghouse and teaching site, educational webcasts, training and research reports at professional meetings, and collaborative projects designed to apply existing evaluation tools and develop new approaches to evaluating Extension programming.

    Several core leaders in this emerging Community of Practice have established links to collaborators in university and community settings who are interested in joining this effort. Such partnerships would significantly enhance the assets and impact of the CoP beyond the Extension community, which is already highly motivated to participate. Finally, core leaders are currently evaluating the effectiveness of a 4-H framework for teaching evaluation through learning circles, funded by National 4-H Council. Such collaborative efforts to improve evaluation resources and training might be anticipated as the membership and opportunities of the CoP grow.

    We look forward to your committee’s review and comments, eventually resulting in formal recognition of our group and support for the important work of documenting Extension and outreach effectiveness. Thank you for your assistance in this regard.

    Sincerely,


    DRAFT for COMMUNITY OF PRACTICE APPLICATION FORM for Feb. 4, 2009 discussion


    B. Silliman 1/20/08 Application to Become an eXtension Professional Development Community of Practice: Evaluation

    Section 1. Program Overview Professional Development Communities of Practice (PDCoP) may apply to address subjects considered to be core competencies for Cooperative Extension System faculty and staff. The expectation is that PDCoPs formed in this manner will meet guidelines and criteria established by the Professional Development Steering Committee, receive the same technical support provided by eXtension as external CoPs, and interact with and serve Communities of Interest within CES.

    Applicant PDCoPs will sign agreements/contracts and follow a scope of work and timeline with technical support of the eXtension staff and eXtension system to become a recognized PDCoP. Once agreement is reached, PDCoPs formed by application will be integrated into the staff support process along with other developing PDCoPs.

    --Evaluation Community of Practice Description-- Section 2. Application Process 2.1. Application

    Each Professional Development Community of Practice application should provide the following information in electronic format (pdf preferred):

    • Cover letter directed to Dan Cotton, Director of eXtension with signatures of the lead contact and his or her Director of Extension. (attached)

    • PDCoP lead contact information including name, title, eXtension ID, address, e-mail, telephone numbers, and other contact details. The PDCoP lead contact must be a land grant faculty member or professional staff with an Extension appointment whose institution is in good standing with eXtension (see Section 4.1 below).

    Lead Contacts: Benjamin Silliman, Department of 4-H Youth Development and Family and Consumer Sciences, P.O. Box 7606, North Carolina State University, Raleigh, NC 27695 ben_silliman@ncsu.edu; PH: (919) 515-8485; FAX: (919) 515-7812; website: nc4-heval.wikispaces.com

    Heather Boyd, 111 Hutcheson, Virginia Cooperative Extension, Blacksburg, VA 24061 Email: hboyd@vt.edu Phone: (540) 231-9423

    • List of PDCoP Core Leadership Members, titles, institutions or business affiliation and e-mail addresses and a description of how the leadership team was formed


    Core Leadership Members

    Suzanne LeMenestral, USDA-CSREES 1400 Independence Avenue SW, Stop 2201, Washington, DC 20250-2201 Email: slemenestrel@csrees.usda.gov; Phone: (202) 720 – 2297; Fax: (202) 720 - 9366

    Mary Arnold, 105 Ballard Hall, Oregon State University, Corvallis, OR 97331 E-mail: mary.arnold@oregonstate.edu ; PH: 541-737-1315

    Michael Newman, College of Agriculture and Life Sciences, Box 9745, MSU, Mississippi State, MS 39762; Email: mnewman@ext.msstate.edu; Phone: 662-325-3462; Fax: 662-325-5858

    Joseph Donaldson, UT Extension Evaluation and Staff Development, 2621 Morgan Circle, 212-D Morgan Hall, Knoxville, Tennessee 37996-4525, E-mail: jldonaldson@utk.edu Phone: 865-974-7245; Fax: 865-974-0882

    Allison Nichols, West Virginia University Extension, 603 Knapp Hall, Morgantown, WV 26506-5031 E-mail: ahnichols@mail,wvu.edu PH: (304) 293-2796

    Sorrell Brown, 101 Curtiss, College of Agriculture and Life Sciences, Iowa State University, Ames, IA 50011Email: sorrel@iastate.edu; URL: http://www.ageds.iastate.edu/undergrad.htm; Phone: 515-294-8802

    Melissa Cater, LSU Extension, Courthouse Building, Room 302, Harrisonburg, LA 71340-0160 E-mail: mcater@agcenter.lsu.edu PH: (318) 744-5442; Fax: (318) 744-0430

    Rebecca White, Family & Child Development, LSU Extension, 221 Human Ecology, LSU, Baton Rouge, LA 70803 E-mail: bwhite@agcenter.lsu.edu Phone Number: (225) 578-3921 Fax Number: (225) 578-2697

    • Description of the Professional Development Community of Practice, including plans to increase and sustain the CoP membership. Include your PDCoP leadership and management plan, organizing events or activities, and methods to achieve the PDCoP's continued productivity.

    Core leadership in the eXtension Evaluation Community of Practice includes 10-15 specialists and agents from all Extension disciplines and geographic regions. Many of these leaders have primary responsibility for evaluation and active membership in the American Evaluation Association (AEA) Extension Education Evaluation Topical Interest Group. This group, whose current membership is over 50 Extension professionals, has a long history of promoting professional development, organizational best practices, and recognition of excellence in evaluation. In addition, most of these leaders are actively engaged in discipline-based (4-H, Agriculture, Family and Consumer Sciences) organizations committed to improving programming and professional development in evaluation. The eXtension Evaluation CoP will foster synergy and innovation among these groups and will build capacity across the Extension system as well as partnerships with other organizations and constituencies.

    Plans to increase and sustain the CoP membership include active engagement of AEA and discipline-based professional organizations, collaboration with Extension administrators groups to expand participation of Extension professionals, and recruitment of colleagues in university and community programs engaged (as leaders and consumers) in program evaluation. At North Carolina State University, for instance, an informal community of practice in evaluation has met to exchange ideas and learn together for the past two years. This group is expanding with support of the Vice Chancellor for Extension, Engagement, and Economic Development, facilitating training, dialogue, and collaboration among university faculty engaged in outreach. Increasing interest in evaluation is also evident among community-based partners of Extension agents. The Evaluation CoP is well-positioned to assist university and community partners as it enhances Extension capacity.

    Leadership and management for the CoP will be grounded in the core leadership team identified above. A leadership team of at least six CoP members will convene annually to establish a Plan of Work, then convene quarterly to review progress of sponsored projects, discuss potential projects, funding, and partnerships in that Plan.

    Events or activities, and methods to achieve continued productivity are specified below. Regular web conference planning meetings led by the core team will facilitate project development, implementation, and reporting; communication and innovation across the membership; and partnership and fund development to support projects and serve audience needs.

    • Description of the Core Competency(ies) being addressed and methods your PDCoP will use to engage the Extension faculty/staff with your educational products and programs.

    Core Competencies of the Evaluation Community of Practice are based on the National Evaluation for Impact Educational Content for Professional Development (Arnold, et al., 2008), providing an educational framework for evaluation skills described in the 4-H Professional and Knowledge Base Competencies (PRKC)(National Professional Development Task Force, 2004):

    1. Program Planning for Effective Evaluations This module introduces tools for understanding and using program development logic models to plan and implement programs. Particular emphasis is placed on identifying short, medium and long term outcomes and their importance in setting the stage for effective evaluation of the program. Participants will also learn to identify other areas of a logic model that can also provide important evaluation information.

    2. Focusing an Evaluation This module covers the different purposes for and types of program evaluation, and helps participants identify various levels of evaluation outcomes. A special emphasis is placed on developing evaluation questions that are linked to the program’s theory or framework. Developing indicators and identifying data sources are also covered. Developing and following an evaluation protocol, timeline, and project management plan is also emphasized.

    3. Evaluation Design This module focuses on developing evaluation questions that appropriate to quantitative and qualitative methods and their relation to outcome indicators and results. A special emphasis will be placed on types of evaluation questions, and matching questions to indicators and outcomes,

    4. Evaluation Methods This module introduces different types of methods typically used in program evaluation. Participants will learn about quantitative and qualitative methods, and their appropriate use. In addition, the relationship between evaluation questions and evaluation methods is covered.

    5. Collecting and Handling Data This module focuses on understanding how to collect and manage quantitative and qualitative data. Content covered includes the ethics and procedures for the collection, storage and processing of data; developing a quantitative data set; data collection methods; developing a data collection methods protocol; and matching data collection methods to evaluation questions.

    6. Analyzing and Interpreting Data This module introduces participants to basic analytic procedures available for both quantitative and qualitative data. Using statistical or qualitative software (such as SPSS or NVivo), participants will learn how to perform descriptive and inferential analyses and will learn how to interpret the results.

    7. Communicating Evaluation Results This module prepares participants to convert evaluation results into forms of communication that are useful to various stakeholders. Participants will learn about the purposes of reporting, the content and types of evaluation products, how to identify stakeholders, and how to present the results that matter most to different stakeholder groups.

    Evaluation Center Standards for Program Evaluation (Ramlow, 2007). Utility Standards The utility standards are intended to ensure that an evaluation will serve the information needs of intended users. U1 Stakeholder Identification--Persons involved in or affected by the evaluation should be identified, so that their needs can be addressed. U2 Evaluator Credibility--The persons conducting the evaluation should be both trustworthy and competent to perform the evaluation, so that the evaluation findings achieve maximum credibility and acceptance. U3 Information Scope and Selection--Information collected should be broadly selected to address pertinent questions about the program and be responsive to the needs and interests of clients and other specified stakeholders. U4 Values Identification--The perspectives, procedures, and rationale used to interpret the findings should be carefully described, so that the bases for value judgments are clear. U5 Report Clarity--Evaluation reports should clearly describe the program being evaluated, including its context, and the purposes, procedures, and findings of the evaluation, so that essential information is provided and easily understood. U6 Report Timeliness and Dissemination--Significant interim findings and evaluation reports should be disseminated to intended users, so that they can be used in a timely fashion. U7 Evaluation Impact--Evaluations should be planned, conducted, and reported in ways that encourage follow-through by stakeholders, so that the likelihood that the evaluation will be used is increased. Feasibility Standards The feasibility standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal. F1 Practical Procedures--The evaluation procedures should be practical, to keep disruption to a minimum while needed information is obtained. F2 Political Viability--The evaluation should be planned and conducted with anticipation of the different positions of various interest groups, so that their cooperation may be obtained, and so that possible attempts by any of these groups to curtail evaluation operations or to bias or misapply the results can be averted or counteracted. F3 Cost Effectiveness--The evaluation should be efficient and produce information of sufficient value, so that the resources expended can be justified. Propriety Standards The propriety standards are intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results. P1 Service Orientation--Evaluations should be designed to assist organizations to address and effectively serve the needs of the full range of targeted participants. P2 Formal Agreements--Obligations of the formal parties to an evaluation (what is to be done, how, by whom, when) should be agreed to in writing, so that these parties are obligated to adhere to all conditions of the agreement or formally to renegotiate it. P3 Rights of Human Subjects--Evaluations should be designed and conducted to respect and protect the rights and welfare of human subjects. P4 Human Interactions--Evaluators should respect human dignity and worth in their interactions with other persons associated with an evaluation, so that participants are not threatened or harmed. P5 Complete and Fair Assessment--The evaluation should be complete and fair in its examination and recording of strengths and weaknesses of the program being evaluated, so that strengths can be built upon and problem areas addressed. P6 Disclosure of Findings--The formal parties to an evaluation should ensure that the full set of evaluation findings along with pertinent limitations are made accessible to the persons affected by the evaluation, and any others with expressed legal rights to receive the results. P7 Conflict of Interest--Conflict of interest should be dealt with openly and honestly, so that it does not compromise the evaluation processes and results. P8 Fiscal Responsibility--The evaluator's allocation and expenditure of resources should reflect sound accountability procedures and otherwise be prudent and ethically responsible, so that expenditu res are accounted for and appropriate. Accuracy Standards The accuracy standards are intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated. A1 Program Documentation--The program being evaluated should be described and documented clearly and accurately, so that the program is clearly identified. A2 Context Analysis--The context in which the program exists should be examined in enough detail, so that its likely influences on the program can be identified. A3 Described Purposes and Procedures--The purposes and procedures of the evaluation should be monitored and described in enough detail, so that they can be identified and assessed. A4 Defensible Information Sources--The sources of information used in a program evaluation should be described in enough detail, so that the adequacy of the information can be assessed. A5 Valid Information--The information gathering procedures should be chosen or developed and then implemented so that they will assure that the interpretation arrived at is valid for the intended use. A6 Reliable Information--The information gathering procedures should be chosen or developed and then implemented so that they will assure that the information obtained is sufficiently reliable for the intended use. A7 Systematic Information--The information collected, processed, and reported in an evaluation should be systematically reviewed and any errors found should be corrected. A8 Analysis of Quantitative Information--Quantitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered. A9 Analysis of Qualitative Information--Qualitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered. A10 Justified Conclusions--The conclusions reached in an evaluation should be explicitly justified, so that stakeholders can assess them. A11 Impartial Reporting--Reporting procedures should guard against distortion caused by personal feelings and biases of any party to the evaluation, so that evaluation reports fairly reflect the evaluation findings. A12 Metaevaluation--The evaluation itself should be formatively and summatively evaluated against these and other pertinent standards, so that its conduct is appropriately guided and, on completion, stakeholders can closely examine its strengths and weaknesses.

    Sources cited: Arnold, A., Calvert, M., Cater, M., Evans, W., LeMenestral, S, Silliman, B., & Walahoski, J. (2008). Evaluating for Impact: Educational Content for Professional Development. and National 4-H Evaluation for Impact Self- Assessment. Washington, DC: National 4-H Learning Priorities Project, Cooperative State Research, Education, and Extension Service, USDA. 

    Online at  www.national4-hheadquarters.gov/library/Indicators_4H_MM.pdf

    National Professional Development Task Force. (2004). New foundations for the 4-H youth development profession. National 4-H Headquarters. Retrieved online January 16, 2009 at http://www.national4-hheadquarters.gov/library/4-Hprkc_study_010605.pdf

    Ramlow, M. E. (2007). Program Evaluation Standards. Retrieved December 27, 2007 from www.eval.org/EvaluationDocuments/progeval.html


    Methods to Engage Clientele with Educational Products and Programs

    The eXtension Evaluation Community of Practice plans to incorporate the following methods into its Plan of Work:

    Web site: Information about the CoP, fact sheets and courses addressing evaluation competencies and standards, announcements of learning opportunities, links to resource sites, project samples, discussion groups, and responses to Frequently Asked Questions

    Webcasts: Monthly national webcasts featuring expert speakers, interactive discussion, and live samples of successful processes and outcomes; quarterly web conferences to set goals and facilitate CoP business.

    Professional conference presentations: Featured presentations on successful processes and outcomes, development of learning communities, and professional development issues at conferences of agents in Agricultural Extension; 4-H; Communities, Families, and Youth-at-Risk; Community Development; and Family and Consumer Sciences, of the American Evaluation Association and other organizations invested in evaluation.

    Pilot evaluation projects: CoP members will seek funding and engage in collaborative projects to enhance training, resources, program evaluation practice, and scholarship. One such project is already underway examining the effectiveness of learning circles using the 4-H National Evaluation for Impact framework with funding from National 4-H Council.

    • Scope of Work for the proposed PDCoP through December 31 of the following year. (e.g., if your application is submitted in August 2008, the Scope of Work should be through December 31, 2009). This Scope of Work should include planned or anticipated deliverables for this time period including, but not limited to: o Number of Moodle courses created: 3

    o Number of published FAQs: 30 2-3 topics per core leader

    o Content outline: 90 4-H Evaluation for Impact Content ACA Program Evaluation Standards

    Number of basic information pages: 8 (overview plus one page for each Evaluation for Impact area)

    • Brief description of planned or anticipated deliverables for years 2 and 3. 2-3 Moodle training conferences 100 Extension bulletins and FAQ sheets

    • Anticipated date of release of content and/or enrollment by Extension staff in Moodle courses. To be determined, but 1 course per year is anticipated, attracting at least 300 staff

    • Resources currently available to this proposed PDCoP Members are engaged in evaluating learning communities using the 4-H Evaluation for Impact framework, funded by $10,000 remaining in a grant from National 4-H Council

    • Potential partnerships or other collaborations American Evaluation Association, Extension Education Evaluation Topical Interest Group NAE4-HA Evaluation and Research Committee USDA

    • Resources or assistance required of eXtension to assure success. As described below. 2.2 Submission

    The application should be submitted to: Dr. Lawrence A. Lippke Coordinator, Internal Communities of Practice 16350 Woodlake Drive College Station, TX 77845-8264 Email: larry.lippke@extension.org Phone: 979.324.7768

    2.3 Submission Timelines

    Applications will be accepted at any time. Reviews of applications by the Professional Development Steering Committee will occur within one month of submission.

    Section 3. General Program

    3.1. Professional Development Community of Practice A Professional Development Community of Practice (PDCoP) is defined as a virtual network of subject-matter content providers consisting of faculty, professional and para-professional staff, county educators, industry experts, and government agency representatives who share knowledge or competence in a specific content area and who are willing to work and learn together over a period of time to further develop and share their knowledge in forms of educational products and programs and electronic interactions with nationwide Extension faculty and staff . PDCoP leadership, however, must be land-grant faculty or professional staff with Extension appointments whose institutions are in good standing with eXtension. Specific functions of a PDCoP include:

    • Interactively helping meet the knowledge needs of Extension faculty and staff. • Stewardship of the available knowledge for the core competency it is addressing. • Best practice development of educational products and programs. • Innovation in content-area knowledge and delivery.

    PDCoPs continually interact with their Communities of Interest. This feedback mechanism allows for the constant improvement of product offerings by the PDCoP to satisfy CoI needs over time. PDCoPs must be open to the continual expansion of their membership, as they will have members who move in and out depending on their interest and time availability. Therefore, membership in a PDCoP is fluid and flexible.

    3.2. Guiding Principles of Communities of Practice CoP membership must subscribe to the eXtension Mission, Vision, Values and Guiding Principles.

    3.3. Support provided by eXtension eXtension will provide a support package including but not limited to:

    • Moodle space, • virtual collaborative work space or environments, • technological infrastructure to facilitate the work of the PDCoP, • team members’ technical competency improvement • communications, marketing, business, and development resources, • training for entry and maintenance of content, • content design/standards for production, • online evaluation/assessment tools, • general and overall marketing strategies for PDCoPs, • assessment mechanisms to define audiences, • CoP leadership professional development, • Periodic Web conferencing for the PDCoP.

    Section 3. General Application and Notification Process 3.1. Eligibility Teams composed of subject-matter content providers consisting of faculty, professional and para-professional staff, county educators, industry experts, clientele, and government agency representatives are eligible to submit applications to create a PDCoP. The lead author(s) in the application process must be an employee of their Cooperative Extension System. The lead institution(s) must have paid the current year eXtension assessment. The applications must be approved by the Extension Director(s) or Administrator(s) whose institution(s) are assuming lead roles in the application process.

    3.2. Acknowledgment of Applications Receipt of all applications will be acknowledged by e-mail once the application is submitted.

    3.3. Application Review Process In general, the purpose of the review process is to increase the likelihood that the PDCoP is successful, and that it creates professional development products and programs that are recognized nationwide for their quality, relevance, and usefulness. Focus during the review will be on improving the plans and operations of the PDCoP. In that regard:

    • The PDCoP application review will be managed by the Coordinator of Internal Communities of Practice, and conducted by the Professional Development Steering Committee. • At least three steering committee members will review each PDCoP application. • All comments will be aggregated by the manager and shared with the PDCoP applicants. If deemed necessary to expedite communications, the manager may initiate an audio call between the review panel members and the PDCoP applicants to discuss applications. • All applications will be reviewed within a month of submission.

    3.4 Conflicts of Interest and Confidentiality During the peer review evaluation process, care will be taken to prevent any actual or perceived conflicts of interest that may impact the review. For purposes of conflict of interest, reviewers who reside at a primary Principal Contact’s home institution shall abstain from evaluating an application from that institution. In addition, the identities of peer reviewers will remain confidential throughout the entire review process.


    3.5. Notification The Coordinator of Internal Communities of Practice will notify the lead contact of successful applications, and will assist in providing the lead contact access to the eXtension tools necessary to support the plans of the PDCoP.


    Section 4. General Information on Community of Practice Continuation

    The philosophy underlying this Cooperative Extension System initiative is that PDCoPs will continue to provide curriculum leadership into the future as long as needs of Extension faculty/staff are being met. The ultimate success of all PDCoPs is the positive demonstrable impact on these faculty/staff. Toward that end, successful applicants are asked to provide an annual progress report and six month updates describing current CoP status, membership, activities, progress, transformation processes observed, and achievements of their CoP. The progress report should be submitted by the timeline indicated. Evaluation of the progress report is a key factor in determining eligibility for subsequent year eXtension support.


    ADDITIONS FROM SORREL BROWN

    Core leadership in the eXtension Evaluation Community of Practice The eXtension Evaluation CoP will foster synergy, sharing and innovation among these groups and will build evaluation capacity across the Extension system as well as partnerships with other organizations and constituencies.

    Leadership and management for the CoP will be grounded in the core leadership team identified above. A leadership team of at least six CoP members will convene (should we be clear convening will be electronic in consideration of budgets?) annually to establish a Plan of Work, then convene quarterly to review progress of sponsored projects, discuss potential projects, funding, and partnerships in that Plan.

    Tags: 
    Evaluation