Updated Fall 2009!
Resource Guide

Programs strengthen quality through an ongoing and integrated process of targeted staff training, coaching and monitoring, and data collection and analyses.

"Continuous Program Improvement" is the glue that holds all of the other guiding principles together. Programs that are continually striving to strengthen quality need to engage in three key practices: 1) continuous and targeted staff training, 2) monitoring and coaching to support implementation on the ground and 3) data collection and analysis of program strengths and weaknesses.

Staff training must be an ongoing process that is built into an organization's culture. While many organizations offer introductory training to new staff or "refresher training" at the beginning of the program, providing training throughout the program cycle allows staff to build upon their prior knowledge and develop further competencies. Training is not effective unless there is some form of monitoring to see if it is being implemented effectively.

Monitoring and coaching fill a gap that sometimes exists between training and program improvement. Monitoring includes conducting program observations of staff "in action" and documenting the findings in a way that allows the information to be shared quickly with staff for "real time" program improvement.

Program directors and staff must review data from program observations, attendance records, surveys or other sources for trends in program strengths and weaknesses. When used well, data collection can increase the effectiveness of direct-service programs. After the program has been up and running long enough, programs can begin to collect and analyze outcome measures that can help staff think about the program in terms of the benefits of participation to the clients instead of program activities, and help an agency understand which practices are more effective than others. Agencies that choose to collect outcome measures must limit the number of outcome measures collected and only survey participants if there are resources to collect and analyze the data (several resources are provided in the list below for agencies interested in surveying youth about outcome measures common to youth development programs).

An organization committed to quality will continually assess its programming in order to improve. Several tools, ranging from self-assessment tools that can be done by program staff to instruments for external observers, have been developed to help agencies accomplish this task. An overview of some of the assessment tools that may be of most use to after-school programs serving preteens is provided at the end of this section.

• Staff are supported through ongoing staff development and training.
• Staff are able to translate what they have learned in workshops into practice.
• Program observations are conducted and the information they yield informs staff training and program improvement.
• Staff development and training is clearly connected to and builds on the program’s vision, purpose and goals.
• Staff retention is high.
• Program goals and objectives are measurable and meaningful.
• Youth outcome measurement data is collected regularly and used to influence program improvements.
• Staff evaluations are regularly conducted with staff and used to guide staff development and program improvement decisions.
• Organization regularly assesses program quality through self-assessment or outside evaluation.

The after-school field has begun to define quality through generally agreed upon guiding principles, including the ones in this guide. Assessment tools associate those standards with specific indicators. There are a multitude of different assessment tools because there are many different types of programs, the nomenclature for the standards vary, organizations are looking to use tools in different ways and the tools have been developed simultaneously by different groups.

Although the number of assessment tools can be overwhelming, a program can benefit from the variety by choosing an assessment tool that works well with the needs of the organization. While none of the assessment tools speaks specifically to the six principles in this guide, most tools will cover all of the principles and allow an organization to begin the continuous program improvement process.

This section provides a link to a comprehensive review of assessment tools and a short description of some of the tools likely to be of the most use to after-school programs serving preteens, including the Assessment of Afterschool Practices Tool (APT), the Program Quality Self-Assessment Tool (QSA), the Quality Assurance System (QAS) and the Youth Program Quality Assessment (YPQA).

Nicole Yohalem and Alicia Wilson-Ahlstrom put together a comprehensive guide, Measuring Youth Program Quality: A Guide to Assessment Tools, which reviews nine assessment tools. They chose only tools that: 1) include setting-level observational measures of quality, 2) are applicable in a range of school- and community-based program settings, 3) include a focus on social processes within programs and 4) are research-based.
http://www.forumfyi.org/node/297

This guide compares the purpose, structure, content and technical properties of the following youth program quality assessment tools:

1. Assessing Afterschool Program Practices Tool (APT) developed by the National Institute on Out-of-School Time
2. Out-of-School Time Observation Tool (OST) developed by Policy Studies Associates, Inc.
3. Program Observation Tool (POT) developed by the National AfterSchool Association.
4. Program Quality Observation (PQO) developed by Deborah Lowe Vandell and Kim Pierce.
5. Program Quality Self-Assessment Tool (QSA) developed by New York State Afterschool Network.
6. Promising Practices Rating Scale (PPRS) developed by the Wisconsin Center for Education Research & Policy Studies Associates, Inc.
7. Quality Assurance System (QAS) developed by Foundations Inc.
8. School-Age Care Environment Rating Scale (SACERS) developed by Frank Porter Graham Child Development Institute & Concordia University, Montreal.
9. Youth Program Quality Assessment (YPQA) developed by High/Scope Educational Research Foundation.

Following is a brief description of four of the nine assessment tools reviewed in the Yohalem and Wilson-Ahlstrom guide. These four are highlighted because they do not rely on outside evaluators, they can be conducted by program staff and they are applicable for the preteen age-group. A fifth assessment tool that is still in draft form is also highlighted as a reference because of its relevance for California after-school programs.

Neither the Foundation nor P/PV is endorsing any of these tools. The following information is intended only as a resource to agencies that are seeking a tool.

1. Assessing Afterschool Program Practices Tool (APT)
   The APT was developed by the National Institute on Out-of-School Time (NIOST) and the Massachusetts Department of Education to assess 21st Century Community Learning Center Programs. The tool is designed for well-established programs that serve a stable group of youth (grades K-8) over an extended period of time.
   
   The APT measures positive program climate, supportive relationships, practices that support individualized needs and interests, and practices that stimulate engagement and skill-building. The tool helps programs identify strengths and weaknesses and guide the development of staff development and improvement goals. There are two tools within the APT: an observational tool that focuses on observable program practices and a questionnaire that examines those aspects of program quality that occur "behind the scenes." The instrument is designed for users to make observations in one full program session (afternoon) and is flexible in terms of administration, use of scales, number of observations, etc. NIOST recommends administering the tool two or three times throughout the program.
   
   While the tool has been primarily used by the Massachusetts Department of Education, the developers have recently made it more widely available. The tools themselves are free and downloadable once a program has completed necessary training. Training is available through an online tutorial or in person. For more information, visit www.niost.org/training/APASbrochureforweb.pdf or call 781-283-2546.
   
   
2. Program Quality Self-Assessment Tool (QSA)
   The Program Quality Self-Assessment Tool (QSA) was developed by the New York State After-school Network (NYSAN) with the 21st Century Community Learning Centers in mind. It was designed for school- and community-based programs offering a broad range of services (serving K-12) and not those with a single activity. After the instrument was completed in 2005, New York State began requiring that all 21st CCLC-funded programs use it twice a year for self-assessment purposes. While the instrument can be downloaded from the Internet, it is intended to be printed and used on paper.
   
   It was designed for self-assessment purposes only, not for external assessment or evaluation. It can be used as a planning tool or for ongoing program improvement. The tool is organized into 10 essential elements of effective after-school programs: 1) Environment/Climate, 2) Administration/Organization, 3) Relationships, 4) Staffing/Professional Development, 5) Programming/Activities, 6) Linkages Between Day and After-School, 7) Youth Participation/Engagement, 8) Parent/Family/Community Partnerships, 9) Program Sustainability/Growth and 10) Measuring Outcomes/Evaluation. Each element has a list of standards of practice that provide guidelines for ensuring quality outcomes. It is intended that program staff rate each indicator based on their observations of the program.
   
   The tool is free and downloadable from: www.nysan.org/section/quality. Programs can call or email NYSAN to for more information on using the QSA Tool and to receive technical assistance: Jennifer Siaca, Coordinator, NYSAN, 646-943-8672, jsiaca@nysan.org.
   
   
3. Quality Assurance System (QAS)
   The Quality Assurance System was developed by Foundations, Inc. to help after-school programs conduct quality assessment and continuous improvement planning. It was designed to be general enough for use in a range of school- and community-based programs serving grades K-12.
   
   It focuses on quality at the site level. Programs using the QAS start with an initial assessment from which observers identify areas in need of improvement and develop specific improvement strategies. Programs then do follow-up assessments. Foundations, Inc. recommends administering the QAS twice a year: an initial assessment at the beginning of the year and a follow-up assessment toward the end of the year. Training observers should take about two to three hours, and the observations would take place in one afternoon.
   
   The QAS can be filled out online. A summary report and charts and graphs are automatically generated from the data. The assessment tool can be tailored to the actual goals and activities of the program. It is based on seven "building blocks" that Foundations, Inc. considers fundamental features that underlie effective after-school programming: 1) program planning and improvement, 2) leadership, 3) facility and program space, 4) health and safety, 5) staffing, 6) family and community connections and 7) social climate. In addition, there are three "program focus building blocks" for users to select from: 1) academics, 2) recreation and 3) youth development. Programs vary and may offer one, two or all three focus areas and are assessed accordingly.
   
   There is a $75 annual site license fee. For more information, contact Foundations, Inc. at qas@foundationsinc.org or go to http://qas.foundationsinc.org/start.asp?st=1.
   
   See an article by the Harvard Family Research Project to find out more about the QAS: www.gse.harvard.edu/hfrp/eval/issue25/pp3.html.
   
   
4. Youth Program Quality Assessment (YPQA)
   The YPQA was developed by the High/Scope Educational Research Foundation. It was designed to evaluate the quality of youth programs and identify staff-development needs for structured programs serving grades 4-12.
   
   The overall purpose of the YPQA is to encourage individuals, programs and systems to focus on the quality of the experiences young people have in programs and the corresponding training needs of staff. The instrument may be used by a trained independent rater such as an outside consultant, researcher, program evaluator or agency administrator. It may also be completed by a team of program staff, such as supervisors, youth workers, teachers, social workers, curriculum directors and parents. It focuses primarily on those features of a program that can be observed and that staff can change. The seven major areas covered include: 1) Engagement, 2) Interaction, 3) Supportive Environment, 4) Safe Environment, 5) Youth-Centered Policies and Practices, 6) Policies and Practices and 7) High Expectations and Access. Information is gathered through observation and interviews and used to score the quality indicators. Item scores are combined to create an overall program quality profile.
   
   A "Starter Pack" can be ordered online for $39.95. The pack includes an administration manual, Form A (which focuses on youth experiences during the school day) and Form B (which assesses the organization's infrastructure). High/Scope has several training options, ranging from one- to three-day trainings, designed to support different uses of the instrument. To order the tool, go to https://secure.highscope.org/ProductCart/pc/home.asp. For more information, see www.highscope.org/index.asp.
   
   A meeting report written by the forum for Youth Investment discusses the YPQA: http://www.forumfyi.org/node/330 Karen Pittman, Executive Director of the Forum for Youth Investment, describes why the YPQA makes such a critical contribution to youth programs nationwide: http://www.forumfyi.org/node/113
   
   
5. California Self-Assessment Tool
   The California Self-Assessment Tool is being developed by the After School Programs Office of the California Department of Education (CDE). It was designed for after-school programs in California as they strive to achieve and maintain quality programs and meet the statutory requirements (program, fiscal, monitoring and evaluation) that are unique to the state.
   
   The California Department of Education envisions that the tool will be used regularly by after-school programs throughout the state to support continuous program improvement. It is an internal self-assessment tool intended for use by the program. The goal is to facilitate self-review and self-analysis to assess the degree to which the program plan is being effectively implemented. CDE recommends that after-school programs use it a minimum of two times per planning/operational program year.
   
   The instrument provides an overview of its purpose, instructions for using the tool and a review page for each of these nine core program content areas, identified through review of current after-school literature and research on program quality: 1) Program Design, 2) Program Environment, 3) Program Leadership/Management, 4) Alignment/Linkages with the Regular Day, 5) Staff Development, 6) Youth Development, 7) Community Partnerships/Collaboration and Family Involvement, 8) Program Finance and 9) Program Accountability Evaluation and Research. The indicators in each content area include research-based "quality indicators" that guide programs toward quality outcomes, as well as indicators that will support programs in meeting statutory, fiscal and audit program requirements. The tool uses a four-point scaled rating system for each indicator. The tool is still in draft form and is free and downloadable from http://www.cde.ca.gov/ls/ba/cp/documents/selfassesstool.doc. Questions should be directed to the After-School Partnership Office, afterschool@cde.ca.gov, 916-319-0923.

If none of the assessment tools detailed above is appropriate for your program, other tools are available. The Harvard Family Research Project has compiled several lists of assessment tools over the last five years. The most recent list can be found in: Little, Pricilla M. Harvard Family Research Project. June 2007. The Quality of School-Age Child Care In After-School Settings. Child Care & Early Education: Research Connections. No 7. http://www.researchconnections.org/SendPdf?resourceId=12576

A more detailed chart, "After-School Program Quality Assessment Breakdown," including the assessment tool, the tool developer, the age targeted, the format and some notes, was created during a Harvard Family Research Project after-school conference in 2005. It has information for most of the tools listed in the more recent article cited above. http://www.gse.harvard.edu/hfrp/content/projects/afterschool/
conference/summit-2005-breakdown.pdf

Promising Practices in Out-of-School Time Professional Development (December 2007)
http://www.sp2.upenn.edu/ostrc/docs/research_publications/
Promising_Practices.pdf
This brief encourages after-school programs to take a more intentional approach to professional development in order to align staff training with program goals. The authors encourage program directors to be thoughtful about professional development goals, incorporate staff feedback into training, use professional development standards from other fields to frame a long-term strategy, and incorporate methods to evaluate the effect of professional development on program quality. Though professional development in the after-school field is often synonymous with workshops, other approaches can make it a richer endeavor, including peer mentoring, small learning communities and observation. The document includes references to other tools that may be useful in developing a professional development strategy.
Author/Publisher: Out-of-School Time Resource Center.

Research Update 3: Highlights From the Out-of-School Time Database (September 2008)
http://www.hfrp.org/out-of-school-time/publications-resources/research-update-3-highlights-from-the-out-of-school-time-database
This brief reviews two main themes that emerged from the 13 articles added to the Harvard Family Research Project Out-of-School Time (OST) Database in September 2008: the use of evaluation to shape program improvements and OST's benefits for older youth. In terms of the first theme, programs are increasingly faced with demands to evaluate their activities, but have found it difficult, as well as time-consuming, to use the results of these efforts to improve programming. The recently added articles, however, include evaluations of three programs that were able to successfully address this challenge. These and other studies found that evaluation can be useful for program improvement because it allows programs to incorporate feedback from stakeholders, seek parents' perspectives and improve the workforce based on the findings.
Author/Publisher: Harris, Erin . Harvard Family Research Project.

Outcome Evaluation: A Guide for Out-of-School Time Practitioners (January 2008)
http://www.childtrends.org/Files//Child_Trends-2008_01_07_OutcomeEvaluation.pdf
Many after-school programs may want to consider undertaking an outcome evaluation to learn whether changes occur for participants and if these changes are associated with a specific feature of the program. Programs may be required by funders to undertake outcome evaluations, but they may also consider such an approach as an indicator of program effectiveness. This document reviews what information might be gained from an outcome evaluation, when such an evaluation should be considered and what the basic steps include. This piece provides a useful introduction to the topic, which will necessarily be followed by extensive follow-up reading before a successful outcome evaluation can be attempted.
Author/Publisher: Allen, Tiffany and Jacinta Bronte-Tinkew. Child Trends.

CDE Quality Self-Assessment Tool. The California After School Program Quality Self Assessment Tool (August 2008)
http://www.afterschoolnetwork.org/qsatool
This tool was developed collaboratively by the California Department of Education and the California Afterschool Network based on research on program quality assessments. The tool can be used by program directors or staff to assess program quality in nine areas: 1) program design and accountability; 2) environment; 3) administration and finance; 4) alignment with the school day; 5) youth development; 6) family involvement; 7) community partnerships; 8) staff development; and 9) provision of diversity and equity. The tool is designed to identify areas for immediate action, stimulate discussion among staff and lead to an action plan for improvement.
Author/Publisher: California Department of Education and California Afterschool Network.

After-School Initiative's Toolkit for Evaluating Positive Youth Development (2004)
http://www.coloradotrust.org/attachments/0000/2849/ASIToolkitJun04.pdf
This toolkit includes evaluation question sets that staff of an after-school program could use to assess youth outcomes. It provides question sets to measure outcomes common to after-school programs promoting youth development. The questions cover 45 youth outcomes in the following eight areas: 1) academic success, 2) arts and recreation, 3) community involvement, 4) cultural competency, 5) life skills, 6) positive life choices, 7) positive core values and 8) sense of self. In addition to questions, the toolkit provides tips on developing and administering surveys.
Author/Publisher: Denver, CO: The Colorado Trust.

Compendium of Assessment and Research Tools (C.A.R.T.) for Measuring Education and Youth Development Outcomes
http://cart.rmcdenver.com
The Compendium of Assessment and Research Tools (C.A.R.T) is an online database that provides information on instruments that measure attributes associated with youth development programs. The database allows a user to locate research tools in specific areas of interest. The instruments are grouped under the following categories: program characteristics, program quality, teachers' role, student engagement, parental/guardian/family support, internal sources of support, external sources of support, school profile, organizational structure, quality of work life, community characteristics, family involvement in school, family characteristics, attention to/awareness of service-learning, national/state/local policies, academic, rationality/connectedness, social role/status, social competence, group interaction/teamwork, school effectiveness, school climate, community involvement, cultural pluralism, student acceptance, school involvement, service-learning program outcomes, reciprocal learning program outcomes, reciprocal learning relationships, social empathy, self-concept, resilience, career exploration/readiness and citizenship.

Quality Time After School: What Instructors Can Do to Enhance Learning (2007)
www.ppv.org/ppv/publications/assets/213_publication.pdf
This report examines youth’s experiences in five of Philadelphia’s Beacon Centers (school-based community centers providing a range of services to all community members and emphasizing after-school opportunities for youth). Grossman et al found that: 1) group management was one of the most important factors in promoting youth engagement, learning, enjoyment and regular participation, 2) positive adult support was critical to enhancing youth learning and engagement and 3) the more input or voice participants felt they had to shape an activity, the more engaged they felt and the more they liked the activity. This report provides some concrete suggestions on staff training that are likely to improve programming.
Author/Publisher: Grossman, Jean, Margo Campbell and Becca Raley. Philadelphia: Public/Private Ventures.

The Evaluation Exchange (Fall 2002)
http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/evaluation-for-continuous-improvement
This issue of The Evaluation Exchange examines the use of evaluation for continuous improvement. It incorporates advice from experts, outlines innovative evaluation practices and provides insights into the evaluations of a wide range of initiatives.
Author/Publisher: Harvard Family Research Project, Vol. VIII, No. 2.

Good Stories Aren't Enough: Becoming Outcomes-Driven in Workforce Development (2006)
http://www.ppv.org/ppv/publications/assets/203_publication.pdf
This report examines six nonprofit workforce development organizations that have adopted a culture of continuous program improvement. The report looks at how organizations got started and institutionalized the process. It highlights how data can be used and what precautions to take when deciding how to collect information.
Author/Publisher: Miles, Martha A. Philadelphia: Public/Private Ventures.

Supporting Success: Why and How to Improve Quality in After-School Programs (2008)
http://www.ppv.org/ppv/youth/youth_major_projects.asp?
section_id=19&initiative_id=29
Sheldon and Hopkins use the evaluation of the Communities Organizing Resources to Advance Learning (CORAL) initiative, a multisite after-school program in California, to discuss using continuous program improvement to support staff and improve the quality of services. Reading this report will help a practitioner gain understanding of how a program can use staff training, monitoring and coaching, and data analysis together to improve program quality.
Author/Publisher: Sheldon, Jessica and Leigh Hopkins. Philadelphia: Public/Private Ventures.

Using Quality Assessment Tools to Evaluate OST Linkages (Fall 2006)
http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/building-and-evaluating-out-of-school-time-connections/using-quality-assessment-tools-to-evaluate-ost-linkages
Helen Westmoreland from the Harvard Family Research Project discusses how out-of-school-time (OST) programs are using quality assessment tools to evaluate and promote linkages with families, schools and communities.
Author/Publisher: Westmoreland, Helen. In The Evaluation Exchange, Harvard Family Research Project, Vol. XII Nos. 1 & 2.

Building Quality Improvement Systems: Lessons from Three Emerging Efforts in the Youth-Serving Sector (2007)
http://www.forumfyi.org/node/299
The report looks at quality improvement efforts in three networks: Girls Incorporated, Michigan 21st Century Community Learning Centers and YouthNet of Greater Kansas City. While these assessment efforts are large-scale, the report provides information about the kinds of quality improvement processes being designed and implemented, and the consequences of the design choices.
Author/Publisher: Wilson-Ahlstrom, A., & N. Yohalem, with K. Pittman. Washington, DC: The Forum for Youth Investment.