There is a vast spectrum of youth development programs in our communities that offer to elevate the character and well-being of children and youth who live in poverty. Efforts to strengthen these programs certainly hold promise in increasing skills, enhancing confidence, broadening new horizons, promoting positive values, and directing youth in making good decisions. At the same time, the success of these programs to help youth rise above their life challenges calls for various ways of evaluating their efficacy. Quite often program developers and directors lack the resources or expertise to use traditional techniques although funding of programs is contingent on showing results (Izzo, Connell, Gambone, & Bradshaw, 2004). In addition, the dynamic interplay among the values of the participants and program leaders, the goals of the program, and the external forces impacting delivery (e.g., schools, neighborhood) must become part of the research agenda.

Various approaches to evaluation that give program funders and directors necessary information for program improvement are clearly needed. Credibility, accountability, defensibility, worth, growth, needs, validity, practical significance, costs, and reliability are just some of the concepts that come into play as one formally evaluates any program (Madaus, Scriven, & Stufflebeam, 1986). Certainly, few would argue that the quality of youth programming and services are dependent on how evaluation is rendered in an effective way.

Historically, we can see that program evaluation has served three important functions: a) to help determine program effectiveness, b) to modify ongoing program operations, and c) to identify what needs to be done to improve the program. Testing, modifying, and affirming ideas have lead several practitioners to use a blend of traditional and non-traditional forms of evaluation (Martinek, 2000). Unfortunately, for many practitioners their approaches to evaluating these three functions have not served them very well. The methods used by practitioners often miss the mark in terms of clarifying the effectiveness of their program. They also provide little guidance on how programs can be modified to improve their effectiveness (Izzo, et al., 2004). Significant time and resources are spent in the evaluation effort with little payback for program planning and improvement.

There are several reasons for this. One of these is, that quite often, practitioners and program planners have little input into the evaluation process. That is, they are often separated from those deliberations that determine their own evaluation questions. Questions are more than likely to be determined by the funders of the program (i.e., a product of funders' agendas). Although the recent focus on community-based research has provided a foothold on involving practitioners in the research process, little has been done to assist in identifying changes that need to be made during program operations. Therefore, continued involvement of practitioners in the evaluation process is critical if results are to become meaningful and useful.

Another reason is that funders focus more on outcome data that determines overall program effectiveness (e.g., did the program cause a decrease in school suspension rates? Or was there a significant improvement in the students' grades?). Little if any is done to assess what the program is actually doing (e.g., is it giving participants leadership opportunities? Are activities providing choices for the participants?). This type of approach has been characterized by Patton (Patton, 1997) as "black box" evaluation since there is lack of attention of what actually goes on during the program being evaluated. Rather, there needs to be attention given to what happens during the program in order to know what caused any changes in the outcomes. This will require a new way that evaluation must take place--a way that involves the practitioner and allows for clarification of what a program is doing or not doing. This will not only insure that relevance of data will be realized by the practitioner, but it will also explain what changes/modifications need to be made to insure future program success. One way to accomplish this goal is to apply logical reasoning in developing an evaluation scheme. The purpose of the article is to describe a logic model as a framework for evaluating youth development programming. It provides a unique approach to how youth programs can be evaluated. First, basic assumptions underlying program evaluation are described. Along with this, basic evaluation designs that have typically been used in evaluation efforts will be presented. Next, a logic model which will embrace the idea of looking at both process as well as product aspects of a program will be presented. A theory of change approach is used here (Connell & Kibrich, 1998; Izzo, et al., 2004) where basic principles and strategies to the evaluation process will be presented. Finally, an example of how this approach is applied to a youth development sport program will be described.

Assumptions Underlying Program Evaluation

The process of evaluating programs operates through three basic assumptions. The first is there is a commitment to making a difference (Martinek, 2000). An obvious rationale for assessing any program is that it will benefit the stakeholders--the children and youth who participate in it. Being committed to making a difference is at the forefront of the evaluation process. There are several challenges to this assumption. One of these is that funders want numbers to show that what they funded is working—this certainly makes sense. And, there is simply the need to be held accountable--something that all program agendas must include.

A second assumption is that youth development programs will more than likely work indirectly to create a number of experiences that are developmental and typically building upon one another (Intrator and Siegel, 2014; MacDonald & Valdivieso, 2000; National Research Council and Institute of Medicine, 2002). This will make it important to identify the unique pathways created by learning experiences along with the strategies that push kids along those pathways. By doing this a particular framework (or theory) is created to insure that positive change will take place. It provides a framework for future program planning and evaluation (Izzo, et al., 2004). This will also make the data useful to all and will enable the program directors, staff, parents interpret the results in a meaningful and informed way.

A final assumption is that it is impossible to prove that a program was totally effective (it proved to produce intended outcomes). There will always be some degree of uncertainty about the program's effectiveness. That is, can we determine if positive outcomes actually occurred and can they be attributed to the program? The very best we can do is to provide some evidence that seems to indicate there is some causal connections between the program experiences and ultimate outcome (Izzo, et al., 2004). Doing this requires an understanding of what the program's experiences are and what they are intended to do. Also, measuring outcomes need to be done in a valid and reliable way and will need to include more than just assessing ultimate outcomes. Rather, there needs to be ways of verifying (determining fidelity) that parts of the program are happening and that they are interconnected to each other and the ultimate outcomes.

Basic Designs

Attributing program outcomes is usually done one of two ways. One way is by using an experimental design where two groups--one group who was exposed to the program and another (an equivalent group) that was not (Shadish, Cook, & Campbell, 2002). If the program group showed positive outcomes over the non-program group, then one could conclude that the outcomes were attributed to the program. The problem with group comparisons, is that it is virtually impossible to have two truly equivalent groups. Although poverty, race, and economic status are common denominators among underserved groups, each youngster brings into the program circumstances and daily experiences unique in severity and scope. And, they also bring various levels of adaptability to hardship conditions. Therefore any differences found may be affected by pre-existing differences between the two groups. Also, there is an ethical issue involved by withholding the program from those in the control group Many program directors are reluctant to do this--making it unattractive to them to adopt this type of model.

A second type of evaluation model is what is called a time series or repeated measure design (Shadish, et al., 2002). In this model, a group of participants is assessed at certain intervals of the program (e.g., pre, mid, & post). The program group serves as its own comparison group. The idea here is to determine whether changes are occurring naturally during the program's operation. The first measure occurs before the program begins. This provides a baseline for subsequent measures. Patterns of change then are explored to determine program impact. For example, if school office referrals (Martinek, McLaughlin, & how , 1999) were found to be high prior to the program, and then gradually decreased during the program, one could conclude (in part) that the program had some impact on this particular outcome. This approach to evaluation has been found to be more appropriate given the restrictions found with two group designs. This design seems to work when access to the data is readily available for measuring at regular intervals. School and program records and surveys that are used over a distinct time period lend themselves well to this type of design (See Martinek, et al., 1999).

Results

In order to fully grasp how the logic model is used several elements will be described in this section. Recall that in applying this model the theory of change approach becomes the foundation for the evaluation process. The following elements are included in this section: a) application of the logic model, b) weekly schedule, c) how it works, d) nature of activities, and d) resources.

Application of the Logic Model

Avoiding the "black box" approach to evaluation requires that processes of the program are taken into account. The logic model applies here. It takes into account the various causal pathways that lead to ultimate outcomes being impacted by the program. These pathways are formed by various theories (or ideas) that support the connections made among the various outcomes. An application of this model is provided and focuses on the Youth Leader Corps (YLC) program at a southern university in the United States (Martinek & Hellison, 2016; Martinek, 2016). The YLC program provides at-risk high school students an opportunity to plan and teach life skills to younger underserved children. The student leaders attend an alternative high school, called a middle college, which is situated on the university campus. Leaders are selected based on their own interest and from referrals from the classroom teachers and the school counselor. The selection criteria identifies those students who chronically struggle academically and socially in the regular classroom setting. Also their past school biographies are plagued with high absenteeism, poor grades, and frequent school suspension. By participating in the YLC key leadership attributes such as strategic thinking, problem solving, communication, emotional control, and conflict resolution (Wright, 2012; Wright, Ding, & Pickering, 2010) can be attained. These qualities have been shown to be important requisites for being more successful in their classroom work (Chafee, 2013; Intrator & Siegal, 2008, 2014; Simonton, 2016;).

Students receive three workshop sessions of training where they learn how to teach life skills through physical activity. They are also familiarized with Don Hellison’s Teaching Personal and Social Responsibility (TPSR) model (Hellison, 2011). The TPSR model serves as a framework for planning and teaching the life skills lessons. Familiarization with the TPSR goals, personal core values, and teaching strategies are covered in the three sessions. The teaching strategies help students create experiences that focus on the responsibility values (i.e., self-control, respect for the rights and feeling of others, trying one’s best, being self-directed, helping and leading others and applying values outside the gym). These strategies include ways for relationship building, empowering students, integrating TPSR values into lesson experiences, self reflection, and transferring values outside the gym. Table 1 illustrates the content of the three training sessions. Emphasis is also placed on having a strong commitment to the program and on representing the YLC in a positive manner in their high school classes. Also, the skills of authentic reflection are woven into the training lessons. Training is provided by graduate students and faculty staff.

The first two sessions focus on familiarization of the TPSR values, program responsibilities, teaching strategies, other pedagogical considerations, and assignment of teaching teams. During the third session the leaders, working in pairs, get together to plan for the first day of the program. Focal points of this meeting are on what to teach and partner responsibility roles. Each lesson is written on a lesson plan guide form. Three areas are described in the plan. First are the goals of the lesson—what do the leaders want to accomplish with the kids and the TPSR goal(s) to be addressed. Next is the activity or activities that will be provided. Leaders are encouraged to be as specific as possible when describing the activities (e.g., types of grouping, actions required by participants, rules to follow, etc.). Finally the equipment needs (i.e., type and number) are entered into the plan.

On the flip side of the lesson plan guide are three areas that the leader is asked to reflect upon after each lesson. The first touches upon the type of contribution the leader made to the lesson. For example, a leader may express how (s)he was able to manage the group effectively throughout the lesson. In response to the second area, the leader is asked about ways (s)he showed leadership for that particular lesson. And finally, many lessons will present some type of challenge for the leader (e.g., dealing with bad behavior, clarifying directions, organizing kids into groups (period placement)). The third area of reflection focuses on those challenges that need to be addressed in future lessons.

Weekly Schedule

The YLC runs twice a week. On one of the days the leaders travel to an elementary school which is located in a public housing community. Here they work with thirty 4th and 5th grade students who have been selected by their school counselor and teachers to participate in the after-school program. These are students who have struggled in their classroom work and have acquired risk behaviors that make them vulnerable to school failure.

The second YLC program is held later in the week where the leaders work with children from three underserved communities-each populated with refugee families from Mexico, Africa, and Indonesia. This program is affiliated with an outreach center located on the university's campus. The center provides services and resources to refugee/immigrant families located in the local community. Approximately thirty-five children (age 8 to 12 years old) are transported to the campus by university vans.

In both programs, stations are set up where the leaders work with groups of students for about 10 minutes. The students then move to another station and set of leaders. The lesson format for each lesson is one that was covered in the leaders' training sessions (i.e., group discussion, activity, group discussion, personal reflection, rotation to next station). At the conclusion of the program, a large group meeting is conducted. Participants are asked about the types of things they learned (i.e., TPSR values), their contribution to the experience, and ways they can apply the TPSR values outside the gym.

A post-program meeting is held during which the leaders, assistants, and faculty leader share perceptions about how things went. The challenges, successes, and strategies often surface during these discussions. The leaders then meet to plan for the next week's program. The program session concludes with everyone going to the university cafeteria for dinner--a great time for continued dialogue about the program experience.

How the Model Works

The following is a description of how the logic model works in evaluating the YLC. It will show how ultimate outcomes are the result of various processes (other outcomes) that are being created during the program. The model should serve to guide evaluation decisions and interpret the reasons for program outcomes. The goal is to develop confidence in being a leader and apply responsible behavior in the classroom. Program structure empowers the students to plan their own lesson, provide opportunities for personal reflection, and provides guidance to the leaders from university staff and graduate students. Besides determining ultimate outcome goals, the model also provides opportunity among staff and leaders to share ways in which the program activities will produce ultimate outcomes.

Figure 1 shows that there are four main elements in the logic model that have been identified by Izzo and her colleagues (2004): a) activities and resources, b) immediate outcome, c) intermediate outcomes, and d) ultimate outcomes.

Activities and resources refer to what is being taught and the various parts of the program that produce certain outcomes. It can also include certain structural and resource aspects that help to support the program's operation. Immediate outcomes are those outcomes that occur immediately from the program activities and are viewed as the experiences that begin to initiate the change process. Intermediate outcomes are the causal aspects that affect the ultimate outcomes. Essentially immediate and intermediate outcomes are the “connectors” between intended goals of the program and its outcomes—some causality is made tenable. During these processes the whole issue of true fidelity becomes a focal point in the evaluation process. It is here where the question “Is the program doing what it is intended to be doing?” is answered. That is, these outcomes of evaluation attempt to dismantle the “Black Box” problem in program delivery. Many times program leaders profess that certain dynamics are occurring—ones that are to be the catalysts to create program outcomes. And yet, quite often, the actual dynamics, when examined closely, belie their intent.

Ultimate outcomes are the criteria that will be used to judge the effectiveness of the program. They will be the direct by-products of the program and its other outcomes. Each of these elements will enable the evaluator to establish the scope and sequence of the evaluation process.

Element one: Activities and resources. This includes describing the actual activities that occurred in the program. The activities listed in Figure 1 illustrate the actual YLC program activities that were created to impact the immediate outcomes. Creating those activities must be in line with the mission and goals of the program. Therefore, a certain amount of theory and intuitive thought go into knowing what will work and not work. In the YLC program having students familiarized with a values-based approach to teaching social and personal responsibility, providing students opportunity to plan and teach values-based physical activity experiences to younger kids, and providing feedback to the leaders and opportunity to reflect upon their teaching experience are the main activities of the program. In addition, the resources such as instructional space, transportation, graduate student assistance/support, and a partnership with youth serving organizations are included as supporting aspects of the program.

Element two: Immediate outcomes. Since the leaders are given the responsibility of planning and teaching the lessons a feeling of empowerment and ownership are gained from the process. Although there may be some guidance ultimately it is their job to plan activities that will keep the younger kids active and at the same time insure that they will experience one or two of the TPSR values during the lesson.

Another important immediate outcome is the relationship that is fostered between the leaders and the kids they teach. Making the right connection with the kids is so vital for the leaders— it draws them closer to the needs of kids and their struggles. It also enables them to challenge the kids without fear of rejection or indifference. This will reinforce the feeling that their leadership is working. Without this foothold the relational aspect of working with their kids is tenable and will present challenges in future engagement efforts. Cultural, behavioral, and skill challenges of the younger kids all intersect with the relationship building attempts of the leaders. All this will help the leaders to start on a social and positive trajectory in their own school lives. It will also helps them to have successful intermediate and ultimate outcomes.

Relationship building is fostered a number of ways. One of the ways is to take time prior to the program's start-up. As kids enter the instructional area, the leaders will meet them at the door and take time to talk to them, engage in a physical activity (e.g., shooting baskets with them or jumping rope), or just say hello and ask them about their day. Beyond these simple interactions, genuine listening, acknowledging their contributions, responding to their fears, successes, and aspirations all serve to form strong and lasting relationships between the leaders and their kids.

A final immediate outcome is the leaders becoming more introspective about the role as a leader and helper of others. The group discussions and written journal entries following each YLC session creates a better feel for what leadership growth and helping others is all about. It places each leader in a position to know better of what he/she did and can continue to do for others.

One simple way to assess whether the immediate outcomes occurred is for the program director to keep a daily journal of how they were evident during each program session. Five minutes can be set aside for this task. In a more formal way, a check system can be utilized (See Table 2) where the specific outcomes are checked off if they had appeared during a lesson. This approach can provide a more objective profile of the fidelity of a lesson as well as a pattern of occurrence over time.

Element three: Intermediate outcomes. It has been theorized that giving kids opportunity to plan and teach others offers the foundation for them to gain a sense of autonomy, competence, and purpose in their lives (Deci & Ryan, 2000; Hellision, 2011; Ryan, Williams, Patrick, & Deci, 2009). Empowering students to lead in the YLC (with guidance) gives them the confidence to try things out—not only in the teaching of kids but also to be more fully engage in their school work. The idea of becoming more self-directed comes into play here. Acquiring sense of responsibility for attending, planning, and teaching in the YLC program become essential elements for engendering a more focused and enthusiastic approach in the leaders’ school work. Additionally, the role that personal reflection (an immediate outcome) plays into all of this cannot be understated. Giving opportunities for students to self examine each teaching experience, discover their personal successes, and acknowledge any failures are so instrumental in building a foundation for personal improvement in everyday life (Hellison, 2011; Melendez & Martinek, 2015). Such opportunities to engage in self growth can solidify the leaders' ability to expand their reach for other possibilities of learning outside of the program. Increases in curiosity, elevating one's focus on effort rather than submission will become instrumental in future achievement (Heckman & Sanger, 2013).

Element four: Ultimate outcomes. The ultimate outcomes were determined to be grade point improvement, lessening of office referrals, and increased confidence in being a leader. The assumption that follows this outcome selection is that given the length of the program (one year) the grades, referrals, and self-confidence were tenable measures. Grade point average can be determined at various grading periods during the year. Typically, grading takes place at 9-week intervals. Office referrals--where teachers send students to the office for disciplinary purposesare kept on a weekly basis. These can also be acquired at 9week interval times. Leader confidence (i.e., how they saw themselves as leaders), was determined by administering a short four-item questionnaire at the end of each 9-week period. Four questions are included in the scale: 1) How confident are you in working with children? 2) How confident are you in providing a TPSR physical activity experience? 3) How confident are you in being a youth leader? and 4) How confident are you about leading a small group discussion at the end of each lesson? The items are answered by checking a response along a 4-point likert scale (i.e., very confident-4, Confident-3, Somewhat Confident-2, Not Confident-1). Certainly looking at long range effects (beyond high school) would be another possibility. But that would require another set of resources--ones that were not available to track students into adulthood.

Nature of Activities

As previously mentioned, a major issue that program leaders face is to know how well the activities are being executed. Knowing this is important since the levels and types of activities will determine any impact on program outcomes. In any program evaluation it is important that the evaluation process focuses on the fidelity of its implementation. The various pathways that function in a program all need to contribute in a wholistic way to the ultimate outcomes of the program. It should also be noted that the degree of fidelity will vary from “full implementation” to some watered down version of the model (Gordon, Jacobs, & Wright, 2016). In fact, some parts of the program may not be functioning at all. But knowing which parts of the program are functioning well and which ones are not will help to explain the level of impact that occurs. It is also important to note that variations may occur for the right purposes. According to Gordon and Doyle (2015) some programmers may have a good grasp of the model but find changes in things like context and/or student disposition may require a “tweak” in the delivery of the program. This can be done while still maintaining the integrity of the program model. So some flexibility is good as long as it doesn’t create program features that belie the intentions of the model. In the end, the benefits from evaluation efforts will depend on the degree to which fidelity is maintained throughout program implementation. Only then, can outcomes be truly attributed to the implementation of the program. Maintaining a record of what the program is doing (or not doing) may require systematic observation. Several researchers/evaluators (Escarti, Wright, Pascual, & Gutierrez, 2015; MacDonald & McIsaac, 2016; Wright & Craig, 2011), have advocated for the use of specific instruments to document and verify the fidelity of program implementation. An example of this was given earlier in the "Immediate Outcomes" section.

A Word about Resources

Part of the model (Figure 1) indicates that available resources are important to consider in the evaluation process. They will determine the extent in which program activities can be effectively carried out. Robert Granger (2002) emphasizes that the need to consider supportive elements is imperative. Doing so will not only affect program quality but ultimately will bring about a greater level of confidence in those who are delivering the program. Program planners will no doubt seek a number of resources to insure that the program’s mission will be effectively carried out. The most obvious resources usually include things like space, transportation, financial support, equipment, and staffing. Some of these key resources were important in the operation of the Youth Leader Corps program. Instructional space was one of the most critical resources needed. For example, although there was ample outdoor space available for part of the program at the elementary school, indoor space was also needed. Unfortunately, the school did not have a gymnasium. So instructional space included a cafeteria, a hallway, and classroom. For the other program that ran at the university, scheduling of a gym was done prior to the beginning of the program. Access to equipment was also a requirement for both programs. This was readily available from the university’s Department of Kinesiology. Funding to cover transportation costs for bringing younger children to the campus site and leaders to the school site was provided by a grant. And, there were the less tangible resources such as the partnerships between the program director and a university outreach center and the public schools. A history of strong collaboration between them and the director helped to solidified scheduling and transportation support. Finally, the opportunity to work with the alternative high school made it possible to have access to student leaders throughout the school year.

Contributions of Author

In this article, an attempt was made to provide a compelling example of how program evaluation can reach beyond just merely looking at ultimate outcomes. Rather it was argued that it is important to examine those mediating qualities that will ultimately impact program outcomes. The logic model approach serves to provide guidance for program leaders on the best ways to deliver a program by identifying the features of the program that are working and those that are not working. In other words, the reasons for failure and success can be identified thus impacting the direction for future planning. Additionally, knowing what data needs to be collected and how they will be interpreted become important aspects of the evaluation process.

One issue that needs to be considered is the causal pathways that determine program impact. Most evaluation efforts find it very difficult to attribute outcomes to a program—it is not feasible for most evaluation designs to determine this (Izzo, et al., 2004). However, if certain aspects of the program clearly focus on enhancing certain outcomes, and these aspects are consistently present throughout implementation (i.e., immediate & intermediate outcomes), attributing outcomes to the program become more tenable. For example, suppose the youth leaders by the end of their first year in the program showed a significant decrease in their office referrals, a significant improvement in classroom behavior, and also saw themselves as future leaders. They also continually experienced opportunities to plan and teach their own lessons, had continued positive relationships with the children they taught, and were able to consistently reflect on leadership experiences. They, in turn, gained a better sense of being responsible, helpful to others, self-directed, and motivated to succeed. In comparison, students from that same school who had high office referrals and classroom issues did not have these same experiences. If this was the case, it would appear that student involvement in the Youth Leader Corps program resulted in positive outcomes. Of course there is no doubt that one can never know if such outcomes would have occurred naturally—without a program. Although providing a true control group comparison would help in this determination, finding an equivalent group is a huge challenge. Therefore, using the logic model approach becomes a nice alternative for determining causal connections.

Another consideration in using the logic model for evaluating programs, is making sure that program pathways are connected well to the program’s ultimate outcomes. Past research, previous experiences, and input from others help to select which of the pathway elements best impact the desired outcome of the program. For example, research has shown that empowerment, connecting with others (relationship building), reflection, and adult guidance (Halpern, 2003; Hemphill, 2015: Hirsch, 2005; Intrator & Siegel, 2014: Melendez & Martinek, 2015; Schilling, Martinek, & Carson, 2007) are significant predictors of school and life success. It is important, therefore, that the choice of what outcomes to include is based on those factors that are found to have the greatest impact.

A final thought to consider is the timing of the evaluation process. That is, before evaluation is implemented it is important to know if the program is running at its intended capacity. It is often found that programs take time to be able to get things right. Consequently, to insure full fidelity of the program, giving enough time for the program to run is needed. It is during this time that some of the immediate outcomes will become focal points of the program directors. Modifications that occur during this time will help to insure success in achieving the ultimate outcomes of the program. Documenting these changes then becomes part of the evaluation process and will serve to inform the evaluator and program director.

Conclusion

In summary, we are reminded that the real purpose of any program evaluation is to provide information to stakeholders so that program quality can be insured. Using the logic model approach provides the opportunity to identify those aspects of the program that impact desired outcomes. Evaluation decisions are made based on relevant information about what is working and what is not thus having direct relevance to practice. Knowing the various aspects of the program places the ultimate findings in context of a coherent program. This improves the credibility and meaning of the findings (Izzo, et.al., 2004). Critical program practices then emerge as by-products of the process

References

Chafee, L.D. (2013). Expanded learning opportunities are key to student learning. In T. Peterson (Ed.), Expanding minds and opportunities: The power of afterschool and summer learning for student success. (pp.17-20). Washington, DC: Collaborative Communications Group.

Connell, J., & Kubisch, A. (1998). Applying a theory of change approach to evaluation of comprehensive community initiatives: Progress, prospects, and problems. In K. Fulbright-Anderson, A. Kubisch, & J. Connell (Eds.), New approaches to evaluating community initiatives: Vol. 2 theory, measurement, and analysis (pp. 66-79) Washington, DC: Aspen Institute.

Deci, E.L., & Ryan, R.M. (2000). The "what" and "why" of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11, 227-268. https://doi.org/10.1207/S15327965PLI1104_01

Escarti, A.; Wright, P.; Paschall, C., & Guitierrez, M. (2015). Tool for assessing responsibility-based education (TARE) 2.0: Instrument revisions, inter-rater reliability and correlations between observed teaching strategies and student behaviors. Universal Journal of Psychology, 3(2), 55-63. https://doi.org/10.13189/ujp.2015.030205

Gordon, B., & Doyle, S. (2015) Teaching personal and social responsibility and transfer of learning: Opportunities and challenges for teachers and coaches. Journal of Teaching Physical Education, 34, 152-161.https://doi.org/10.1123/jtpe.2013-0184

Gordon, B.; Jacobs, J.M., & Wright, P.M. (2016). Social and emotional learning through ateaching personal and social responsibility based after-school program for disengaged middle-school boys. Journal of Teaching Physical Education, 35(4), 358-369. https://doi.org/10.1123/jtpe.2016-0106

Granger, R.C. (2002). Creating the conditions linked to positive youth development. InR.M. Lerner, C.S. Taylor, & A. von Eye (Eds.), New directions for youth development: Pathways to positive development among diverse youth. (pp.149-164). San Francisco: Jossey-Bass.https://doi.org/10.1002/yd.20

Halpern, R. (2003). Making Play Work: The Promise of After-School Programs for Low Income Youth. New York: Teachers College Press.

Heckman, P.E., & Sanger, C. (2013). How quality afterschool programs help motivate and engage more young people in learning, schooling, and life. In T. Peterson (Ed.), Expanding minds and opportunities: The power of afterschool and summer learning for student success. (pp. 31-33). Washington, DC: Collaborative Communications Group.

Hellison, D. (2011) Teaching Personal and Social Responsibility Through Physical Activity(3^rd Ed.). Champaign, IL: Human Kinetics.

Hemphill, M. (2015). Inhibitors to responsibility-based professional development with inservice teachers. Physical Educator, 72, 288-306.

Hirsch, B. (2005). A place to call home: After school programs for urban youth. New York: Teachers College Press. https://doi.org/10.1037/11087-000

Intractor, S., & Siegal, D. (2008). Project coach: Youth development and academic achievement through sport. Journal of Physical Education, Recreation, and Dance, 79(7), 17-23.https://doi.org/10.1080/07303084.2008.10598210

Intrator, S.M., & Siegel, D. (2014). The Quest for Mastery: Positive Youth Development Through Out-of-School Programs. Cambridge, MA: Harvard Educational Press.

Izzo, C. V.; Connell, J. P.; Gambone, M. A., & Bradshaw, C. P. (2004). Understanding andunderstanding and improving youth development initiatives through evaluation. In S. F. Hamilton and M.A. Hamilton (Eds.), The youth development handbook: Coming of\age in American communities (pp. 301-325). Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781452232560.n13

MacDonald, D. J., & McIsaac, T. (2016). Quantitative assessment of positive youth development in sport. In N. Holt (Ed.). Positive youth development through sport (pp. 83-96). London: Routledge.

MacDonald, G. B., & Valdivisso, R. (2000). Measuring deficits and assets: How we track youth development now and how we should track it. In Public/Private Ventures (Ed.), Youth development: issues, challenges and directions (pp. 149-184). Philadelphia: Public/Private Ventures.

Madaus, G. F.; Scriven, M., & Stufflebeam, D. L. (1986). Evaluation Models: Viewpoints on Educational and Human Services Evaluation. Boston: Kluwer-Nijhoff.

Martinek, T. (2000). Program evaluation. In Hellison, D. Cutforth, N., Kallusky, Martinek,T. Parker, M., & J. Stiehl (Eds.), Serving underserved youth through physicalactivity: Toward a model for school-university collaboration (pp. 211-228), Champaign, IL: Human Kinetics.

Martinek, T. (2016). Project Effort: Creating responsible leadership through values based learning. Active and Health Magazine. 23(2/3), 43-46.

Martinek, T., & Hellison, D. (2016). Learning responsibility through sport and physical activity. In N. Holt (Ed.). Positive youth development through sport (pp. 49-60). London: Routledge.

Martinek, T.; McLaughlin, D., & Schilling, T. (1999). Project effort: Teaching responsibility beyond the gym. Journal of Physical Education, Recreation, and Dance, 70, 12-25. https://doi.org/10.1080/07303084.1999.10605954

Melendez, A., & Martinek, T. J. (2015). Life after project effort: Applying values in a responsibility-based physical activity program. RICYDE. Revista internacional de ciencias del deporte, 41(11), 258-280.https://doi.org/10.5232/ricyde2015.04105

National Research Council and Institute of Medicine (2002). Community Programs to Promote Youth Development. Washington, DC: National Academy Press.

Patton, M. (1997). Utilization-Focused Evaluation (3rd Edition). Thousand Oaks: Sage.

Ryan, R.M., Williams, G.C., Patrick, H., & Deci, E.L. (2009). Self-determination theoryand physical activity: The dynamics of motivation in development and wellness. Hellenic Journal of Psychology, 6, 107-124.

Schilling, T.; Martinek, T., & Carson, S. (2007). Developmental processes among youth leaders in an after-school, responsibility-based sport program: Antecedents andbarriers to commitment. Research Quarterly for Exercise and Sport. 78, 48-60. https://doi.org/10.1080/02701367.2007.10599403

Shadish, H.W.; Cook, T., & Campbell, D.T. (2002). Experimental and quasi-experimental design for generalized causal inference. Belmont, CA: Wadsworth

Simonton, S. (2016). Keeping adolescents engaged: What can after-school programs do?Youth Today. 25(2), 1, 23-24.

Wright, P. (2012). Offering a TPSR physical activity club to adolescent boys labeled "at risk" in partnership with a community-based youth serving program. Agora for Physical Education and Sport, 14, 94-114.

Wright, P., & Craig, M. (2011). Tool for assessing responsibility-based education (TARE). Instrument development, content validity, and inter-rater reliability. Measurement in Physical Education and Exercise Science. 15(3), 204-219. https://doi.org/10.1080/1091367X.2011.590084

Wright, P.M.; Ding, S., & Pickering, M. (2010). Integrating a personal and social responsibility program into a lifetime wellness course for urban high school students: Assessing implementation and educational outcomes. Sport, Education, and Society, 15, 277-296.https://doi.org/10.1080/13573322.2010.493309

Additional information

Correspondence/correspondencia:: Thomas Martinek University of North Carolina at Greensboro. USA. Email: tjmartin@uncg.edu