Peer Review: Evaluation of Readiness for Wrestling Program

Previously published in Volume 82, Issue 4

Abstract
Background: Youth participation in extracurricular activities such as wrestling and boxing has positive affects on their mental and physical wellbeing. However, these sports are not offered in many rural communities in Canada where youth physical and mental wellbeing is lower than Canadian urban communities. Objectives/purpose: To determine if an extracurricular wrestling and boxing program is perceived to be acceptable, feasible and sustainable by the members of the highly successful, rural aboriginal community of Conne River, Newfoundland and Labrador. Methods: A modified Delphi method was used to engage community stakeholders and experts from the fields of kinesiology, combat sports and aboriginal affairs to develop a survey to accurately determine the community of Conne River’s needs, as well as factors contributing to building an acceptable, feasible, and sustainable wrestling and boxing program. The survey consisted of free text questions as well as a 5-point, anchored Likert scale and was handed out to parents of youth attending the community school, the Band Council of Conne River, and school employees. Results: There was a 62% response rate from participants. Average responses indicated a high level of agreement to the questions asked and indicated the program would be feasible and sustainable in the community. The parent group had the highest level of agreement in all categories while the teacher group was lowest in all categories except for questions related to the appropriateness of the program in the rural, aboriginal context. Conclusions: The community of Conne River, NL is receptive to having a wrestling and boxing program implemented in their community and has the resources and motivation to make the program sustainable.

Youth who do not participate in afterschool extracurricular activities, such as athletics, fine arts and academic clubs have been shown to have a higher risk of engaging in unhealthy behaviors that could have detrimental effects during their time in school and after graduation (Broh, 2002; Darling, 2005; Eccles et al., 2003; Feldman and Matjasko, 2005; Fredricks and Eccles, 2006; Gardner et al., 2008; Holland and Andre, 1987; Lareau, 2003; Lerner, 2005; Mahoney, 2000; Mahoney and Cairns, 1997; Mahoney et al., 2003; McNeal, 1995; Simpkins et al., 2005; Zaff et al., 2003). Several studies have shown that participation in extracurricular activities such as sports and fine arts can reduce low levels of mental health, and can reduce the prevalence of school drop outs, substance abuse and anti-social behaviors in youth (Bohnert and Garber, 2007; Busseri et al., 2006; Darling 2005; Dotter et al. 2007; Gilman et al., 2004; Mahoney, 2000). The skills that students learn in extracurricular activities such as physical and social competency, self-management, integrity and respect, goal-setting, decision-making, communication and teamwork are typically carried well beyond those learned in the classroom (Weiss and Wiese-Bjorns, 2009). By being part of a group or club students gain a sense of belonging and community that has shown to have positive benefits in school commitment (i.e. attendance and work ethic) and achievement, and promotes healthy choices (Eccles et al., 2003; Marsh and Kleitman, 2002).

As it relates specifically to sports, literature suggests that involvement in afterschool activities increases youths’ levels of self-esteem (Daniels and Leaper, 2006; Erkut and Tracy, 2002; Holland and Andre, 1994; Tracy and Erkut, 2002) by providing a context for self-assessment outside of the norms of school and family life (Barber et al., 2001). This increased level of self-esteem has been shown to carry into adulthood and can prevent problems such as life dissatisfaction, physical health problems, depression, substance abuse, suicidal behavior, and aggressive behavior (Crocker and Wolfe, 2001; Kort-Butler and Hagewen, 2010; Trzesniewski et al. 2003).  Although the positive effects of sports on the future development of students as highly functioning individuals has been demonstrated many times, in certain populations, access to sports such as hockey, basketball, football and soccer is limited or prohibitive due to costs of participation (i.e.: costs of obtaining equipment or rink/court/field time) (Humbert et al. 2006).

Combat sports such as wrestling and boxing increase self-esteem like other sports, but they are universally accessible (Hanson et al., 2001; Oh et al., 2015; Rachele et al., 2014; Sibold et al., 2015; Silvestri, 1997; Warner & Dixon, 2013). In both wrestling and boxing, there is no gender bias (women compete against women and men compete against men, but they train together); cost is not a factor (individuals can wrestle in any style of clothing and boxing gloves are of low cost and easily shared among participants); and they do not require any particular genetic or physical criteria due to separate weight classes for competition (Channon, 2014; Hayhurst, 2014; Kamble et al., 2015; Ousley et al., 2013; Pettersson et al., 2013; Terry et al., 2014; Velija et al., 2013; Woodland, 2009). This inclusiveness makes combat sports, such as wrestling and boxing positive candidates for a universally accessible extracurricular program.

Youth from rural, aboriginal and remote communities, such as those in Newfoundland and Labrador often face a lack of afterschool sport activities (Newfoundland and Labrador Poverty Reduction Strategy Progress Report, 2014; Provincial Wellness Review, 2014; UNICEF Report Card, 2016).  Frequently these communities do not have the infrastructure to support team sports such as hockey, basketball, football or soccer. In addition, these communities also lack the numbers of students who would be willing to participate to make up the teams. Thus, youth are often criticized for not engaging in sport activities, yet in rural, aboriginal and remote communities it is often the lack of infrastructure and number of students available to participate in organized programs that are the limiting factors (Roth and Brooks-Gunn, 2003). Furthermore, in rural communities and aboriginal communities in Canada there are fewer extracurricular activity programs offered due to a lack of facilities, lower household income, lack of volunteers, and/or little community cohesion (Clark, 2008; Guèvremont et al., 2008; Hoefer et al, 2001; Huebner and Mancini, 2003; Kremarik, 2000; La Torre et al., 2006; Offord et al., 1998; Simpkins et al., 2005; Wen et al., 2007; White and Gager, 2007; Wimer et al., 2008; Xu et al., 2009).

Currently, schools and community centers, are endeavoring to offer more afterschool activities. However, many new programs that are implemented often fail due to a lack of youth participation (Roth and Brooks-Gunn, 2003). The unwillingness to participate, or the high drop out rates are due to several factors such as programs offered irregularly, a lack of established programs goals, lack of challenges for youth, or a lack of coordination with the community and implementers of the program (Roth and Brooks-Gunn, 2003). Before spending large amounts of time and money in implementing an extracurricular program, the program must first be evaluated by all interested parties to determine if the community is prepared to work with implementers to ensure program sustainability.

The purpose of this study was to determine if an extracurricular after school wrestling and boxing program would be acceptable, feasible and sustainable in the rural indigenous community of Conne River, Newfoundland and Labrador (NL), Canada. Specifically, the objectives of the study were first to create and validate a survey tool using the Delphi method, and second, to use the tool to determine the acceptability, feasibility and sustainability of an extracurricular after school wrestling and boxing program in Conne River, NL, Canada.

Methods
Prior to starting the study, ethics approval was obtained from the Health Research Ethics Board, of Newfoundland & Labrador. The Conne River, NL Band Council granted permission and participant consent was informed and implied through the return of a completed survey. Results of all phases will be discussed in the results section of this paper.

Phase 1: Panelist Selection
The goal of the first phase of this study (panelist selection) was to recruit a representative sample of highly trained experts with first-hand knowledge of the content domain, whose opinions were respected by their peers (Clayton, 1997; Fink et al., 1984; Hsu and Sandford, 2007; de Villiers et al., 2005) for a subsequent expert consensus building exercise (Phase 2: Survey Development).  Experts were defined as practicing provincial school educators, individual aboriginal people currently active in their communities, elite-level combat sport coaches, and kinesiology researchers at the university level. The panel was restricted to current active aboriginal community members, educators and coaches because the authors felt their responses would incorporate relevant ideas about what aboriginal youth are in need of, and reference school curricula in their responses.

Prospective panelists were identified through purposive sampling to determine the initial panelist (Morgan et al., 2007) after which a ‘snowball’ technique was used to ensure adequate representation of the individuals who would understand the practicalities of the implementation of a wrestling and boxing program (Penciner et al., 2011; Valente and Pumpuang, 2007). Specifically, key informants (elite combat sport participants, university professors, high school teachers and aboriginal community members) were asked to nominate individuals considered to be experts based on the aforementioned criteria (Clayton, 1997). Each nominee was contacted electronically, provided with a letter detailing the purpose, methods and anticipated time commitment of the study, and asked to participate if he or she was able to complete all survey rounds.

Although there is no consensus on an appropriate panel size for a Delphi process, (Hsu and Sandford, 2007) a sample of between 10 and 15 is considered adequate for most purposes (Gustafson, 1975) and most authors recommend using the minimal number of participants required to balance representativeness and the likelihood of participants completing all survey rounds (Hsu and Sandford, 2007; de Villiers et al., 2005). In this study, recruitment continued until a minimum of three panelists from each specialty (12 panelists in total) agreed to participate. Each of these individuals was provided with a unique subject identification number to ensure that each person's identity remained confidential throughout the Delphi process.

Four initial panelists were identified, one from each of the four key areas (aboriginal communities, elite combat sport athletes/coaches, university professors, and high school teachers). They each then provided additional panelists from their areas until the final panel was formed, which consisted of 12 panelists, three high school teachers, three combat sport athletes/coaches, three active aboriginal community members and three university professors.

Phase 2: Survey Development/Modified Delphi Method
The aim of this phase was to use a modified Delphi method to develop and refine a Utilization-Focused Evaluation (UFE) (developed by: Michael Quinn Patton, 2012) survey that would be used to evaluate the Aboriginal communities’ contextual factors influencing readiness for the program, existing infrastructure and process, as well as expected results if such a program was to be established.

Item Generation

The initial idea for the wrestling and boxing program was based on other recreational and elite level wrestling and boxing programs that are currently offered in urban areas in Canada and would have elements of wrestling, boxing and conditioning. Theories gained from examining current programs were then taken to the community of Conne River on March 31st 2015, and the authors discussed the program with stakeholders in the community. In accordance with the modified Delphi method (Graham et al., 2003; Hsu and Sandford, 2007; de Villiers et al., 2005) the authors used the previous information gained to generate an initial list of items to be presented to Delphi panelists. The initial set of survey questions was compiled based on Stufflebeam’s Context, Input, Process and Product (CIPP) model (Hakan & Seval, 2011). The total of number of questions in each of the CIPP areas were: Context = 8, Input = 6, Process = 10, Product = 6 for a total of 30 initial questions to be evaluated.

Survey instrument

The survey website “FluidSurveys” was used by the authors to house the structured electronic questionnaire and record the data. The survey began with a detailed set of instructions. Panelists were then asked to rate their level of agreement with the inclusion of each of the 25 questions in the item on a five-point Likert scale, where 5 indicated strong agreement that the item was relevant to the purpose of the study and 1 indicated a strong disagreement that the item was relevant. Free-text boxes were provided after each condition and at the end of each section to allow panelists to comment, add, or suggest revisions to the listed items.

Modified Delphi Method

The structured questionnaire was presented electronically to the expert panel in multiple iterative rounds (Hsu and Sandford, 2007; de Villiers et al., 2005). In round one, panelists completed all parts of the questionnaire, identified additional questions that were not included in the item, and suggested revisions to the wording of questions that were ambiguous or confusing. In round two, panelists were sent a revised version of the questionnaire that included suggested additions and revisions, and were instructed to rate the conditions again. If their ratings deviated from the group mean by more than 20%, they were also asked to provide a brief comment outlining their reasons for remaining outside the consensus (Hsu and Sandford, 2007; Lambe and Bristow 2010; Scavone et al., 2006). Weekly reminders were sent to panelists who had not completed the questionnaire after one week until panelists responded.

All statistical analyses were conducted using Excel for Mac 2011 (Microsoft, Redmond, WA, USA). The opinions of all panelists were weighed equally. After each round, the panel's mean ± SD response was calculated for each condition. A final list of the questions to be included in the final item is presented in the results.

In this study, consensus was defined based on the homogeneity or consistency of opinion among panelists across survey items, which was quantified using a an SD < 1. If the SD was > 1 then it was determined that the question was not clear to all members of the panel and was reworded based on feedback or removed. Finally, the panel's mean ratings for each question were calculated to determine the relationship between panelists’ ratings of each question. Questions were kept in the survey if their mean score of all panelists’ answers was > 3.5. The values of these measures observed in each round of the survey are reported in Table 1 and Table 2 to demonstrate the degree of consensus throughout the Delphi process.

The process was terminated after the second round, as the minimum consensus criterion was achieved, and only minor changes in panelist responses between rounds one and two of the survey were observed.

Phase 3: Survey
Between December 12th, 2015 and January 21st, 2016, individuals that were part of our three specific categories and were living in the community of Conne River Newfoundland, Canada (Miawpukek First Nation) were invited via the call out program Synrevocie to come to the local school to complete the survey assessing the sustainability of a wrestling and boxing program in their community. Employees of the local private school (St Anne’s School) completed the surveys (n=18) as did all Band Council members (n=5) and several parents with eligible youth (total of 54 eligible youth) attending the school (n=8). The survey was given out to 8.3% of total population of Conne River who were selected for being in one of the three categories of stakeholders vital to the program: school employees, parents of youth that were eligible to participate in the program, and Band Council members of the community. A total of 31 people out of 50 (response rate of 62% and 5.2% of the total population) completed the survey. Descriptive statistics were used to summarize the findings of the survey and to provide data on the feasibility and sustainability of a wrestling and boxing program. Each group (school employees, band council members and parents) was analyzed separately and all groups were then analyzed as one.

Analysis
Phase 1: Panelist Selection
This phase did not require any statistical analyses.

Phase 2: Delphi method
Descriptive statistics (mean and SD) for each of the questions, in each of the rounds were calculated using Statistical Package for the Social Sciences (SPSS) software.

Phase 3: Survey
Questions were grouped into four categories based on subject matter: context, input, process and product. A mean score for each category was created for each participant by summing responses and dividing by the number of questions answered by that participant. The mean was calculated from 1 to 5 with a mean closer to 1 being representative of a prepared community and a score closer to 5 being representative of an unprepared community. If a participant did not answer a minimum of half the questions in a given category, their data was removed for that category.

Four separate one-way analyses of variances (ANOVAs) were performed to compare mean scores of each category between participant groups (school employees, parents and band council) to determine if there were any significant differences between groups on any of the measures. Bonferroni corrections were applied to control for multiple comparisons.

Results
Phase 2: Modified Delphi Method
All 12 of the panelists completed Round One of the Delphi process and ten (83%) completed Round Two. As seen in Table 1, following Round One, three questions were removed, one due to lack of importance (Mean < 3.5), and two were removed due to having both a high level of ambiguity (SD > 1) and panelist comments. Four questions were reworded due to high levels of ambiguity (SD > 1).

As seen in Table 2 after the completion of round two, four questions were removed due to lack of importance (Mean < 3.5), one question was reworded due to a high level of ambiguity (SD > 1), two questions were removed due to panelist comments, and one question was split into three questions in order to allow for a more concise response.

The final survey following the CIPP model consisted of 23 questions: context = 7, Input = 3, Process = 8, Product = 5 (Table 3). Only minor modifications to the questions were made based on the participants’ written feedback. Because the minimum criteria were met after round two, the final survey was not circulated for round three.

Phase 3: Survey Results
The final survey was sent out to three groups (school employees, parents and band council) equaling a total of 8.4% of the total population of Conne River (595 individuals) and among all three groups there was a response rate of 62% (5.2% of the total population). Mean responses by category and participants are shown in Table 4. Average responses range from 1.35-2.07 indicating a high level of agreement to the questions asked in the survey. The parent group had the highest level of agreement in all categories (context = 1.70, input = 1.58, process = 1.91, and product =1.35) while the teacher group was lowest in all categories except for context (teachers =1.84, band council members = 1.87). One-way ANOVAs were used to compare the three participants groups (school employees, parents, and band council) on the four question categories (context, input, process, and product). Results showed no significant differences in the context, input, and process category at p < .05 (Context F(2,28) = .301, p = .742; Input F(2,28) = 1.168, p = .326; Process F(2,28) = 1.087, p = .351). There were significant differences found between participant groups in the product category (Product F[2,28] = 3.751, p = .036). A Bonferroni correction was applied, which showed no significant differences in the product category between band council (M = 1.52, SD = 0.438) and either other participant group, but parents (M = 1.35, SD = 0.351) answered significantly more positively than teachers (M = 1.77, SD = 0.358).

As seen in Table 5 all response range from 1.1 to 3.1 showing a high degree of agreeability. The lowest agreeability was seen when participants were asked “Would you be willing to get coaching certified in order to help coach combat sports in Conne River?” The highest agreeability scores were seen during questions of outcomes of the implementation of the program.

Question 7 was answered via a written response and answers are given in Table 6. A total of 14 school employees responded to the question, five parents responded and one band council member.

Discussion
The most important finding of the present study is that the residential stakeholders of Conne River, NL, Canada do agree that a wrestling and boxing program is needed, feasible, and would be sustainable after implementation in the community. The second finding is that a modified Delphi method was successful in creating and validating a survey tool in order to determine the first finding.

In urban and rural environments alike, new extracurricular school programs often face many challenges. For example, finding coaches for the programs can be difficult (Magnotta, 1990). When more youth want to participate in programs this requires more supervising and coaching staff, and the responsibility of these roles usually falls on school employees. Magnotta (1990) found that school employees are often not compensated for giving up their after-school time, or for the weekends during which competitions can occur. School employees feel that giving up their free time can affect preparation for classes and prevent their own leisure activities. With fewer school employees willing to take on the roles of coaches/supervisors, many schools have started hiring coaches from outside sources. However, this solution often generates another challenge; funding. Without the necessary funding to initiate the program and to increase funding as the program grows the program may fail (Parent and Harvey, 2016; Stott, 2013). In addition, if an extracurricular activity is developed and implemented by an external body, such as the government, university, or school board without proper early engagement with the school, the program is more likely to encounter challenges (Parent and Harvey, 2016). Rather than work on a top down approach Mintzberg (2006) recommends using a bottom up approach. This approach allows for external bodies and internal stakeholders to communicate and build the program to fit the needs of the stakeholders. In order to have a successful and sustainable program it is important to have coaches, volunteers and funding set up prior to implementation and it is vital to have the community involved during the creation, implementation and evaluation of the program (Secret et al., 2011). Thus in order to ensure participation early on from stakeholders and to create a successful and sustainable program a bottom up approach was used in this study. We have used a Utilization-Focused Evaluation (UFE) and the elements of Context, Input, Process, Product (CIPP) to develop the program in a way that was best fitting with the community. To achieve this the stakeholders of the program located in the community of Conne River, NL were asked what they want, how it will fit in with the local context, what resources and processes will be required and if they were available, and finally, what measures of success would help determine if the program matches the vision the community had of the program.

All three groups surveyed had a high level of agreement in all four categories; context, input, process and product. This suggests the survey responders believe the wrestling and boxing program will be sustainable and are willing to help make this happen. There were no differences in the three categories of responders in how they perceived the best way to build and implement the program. The parent group had the highest level of agreement in all categories and comments g iven about the program from parents were positive. A significant difference found between school employees’ and parents’ responses in the product category could be linked to the fact that many school employees have been shown to have little interest in coaching or teaching after school activities. As mentioned previously Magnotta (1990) suggests that this is due to three reasons, a lack of remuneration for additional hours worked, teachers not wanting to give up their weekends to attend competitions, and also many teachers do not feel they have the skills to coach or teach an extracurricular activity. Thus a “train the trainer” model would be best to instruct volunteers/coaches from the community who may or may not work in the school system but still are stakeholders (such as parents of youth attending the school) on how to coach wrestling and boxing, and would initiate change while simultaneously building both capacity and sustainability (Orfaly et al., 2005).

Thematic analyses of free text comments (Question 7) revealed a number of possible local barriers to the success of the program. The major issues brought up by the individuals surveyed were: youth interest, lack of volunteers/coaches and funding. This is line with the views of Magnotta (1990) and Parent and Harvey (2016), who discuss the importance of having several volunteers and sustainable funding prior to implementation. Also, if the program is not run during regular intervals, youth will lose interest due to a lack of consistency and reliability (Roth and Brooks-gunn, 2003).

The second main finding of this study was that the modified Delphi method allowed the authors to develop a tool for gaining stakeholder input that was accurate and had been reviewed by experts, thus creating a far superior tool than was created by the authors themselves (Hsu and Sanford, 2007). The panel of experts examined the survey through the modified Delphi method to refine the survey and make it concise and easily understood. The removal of seven questions made the survey more easily understood and geared the survey to the important issues surrounding the program.

Limitations
Limitations to this survey first include only surveying 5.2% of the total populations due to limiting our participants based on being part of three cohorts (parents, school employees and band council members). These three cohorts had a response rate of 62%. The information that could have been gained by the remaining 38% may have been useful in helping determine a more accurate picture of what exactly the community of Conne River NL wanted/needed. It is also important to stress that these results cannot be generalized for any rural or aboriginal community in Newfoundland and Labrador. Each community has its own specific needs and these needs should be examined before making any assumptions of what the community requires or is interested in receiving. However, the model used in this study is transferable to other communities as long as the specific needs of those communities are the ones being examined and the survey is adapted to those needs.

Secondly, the use of a modified Delphi Method may have allowed for bias to be introduced into the initial survey given to the panel. There were also only two rounds of review conducted by the panel. While literature suggests three rounds is optimal, due to time constraints and an agreement across the panel for all questions the Delphi process was stopped after only two rounds (Goel et al., 2013).

Conclusion
Based on the information gained in this study, an extracurricular wrestling and boxing program is perceived as acceptable, feasible and sustainable by the members of the community of Conne River, NL, Canada and the community of Conne River, NL, Canada has the resources and motivation to make the program sustainable. Through the survey which was successfully created via a modified Delphi Method, the community demonstrated their understanding of what would be required to implement and sustain the wrestling and boxing program in their community and suggests they are willing to do what it takes to keep the program sustainable. This understanding and willingness within the community should allow implementers from outside and inside the community to avoid the pitfalls that cause many programs to fail during the early stages of implementation.

References

Barber, B., Eccles, J., & Stone, M. (2001). Whatever happened to the jock, the brain, and the princess? Young adult pathways linked to adolescent activity involvement and social identity. Journal of Adolescent Research, 16: 429–455.

Bohnert, A., & Garber, J. (2007). Prospective relations between organized activity participation and psychopathology during adolescence. Journal of Abnormal Child Psychology, 35: 1021–1033.

Broh, B. (2002). “Linking extracurricular programming to academic achievement: Who benefits and why?” Sociology of Education 75(1): 69-91.

Busseri, A., Rose-Krasnor, L., Willoughby, T., & Chalmers, H. (2006). A longitudinal examination of breadth and intensity of youth activity involvement and successful development. Developmental Psychology, 42: 1313–1326.

Channon, A. (2014). Towards the "Undoing" of gender in mixed-sex martial arts and combat sports. Societies, 4(4): 587-605.

Clark, W. (2008). “Kids’ sports”. Statistics Canada — Catalogue No. 11-008: 54-61, http://www.statcan.gc.ca/pub/11-008-x/2008001/article/10573-eng.pdf, accessed on April 23, 2016.

Clayton M. (1997). Delphi: a technique to harness expert opinion for critical decision-making tasks in education. Educ Psychol 17 (4): 373–86.

Crocker, J., & Wolfe, C. (2001). Contingencies of self-worth. Psychological Review, 108: 593–623.

Daniels, E., & Leaper, C. (2006). A longitudinal investigation of sport participation, peer acceptance, and self-esteem among adolescent girls and boys. Sex Roles, 55: 875–880.

Darling, N. (2005). “Participation in extracurricular activities and adolescent adjustment: cross-sectional and longitudinal findings.” Journal of Youth and Adolescence 34(5): 493-505.

Dotterer, A., McHale, S., & Crouter, A. (2007). Implications of out-of-school activities for school engagement in African American adolescents. Journal of Youth and Adolescence, 36: 391–401.

Eccles, J., Barber, B., Stone, M., & Hunt, J. (2003). “Extracurricular activities and adolescent development.” Journal of Social Issues 59(4): 865-889.

Erkut, S., & Tracy, A. (2002). Predicting adolescent self-esteem from participation in school sports among Latino subgroups. Hispanic Journal of Behavioral Sciences, 24: 409–429.

Feldman, A., & Matjasko, J. (2005). The role of school-based extracurricular activities in adolescent development: A comprehensive review and future directions. Review of Educational Research, 75: 159–210.

Gardner, M., Roth, J., & Brooks-Gunn, J. (2008). “Adolescents’ participation in organized activities and developmental success 2 and 8 years after high school: Do sponsorship, duration, and intensity matter?” Developmental Psychology 44(3): 814-30.

Gilman, R., Myers, J., & Perez, L. (2004). Structured extracurricular activities among adolescents: Findings and implications for school psychologists. Psychology in the Schools, 41: 31–41.

Goel, S., Kumar, R., Lal, P., Sharma, D., & Singh, R. (2013). Refining compliance surveys to measure the smokefree status of jurisdictions using the Delphi method. Public Health Action, 3(4): 342-5.

Graham, B., Regehr, G., & Wright, J. (2003). Delphi as a method to establish consensus for diagnostic criteria. J Clin Epidemiol 56(12):1150–6.

Guèvremont, A., Findlay L., & Kohen, D. (2008). “Organized extracurricular activities of Canadian children and youth.” Component of Statistics Canada Catalogue No. 82003-X, Health Reports. http://www.statcan.gc.ca/pub/82-003-x/2008003/article/10679-eng.pdf accessed on April 23rd, 2016.

Hakan, & Seval. (2011). CIPP evaluation model scale: Development, reliability and validity. Procedia - Social and Behavioral Sciences, 15: 592-599.

Hanson, C., Nabavi, D., & Yuen, H. (2001). The effect of sports on level of community integration as reported by persons with spinal cord injury. The American Journal of Occupational Therapy: Official Publication of the American Occupational Therapy Association, 55(3): 332-8.

Hayhurst, L. (2014). The ‘Girl Effect’ and martial arts: Social entrepreneurship and sport, gender and development in Uganda. Gender, Place & Culture, 21(3): 297-315.

Hoefer, W., McKenzie, T., Sallis, J., Marshall, S., & Conway, T. (2001). “Parental provision of transportation for adolescent physical activity.” American Journal of Preventive Medicine 21(1): 48–51.

Holland, A., & Andre, T. (1994). Athletic participation and the social status of adolescent males and females. Youth and Society, 25: 388–407.

Holland, A. & T. Andre .(1987). “Participation in extracurricular activities in secondary school: What is known, what needs to be known?” Review of Educational Research 57(4): 437-466.

Hsu, C., & Sandford, B. (2007). The Delphi technique: making sense of consensus. J Pract Assess Res Eval 2007;12 (10):1–8.

Humbert, M., Chad, K., Spink, K., Muhajarine, N., Anderson, K., Bruner, M., Mark., Girolami, T., Odnokon, P., & Gryba, C. (2006). Factors that influence physical activity participation among high- and low-SES youth. Qualitative Health Research, 16(4): 467-83.

Huebner, A., & Mancini, J. (2003). “Shaping structured out-of-school time use among youth: The effects of self, family, and friend systems.” Journal of Youth and Adolescence 32(6): 453-463.

Kamble A., & Wangwad V. (2015). Relationship between socio-economic statuses and performance on wrestling players in Solapur University. Indian Streams Research Journal, 5(4): 1-6.

Kort-Butler, Lisa A., & Hagewen, Kellie J. (2011). School-Based Extracurricular Activity Involvement and Adolescent Self-Esteem: A Growth-Curve Analysis. Journal of Youth and Adolescence, 40(5): 568-581.

Kremarik, F. (2000). “A family affair: Children’s participation in sports.” Canadian Social Trends No. 58: 20-24. Retrieved on April 23rd, 2016 http://www.statcan.ca/english/freepub/11-008-XIE/2000002/articles/5166.pdf

Lambe, P., & Bristow, D. (2010). What are the most important non-academic attributes of good doctors? A Delphi survey of clinicians. Med Teach 32 (8): e347–54.

Lareau, A. (2003). Unequal childhoods: Class, Race, and Family Life. Berkeley, CA: University of California Press.

Lerner, R. M. (2005). “Foreword: Promoting positive youth development through community and after school programs.” In J. L. Mahoney, R. W. Larson, & J. Magnotta, J. (1990). Why Teachers Won't Coach. Strategies, 3(5): 19-22.

Mahoney, J., Cairns, B., & Farmer, T.(2003). “Promoting interpersonal competence and educational success through extracurricular activity participation.” Journal of Educational Psychology 95(2): 409-418.

Mahoney, J. (2000). “School extracurricular activities participation as a moderator in the development of antisocial patterns.” Child Development 71(2): 502-516.

Mahoney, J. & Cairns, R., (1997). “Do Extracurricular Activities Protect Against Early School Dropout?” Developmental Psychology 33(2): 241-253.

Marsh, H., & Kleitman, S. (2002). Extracurricular school activities: The good, the bad, and the nonlinear. Harvard Educational Review, 72: 464–514.

McNeal, R. (1995). “Extracurricular activities and high school dropouts.” Sociology of Education 68(1): 62-80.

Mintzberg, H. (2006). Developing leaders? Developing countries? Development in Practice, 16(1): 4-14.

Morgan, P., Lam-McCulloch, J., Herold-McIlroy, J., & Tarshis, J. (2007). Simulation performance checklist generation using the Delphi technique. Can J Anaesth 54(12): 992–7.

Newfoundland and Labrador Poverty Reduction Strategy (2014). Department of advanced education and skills

Offord, D., Lipman, E., & Duku, E. (1998). Sports, the arts and community programs: Rates and correlates of participation. Applied Research Branch Strategic Policy Human Resources Development Canada.

Oh, H., Yoshino, A., Rana, S., Lee, M., & Hovatter, R. (2015). Physical Activity Level and Well-Being: Youth Living in Rural Areas. Research Quarterly For Exercise And Sport, 86: A57-A58.

Orfaly, R., Frances, J., Campbell, P., Whittemore, B., Joly, B, & Koh, H. (2005). Train-the-trainer as an educational model in public health preparedness. Journal of Public Health Management and Practice, 11(6): S123.

Ousley, C., Shuford, R., & Roberts, T. (2013). How to incorporate self-defense instruction into physical activity programs. Strategies, 26(3): 25-28.

Patton, M. (n.d.). A utilization-focused approach to contribution analysis. Evaluation, 18(3):364-377.

Pettersson, S., Pipping, M., & Berg, C. (2013). Practices of weight regulation among elite athletes in combat sports: A matter of mental advantage? Journal of Athletic Training, 48(1): 99.

Poverty Reduction Strategy Progress Report (2014). Newfoundland and Labrador centre for health information.

Rachele, J., Cuddihy, T., Washington, T., & McPhail, S. (2014). The association between adolescent self-reported physical activity and wellness: The missing piece for youth wellness programs. Journal of Adolescent Health, 55(2): 281-286.

Roth, J., & Brooks-gunn, J. (2003). Youth development programs: Risk, prevention and policy. Journal of Adolescent Health, 32(3): 170-182.

Scavone, B., Sproviero, M., McCarthy, J., Wong, C., Sullivan, J., Siddal,l V., & Wade, L. (2006). Development of an objective scoring system for measurement of resident performance on the human patient simulator. Anesthesiology 105(2): 260–6.

Secret, M., Abell, M., & Berlin, T. (2011). The Promise and Challenge of Practice-Research Collaborations: Guiding Principles and Strategies for Initiating, Designing, and Implementing Program Evaluation Research. Social Work, 56(1): 9-20.

Sibold, J., Edwards, E., Murray-Close, D., & Hudziak, J. (2015). Physical activity, sadness, and suicidality in bullied US Adolescents. Journal of the American Academy of Child & Adolescent Psychiatry, 54(10): 808-815.

Silvestri, L. (1997). Benefits of physical activity. Perceptual and Motor Skills, 84(3 Pt 1): 890.

Simpkins, S.D., Ripke, M., Huston, A., and Eccles, J., (2005). “Predicting participation and outcomes in out-of-school activities: Similarities and differences across social ecologies.” New Directions for Youth Development 105: 51-69.

Stufflebeam, D. (1983). The CIPP Model for Program Evaluation. Evaluation in Education and Human Services, 6: 117-141

Terry, P., Hahn, A., & Simjanovic, M. (2014). Effects of a sport program (Box'Tag ®) on disadvantaged youth participants. International Journal of Sport and Exercise Psychology, 12(3): 258-272.

Thompson, M. (2009). Considering the implication of variations within Delphi research. Fam Pract 26(5):420–4.

Tracy, A., & Erkut, S. (2002). Gender and race patterns in the pathways from sports participation to self-esteem. Sociological Perspectives, 45: 445–466.

Trzesniewski, K., Donnellan, M., & Robins, R. (2003). Stability of self-esteem across the life span. Journal of Personality and Social Psychology, 84: 205–220.

UNICEF Report Card 13: Fairness for Children. (2016).

Velija, P., Mierzwinski, M., & Fortune, L. (2013). ‘It made me feel powerful’ : Women’s gendered embodiment and physical empowerment in the martial arts. Leisure Studies, 32(5): 524-541.

Warner, S., & Dixon, M. (2013). Sports and Community on Campus: Constructing a Sports Experience that Matters. Journal of College Student Development, 54(3): 283-298.

Weiss, M., & Wiese-Bjornstal, M. (2009). Promoting positive youth development through physical activity. President's Council on Physical Fitness and Sports Research Digest, 10(3): 1–8.

Wen, M., Browning, C., & Cagney, K. (2007). “Neighbourhood deprivation, social capital and regular exercise during adulthood: A multilevel study in Chicago.” Urban Studies 44(13): 2651-2671.

White, A. & Gager, C. (2007). “Idle hands and empty pockets? Youth involvement in extracurricular activities, social capital, and economic status.” Youth & Society 39(1): 75-111.

Wimer, C., Simpkins, S., Dearing, E., Bouffard, S., Caronongan, P., & Weiss, H. (2008). “Predicting youth out-of-school time participation: Multiple risks and developmental differences.” Merrill-Palmer Quarterly 54(2): 179-207.

Woodward, T. (2009). A review of the effects of martial arts practice on health. WMJ: Official Publication of the State Medical Society of Wisconsin, 108(1): 40-3Ω

Xu, L., Gauthier, A., & Strohschein, L. (2009). Why are some Children Left Out? Factors Barring Canadian Children from Participating in Extracurricular Activities. Canadian Studies in Population, 36(3/4): 325.

Zaff, J., Moore, R., Papillo, A., & Williams, S. (2003). “Implication of extracurricular activities participation during adolescence on positive outcomes.” Journal of Adolescent Research 18(6): 599-630.