KnowledgePoints

Educational Management System

Results for an expanding program: 2005

Jerry Litzenberger, PhD - Litzenberger Consulting

Executive Summary

When Congress passed the Elementary and Secondary Education Act (ESEA) also known as No Child Left Behind, the standards for effective educational programs were raised dramatically. Programs are now asked to demonstrate overall improvements in the academic performance of program participants and to ensure equity in that performance. The evaluation of the KnowledgePoints program as presented in this document was designed to determine whether this program meets these high standards of excellence.

In the first of these evaluations (2001), testing information was gathered from tutoring centers that had been in operation for more than one year. These centers were the first to implement this new tutorial program and served as pilots for a franchise model intended to have a nationwide impact. The subsequent evaluations conducted in 2003 and 2005 were conducted in centers across the nation to confirm the findings from the first evaluation and to determine whether this program could be disseminated nationwide with effectiveness equal to the results found in 2001. Each study conducted consequently produced increasing results.

Upon entering the KnowledgePoints program a pretest is administered as part of the diagnosis and intake processing for each student. At the completion of 40 tutoring sessions a progress test is administered to measure a student’s growth and improvement. The evaluation used the data gathered from the reading and math subtests of the California Achievement Test (Edition 5), in order to determine whether the goals of equity and excellence required by ESEA had been met.

A series of evaluation perspectives were analyzed. These perspectives included analyses of improvements in performance from pre to progress testing for all students in reading and math and additional analyses that addressed the following questions:

• Have all students improved in the basic skills areas of reading and math?
• Have the gains of students been equally shared by male and female students?
• Has the program been effective for students of various ethnicities?
• Has the equity gap been narrowed
• Is the program effective for students in each age group and at all grade levels?

Program Effectiveness 2005

Table 1 represents the pre and progress test scores for students included in this program evaluation. Both reading (N=834) and math (N=349) showed gains that were extremely significant over these 40 sessions of tutoring. Gains of 18 NCE’s in vocabulary, 20 in comprehension are exceptional scores when compared to other remedial education programs and represent gains beyond the .001 level of significance. Similar gains were achieved in math with increases of 24 points in computation, 19 points in concepts and applications and 24 points in total math.

As an additional comparison, program effectiveness can be claimed when NCE gains approach 5 points. National model programs are defined by gains of 10 to 15 NCE’s over a period of one school year. Using these reference points for program excellence, the gains achieved by the centers for KnowledgePoints are truly remarkable. These results document the effectiveness of the KnowledgePoints program and the success at disseminating this instructional model to centers across the nation.

TABLE 1

2005 NCE Gains by Subject for all Students

Equity Analysis 2005

One major challenge to education in the twenty-first century is to develop programs that are not only effective, but equitable. The lack of equity is evident in national tests of basic skills, in preparation of students for college admissions tests, in the proportion of students who drop out of school and nearly all other achievement measures. Programs that can narrow these achievement gaps add a new dimension to the measurement of effectiveness.

To determine the impact of this program on minority students, disaggregations of all scores were conducted and results are presented in Table 2. While some differences in gains are apparent from reviewing this information, it is also apparent that all comparison groups made significant gains. In math, significant differences existed between majority and minority groups on the pre-test, and these differences were also observed on progress test measurements. This suggests the program was effective in producing gains for all students.

TABLE 2

2005 NCE Gains by Gender & Ethnic Group

Grade Level Analysis 2005

The final evaluation perspective that was completed was to determine whether the program was effective for students of all ages and grade levels. This could help determine whether the instructional tools might need revisions for some age groups.

Reading comprehension gain scores for each grade level ranged from a 32 NCE point gain for students in 1st grade to a 10 point gain for 9th grade students. There was no trend of increasing or decreasing gains evident across grades. This indicates the program worked well for students in all the grades served.

In mathematics, students in each grade level achieved gains of 12 NCE points or more. There was some trend evident with students in early elementary grades (2nd—4th) achieving gains of more than 30 points and gains for students enrolled in higher grades achieving 12 to 20 NCE gains. These represent excellent gains using any of the standards presented above. This means that in every grade students enjoyed an acceleration of learning at a level that meets the criterion for consideration as a national model program.

TABLE 3

2005 NCE Gains by Grade

Evaluation Summary

The tutorial program developed by KnowledgePoints has now achieved nationwide dissemination evaluation information from nearly 100 sites across the United States. The purpose of this evaluation was to determine whether the consistency of the program implemented in these centers has produced the high quality educational gains that were found in earlier research based on smaller numbers of centers. The gain scores achieved by the approximately 1200 students completing 40 lessons in these centers demonstrates that the KnowledgePoints program is effective from each of the evaluation perspectives analyzed in this study. As the number of centers have expanded, additional achievement information has become available. Other findings indicate confidence in results improves as the number of students increase. In all respects, this program is effective at addressing overall improvements in academic achievement for participants and in successfully addressing the equity issues that confront our national education system.