Research Report September 2000

Executive Summary

Academic achievement in schools is currently one of Florida's most important social and fiscal policy issues. Education is seen as the one of the most important factors enabling Floridians to thrive in an information age and is considered vital for the economic and democratic future of our state and nation. However, parents, community leaders and policymakers remain concerned that many of Florida's children are not receiving the type of classroom instruction that will enable them to thrive in today's information society. To help to get to the bottom of the issue, Florida TaxWatch undertook a controlled, comparative evaluation of Project CHILD, an innovative computer-integrated instructional program designed to increase academic achievement.

Developed in 1988 at Florida State University by Dr. Sally Butzin and currently under her executive directorship of the Institute for School Innovation, Project CHILD is comprised of fully developed instructional materials, correlated with state standards and intensive training provided by Institute for School Innovation staff and certified consultants. This is designed to enable Project CHILD teachers to transform the text-dominated traditional classroom into multi-dimensional learning stations.

The evaluation of Project CHILD by Florida TaxWatch was both "process" and "outcome" oriented and was composed of five components. The first component was a review of previous research conducted on Project CHILD. The eight years of research reviewed by TaxWatch suggests that Project CHILD students attain higher achievement levels as measured by report card grades and standardized tests. For example, a 1995 longitudinal study by FSU's Department of Education indicates that Project CHILD students had better grade point averages on mid-semester report cards and had higher test scores than their peers on the comprehensive Test of Basic Skills. Appendix A of this report consists of a list of the studies independently reviewed by Florida TaxWatch.

The second component of the evaluation was undertaken by the Institute for school innovation staff and focused on the actual implementation process of Project CHILD. It suggests that all three research schools (John D Floyd Elementary in Hernando County, Tedder Elementary in Broward County, and Windy Hill Elementary in Duval County) have, to varying degrees, implemented the Twenty Essential Components of the Project CHILD program. John D Floyd Elementary most fully implemented the program, with the others making a good faith effort at implementation. The success at John D Floyd was due mainly to the work of the Project CHILD coordinator who put in significant time and effort to assist the Project CHILD teachers, the results of which are indicated in both the improved implementation process and the higher test scores of her school's Project CHILD students.

The third component of the evaluation focused on a qualitative assessment of the academic achievements of students in Project CHILD classes. It included anecdotal evidence from Project CHILD principals, teachers, students and parents regarding the way in which Project CHILD has enhanced the educational experiences of students. The anecdotal data included testimony from Tedder Elementary's "Teacher of the Year," Shonner Gainer, who stated that the students in her Project CHILD classes have shown tremendous academic progress as measured by the Stanford Achievement Test Version 9 (SAT 9) scores, while the growth of the other students at her school using the same measures have been insignificant.

The fourth component of the evaluation focused on a cost-effectiveness analysis of the implementation of Project CHILD. It suggests that, if fully implemented statewide, the Project CHILD model with class size 30 students currently mandated by the Florida Legislature could accrue an annual cost saving or cost avoidance of more than $4 billion over the traditional model of class size 20.

The final component was the class size comparative statistical evaluation. The three Project CHILD research schools were compared to three demographically similar schools with smaller class sizes. Comparisons were based on Florida Comprehensive Achievement Test (FCAT), the SAT 9 test and the Norm Reference Test (NRT). Independent t-Test and the Mann Whitney non-parametric statistical tests were used to analyze whether there was a statistically significant difference between the test scores from the Project CHILD schools and the comparison schools with smaller class sizes. The results indicate that classes at two of the three Project CHILD schools scored statistically significantly better than their counterparts at traditional schools with smaller class sizes.

These results will be bolstered as more data are gathered over subsequent years; however, they are a testament to Project CHILD taking into consideration that this was the first year of its implementation at the schools.

Overall, the evaluation suggests that Project CHILD programs have made a positive difference according to some of its teachers, students, parents and administrators and have the potential to generate significant taxpayer cost savings if expanded. Project CHILD also appears to make a difference in improving academic results, as indicated by statistically significant higher test scores in two of the three Project CHILD schools as compared to their counterparts at traditional schools with smaller class sizes. This is especially true in schools such as John D. Floyd Elementary where the technology and the Project CHILD coordinators fully complemented the Project CHILD teachers.

Recommendations

Additional technology training should be provided for the para-professionals who assist the teachers in Project CHILD classes. The para-professionals have asked for training from beginner to advanced on all hardware and software in the classrooms. They want to be able to efficiently assist the students on the available software titles and be able to perform simple troubleshooting. Training should be ongoing through Year Two for all five sites.

Both on-site coaches and para-professionals should be funded full-time in year two. The teachers requested that their para-professionals be funded all day. This will allow for assistance as the teachers work with larger class sizes. The para-professionals worked for an average of 3 « hours a day at Windy Hill and Tedder, and the teachers found they also needed help to successfully continue with instruction in the afternoon. It also was not possible this first year to collect the necessary implementation data from coaches. They had too many other responsibilities which kept them from fully monitoring the degree of implementation. As full funding is provided for site coaches, the Institute should require that all required reports be received in a timely manner as outlined in each school's grant agreement.

The Institute for School Innovation should build upon the successful strategies of John D. Floyd Elementary's on-site coach. The steady increase in the percentage of both Teacher Collaboration Components and Instructional Design Components was a direct consequence of the coach's hands-on involvement [Because, after the first semester the coach was concerned with the low amount of reports turned in, each quarter a more efficient way of monitoring and tracking was instituted to assure their being submitted on time.].

An effort should be made to gather substantive input from Project Child teachers on the implementation of Project CHILD and their recommendations for improvement. This should be done quantitatively (through the implementation of an annual survey) and quantitatively ( by asking teachers to provide artifacts documenting innovative results that they may have had with Project CHILD through the course of the year). An effort should be made to institutionalize this process for continuous improvement through the use of a rudimentary management information system.

An effort should be made to gather longitudinal data as well as cross-sectional data over the next two years. It should build upon this year's documentation of results by both comparing it to next year's and the third year and determining whether a trend exists in terms of growth. It would be prudent to pilot test this method at John D. Floyd first, since it has the lowest mobility rates of the three research schools included so far. High mobility rates are one of the biggest impediments to longitudinal research.

The process of institutionalizing the evaluation process at the Project CHILD schools also should begin in the second year. An effort should be made to train Project CHILD teachers, on-site coaches and para-professionals in collecting, inputting and analyzing the data used in this research for their own schools. This will enable them to track the growth of their students and also provide them with a tangible indicator of the results that they are achieving. A training schedule should be put in place and Statistical Package for the Social Sciences (SPSS) should be purchased for use in each of the schools.

Introduction

Academic achievement in schools is currently one of Florida's most important social and fiscal policy issues. Education is seen as the most important factor in enabling Floridians to thrive in an information age and is considered vital for the economic and democratic future of the state and nation. However, parents, community leaders and policymakers remain concerned that many of Florida's children are not receiving the type of classroom instruction that will enable them to succeed in today's society. To get to the bottom of the issue, Florida TaxWatch is presently undertaking a multi-phased, controlled, comparative evaluation of Project CHILD, an innovative computer-integrated instructional program designed to help to correct this situation.

Developed in 1988 at Florida State University by Dr. Sally Butzin and currently under her executive directorship at the Institute for School Innovation, Project CHILD has received numerous awards and has been recognized as an effective program by such groups as the National Diffusion Network. The Project CHILD model is designed to enable elementary schools (principally grades __ to __)to integrate technology into their reading, language arts, and mathematics curriculum. It incorporates a variety of school improvement strategies to improve student instruction and student learning: brain-based research, cooperative learning, continuous progress instruction, authentic assessment, and hands-on active learning. Fully developed instructional materials are correlated with state standards and intensive training provided by the Institute for School Innovation staff. Certified consultants enable teachers to transform the text- dominated traditional classroom into multi-dimensional learning stations.

Florida TaxWatch's evaluation is using both expert opinion and statistical analyzes to assess the efficiency and effectiveness of the Project CHILD program. The study is designed to determine the effects of class size on academic achievement in reading, writing and mathematics. It also assesses the cost-effectiveness of the implementation of Project CHILD in elementary schools in Florida. The evaluation was applied to at least three diverse and geographically dispersed elementary schools throughout the state. The final draft of the evaluation results were submitted September 29, 2000.

Purpose

The purpose of this evaluation is to determine what effect participation in the Project Child program has on student learning, even in classrooms that are typically larger (30 or more) than traditional elementary school classes. The study also evaluates the potential cost- effectiveness of statewide implementation of Project Child. This is achieved using quantitative data collection and analysis methods including parametric, nonparametric and cost-effectiveness methods, as well as qualitative anecdotal data from parents, teachers, students and administrators regarding their experiences with Project CHILD.

Research Design

The Florida Legislature provided funding in FY 1999-2000 to the Institute for School Innovation to design and implement a research study utilizing the Project CHILD model to determine the effect that participation in Project child classes have on academic achievement even in classrooms of 30 or more. It was conducted by Florida TaxWatch using students in three diverse and geographically dispersed elementary schools throughout the state of Florida.

The study is comprised of five components:
1) The first component is a review of previous, similar, independent research conducted on Project CHILD.
2) The second component is an analysis of the checklist developed by Project CHILD to determine the degree of implementation of the Project CHILD Essential Components in each of the three research schools during the 1999-2000 school year.
3) The third component of the evaluation focuses on a qualitative assessment of the academic achievements of students in Project CHILD classes. It includes anecdotal evidence from Project CHILD principals, teachers, students and parents regarding the way in which Project CHILD has enhanced the educational experiences of students. This was completed by the Institute for School Innovation Staff.
4) The fourth component is an examination of the cost-effectiveness of the enhanced Project Child model in increasing academic achievement as compared to traditional classroom teaching models with fewer students.
5) The fifth component includes a quantitative assessment of academic achievement (as measured by standardized test scores) of students in Project CHILD classes compared to the academic achievements of comparable students in traditional classes. It compares student scores on the Florida Comprehensive Achievement Test (FCAT), the Stanford Achievement Test Version 9 (SAT 9) and the Norm Referenced Test from Project CHILD classes of 28 or more to student scores from analogous, traditional non-Project CHILD schools of 25 students or less. The Independent Samples t-Test and the Mann Whitney non-parametric test for comparing two populations were used in this analysis.

Methodology

Current researchers in the social sciences tend to agree on two points: (1) the best research design is one that uses several different research methods (a triangulation of methods); and, (2) qualitative and quantitative methods are complimentary and both should be used as appropriate. Since each research method has its particular limitations and weaknesses, the use of multiple methods can compensate for potential blind spots that could occur if only a single method is used. Moreover, if findings from different methods support one another, more sound and dependable conclusions can be drawn. On the other hand, relying upon an individual research method to find answers to a research question might be dangerous, because it is possible that the findings might be an artifact of the research method used.

Accordingly, multiple methods (and multiple data sources) were used in Florida TaxWatch's investigation of the impact of Project CHILD on classroom achievement:

I. Previous Analysis

Previous evaluations of Project CHILD were analyzed for two purposes. First, these assisted in the current analysis of the implementation of Project CHILD in Florida Schools. Second, the previous studies' methodologies were analyzed to seek methodological improvement for the current analysis. Based on this, it was decided to use both the Independent-Samples t Test and the Mann Whitney non-parametric test to compare the two populations of students.

A variety of sources were analyzed and are referenced in Appendix A. They include studies done by Project Child researchers themselves as well as studies done by independent researchers.

II. Analysis of Current Implementation

This section of the report includes issues related to implementation of the program during the first year and was completed by Institute for School Innovation staff. The first year of the research study was designed to develop and refine monitoring instruments to gauge the degree of program implementation. The first year also involved developing procedures to train new teachers and classroom assistants, recruit an additional school to participate in the research, and begin collaboration activities between the research schools. Second, checklists were used to determine the degree of implementation of the Project CHILD Essential Components in each of the research schools during the 1999-2000 school year. Staff designated as Project CHILD site/school coordinators and the Institute for School Innovation Staff provided the appropriate assistance in completing such checklists and wrote the summary that was included in this report.

III. Qualitative Analysis

This section of the report contains anecdotal evidence of the improvements made by Project CHILD as seen through the eyes of principals, students, teachers and parents of Project CHILD schools.

IV. Cost Effectiveness Analysis

This section contains a brief spreadsheet analysis of potential statewide cost savings associated with the Project CHILD program based upon data provided by the Florida Department of Education. The first spreadsheet analysis compares all K-5 students in a classroom size of 20 to the Enhanced Project CHILD model of 30 students per classroom. The enhanced application included one classroom assistant and one FTE teacher involved with 30 students per class. The second spreadsheet analysis is of potential cost savings for the Project CHILD Model of 22 students per classroom (excludes enhancements) as compared to the classroom of 20 students.

IV. Class Size Analysis

1) School Selection/Student Population Selection - The three schools selected for participation in Phase 1 of the research project by the Institute for School Innovation were:

a) Exemplary Schools (schools with school-wide implementation of the Project CHILD model):
Tedder Elementary School, Pompano Beach (Broward County) - urban, school wide Title I, 84% minority, 84% free/reduced lunch.
Floyd Elementary School, Spring Hill (Hernando County) - suburban, 12% minority, 39% Free/reduced lunch.

b) Evolving School (implements one-two full "strands", K-5 classes)
Windy Hill Elementary, Jacksonville (Duval County) - urban, Title I inclusion, 36% minority, 54% free/reduced lunch.

2) Selection of Comparison Schools - Researchers from Florida TaxWatch, with the assistance of school district officials in each of the school districts, Project CHILD school principals and on site coaches and staff in the Department of Education, selected comparison schools with similar characteristics and demographics for the evaluation. Indicators such as urban/rural location, inclusion of Title I Program, percent of minority and ESOL population, percent of free and reduced lunch and mobility rates were used as the basis for the selection of the comparison schools. Availability of traditional classrooms with appropriate class size ranges was also a factor in selection of the non-Project CHILD schools ¹.

3) Data Selection/Collection/Analysis - Standardized test scores were used to compare academic achievement in reading, writing and math in those grades where such test scores were available. No test scores were available for kindergarten, so it was not a school evaluation category. SAT 9 standardized test scores were used as the basis for comparison in Grades 1 & 2. FCAT and NRT scores were used as the basis for comparison for Grades 3-5.

The Data Analysis component of this study used the Independent-Samples t Test and the Mann Whitney non-parametric test to compare two populations of students (the Project CHILD classes and the classes from the comparison schools). The Independent Samples t- test procedure compares means for two groups of cases (in this case, the Project CHILD classes and the comparison school classes). Ideally, for this test, the subjects should be randomly assigned to two groups so that any difference is due to the treatment (in this case, the exposure to Project CHILD) and not to other factors. As this was not feasible, care was taken to select comparison schools that were demographically similar to the Project CHILD schools in order to avoid bias as much as possible. However, no claims are made that the study totally avoids such bias.

Average standardized test scores for enhanced Project CHILD classes were compared to average scores from traditional classes at grades 1-5 levels in the areas of reading, writing and mathematics. The objective of the comparison of mean scores was to determine whether or not there was a statistically significant difference between the mean test scores of the enhanced Project CHILD classes and comparison classes.

The Mann Whitney non-parametric Test is the most commonly used alternative to the independent samples t-test when the sample size is too small to assume normality. It was used in this case to both bolster validity of the results from the Independent t-Tests and in cases where only one class was compared to another; therefore, the sample size was too small for the proper use of the Independent t-Test.

The null hypothesis being tested in this study is that there is no difference between the average scores for 1-5 grade levels at the Project Child schools and the comparison schools despite differences in class size. In other words, the study assumes that the differences between the average test scores at the Project CHILD schools and the comparison schools are statistically insignificant.

The research hypothesis being tested is that both the Independent Samples t-test and the Mann Whitney test will produce results which indicate a statistically significant difference between the two groups of scores. Specifically, it is hypothesized that the Project CHILD schools will score higher as a group than will the comparison schools.

V. Dissemination of Findings

Florida TaxWatch will assist the Institute for School Innovation in disseminating information about the project to the news media and the expert panel described in the research project. An extract of findings and conclusions will be posted on the Web site of Florida TaxWatch and will be published in print form for use by the Florida Legislature, Florida TaxWatch membership and other interested parties.

VI. Time lines

1. A written evaluation plan which outlines the framework and procedures for accomplishing the tasks described above no later than September 15, 1999.

2. A draft of the final report which states the findings and conclusions of the first year of the project. Such draft shall be delivered to the Institute for review and sign- off no later than September 26, 2000.

3. A final report and executive summary suitable for release to the Florida Department of Education, news media, and other interested parties shall be delivered to the Institute no later than September 28, 2000.

4. Periodic and ongoing status reports will be provided to the Institute for School Innovation for input and information as the research project develops.

Review of Previous Research on Project CHILD

Florida TaxWatch's preliminary review of previous research on Project CHILD suggests that the Program can make an important difference in improving student performance. The evaluation of previous studies by TaxWatch of student performance in larger classrooms using the Project CHILD model as compared to smaller classrooms not incorporating the model will be an important preliminary test of the program's impact on student achievement and program cost effectiveness.

The eight years of research reviewed by Florida TaxWatch suggests that Project CHILD students attain higher achievement levels as measured by report card grades and standardized tests. For example, a 1995 longitudinal study by FSU's Department of Education, indicates that Project CHILD students measure above their peers. The study involved 360 sixth grade students at 2 middle schools in Oskaloosa County, Florida. The two schools are feeder schools for nine elementary schools which have been implementing the Project CHILD program for several years. The study matched 180 students who had been in the Project CHILD program with 180 students who had been in traditional classrooms. The students were matched by socioeconomic status (SES), ethnicity, sex, and at-risk factors. The results attained were impressive.

Project CHILD students had better grade point averages on mid-semester report cards in language arts and mathematics (the curriculum focus of Project CHILD) and higher test scores on the Comprehensive Test of Basic Skills (CTBS) administered in the Spring in reading, math, and the total battery. In addition, 41.6% of the Project CHILD students were enrolled in advanced mathematics courses as compared to only 25.5% of the traditional students (See Appendix A for a list of other past studies which point to increasing academic achievement by Project CHILD students.)

Implementation Issues ²

Project CHILD is an instructional model with 20 essential components (see attachments in Appendix B of this report). The degree to which these components are implemented has been determined to impact academic achievement. The more fully the components are implemented as designed, the stronger the academic achievement of students.

The Project CHILD program has redesigned the traditional educational delivery system. The teachers work in teams of three with each teacher becoming a specialist in one of three core academic subjects, reading, writing or math. Three classrooms form a cluster. A cluster consists either of primary grades K-2 or intermediate grades 3-5. Each teacher is responsible for one grade level as his/her home-room. This group of students receive instruction in other areas such as science and social studies after they have rotated to the other two subject area classrooms for a period of one hour in each room. The students stay with this team of teachers for three years. This allows for teachers to become very familiar with each student's individual strengths and needs so they are able to work to their highest academic potential.

The first year of the research study was designed to develop and refine monitoring instruments to gauge the degree of program implementation. The first year also involved developing procedures to train new teachers and classroom assistants, recruit an additional school to participate in the research, and begin collaboration activities between the research schools.

The implementation analysis began in August 1999 at three schools throughout Florida: Tedder Elementary in Broward County, Floyd Elementary in Hernando County and Windy Hill Elementary in Duval County. Thirty six teachers twelve at each site were asked to have at least thirty children in their classrooms. The purpose of the project was to determine the outcome of student learning in Project CHILD classrooms with larger class sizes.

The average enrollment for research classrooms was 29.03 at Floyd Elementary with kindergarten averaging 25.6 students. At Tedder Elementary, the average enrollment in the twelve research classrooms was 30.02 with kindergarten, averaging 30.95 students the whole year. Windy Hill Elementary averaged 27.39 with their kindergartens averaging 25.65. The enrollment fell below the required 30 students due to the high mobility rate of the students population.

It should be noted that, after the first quarter, the teachers shared that they felt thirty in kindergarten was too much. Their administrators were told as children moved out they could consider not filling the vacated position with other students. Each school was allowed to determine class size for kindergarten but was asked not to go below 24 or 25 for research purposes.

The sites chosen varied in several ways: social economic make-up, mobility rate, and the percentage of free and reduced lunch. Two schools, Floyd and Tedder, were "Exemplary Schools" and implemented Project CHILD school wide, although only 12 of the classrooms were considered "research classrooms" with 30 or more students. Windy Hill implemented Project CHILD in 12 research classrooms and was designated as an "Evolving School" since the rest of the school remained traditional self-contained.

Two schools, Mascotte and Wilkinson, were considered "Emerging Schools" and began partial implementation in the second semester in six classrooms with their home room students. No implementation data were collected from these sites during the first year.

The Project CHILD research classrooms were enhanced with half-day teacher assistants for each classroom and monies to purchase new computers. Providing the extra staff and new computers took away two barriers classroom teachers felt would exist as we increased their class size. In addition, the three fully implementing schools received funds to support a full-time, on-site coach.

The para-professionals at Floyd Elementary were hired full-time and trained in August. The grant funded these positions half-time and the school budget covered the other half of their day. At two sites, Windy Hill and Tedder, the school budget could not cover the balance for full-time assistants, so part-time employees were hired. Both schools reported difficulty in finding qualified persons who were willing to work only half a day.

As soon as all twelve assistants were hired at these sites the project manager provided a one-day intensive training on the philosophy behind and the procedures in a Project CHILD classroom. The purpose of the training was to make the assistants a vital partner in the educational environment of the classroom.

Each on-site coach was provided with a detailed job description and a time-line for monitoring implementation of the Twenty Essential Components. The coach was expected to visit the classrooms every two weeks, fill out a monitoring report sheet and regularly send them into the Institute (see attachments).

At two sites, the coaches found it difficult to find the required time to visit the classrooms. One coach was also the assistant principal and turned in only 28 % of the required documentation. The other coach was also a resource teacher, and she turned in 27% of her required reports. Not being funded fully by the project meant they were required to perform other duties, which kept them from providing the necessary documentation to truly determine the full degree of implementation (see attached individual school data sheets).

At the third site, however, the coach had no other duties, except thirty minutes of lunch duty. She was able to provide the Institute with 60% of the required documentation. It should be noted that throughout the year she too ended up being responsible for extra duties because a portion of her salary was supplemented by the school budget.

Through her frequent classroom visits, this coach was able to offer continued support over the course of the year. Monthly meetings were set-up for disciplines to meet and share ideas for station activities. Classroom assistants and parent volunteers, under her guidance, were organized and made a minimum of thirty hands-on station activities each semester for each teacher. The teachers in their year-end survey felt their on-site coach's knowledge as an experienced Project CHILD teacher, as well as her understanding, helped them keep focused on student learning.

At the three sites the degree of reporting by the on-site coach coincides with the degree to which the teachers followed two of the components that reflected implementation. One is the Weekly Cluster Meeting, which is to be recorded on a form provided in each teacher's manual, and Observation Reports to be done in a teammate's classroom each nine weeks (see attachments).

At Floyd where the coach provided 60% of her reports the teachers turned in documentation for 68% of cluster meetings and 54% of observation reports, as Tables 1, 2 & 3 indicate. At Tedder, where the coach provided 27% of her reports, the teachers turned in documentation for 36% of cluster meetings and 30% of their observation reports. At

Windy Hill, where the coach provided 28% of the required reports, the teachers turned in 26% of their cluster meeting reports and 12.5% of their observation reports.

To determine the actual degree of implementation it would have been necessary to have received 100% of all required reports from each individual research site. As previously mentioned, the on-site coaches found that at times they were asked to perform other school duties. These other duties resulted in a low percentage of documentation being done, and, therefore, the degree of implementation was difficult to determine.

In addition to the teacher collaboration components of the CHILD implementation plan, the instructional design components require teachers to create the necessary multi-modality classroom stations. Project CHILD classroom stations should comprise 50% hands-on activities, 40% paper/pencil, and 10% computer-based.

The following tables reflect the data reported in the coaches' reports. Whether they are a true picture of classroom implementation is questionable given the inadequate reporting mentioned earlier. These data should therefore be viewed with caution.

The following is a time-line of support activities provided at the five sites by the project manager from the Institute for School Innovation:

August-September, 1999
Began to outline monitoring and reporting strategies and created the necessary forms for data collection. Trained para-professionals at Floyd and Windy Hill. Visited all research sites to assure implementation of program for first quarter.

September-October, 1999
Worked closely with sites to monitor and assist in first stages of the implementation process. Refined reporting process with on-site coaches. Participated in parent awareness meeting at Mascotte.

October-November, 1999
Had first leadership meeting with research site principals and coaches. Arranged and hosted Regional Support Days at Floyd, Tedder and Windy Hill. Worked with teacher assistants at all sites on remediation strategies. Trained teachers at Mascotte Elementary. Arranged for new Project CHILD teachers at all sites to visit another school.

December, 1999
Wilkinson selected as final school for research project. Trained new teachers at Floyd. Provided on- going support to all sites.

January-February, 2000
Visited all sites to meet with teachers and para-professionals to determine successes of first semester and needs for second semester. Trained teachers at newly selected research site: Wilkinson Elementary in Sarasota County. Held first Expert Panel Meeting at Pompano and toured Tedder with panel on January13-14. Attended Project CHILD 10th reunion in conjunction with FETC. Also, held leadership meeting with research site principals and coaches.

March-April, 2000
Visited all sites and met with individual teachers and para-professionals. Attended School Advisory and PTA meetings at Mascotte. Worked with teachers at Mascotte as they began making and setting up station activities for their students.

May-June, 2000
Visited all sites to gather fourth-quarter and summative information from principals, on-site coaches, teachers and para-professionals. Arranged and hosted Regional Support Days at Wilkinson, Panther Run, and Moton Elementary. Worked with Institute staff at FCAT workshop to organize and develop station activities, which require higher-order thinking and problem-solving skills.

The year ended with all sites having, to varying degrees, implemented the Twenty Essential Components. It appears that Floyd most fully implemented the program, with the others making a good faith effort as the first year got underway.

Tedder and Floyd experienced similar difficulties having already transformed their schools into total Project CHILD sites. They discovered that scheduling for the teachers to have common planning and lunch times for a faculty of thirty plus being quite an undertaking and requiring much forethought. After studying and planning for the needs of the primary clusters and intermediate clusters in year two, the schedule will be changed to include longer class periods for the intermediate clusters.

This first year the teachers at Windy Hill did not have their new computers purchased or in the classrooms until November, 1999. This was mainly due to the purchasing procedures required by the county and the chosen vendor. As the 2000-2001 school year begins all schools will have computers and printers ready for student use.

To determine the ongoing climate of the school, research teachers were surveyed each nine weeks. The para-professionals were surveyed at the end of the second semester. The para- professionals in their surveys said that they believed that they had truly made a difference in the lives of the students with whom they worked. Seasoned para-professionals shared comments such as, "this has been my best year out of ten as a paraprofessional" and "I'm never at a loss for things to do because of the excitement in the classroom." Overall, through regular meetings with the project manager and the on-site coach they felt as if their insight was valuable as they viewed things from a different perspective than the teachers.

The teachers ended the year with many ideas as to how to improve the performance of their students through more advanced applications at the stations. They planned on working on their own time over the summer with their team to enhance what they had done during the first year of this project.

Year one has ended with a high degree of satisfaction as evidenced by the documentation gathered by the Project Manager from teachers, para-professionals, and administrators. As year two begins, our exemplary sites will be expanding the number of classrooms in the research project. Tedder Elementary will be expanding the research classrooms from twelve to eighteen, and Floyd will be expanding from twelve to fifteen classrooms. Windy Hill will have the same number of research classrooms, but, through awareness activities during the first semester, we anticipate that other teachers will begin to implement Project CHILD as emerging classrooms.

The two emerging sites, Mascotte Elementary in Lake County and Wilkinson Elementary in Sarasota County, will each have six research classrooms as they begin to fully implement Project CHILD. At both of these sites awareness activities will be organized for faculty as well as for parents. The Project Manager will be working closely with the new on-site coaches at each emerging site, as they have no previous Project CHILD experience. It will be a focus of the Institute to provide ongoing awareness activities at all research sites in the hope that many questions will be answered as the face of education is changed one-child-at-a-time.

It should be said the first year ended very successfully, but not without a few lessons being learned by all. Based upon the lessons we learned from the first year of implementation, the following changes and enhancements will occur during year two:

Additional technology training will be provided for the para-professionals. The para-professionals have asked for training from beginner to advanced on all hardware and software in the classrooms. They want to be able to efficiently assist the students on the available software titles and be able to perform simple troubleshooting. Training will be ongoing through year two for all five sites.

Full-time para-professionals will be provided by the grant. The teachers requested that their para-professionals be funded all day. This will allow for assistance as the teachers work with the larger class sizes. The para-professionals worked for an average of 3 « hours a day at Windy Hill and Tedder, and the teachers found they also needed help to successfully continue with instruction in the afternoon.

The kindergarten classes will not be expected to begin the school year with thirty students. Each school will determine the number of students in their kindergarten classrooms. This should not affect the research as they are generally not tested.

Intermediate teachers will investigate the possibility of extending their class periods to seventy minutes. The intermediate teachers found that, to prepare their students for Florida Writes and FCAT, they need to have more time for whole group instruction before the students move to stations. This change will need to be determined by each individual research site.

The on-site coach will be funded full-time by the grant for year two. It was impossible this first year to collect the necessary data from the coaches. They had too many other responsibilities which kept them from fully monitoring the degree of implementation. As full funding is provided for the site coach, the Institute will expect 100% of all required reports to be received in a timely manner as outlined in each school's grant agreement.

The successful strategies of Floyd's on-site coach will be emulated. The steady increase in the percentage of both Teacher Collaboration Components and Instructional Design Components was a direct result of the coach's hands-on involvement. After the first semester she was concerned with the low number of reports turned in to her, so each quarter she instituted a more efficient way of monitoring and tracking to assure adherence to these components.

The fourth quarter shows a marked improvement from the first. A successful method of gathering the documentation was accomplished by putting the necessary forms in the hands of the teachers and requiring them to be turned in by a specific date. If the teachers failed to meet the deadline another form was given. These strategies are very replicable and will be shared with the other on-site coaches as a successful method to be initiated during the second phase of the research project:

Class size will be closely monitored in grades one through five, and, if the enrollment drops below thirty at any site, the administrator will be expected to add students in an expedient manner as the school population increases.

A kickoff meeting took place in late July, 1999 to build collaborative relationships between the research sites, as well as to set implementation expectations for the 2000-2001 school year.

Anecdotal Evidence of Project CHILD's Effectiveness

A Principal's Story
The following is a verbatim, e-mail letter received by Florida TaxWatch from Principal Tom Wallon of Windy Hill Elementary School in Jacksonville, Florida. Mr. Wallon describes in vivid terms the differences that the Project CHILD program already is making at his school. The program has been implemented at Windy Hill since 1992. The letter is presented here with Principal Wallon's permission and is characteristic of the kinds of testimonials being made by other Florida elementary school principals and teachers who have actively been engaged with the Project CHILD program:

Students in Project CHILD classes have also had positive experiences with the classroom method, as evidenced by the account below from one student (Bethany) at John D. Floyd Elementary School (edited somewhat for grammar):

Parents of Project CHILD students have also indicated that they are very satisfied with the quality of instruction that their children receive in Project CHILD classes. Doug and Terri Miller, parents at John D. Floyd Elementary School had this to say:

Project CHILD teachers have also lauded praises on the effects of the program on their students' learning and on themselves. The testimonial presented below is from Tedder Elementary School's "Teacher of The Year" and Project CHILD teacher Shonner Gainer:

Spreadsheet Analysis of
Potential Cost-Related Impacts of Project CHILD

The following data represent a brief, spreadsheet analysis of potential statewide cost savings associated with the Project CHILD program based upon data provided by the Florida Department of Education. The first spreadsheet analysis compares all K-5 students in a traditional classroom size of 20 to the Enhanced Project CHILD model of 30 students per classroom. The enhanced application includes one classroom assistant and one FTE teacher involved with 30 students per class. The second spreadsheet analysis is of anticipated cost savings for the Project CHILD Model of 22 students per classroom (excluding enhancements) as compared to the traditional classroom of 20 students. It should be noted that , in this spreadsheet analysis, student enrollment, average teacher salary and average student workstation cost were obtained from the latest available FDOE data. Also, potential cost savings would accrue over time through attrition and cost avoidance.

Test Scores Comparison

The following is a brief analysis of pairs of schools (a Project Child School and a comparison school) in three counties (Broward, Duval and Hernando) in Florida. The schools vary in average class sizes and in the fact that one of each of the three pairs is a Project Child School and the other a traditional school. The individual traditional schools henceforth will be referred to as a "Comparison School."

Hernando County Schools Analysis

Grade-Level Comparison
The first Project Child School analyzed was from Hernando County: John D. Floyd Elementary. The demographic characteristics both for this school and for the Comparison School used in this analysis appear in Table 8 below.

The schools in this county were compared at two levels. First , they were compared by grade levels for grades 1 to 5. Second, a Grade 4 class of 28 students (excluding the special students who were not tested for the FCAT) from John D. Floyd was compared with a Grade 4 class from the comparison school (class size of 21) in order to further validate the results for the first test. The basis for both comparisons were FCAT and SAT 9 scores.

The type of comparative statistical analysis used in comparing the two schools at the grade level was the independent t Test. First, the mean test scores for Reading and Math for each school at each of the five grade levels were generated. Then the t-Test was utilized to determine if it was possible to be at least 95% confident that the differences in the mean scores from the two schools were not just coincidental (due to chance) but statistically significant. The results of this analysis appears in Table 9.

Table 9 presents the average test scores for John D. Floyd (The Project CHILD School) and the Comparison School. In the first two grades, the average scores used were SAT 9 scores as this is what is normally given to these students. In the other three grades, the FCAT scores were used. As Table 9 indicates, about 80% of the differences in mean test scores between John D. Floyd and the Comparison School were statistically significant. In other words, the Project CHILD school scored higher than the Comparison School about 80% of the time, and one can be at least 95% sure that these results are statistically significant. Therefore, in the case of these two schools, there is a scientific indication that Project CHILD does make a statistically significant difference.

Class-Level Comparisons
Secondly, data from these two schools comparing the FCAT scores of a 4th Grade Project CHILD class of 28 at John D. Floyd with a 4th grade class of 21 at the Comparison School were used to bolster the results generated above. In order to use the independent t-Test with such a small sample size, the sample distributions for the Reading and Math scores were first graphed and a normal probability line was super-imposed on the graph. This revealed that both samples did not grossly violate assumptions of normality necessary to use the t-Test. Therefore, it was deemed proper to proceed with the t-Test. The results for the 4th Grade Math scores using the t-test are presented in Table 10 below.

The t-Test analysis revealed that there was a statistically significant difference between the mean scores for the 4th grade classes in John D Floyd and the Comparison School. In particular, the significance level score (.042) indicates that there is less than a 5% chance that the difference between the mean FCAT scores from the two classes are not statistically significant. This means that, with regards to these two classes, the Project CHILD class at John D. Floyd Elementary scored better as a group in math than the class at the Comparison School, even though the comparison class had fewer students.

The results for the 4th Grade reading scores are presented in Table 11 below. The t-Test analysis in this case revealed that there was a statistically significant difference between the mean reading scores for the two 4th grade classes at John D Floyd and the Comparison School.

In particular, the significance level score (.051) indicates that there is only about a 5% chance that the difference between the mean FCAT scores from the two classes are not statistically significant. This means that, with regards to these two classes, the Project CHILD class at John D. Floyd Elementary scored better as a group in Reading than the class at the Comparison School, even though the class at the Comparison School had less students.

In order to enhance our scientific confidence that the small sample size from the two classes did not skew the results presented above, a non-parametric test was generated. Non- parametric tests are less stringent with regards to the assumptions of normality than parametric tests such as t-Test; therefore, they are useful when small sample sizes are involved. The Mann Whitney test is the non-parametric counterpart to the independent t-Test and was used to both validate the results described above for the class comparison and to compensate for the small sample size.

The results of the Mann Whitney test for the math scores from the two classes are presented below in Table 12.

The Mann Whitney Test revealed only borderline statistical significance at the .05 level between the mean Math scores for the 4th grade classes in John D. Floyd and the Comparison School⁴. The significance level of the Z score (.061) revealed that there could be more than a 5% chance that the difference between the mean FCAT scores from the two classes are not statistically significant. Therefore, the Mann Whitney non-parametric test did not fully support the results of the Independent t-Test at the .05 level of significance. This means that some caution should be used and these results noted when interpreting the results of the Independent t-Test for the Math scores. In this case, sample size may matter.

The results of the Mann Whitney test for the reading scores from the two classes are presented below in Table 13.

In this case, the Mann Whitney test fully supports the results of the t-Test for the reading scores described earlier in Table 11; viz, the differences in mean scores between the two classes at John D Floyd and the comparison School are statistically significant at the .05 level. The significance level of the Z score (.021) revealed that there is less than a 3% chance (normally, less than a 5% chance is considered statistically significant) that the difference between the mean FCAT scores from the two classes are not statistically significant. Therefore, the Mann Whitney test supports the assertion that the students at John D. Floyd did better as a class in terms of FCAT scores than those at the comparison School, even though the class at the comparison School had a smaller class size.

Overall, the results of the analysis at the two schools in Hernando County indicate that Project CHILD has made some difference in terms of higher FCAT and SAT 9 scores at the Project CHILD school. These results are especially noteworthy when one takes into account that this is just the first year in the implementation of Project CHILD at this school, and, as the implementation section illustrates, there were many "growing pains." At the same time, it must be acknowledged that these results are cross-sectional in nature (that is, they represent only one year), and a longitudinal analysis (one involving three or more years) needs to be done to get more conclusive results.

Broward County Schools Analysis
The second Project CHILD school used in this analysis was from Broward County: Tedder Elementary. The demographics for this school and the Comparison School chosen are presented in Table 14.

The two schools were compared by grade levels for grades 1 to 5. It was deemed that there was no need to compare the schools at the class level since the Comparison School had an average class size below 20. The basis for the comparisons were SAT 9 scores for Grades 1 and 2 and FCAT and NRT scores for grades 3, 4, and 5. The results are presented in Table 15.

The type of comparative statistical analysis used in comparing the mean scores by grade level for the two schools was the independent t Test. First, the mean test scores for reading and math for Project CHILD classes and for classes at the Comparison School at the same grade levels were generated. Then the t-Test was utilized to determine if it was possible to be at least 95% confident that the differences in the mean scores from the two schools were not just coincidental (due to chance), but statistically significant.

As Table 15 indicates, classes at Tedder Elementary, the Project CHILD school, scored higher in 7 of the 10 categories analyzed, with 6 of these categories being statistically significant. In other words, we can be at least 95% confident that the Project Child classes at Tedder scored statistically significantly higher than the Comparison School classes in 6 of the 10 categories analyzed.

Overall, the results of the analysis at the two schools in Broward County indicate that Project CHILD has made a statistically significant difference in terms of higher FCAT, NRT and SAT 9 scores in six of the 10 categories analyzed. It bears repeating that these results are even more noteworthy when one takes into account that this is just the first year in the implementation of Project CHILD at this school, and, as the implementation section illustrates, these results were attained despite the many "growing pains" that typically apply in the first year classroom experience with Project Child. Also, it must be pointed out that these results are cross-sectional in nature (that is, they represent only one year); a longitudinal analysis (one involving three or more years) needs to be done to get more conclusive results.

Duval County Schools Analysis
The third Project CHILD research school used in this analysis was from Duval County: Windy Hill Elementary. The demographics for this school and the Comparison School chosen are presented in Table 16 below.

Because no significant test was given at Grades 1 and 2 at the Project CHILD school, there was no means for comparison at these grade levels. Therefore, the two schools were compared by grade levels only for grades 3 to 5. The basis for the comparisons were FCAT and NRT scores for grades 3, 4, and 5.

The type of comparative statistical analysis used in comparing the mean scores by grade level for two schools was the independent t Test. First, the mean test scores for reading and math for Project CHILD classes and for classes at the comparison school at the same grade levels were generated. Then the t-Test was utilized to determine if it was possible to be at least 95% confident that the differences in the mean scores from the two schools were not just coincidental (due to chance), but statistically significant. The results of this analysis appear in Table 17.

As Table 17 indicates, classes at Windy Hill Elementary, the Project CHILD school, only scored higher in two (one in Reading and one in Math) of the six categories analyzed with one category tied. None of the two categories in which the Project CHILD school scored higher were found to be statistically significant differences. On the other hand, the Comparison School scored better than Windy Hill in three of the six categories analyzed; of these, two of the three were statistically significant differences.

Therefore, the results of the analysis at the two schools in Duval County indicate that Project CHILD did not make a statistically significant difference in terms of higher FCAT, NRT and SAT 9 scores. It should be noted that these results only took into account grades 3 to 5 and that this is just the first year in the implementation of Project CHILD at this school. Also, it is worth repeating that these results are cross-sectional in nature (that is, they represent only one year) and that a longitudinal analysis (one involving three or more years) needs to be done to get more conclusive results.

Conclusion

The evaluation of Project CHILD by Florida TaxWatch was both "process" and "outcome" oriented and was composed of five components. The results of each component of the evaluation are summarized below:

The first component was a review of previous research conducted on Project CHILD. The eight years of research reviewed by TaxWatch suggests that Project CHILD students attain higher achievement levels as measured by report card grades and standardized tests. For example, a 1995 longitudinal study by FSU's Department of Education indicates that Project CHILD students had better grade point averages on mid-semester report cards and higher test scores on the comprehensive Test of Basic Skills than did their peers. Appendix A consists of a list of the studies reviewed.

The second component of the evaluation was undertaken by the Institute for school innovation staff and focused on the actual implementation process of Project CHILD. It suggests that all three research schools (John D Floyd Elementary in Hernando County, Tedder Elementary in Broward County, and Windy Hill Elementary in Duval County) have, to varying degrees, implemented the Twenty Essential Components of the Project CHILD program. John D Floyd Elementary most fully implemented the program, with the others making a good faith effort at implementation. The success at John D Floyd was due mainly to the work of the Project CHILD coordinator who put in significant time and effort to assist the Project CHILD teachers, the results of which are indicated in both the improved implementation process and the higher test scores of her school's Project CHILD students.

The third component of the evaluation focused on a qualitative assessment of the academic achievements of students in Project CHILD classes. It included anecdotal evidence from Project CHILD principals, teachers, students and parents regarding the way in which Project CHILD has enhanced the educational experiences of students. The anecdotal data included testimony from Tedder Elementary's "Teacher of the Year" Shonner Gainer, who stated that the students in her Project CHILD classes have shown tremendous academic progress as measured by the Stanford Achievement Test Version 9 (SAT 9) scores, while the growth of the other students at her school using the same measures have been insignificant.

The fourth component of the evaluation focused on a cost-effectiveness analysis of the implementation of Project CHILD. It suggests that, if fully implemented statewide, the Project CHILD model with class size 30 would accrue an annual cost saving of more than $4 billion over the traditional model of class size 20.

The final component was the class size comparative statistical evaluation. The three Project CHILD research schools were compared to three demographically similar schools with smaller class sizes. Comparisons were based on Florida Comprehensive Achievement Test (FCAT), the SAT 9 test and the Norm Reference Test (NRT). Independent t-Test and the Mann Whitney non-parametric statistical tests were used to analyze whether there was a statistically significant difference between the test scores from the Project CHILD schools and the comparison schools with smaller class sizes. The results indicate that classes at two of the three Project CHILD schools scored better than did their counterparts at traditional schools with smaller class sizes.

These results will be bolstered as more data are gathered over subsequent years; however, they are a testament to positive influence of Project CHILD taking into consideration that this was the first year of its implementation at the schools.

Overall, the evaluation suggests that the Project CHILD Program has made a positive difference on student performance according to many of the teachers, students, parents and administrators involved with the Program and has the potential to generate significant cost savings if expanded. It also appears to make a difference in results, as indicated by statistically significant higher test scores in two of the three Project CHILD schools as compared to their counterparts at traditional schools with smaller class sizes. This is especially true in schools such as John D. Floyd Elementary where the technology and the Project CHILD coordinators fully complemented the Project CHILD teachers.

Recommendations

Additional technology training should be provided for the para-professionals who assist the teachers in Project CHILD classes. The para-professionals have asked for training from beginner to advanced on all hardware and software in the classrooms. They want to be able to efficiently assist the students on the available software titles and be able to perform simple troubleshooting. Training should be ongoing through year two for all five sites.

Both the on-site coach and the para-professionals should be funded full-time in year two. The teachers requested their para-professionals be funded all day. This will allow for needed assistance as the teachers work with the larger class sizes. The para-professionals worked for an average of 3 1/2 hours a day at Windy Hill and Tedder, and the teachers found they also needed help to successfully continue with instruction in the afternoon. It also was impossible this first year to collect the necessary data from the coaches. They had too many other responsibilities which kept them from fully monitoring the degree of implementation. As full funding is provided for site coaches, the Institute will expect 100% of all required reports to be received in a timely manner as outlined in each school's grant agreement.

The Institute for School Innovation should build upon the successful strategies of John D. Floyd Elementary's on-site coach. The steady increase in the percentage of both Teacher Collaboration Components and Instructional Design Components was a direct result of the coach's hands-on involvement. After the first semester she was concerned with the low amount of reports turned in to her, so each quarter she instituted a more efficient way of monitoring and tracking to assure adherence to these components.

An effort should be made to gather substantive input from the Project Child teachers on the implementation of Project CHILD and their recommendations for improvement. This could be done quantitatively (through the implementation of an annual survey) and quantitatively ( by asking them to provide artifacts documenting innovative results that they may have had with Project CHILD through the course of the year). An effort should be made to institutionalize this process for continuous improvement through the use of a rudimentary management information system.

An effort should be made to gather longitudinal data as well as cross-sectional data over the next two years. This year's results should be built upon by comparing it to next year's and the third year and describing whether a trend exists in terms of growth. It might be prudent to pilot-test the method at John D. Floyd first since it has the lowest mobility rates of the three research schools included so far. High mobility rates are one of the biggest impediments to longitudinal research.

The process of institutionalizing the evaluation process at the Project CHILD schools should begin in the second year. An effort should be made to train the Project CHILD teachers and the on-site coaches in collecting, inputting an analyzing the data used in this research for their own schools. This will enable them to track the growth of their students and also provide them with a tangible indicator of the results that they are achieving. A training schedule should be put in place and an SPSS statistical package purchased for the schools.

4. It does reveal, however, statistical significance at the .1 level, which is an acceptable level of scientific observation in most cases. Additionally, when choosing between the acceptability of parametric (like the t-Test) and non-parametric (like the Mann Whitney Test) results, statisticians normally opt for the parametric test. However, with the small sample size, one needs to be extra careful; hence the use of both in this study.

1. A 1991 random sample study by Evaluation Systems Design, Inc. and presented at the Southeast Evaluation Association, January, 1991 indicates that Project CHILD students in Volusia County, Florida scored significantly higher than their counterparts in traditional classes.

2. A 1991 six county study published in Florida Technology in Education Quarterly, Vol. 4, No.4, Summer 1992 and conducted by the FSU Center for Instructional Development compared mean standardized test scores for Project CHILD students compared to non- Project CHILD students at each school and found 11 positive effects, 2 negative effects and large positive effects for long-term students.

3. A 1992 six county follow-up study published in the Journal of Research on Computing in Education, Vol. 26, No. 1, Fall 1993 by O. Kromhout compared mean standardized test scores for Project CHILD Students compared to non-Project CHILD students at each school and found 12 positive effects, 2 negative effects, and that large positive effects continue for long term students.

4. A1993 six-county follow-up report to the Daniel Memorial Institute, Jacksonville, Florida by O. Kromhout compared mean standardized test scores for Project CHILD students compared to non-Project CHILD students at each school and found 15 positive effects, 0 negative effects, and that large positive effects continue for long term students.

5. A1994 middle school follow-up report to the Daniel Memorial Institute, Jacksonville, Florida by B. Gill compared standardized 6th grade test scores at two middle schools which receive students from Project CHILD schools.

6. Descriptive statistics (means and standard deviations) were calculated for percentile scores, then independent t-tests were conducted to determine differences between 180 former Project CHILD and non-Project CHILD students. The results indicate that percentile scores for CHILD students were 5 and 10 percentiles higher than matched sample of non-CHILD for reading and math.

7. 1997 longitudinal effects study through fifth grade preliminary report to the Institute for School Innovation, Tallahassee, Florida compared standardized test scores for students who participated in Project CHILD (K-5) at 2 elementary schools to a matched sample of traditional students. Descriptive statistics (means and standard deviations) were calculated for percentile scores, then independent t-tests were conducted to determine differences between 25 former Project CHILD and non-Project CHILD students. The results indicate that Project CHILD students had higher NCE and percentile scores in all areas: reading math, language and total battery, with the largest differences in subtests of reading comprehension and math concepts.

This report was researched and written
under the direction of Dr. Keith G. Baker, Senior Vice President and Chief Operating Officer.
T.O'Neal Douglas, Chairman; Dominic M. Calabro, President and Publisher
© Copyright Florida TaxWatch, September 2000


To obtain more information on this or any TaxWatch report,
E-Mail:
.

Return to the TaxWatch main page.