Luis Fernando Gamboa
University of Rosario

Abstract This paper examines whether the importance of the school has changed with the introduction of the new ICFES test. It is found that importance of the school, which remained relatively stable between 1997 and 1999 (around 27% and 37%), is reduced considerably with the introduction of the new ICFES examination, to levels between 10% and 27% for the year 2000. Resumen Este trabajo estudia si la eficacia de la escuela ha cambiado con la introducción del nuevo examen ICFES. Hemos observado que dicha eficacia, que había permanecido relativamente estable entre los años 1997 y 1999 (aproximadamente un 27% y un 37%) se ha reducido considerablemente con la itroduccción del citado examen ICFES, a niveles comprendidos enter el 10% y el 27% en el año 2000.

1.- Introduction    Since Coleman´s Report (Coleman 1966), several studies in school effectiveness and school improvement, (Murnane 1975; Summers and Wolfe 1977; Teddlie and Stringfield 1985; Scheerens and Creemers 1989; Teddlie and Reynolds 2000; Thomas, Smees et al. 2001) recognize the clustered structure of educational data. In Colombia there are few studies in this field. Mision Social (1998) and Piñeros and Rodriguez (2000) indicate that schools only explain one third of the students outcomes (i.e. the school effect), and this percentage is higher in private schools than in public ones.    The purpose is to study whether the school effect changed with the introduction of the March 2000 reform in the ICFES test. The ICFES test is a National mandatory exam for all students finishing secondary education. The next section summarizes the theory behind school effectiveness. Then, I estimate the intra-school correlation coefficient with a composed socioeconomic index to control for student´s outcomes. This shows that the variance changed with the introduction of the new type of ICFES examination. Finally, I show how these results can be used by schools in order to improve the examination results of their students. 2.- Theoretical framework    2.1 The School Effectiveness Theory    School effectiveness (SE) refers to how the outcome of schools is influenced by modifiable conditions (financial or material inputs, activities and more complex processes such as school management, curriculum and teaching), in terms of the academic success of its students. Several theories, based on Carroll´s school learning model (1963), try to explain the school´s results in relation with the quality of education that they provide.    The first SE studies were interested in determine the combination of school inputs that maximize the achievement of the students.(input-output theory). The Coleman Report (1966), found that only 10% of the total variance of the student´s achievement was explained by schoolfactors, while the remainder was attributed to aspects related to the context of the student and their families. These results would be confirmed by Jencks, et al. (1972).    Scheerens (2000a) shows a synthesis of several studies carried out on this field. The last stage of effectiveness and improvement´s studies, started at the beginning of the 90´s with an integral interpretation about how processes and inputs interact, in order to explore the school effects in different school contexts.    Integrated models adopt economic rationality as a theoretical framework, and the concept of productivity as the key element to evaluate their fulfillment. Moreover, integrated models incorporate elements of the organizational theory for studying how inputs and processes produce certain educational outcomes, (Figure 1). These models give a principal role to the school´s context as an important factor of its effectiveness. A schools is not an isolated and closed unit, but a system in interaction with the environment.    The multilevel perspective in education explain why the upper levels should provide helpful conditions to develop the central processes in the lower ones. In general, these include a level of organization and school management; another level for the teacher or classroom, and finally a level for the student. Another approach is the model of educational effectiveness (Creemers, 1997). It includes not only aspects related to the course, but also to the upper levels that have an incidence on the effectiveness of the teaching. The model defines four basic levels (educational system, school, classroom and student) and three fundamental dimensions (quality, time and opportunities of learning).    Ridell (1997) shows the little use of the new methodologies and the scarce utilization of its conceptual base and emphasize the need of tightening the link between the SE and school improvement. Velez, et al. (1993) analyze the results of eighteen Latin-American studies concerning factors of SE in primary education. Estimating production functions for education, Velez et al. find that the percentage of the achievement explained by the school was between 60% and 49%.    Fuller and Clark (1994) indicate that there appear to exist more studies about primary education than on secondary education. The most investigated variables have been the availability of physical, material and financials inputs. Another aspect that is also taken into account is the cultural context in which the schools and the teachers develop their activities. Heneveld and Craig (1996), based on studies in the Sub-Saharan Africa, enumerate a list of the factors that have a greater incidence on the academic performance of the students. They notify that though such models are intended to be general, in the sense that they would be able to be applied in different contexts, the different factors should not be seen as independent, because of their interactions. Likewise, the combination of characteristics, and the form in which they interact, depend on the context (institutional, cultural, social and political) in which the schools operate.    2.2 The methodology of Multilevel Analysis    Unlike traditional statistical techniques, multilevel analysis recognizes the nesting of students in classrooms and of these in schools. When students characteristics, input provision, and teaching and managerial practices are school specific, the use of the traditional statistical techniques could result in a biased estimator. The estimation of the fixed effects model, or empty model, constitutes the initial point in multilevel analysis.    The Fixed Effect Model.    This model estimates the intercept parameters (student and schools). These coefficients are the means of students and schools achievement. The model is given by the following equations:    Equation (1) (1-level ) indicates that the score of each student (Yij) is a function of their school average achievement.( ßoj - intercept) plus an error term. The second equation represents the 2-level model. In this equation each school´s average achievement (ßoj) is a function of the overall average and its error. Finally, the third equation derives from the combination of the previous two.    The Importance of School.    The Intra school Correlation Coefficient (ICC), which is the result of the analysis of variance of resid, defines the importance of student and schools´ factors in the explanation of achievement levels. The variance of the achievement of a student will be:    Where represents the variance within the school (between students) and the variance between schools. The piece of the variability in the achievement of student explained by school factors or ICC is given by:    The Socioeconomic Status´ Effect.    The study of the socioeconomic status (SES) of the students constitutes a fundamental piece in the analyses so much of the students access to the educational services as of school effectiveness. The first one relates to the equity in the access and the function that the State complies in the provision of education, bearing in mind the premise that the government is committed to guaranteeing the right to education to all and, at the same time, favoring the less-favored sectors of the population.    On the other hand, the effectiveness school concept, defines two fundamental objectives of the educational policy: the quality and the fairness of the service. Quality is given by the performance of the students and it is sought that it be as higher as possible. Fairness represents the compensatory power of each school with regard to the SES of its students, that is to say, the capacity of the school to neutralize the SES´ influence of each student on its achievement (Bryk and Raudenbush, 1992; Brandsma and Knuver, 1989).    Both dimensions can be illustrated in Figure 2, which shows the existing relation among the academic achievement (Logro) and the SES. The quality is represented by the point in which the line achievement-SES cut the achievement´s axis. The equity is given by the slope of this line, which is equal to the effect of the SES on the academic achievement. In the figure one can observe that the students from schools A and B have a greater average performance to that of students in schools C and D. Likewise, schools B and D are fairer than schools A and C. This means that the SES has smaller influence on the achievement in the schools B and D.   Since this perspective, the challenge of an educational system is that all their students reach a level in which the average achievement is equal to the achievement of the school with greater performance and, at the same time, neutralize the students SES´ incidence on the achievement. That is to say that the intercept be the highest possible one and the slope nil.    The Students SES´ Effect.    Just as it remained registered in the graphs that illustrate SE´s models, in estimating the real impact of the different factors on the performance of the students, we isolate the SES effect on the academic results. The model has the following functional form:    ßoj is the mean of the i-student achievement in the j-school (which is a function of the achievement average of the school, is equal to the SES effect on his own achievement.    The School SES´ Effect.    This model seeks to estimate the student´s average performance eliminating the effect that has the students SES mean in their school on the scores. In other words, this effect tries to purify the performance´s overall average, isolating the effect that it has upon the SES of each establishment, expressed in terms of the average SES of its students. That it would be given by the following equations:    As in the Empty Model, ßoj is the mean of the i-student achievement at the j-school j. ßoj, at the same time, is function of the achievement average of the school and the SES effect on the level of performance of its students 3.- The School Effect 1999 Vs. 2000    During the last years the analysis of the quality of the education in Colombia have enlarged considerably. Besides the study of the academic results, another dimension that has captured the attention of the investigators relates to the measurement of the importance of the educational institution in the explanation of the levels of performance of the students.    The importance of the school in the explanation of the performance of the Colombian educational system has been situated around 29%, including both public and private establishments. This is certain independently of the type of test (SABER/ICFES) and the degrees that are being evaluated. Bogota´s estimates are the lowest, but it should be kept in mind that in this case the scores were weighted for each level (including not only the areas but also the course level), which could have affected their variability.    We can also see that the school effect is greater in the private sector that in the public one, and once the effect of the SES is controlled, the school effect is reduced, being situated around 14%.    In order to determine how the school effect has changed with the new design of the test, the exercise starts with the empty model estimation for establishing the "gross" importance of each one of those groups as the two large factors for the years 1997,1999 and 2000)1 . Subsequently, the students effect and their family´s SES are controlled to estimate the school´s net effect.    Each area (mathematics, language, natural and social sciences) has its own estimated coefficient for three years (Table 2). Only the year 2000 includes data for the new exam. The table shows that between 1997 and 1999 the school importance is stable in all areas (being located between 26% and 37%), but for the new exam (2000) we found a drastic reduction mainly in mathematics (from 31% in 1999 to 10% in 2000). Social sciences did not reduce during the period. The change in the exam modified one indicator that had been stable in other studies and other test which is the exam´s discriminatory power.    The school effect (school variables) can be separated from the other student features by means of the socioeconomic effect.2 Figure 3 shows the relation between the SES and the students achievement. The Y-axis shows the standardized scores with mean zero. In the Xaxis we plot the SES in deciles from the lowest (1) to the highest SES (10). The performance of the students, according to their SES, is similar during these three years. In contrast, we found a lower score in mathematics in the 2000 year than in the others.    The effect of the socioeconomic Status in each student on his/her achievement was measured with equations 6-8. SES of the student (ses_stud), is the explanatory variable and its variance is the residual variance after controlling by ses_stud. SES effects are significant and not high enough. Means in each area are similar in both models (empty and intercept models). In brief, after controlling for the SES effect, the importance of the school does not reduce considerably. However, it is important to highlight the differences between 1999 and 2000, because the reduction is not the same in all areas. (Table 3)    After estimating the SES effect on the achievement of students, we separate the schools SES effect from the students one. In this case, the model has the functional form described in 9-10 equations. Now, there are two variables: the student´s SES and the SES mean of each school (mean_ses_sch), and the new variance is the residuals after controlling for both variables.    Table 4 shows that SES has a higher impact on language than the others. It reflects so much in the magnitude of the estimated coefficients, as in the reduction of the variance when its effect is controlled and the ICC that of it is derived. We found that estimated coefficients of the school SES are higher than students SES one. For instance, in 1998 increase in 1% in student SES increase his achievement between 0.05 y 0.08, while an increase in the same level of the school SES means an increase in 0.27 and 0.38 in the achievement according to the area. Here, ICC is equivalent to the ´net´ importance of the school and changes from the 13 and 20 percent in 1999 to 6 and 11 percent in 2000. In mathematics, goes from 17 percent in 1999 to 6 percent in 2000.    Finally, we measure the goodness of fit by using the variance reduction in each model with respect to the empty model. (this type of models does not have analogous goodness of fit statistic as the coefficient of determination R2 in regression models). (Snijders y Bosker, 1999). The final model has an increment in its predicted power between 16 and 22 percent before 2000 and an increment of 4 and 19 percent for the new exam. It is especially important the case of mathematics, where the final model exhibits an additional 4 percent in its explicative power. 4.- Conclusions    The use of multilevel analysis allows us to separate the school effect from socio-economic factors in the explanation of academic achievement of Colombian students. With the new design of the test, we found drastic changes in those effects. It is important to study all the effects of the new test in order to help policy makers design the most appropriate decisions in countries that exhibit strong differences in their basic private and public education. Notas a pié de página 1 The variance among schools or CCI would be able to be interpreted like indicator of the difference that represents, for a student to attend specific educational center. In this part should be keep in mind that in these moments splits of the estimations of the total or rough variance, which includes the variation originated in the differences of context and other students entrance´s variables (such as the prior achievement, the aptitudes and the SES), fundamental to control when is a matter of ´aggregate value´s studies. For OECD´s countries, the CCI without adjusting for the effect of the student´s context variables does not represent more than the 10% or 15% of the student´s performance total variance. In the developing countries those differences are greater, doing that the CCI be situated among the 30% and 40%. 2 There are many things that constitute the SES of the people. (familiar income, parental education, occupational status, economic dependence, among others). We use the inscription form to extract information about the student´s life conditions. However, like most of this are qualitative we construct a numerical index which change categorical variables into numerical. The Qualitative Principal Components Methodology and Optimal Scaling give us the opportunity of getting a numerical index (See Young, F. (1975), Young, F. W., Y. Takane, et al. (1978), Young, F. (1981), Kuhfeld, W., W. Sarle, et al. (1985), for details). The variables included in the index are: the occupation of the father´s student, educative levels of the parent´s student, property of the home, quantity of familiar income in minimum legal wages, number of people per income receiver, number of brother and sisters, employ of the student and family support. With this information, we estimated our index for 431.704 students in 2000, 417.851 students in 1999 and for 1997, and getting our SES index in a range from 0 to 100 (the best socioeconomic level). References Coleman, J. e. a. (1966). Equality of educational opportunity. Washington, DC, US Government Printing Office. Kuhfeld, W., W. Sarle, et al. (1985). Methods for Generating Model Estimates in the PRINQUAL Macro. SAS users group International Conference Proceedings, SAS Institute. Murnane, R. (1975). The impact of school resourses on the Learning of Inner City Children. Cambridge, MA, Ballinger. Scheerens, J. and B. P. Creemers (1989). "Conceptualizing school effectiveness." International Journal of Educational Research 13(7): 691-706. Summers, A. and B. Wolfe (1977). "Do Schools Make a Difference?" American Economic Review 67(4): 639-652. Teddlie, C. and D. Reynolds (2000). School effectiveness Research and the Social and Behavioral Sciences. The International Handbook of School Effectiveness Research. C. T. y. D. Reynolds. Londres, Falmer Press. Teddlie, C. and S. Stringfield (1985). "A differential analysis of effectiveness in middle and lower socioeconomic status schools." Journal of Classroom Interaction 20: 38-44. Thomas, S., R. Smees, et al. (2001). Attainment, progress and added value. Improving School effectiveness. M. a. Mortimore. Buckingham, Open University Press. Young, F. (1975). "Methods for Describing Ordinal Data with Cardinal Models." Journal of Mathematical Psychology 12: 416-436. Young, F. (1981). "Quantitative Analysis of Qualitative Data." Psychometrika 46: 357-388. Young, F. W., Y. Takane, et al. (1978). "The Principal Components of Mixed Measurement Level Multivariate Data: An Alternanting Least Squares Method with Optimal Scaling Features." Psychometrika 43: 279-281. About the Author Autor: Luis Fernando Gamboa Dirección: Department of Economics. University of Rosario Correo electrónico: lgamboa@urosario.edu.co *Acknowledgements: I would like to thank the University of Rosario and Colciencias for their financial support. I would also like to thank Javier Perez, Andrés Casas, Luis Piñeros, Jesús Otero, Hernán Jaramillo, Diego Moreno for their constructive comments. Finally, I would like to thank ICFES for access to data.

DOCUMENTOS DE TRABAJO EN ANÁLISIS ECONÓMICO (EAWP)
Derechos reservados 2002. El permiso para reproducir algún artículo está garantizado si Documentos de Trabajo en Análisis Económico lo acredita, las copias no son vendidas y es en acto de mayor difusión del documento.

Editor: Fernando González-Laxe. (Universidade da Coruña)
Director: Venancio Salcines. (Universidade da Coruña)
Subdirector: Andrés Blancas. Instituto de Investigaciones Económicas (UNAM)
Editor Asociado para America Latina: Luis Miguel Galindo. Facultad de Ecomomía (UNAM)