.2008 - Volumen 1, Número 1
 
       
       
   
Educational Accountability in Brazil
 
       
    Nigel Brooke  
       
   

1. INTRODUCTION

The expanding use of external measures of pupil performance in large-scale systems of assessment, and the growing reliability of the instruments employed, has facilitated the development of educational accountability policies that ascribe school staff with responsibility for the results of their schools (Linn, 2000; West and Peterson, 2003). In some countries, most notably the US, both positive and negative consequences have been made contingent on these results, thereby raising the stakes associated with the assessments (Hess, 2003).  These policies have not met with widespread support among teachers for a variety of reasons including the real educational risks associated with test-driven curricula, the apparent threat to the professional autonomy of teachers and doubts as to the validity of conclusions regarding school performance drawn from large-scale systems that are less sensitive to the specific characteristics of schools or school districts. Nonetheless, there is evidence to suggest that accountability measures can promote pupil performance gains (Carnoy, Loeb and Smith, 2001; Jacob, 2003; Carnoy and Loeb, 2004;).

The purpose of this paper is to offer a preliminary answer as to whether and in what form similar policies of accountability are following the expansion of educational assessment in Latin America by looking at the case of Brazil. Over the past two decades, most Latin American countries have adopted some form of assessment of their national education systems, in addition to participating in international assessments such as TIMSS, PISA and PIRLS (Ferrer, 2006). Brazil is an especially interesting case, however, with national assessments existing at every level of education, including higher education.

More specific questions are whether the process of policy implementation is subject to the same resistance by school teaching and administrative staff and whether there is any evidence as to the beneficial effects of accountability policies regarding pupil performance in a developing world context. Accountability policy is defined here as an attempt to improve school results by establishing school or individual teacher consequences, either material or symbolic, in accordance with pupil performance as measured by state or municipal education assessment procedures.

State-wide, externally-administered education assessment systems exist in 13 of the 26 Brazilian states and the oldest of these, in Minas Gerais State, goes back to 1991. There is also a national, sample-based assessment system known as SAEB{1} which in its thoroughness and technical sophistication is among the most complete of its kind. First launched at the end of the 1980s, SAEB collects academic performance and socioeconomic data from pupils from three different grade levels as well as school data from teachers and school principals from the country-wide sample of both public and private schools. In addition, the federal government has pioneered a non-compulsory national secondary school leaving exam, now taken by almost three million students in their final year of mandatory schooling, and unique national tests for the assessment of all higher education courses for both first and final year undergraduate students. In both cases, government interest was to create mechanisms for the external monitoring of education quality and, through the publication of results, to stimulate demand for improvement (Castro, 1999). 

There is sophistication in the methods of test construction, in the techniques to guarantee comparability over time, in the logistics involved in applying tests simultaneously from north to south, and in the production of school and individual-level reports (Ferrer, 2006). In 2005, the first national, census-type assessment of elementary Portuguese and maths education, called Prova Brasil, was carried out in the 4th and 8th grades of all urban schools, both public and private. In sum, Brazil has accumulated considerable technical expertise through almost two decades of experience in large-scale educational assessment and is among the leaders in this field in Latin America.

What the basic education (Grades 1-12) assessments in Brazil have shown is that average school results are extremely poor. As reported by the Ministry of Education, the 2001 SAEB results indicated that 59% of 4th grade pupils could only read the simplest of phrases or could not read at all and only 4.8% in this grade performed at the level considered appropriate for their age (MEC 2003). The results from the SAEB tests applied in 2005 show a fractional improvement over previous years in the performance of 4th graders but scores in the 8th grade and the 3rd year of secondary school in both Portuguese and Maths continue to fall and are now between 3.8% and 11.2% lower than in 1995 (MEC, 2007).

The government’s impression that the quality of basic education is failing to reach even minimum standards is corroborated by Brazil’s poor performance in large scale international assessments involving countries from other regions. Brazil has participated in these since the second International Assessment of Educational Progress carried out in 20 countries in 1990-91 to study the maths and science performance of 9 and 13 year olds (Lapointe, Askew and Mead, 1992a, 1992b).  In the case of Brazil this study was restricted to 13 year olds in the cities of São Paulo and Fortaleza and notwithstanding concerns regarding the representativeness of the samples, showed the country to be only marginally ahead of Mozambique, a predominantly rural country with 80% adult illiteracy at that time.

Brazil was also among the 32 countries to take part in PISA in 2000, securing the lowest average scores on the combined reading literacy scale and on the mathematical literacy scale, behind the other Latin American participant, Mexico. In order to correct for the differences between countries regarding the average number of school grades to which the sample students had been exposed--the result of later enrolment and higher rates of grade repetition in the poorer countries--those responsible for the Brazil report carried out comparisons between students with the same number of school grades (INEP, 2001). Even with these controls, the Brazilian disadvantage was clear. For example, the proportion of Brazilian students with 9 or more school grades performing at proficiency level three or above on the reading literacy scale was only 25% as compared with 76% for South Korea, 59% for Spain and 30% for Mexico. For one observer: “what PISA shows us is that of all the thousand things that a [Brazilian] school does or tries to do, it forgets the most important; to teach mastery of the language” (Castro, 2001). The PISA 2003 results were no more comforting with between 22.5% and 28.2% of Brazilian students reaching proficiency level 2 or higher on the four maths scales in comparison with averages of between 75.2% and 79.3% for the OECD countries (OECD, 2004).

Within the legal framework of the Brazilian federation, responsibility for this situation lies with state and municipal governments. The current division of attributions, defined by the 1988 constitution and then ratified by the Education Act of 1996, gives prime responsibility for public pre-school and elementary education to municipal governments and secondary education to state governments while emphasising the responsibility of the federal government for public higher education. Table 1 shows the current distribution of student enrolments between the different spheres of government as they work towards their constitutional responsibilities, at the same time as demonstrating the importance of private schooling, principally in higher education. While most public pre-school and elementary students are enrolled in municipal schools, more than 12 million are still in state schools. At the same time, there are some 186.000 students enrolled in municipal rather than state secondary schools (MEC, 2006).

Given the general scenario of poor elementary and secondary school results along with relatively high levels of expertise in the area of educational assessment, it would be reasonable to expect the implementation of school accountability policies based on measures of pupil performance at those levels of government with responsibility for basic education. In Brazil, this would mean the growth of accountability policies within state and municipal education systems, rather than as a component of federal policy, with a diversity of policies reflecting the varying conditions of the local secretariats of education. 

TABLE 1. ENROLMENTS BY LEVEL OF EDUCATION AND TYPE OF STABLISHMENT. BRAZIL, 2006.

Level

Federal

State

Municipal

Private

Total

Pre-School

   2.471

   242.979

4.820.236

1.950.409

7.016.095

Elementary

25.031

11.825.112

17.964.543

3.467.977

33.282.663

Secondary

67.650

 7.584.391

    186.045

1.068.734

8.906.820

Adult

  1.203

3.226.780

2.180.391

   207.917

5.616.291

Higher* (undergrataduate)

579.587

   477.349

   135.253

1.753.184

4.453.156

Total

675.942

23.356.611

25.286.468

8.448.221

59.275.025

Data for 2005
Sources: MEC/INEP, Censo Escolar – 2006; MEC/INEP, Sinópse da Educação Superior – 2005.


This expectation derives from the recent history of accountability in the UK and United States. In these countries, the belief that educational standards were under threat pushed education to the top of the political agenda and led governments to demand greater responsibility from their schools (Jones, 1989; West and Peterson, 2003; Gillard, 2004). It also recognises the ready transmission of ideas and education policy alternatives across national boundaries as the result of the influence of international agencies and the growing circulation of public policy information.

However, what we find is that there are few examples of formal accountability policy in Brazil and of those that have emerged, some have already been abandoned while all the others have undergone major modifications since making their first appearance.

There is no national data collection on state or municipal education policy in Brazil. This is not seen to be the role of the Ministry nor of the Council of State Secretaries of Education (the equivalent of the Council of Chief State School Officers) nor of the Union of Municipal Education Leaders.  This is an example of the considerable autonomy of state and municipal governments, founded on federative principles regarding the non-interference in matters pertaining to other spheres of government, which until recently has plagued the coordination of public education policy (Brooke, 1992.)

What information is available indicates that accountability policies have existed or are still in place in the state systems of Rio de Janeiro, São Paulo, Ceará and Paraná and in the municipal systems of Sobral and Belo Horizonte. Of these the Rio de Janeiro, São Paulo, Ceará and Sobral versions can be defined as high stakes systems, offering salary incentives or monetary prizes, while the cases of Paraná state and the municipality of Belo Horizonte are low-stakes, report card systems, which involve the publication and dissemination of performance indicators without explicit material consequences for school personnel.

The study proposes to look at the salient details of each accountability system to determine whether accountability has met with resistance and, if so, what it is that has impeded both the implementation and the subsequent maintenance of accountability policies in Brazil. Any evidence as to the possible impact of existing policies is also presented.

2. METHODS

The data on the cases included in this study is drawn from private correspondence with government officials responsible for the management of state and municipal policies, information shown in meetings or available on the internet and the rare publications produced either for schools participating in accountability programs or by government officials as a record of their actions. Data collected through private correspondence was obtained through a sort of snowball sampling, in which I relied on referrals from initial contacts to generate additional informants.  In summarizing and analyzing this information, I bring to bear my own experience and observations as a member of the Faculty of Education of the Federal University of Minas Gerais and nine years as Program Officer for Education with the Ford Foundation in Rio de Janeiro. While the search for secondary information was reasonably thorough, this survey does not pretend to be exhaustive. The brief story of each accountability program, based on the more salient features, is qualitative and designed as a preliminary description rather a structured summary of common elements.

3. HIGH STAKES ACCOUNTABILITY PROGRAMS

3.1. Rio de Janeiro State

The New School Program (Nova Escola) is Brazil’s longest running education accountability policy. It was created in January 2000, together with the state’s Permanent Public School Assessment System, to be applied to both school administrators and teachers. The long list of changes since the program’s creation gives an idea of the difficulty in establishing consensus as to the purpose and methods for comparing school performance.

Designed to be a longitudinal, cohort-gain model starting with pupils in the 3rd elementary grade, to give a more accurate measure of school contributions to pupil learning, the lack of any external assessment in the program’s third year, 2002, due to a change in the governorship, provoked a change of plans. New instruments, in line with the national SAEB scale (to permit comparisons with other states), were applied to 180,000 4th and 8th grade elementary pupils and 3rd grade secondary pupils at the end of 2003. Measures of school governance and efficiency (based on pupil flow statistics) were included and five comparison groups of schools with similar socioeconomic intakes were created.

In 2004, the state government awarded the contract for the design and management of the Nova Escola program to a different educational testing organization where the criteria were reformulated to concentrate on school administration, pupil performance and pupil flow. The comparison groups were also scrapped but the SAEB model of testing the 4th, 8th and 11th grades has been retained as has the awarding of two different types of bonus, one for absolute performance and the other for the level of progress.

Bonuses are awarded to schools based on the accumulation of points. The maximum number of points a school can accumulate is 25, of which 10 relate to pupil flow (90% or more of pupils still enrolled at the end of the school year), 5 to school management (with criteria dealing with transparency, community relations, staff frequency and enrolment management), and 10 (or 40%) with pupil performance on the state assessment system. In the case of pupil performance, the school achieves the maximum number of points if 80% of its pupils in the tested grades achieve the level of performance described as satisfactory on the SAEB scale.

The bonus given school staff has varied according to level of performance. In 2004, the top level of a five level scale was equivalent to R$500 for full time teachers and R$250 for support staff. The bonus money, two thirds of which was awarded for the overall score performance of the school and one third for school score ‘progress’ between school years, was paid out to a total of 58,008 teachers and 19,843 support staff.

One important aspect of the program is that it is the only one to have been evaluated by means of a comparison between the results achieved by secondary schools in Rio de Janeiro and those of two other states in the Brazilian southeast from 2000 to 2005 (Rodrigues, 2007). The results in question are the average SAEB scores for the three states (collected every two years between 1995 and 2005) and the average pass/fail and dropout rates collected every year as part of the school census data collection. Despite the lack of controls for other possible educational policy differences or for any changes in the relative economic status of the three states, the study shows greater gains in Portuguese performance in Rio relative to the other states, inverting a position of inferiority during the period preceding the creation of the Nova Escola program (Rodrigues, 2007).  Although showing losses rather than gains, flow statistics also show that average end-of-year pupil fail rates rise more slowly in Rio de Janeiro than in the other states. However, other results make these findings less conclusive. The results for maths performance show the opposite tendency to Portuguese, with Rio de Janeiro losing ground in relation to the other states. As maths is more of a “school” subject than Portuguese (Brandsma e Knuver, 1989), the expected result would have been the opposite. It is also shown that the level of dropout has risen in comparison to other states, thus raising the doubt that the lower than expected fail rates could have been due to self-selection by pupils. Although a brave effort to subject the program to an objective evaluation, the ambiguity of the results indicates the need for further studies.

Despite recent changes in the program, including the reduction in the weighting of the governance criteria and more specific criteria regarding pupil flow, resistance to the program is still high and technical, political, and doctrinaire criticisms abound. One such criticism is that the program reflects neo-liberal concern with the demands of employers: “The New School program is an impoverishment of education. It is more than this, it is an attempt to satisfy the demands of the productive labor market” (Gama, 2004). Up until 2006, the continuity of the program was made possible by the support given in the state congress by representatives of the popular, evangelical, low-income sectors that were unconnected with the teachers’ unions or other organized social groups. The newly elected governor has yet to declare support for the program and there is uncertainty as to its survival in the new political environment. In August 2005, the teachers union was responsible for 5 legal actions against the then governor, of which 3 were related and hostile to the New School Program, mainly due to its interference in the official teachers career and salary scales. If the new governor sees the need for political support from the teachers’ state congress representatives, the program is unlikely to survive in its current form.

3.2. São Paulo State

Between the years 2001 and 2003, school principals and teachers in São Paulo state received a lump salary supplement, announced at the end of the school year, a little before Christmas. This bonus was calculated according to average staff attendance levels, overall school performance results as measured during the previous school year by the State assessment system,  SARESP,{2} and other indicators of quality, principally the schools’ pupil retention rates. This accountability policy, involving calculations to determine individual payments, was an expression of the State Secretariat’s outspoken belief in the importance of making public the results of its annual assessments:

“School performance results should be given wide publicity to ensure that school staff and the community are able to identify where the school is placed in relation to other schools in the region, neighbourhood and municipality. This will allow the school to find ways to diversify its activities with the purpose of improving its results. At the same time, it will give the population the information it needs to oversee, participate and demand quality of service.” (São Paulo, 1995, p.310).

In 2002 the state government paid out a total of R$370 million{3} to 170,974 teachers and 18,744 school principals and supervisors, a 58% increase over the previous year. The highest bonus for teachers was R$4,500 and for principals R$5,500 and the lowest R$1,000 and R$1,500, according to which of the six performance levels was attained by the school. In 2003, 42.000 school auxiliary staff members were also included in the program and received a fixed bonus of R$500 that did not depend on performance criteria. In 2004, the amount disbursed increased to a total of R$514 million, paid out to 194,348 teachers receiving between R$1,220 and R$6,000, to school principals and other school professionals receiving between R$1,200 and R$7,000 and to auxiliary staff receiving the same fixed bonus of the previous year. However, in this year the criteria were altered, first, to include a minimum of 200 days of effective service and the development of projects involving the school community and, second, to exclude the use of SARESP performance results. In 2006, SARESP was suspended and pupil performance results again removed from the calculation of the bonus. This was largely due to the application the previous year of the new Prova Brasil, the federal government’s assessment of 4th and 8th grade pupils from all urban elementary schools (APEOESP, 2006). The simultaneous decision to hold the SARESP exam every two years rather than every year makes it uncertain whether there will be a future return to the use of pupil performance data in the bonus program to determine the level of teacher pay supplements.

Opposition to the annual bonus has been strong. The most vociferous opposition, from the official teachers union, has expressed a rejection of the “abusive transfer of market logic” to education (Cavaliere, 2003, p.30) and the competition between schools and school principals that the policy is seen to foster (CDEP, 2006). Other criticisms have had more to do with the problems of cheating, there being opportunity for the teacher to help pupils fill in the answer sheets or to doctor pupil answers given that teachers are asked to apply and correct the tests. These cases are considered “well known” among state school teachers (Dissidência, 2006) as are the bureaucratic problems associated with making the individual teacher calculations and payments. A further argument, apparently well justified, concerns the use of average pupil performance results without any attempt to control for socioeconomic differences between schools or for the impact of prior learning levels.

Despite initial intentions, the bonus program appears to have become more a way to justify a Christmas handout than to forge a clear connection between school performance levels and staff pay supplements. Indeed, the Secretariat has been accused of a lack of transparency over the way in which the bonus is calculated (Dissidência, 2006) which would also contradict the idea that it is intent on establishing performance as a clear-cut component of teacher pay. It would seem, therefore, that without any statement to this end or any explanation as to whether the results were deemed satisfactory or not, the accountability program that was started in 2001 has effectively ended. The continued use of minimum teacher attendance rates or the use of teacher attendance at training courses, as included among the 2005 criteria, are a reflection of ongoing problems of teacher absenteeism rather than an expression of policy to attribute responsibility for school results to teachers or to other members of school staff (Forum Estadual em Defesa da Escola Pública, 2003).

3.3. Ceará State

The connection between the state education assessment system, SPAECE{4}, and the New Millennium School Prize was legally established in 2001. The 100 best schools were to get a prize on the basis of the average scores in maths and Portuguese of the 4th and 8th grade students included in the state sample. For the top 50 schools in 2002, the prize was worth R$800 to all full-time teachers and R$300 for administrative personnel. For the next 50 schools, the prize was worth half this amount. There were also prizes for the best students on the basis of their test performance. The explicit purposes of the prizes were to promote recognition of the achievements of public schools as measured by SPAECE, improve the school environment and impact results by creating a climate of quality, and raise the standard of public education (Law Nº 13.203,  February 21st , 2002).

Results from the 2003 SPAECE assessment of 8th grade elementary and 3rd grade secondary students caused considerable concern insofar as they showed a marked drop in average performance. In Portuguese, 8th grade results were 7.5% down on 2002 and while maths results in 2003 were equal to 2002, both were 9.4% lower than in 2001. By level of performance in Portuguese the proportion of pupils defined by the Secretariat to be either at the critical level or worse reached 60.8%, with a corresponding drop in the proportion of pupils reaching intermediate or adequate levels. In maths, 97.7% were classified as at the critical level or below and only 0.2% achieved the level of performance considered adequate for the grade (Lima et al, 2005). Although the government offered no explanation for these results nor discussed the possible impact of its accountability program, it is evident that the New Millennium School Prize had been unable to counteract the broader tendencies leading to the drop in pupil performance. Whether this was due to the value of the prizes or other aspects of program configuration or whether the prize system was simply irrelevant is not clear but the fact that the accountability policy survived indicates that the government believed it still had a role to play.

As part of the review of ongoing policies, in 2004 and 2005 the New Millennium School Prize program was overhauled through laws that revoked previous procedures, created a school certification program and, for certified schools, the Outstanding School of the Year Award with new procedures for awarding prizes. These stipulated that prizes would go to the best 50 schools on the basis of a variety of indicators including pass rates, dropout rates and pupil scores in maths and Portuguese as determined by SPAECE. With these new procedures a total of 41 prizes are now to be awarded to schools on the basis of school improvement from one year to the next and 9 for absolute levels of average school performance (Ceará, 2006).

The reformulated policy has as its stated objective “to reward school management committed to the success of its pupils; to promote public recognition of the quality of state schools; to develop a culture of school monitoring and assessment, and thereby contribute to improvement in the quality of teaching” (Ceará, 2006, p.3). Three categories (Only Elementary, Elementary and Secondary, and Only Secondary) have been created to permit schools to compete with similar institutions in terms of the level of education offered, and within each category the number of quality indicators varies from 10 to 22, with the SPAECE pupil performance indicators representing 60% of the total score. Other indicators include average pass rates for each school grade and dropout. Each category of school is eligible for 3 of the overall performance prizes and a number of school improvement prizes that varies from 8 to 18 according to the number of schools in the category.

With these changes some of the previous problems associated with the comparison of schools with different levels of education have been ameliorated.  The awarding of prizes for school improvement from one year to the next also reduces other risks associated with cross-sectional approaches to accountability based solely on one-time average performance measures, given that the composition of the student body is unlikely to change to any significant degree. However, the major problem of comparing schools with pupils of different socioeconomic backgrounds has yet to be resolved. While this “status change” model of accountability reduces the impact of previous levels of learning (Hanushek and Raymond, 2003), it does not control for the different rates of learning of different social groups or for between measurement changes in other factors associated with student learning.

There has been no external evaluation of the Ceará government’s school improvement policies but the survival of the prize-giving idea, albeit in a reformulated version, suggests that this type of accountability, in which schools are not obliged to participate, provokes less resistance than system-wide programs in which all schools must be compared, such as the program in Sao Paulo state, even if there is no disincentive for those schools with lower evaluations. 

3.4. The Municipality of Sobral (Ceará)

During the last year of the 1997-2000 municipal government’s period in office, the level of literacy of schoolchildren in Sobral was measured twice (INEP, 2005). The first test, carried out by a specialist from the Federal University of Ceará, showed that half of the children at the end of the second grade were still unable to read. In the second evaluation, carried out on all pupils in the 1st to 4th grades by the Ayrton Senna Foundation, Sobral came last but one among the more than 400 municipalities assisted by that Foundation through its Accelerate Brazil Program. On the basis of these results, the incoming government established the improvement of literacy teaching as its prime objective and created new policy based on the reorganization of the education secretariat’s pedagogical support system and on greater school autonomy. The first involved the setting of goals, the continual supply of materials and training, the hiring of external assessments and the inculcation of a new culture for the monitoring of pupil progress. With their new autonomy, schools were to select the best teachers for the literacy classes, deal individually with the problems of each pupil, constantly evaluate results and mobilize the families of the pupils for the task of reaching school goals.

A monthly bonus for literacy teachers worth R$100 (approximately US$47) was also created in 2001 as part of the new policy to improve reading and writing results. Teachers had to achieve 75% literacy in the 1st basic grade (6 yr olds) or 90% literacy in the following grade (7 yr olds) or 100% in the 2nd to 4th grades. The school with 90% literacy among children from the 1st to the 6th grades, providing it reached 75% in the 1st basic grade and 70% in the school as a whole, was also eligible for the School Literacy Prize, also created in 2001.

There have been multiple problems with this incentive system, such as what to do with teachers who satisfy the criteria one year but fail to do so the next. In response to this, a new municipal law was passed permitting reductions of 25% and 50% to the bonus. A second problem, in response to the increased pressure to post high marks, was that schools began to withhold children from the test by saying they were absent or had been transferred (INEP, 2005). This was addressed by the creation of a new law requiring schools to prove that students had been transferred and mandating a minimum 98% turnout on the day of the external test.
The use of external assessments, as developed by specialists from the Federal University of

Ceará and the well-respected testing organization the Carlos Chagas Foundation, has permitted the posting of good overall results for Sobral’s literacy policy. According to one source (INEP, 2005), between 2001 and 2004 the level of literacy achieved in the first basic grade (6 yr olds) increased from 33.7% to 89.1% and in the first regular grade for 7 yr olds from 49.1% to 92.2%.
As components of a broader policy containing numerous other investments, the exact contributions of the bonus for literacy teachers or of the School Literacy Prize are difficult to gauge. However, as neither a recent report on the Sobral experience (INEP, 2006) nor the Municipal Secretariat’s site on the internet make any mention of the teacher bonus, it seems likely that having created widespread discontent this part of the accountability policy has now been shelved. The continuation of the School Literacy Prize indicates that the authorities see it as having more positive effects than negative and as a valid contribution to the overall literacy policy. 

4. LOW STAKES, REPORT CARD PROGRAMS

4.1. Paraná State

A pioneer report card program called the School Bulletin was created in 2001 by the State Secretariat of Education of Paraná as part of a strategy to increase accountability by schools and empower parents to demand results (Ayres, 2003). A less publicized purpose was to strengthen the parent associations and counterbalance the power of the teachers union that was then at loggerheads with the State government.{5} The underlying theory of this strategy appears to be that schools would feel pressured for better results if parents were better informed as to how their schools compared with others.

Distributed to all parents and members of school staff, the bulletin was organized in a four-page newsletter format that provided three blocks of information. The first block, with results from the state assessment program for 2000, contained the school, municipality and state averages for 4th and 8th grade maths and Portuguese, all classified in accordance with 4 levels of performance. The meaning of these performance levels was explained in a supplement that described the abilities in maths and Portuguese associated with each level. In 2002 the School Bulletin added an indication of whether the school had achieved the ‘expected’ level of performance given the average SES of pupils for each grade. The different levels were established on the basis of state-wide calculations using pupil socioeconomic data drawn from a pupil questionnaire applied at the time of the assessment and then included in the Bulletin as what was called a “value-added” measure of school performance.

The second block of information contained official school census data, including pass, fail and dropout rates, also with municipal and state comparisons. The third block presented subjective information from pupil, principal and parent questionnaires regarding the school (Ayres, 2003).  The voluntary parent survey, also introduced in 2001, represented the first attempt to capture parental opinions and perceptions on key elements of the school environment, including their own involvement.  In the words of the then Secretary of Education, it was to “trigger a process through which parents increase their awareness about their role as educators, as citizens and as consumers of education services – for their children” (Saliba, 2003). In 2001, 53,000 parents agreed to visit their children’s schools to answer the questionnaire, a number that increased by 50% when the survey was repeated the following year. 

Parents were also asked to give an overall grade for the school but as school principals became wary that the grade might effect their performance evaluations and sought to influence parent responses, the average grade could not be taken as a true reflection of parent opinions (Ayers, 2003). With the change in state administration in 2003 and the nomination of a Secretary of Education with close ties to the teachers unions, the report card system was completely abandoned. The difficulties of instructing parents in the interpretation and use of the report card, the lack of real communication between parents and teachers and the overall lack of confidence of teachers and principals regarding the purpose of the policy likely led to its early abandonment, although no studies have been done. Despite the considerable efforts in training parents, and in establishing a multiplier process whereby those parents would train others, the demise of the Paraná School Bulletin shows that while the parent associations had been a key element in the design of the accountability strategy they were powerless to defend it in the face of teacher union opposition.  

4.2.  The Municipality of Belo Horizonte (Minas Gerais)

Created in 2006 and designed for use by school administrators and teachers, the Belo Horizonte School Bulletin is the most recent of the low-stakes educational accountability systems for which information is available. In its first version, the four-page report card was distributed to the schools rather than to parents and prior training given only to the school liaison officers who act as intermediaries between the schools and the municipal secretariat of education.

The first Bulletin was the first ever attempt by the secretariat to give schools an overview of their own results as well as an opportunity to compare themselves with other schools with similar characteristics. This followed from the explicit purpose of the School Bulletin to furnish schools with the necessary data for self-evaluation, to promote the review of school plans and objectives and establish parameters for comparisons over time (SMED, 2006).  General indicators, to supply school context, included the size and level of complexity of the school, the race/ethnicity and SES of students and the number of pending administrative/financial matters as an indicator of administrative efficiency. Teacher information included level and type of education and, for the first time, the average number of days lost by the school through health-related teacher absences per year per class. The inclusion of this indicator, ranging from 3.3 days/class to an astounding 98.5 days/class is a further indication of the problem of public sector teacher absences.

Pedagogical indicators include external exam results on State (SIMAVE) and national tests (Prova Brasil) with the percentage of pupils at each of four levels of performance and comparisons only with schools with similar SES. Further indicators show the degree of socioeconomic and racial equity achieved by the school, calculated on the basis of pupil performance data for the city of Belo Horizonte as a whole. These indicators were considered of special importance given the objective of social inclusion of the municipality’s ‘Plural School’ education policy, widely held as the hallmark of the workers’ party government in Belo Horizonte. The differences in the levels of performance for pupils from different socioeconomic groups and different racial groups within the school are compared with the differences in the city-wide school population. The schools are informed whether they had achieved a level of equity that reflected the average for the city or whether the differences between their pupils showed greater or lesser equity than in the population as a whole. By placing equity indicators in the report card beside traditional pupil performance indicators, the municipality made equity a component of its accountability policy and demonstrated the importance it attaches to teaching strategies that are socially and racially inclusive (SMED, 2006).

It is too early to evaluate the program’s impact but despite a highly politicized environment there have been no signs of a backlash. This is relatively surprising given that the local public sector teachers union has been a long-time critic of external assessment and the city of Belo Horizonte has been the seat of a radical Workers Party education reform to improve social inclusion by eliminating end-of-year pupil assessments. Reports by school inspectors suggest there has been interest and engagement with the School Bulletin rather than indifference, which may be attributed to the absence of any explicit consequences for below average performance and the careful efforts of the secretariat to explain the usefulness of the information for school planning purposes (SMED, 2006).

5. DISCUSSION/CONCLUSIONS

The first observation to make is that despite the widespread use of pupil performance assessment in Brazil there are very few accountability programs that use this data in such a way as to hold schools or teachers accountable for their results. For the accountability programs that exist there is no empirical data on their possible consequences, except in the case of Rio de Janeiro. The study of these programs from the point of view of their impact on school results cannot be an easy task given that most have undergone continual change since implementation and that three of the six examples have now been eliminated after between two and four years of  existence.

Regarding the characteristics of the different programs, it is significant that all four of the high stakes accountability policies have established some form of salary incentive in connection with school and teacher results. There are no examples of programs working with other forms of incentive, such as additional resources or career advantages for members of school staff. There are also no examples of programs using disincentives, such as school closure, lower teacher evaluations or high profile methods for the dissemination of school results. The programs that exist have all chosen to give pecuniary rewards to some schools (or teachers, in the case of Sobral) while leaving other schools without, on the assumption that school staff will make an additional effort to secure the prize or the higher bonus. The drop in performance levels in Ceará certainly raises questions as to the effectiveness of even this type of accountability program but with results dropping also in other states (Rodrigues, 2007), it is possible that the Ceará results would have been even worse without the Millennium School Prize.

Of the four high stakes accountability programs described, two have taken the form of prizes, and a third of an annual salary bonus, awarded close to Christmas. This would seem to suggest that while the state or municipal governments are keen to establish a connection between pupil results and the final pay of teachers, a way to create a more direct association for all schools, as in the case of Rio de Janeiro, either has not presented itself or has not been politically viable.

As prizes are a growing part of the educational scene, and represent a way of rewarding superior performance without the divisiveness created by other forms of calculating differential pay supplements, it is easy to see how they have become a central aspect of accountability policy.

A further characteristic is that only the more recent versions of the Ceará and Rio de Janeiro programs have incorporated measures to determine pupil and school performance change over time. Without being true measures of the ‘value added’ by schools, as neither employs a longitudinal model to accompany the same pupils, the degree of school progress over time is clearly a more adequate indication of effectiveness and a better basis for the calculation of salary supplements than one-time school averages. This means that the other programs, as well as the earlier versions of the Ceará and Rio programs, are all open to criticism regarding their simplicity concerning the crucial measure of academic performance. The collection of data on other aspects of school performance has also been fairly rudimentary. There has been difficulty, therefore, in legitimating and maintaining procedures for the collection of data as well as for establishing the connection between evaluation indicators and their consequences.

One reason for the slow and apparently hesitant advance of accountability is the opposition of the teachers’ unions. This opposition has been both reactive, as in Rio de Janeiro, Paraná and São Paulo states, but also pre-emptive. In the case of Minas Gerais state, for example, the use of information derived from pupil assessments for the purpose of accountability is expressly excluded by Article 126 of the 2000 legislation creating the Minas Gerais System for the Evaluation of Public Education (SIMAVE){6}. This article prohibits the use of the results of the state system of assessment for the purpose of “classifying schools or other components of the Minas Gerais System of Education with a view to altering the educational process”. In this explicit rejection of the use of test results for the purpose of accountability, it appears the state legislators were swayed by the political power of the teachers’ union and other critics of external assessment.

When the public sector teachers and their unions are part of the governments’ power base, accountability policies based on pupil performance are unlikely to be implemented because of continuing suspicion of external assessment and active resistance to school performance evaluation. Even low stakes systems can be combated because of fear of negative consequences. When public sector employees, especially teachers, do not represent a significant source of government support they can even be cast as opposing government efforts to improve education quality for the poorest segments of the community, as in Rio de Janeiro, and their opposition ignored.  The survival of the Nova Escola policy in Rio is not a measure of its popularity nor indeed of its efficacy but of the decision by government to seek political support for its policies outside the education sector

The poor initial design of state and municipal accountability policies, often put into practice without adequate study and preparation, would seem to indicate another common problem, the lack of technical assistance and/or funding from central education authorities. As already suggested, this is the consequence of a division of responsibilities between the different spheres of government comprising the Brazilian federation that inhibits central government involvement in the specifics of state and municipal education policy. Beyond insuring compliance with the education reform act passed by Congress in 1996, the Ministry of Education in Brazil has no executive or legislative control over state and municipal secretariats of education or of their teachers. What financial influence it might exert is also limited, being responsible for only a minor part of basic education funding through the transferral of federal resources in accordance with fiscal equalization criteria. Under the currently accepted distribution of duties and responsibilities and the high degree of local political and financial autonomy, the Ministry has pursued the expansion of its own education assessment policies without attempting to mandate the use of assessment by other spheres of government. Under these rules, no technical assistance in the creation of accountability programs at the state or municipal level has either been requested or offered.

A further key to understanding the incipience of accountability policy is the level of awareness regarding the quality of education amongst those whose children are enrolled in public elementary and secondary schools. Despite the progressive decentralization of school management since the return to democracy in the early-1980s and the ensuing efforts to involve parents in school affairs, the flight of the middle classes to the private education sector (Chaui, 1999) has meant lower levels of mobilization and less pressure for change from those whose children continue to attend public schools.  With the lack of a regular supply of information in the press beyond the yearly publication of the national assessment results,  reflecting the absence of any larger political movement in favour of educational change, those parents with the most to gain from accountability programs concentrating on pupil results are the least able to demand such policies.

Despite the absence of academic debate and only limited expressions of public support for existing accountability policies, there have been signs of interest from the National Education Research Institute belonging to the Ministry of Education (INEP/MEC). These signs have yet to become anything approaching guidelines for state and municipal governments but given that the volume of resources to be transferred to lower levels of government is set to increase under new legislation, the Ministry can perhaps be expected to play a more substantial role in the inducement of local accountability policies. Until now, the Ministry has played an entirely subsidiary role in the determination of educational standards by distributing text books and other material resources and by fostering adherence to national curriculum guidelines. Now, if concerned to give greater leadership, it could well use its resources in line with the pupil performance results uncovered by its own recently-created Prova Brasil and establish criteria with which to compare the performance not just of the schools but also of the local secretariats of education.

Although the Paraná report card did not survive for very long, the successful implementation of a similar instrument in the city of Belo Horizonte in what was a potentially hostile environment suggests that this might be the way forward in Brazil. There is still no objective evidence of change, but it is easier to imagine the acceptance and use of low-stakes information for the purpose of promoting school-level discussions of performance than external grading exercises that soon lose their diagnostic capacity due to the overarching importance of the material benefits with which they are associated.

While many of the criticisms regarding the technical shortcomings of existing accountability systems seem justified, it is early to discard the policy as inappropriate or ineffectual. In Brazil, legitimate sources of pressure for the improvement of educational results, either from parents, pupils or from teachers themselves, are largely absent. Without these primary sources of pressure for the improvement of standards there is good reason for state and municipal governments to continue to experiment with accountability policies that hold out the possibility of better results. In a dramatic scenario of sub-standard school performance that is disproportionate in its effects on the life chances of the poor, local government must re-evaluate its own activities as well as play a leadership role in the improvement of its schools.

 

REFERENCES

APEOESP (2006). Sindicato dos Professores do Ensino Oficial do Estado de São Paulo. Fax nº 48, 22/08/2006 At: http://www.apeoesp.org.br/teste/Fax/48_06.htm. Access 1/04/07

Ayres, S. (2003). Boletim da Escola: Diagnóstico Multidimensional das Escolas do Paraná. Ensaio, 38(11), jan-março. p. 33-44.

Brandsma, H. and Knuver, J. (1989). Effects of School Classroom Characteristics on Pupil Progress in Language and Arithmetic. International Journal of Educational Research, 13.

Brooke, N. (1992). The Equalization of Resources for Primary Education in Brazil, International Journal of Educational Development, 12(1) pp. 37-49.

Carnoy, M. e Loeb, S. (2004). A Responsabilidade Externa tem Efeito nos Indicadores Educacionais dos Alunos? Uma análise entre os estados dos E.U.A. Documento 29. Santiago de Chile: PREAL.

Carnoy, M., Loeb, S. y Smith, T. (2001). Do Higher State Test Scores in Texas Make for Better High School Outcomes? Consortium for Policy Research in Education, University of Pennsylvania (CPRE). Research Report Series RR-047.

Castro, M.H.G. (1999). Educação para o século XXI: o desafio da qualidade e da eqüidade. Brasilia : Instituto Nacional de Estudos e Pesquisas Educacionais.

Cavaliere, A.M. (2003). Entre o Pioneirismo e o Impasse: A reforma paulista de 1920. Educação e Pesquisa, 29(1),  pp.27-44.

CDEP (2006). Comitê em Defesa da Escola Pública. http://www.adusp.org.br/PEE/PEE.rtf Accessed:  12/12/2006.

Ferrer, G. (2006). Educational Assessment Systems in Latin America: Current practice and future challenges. Washington: PREAL.

Fórum Estadual em Defesa da Escola Pública (2003). Plano Estadual de Educação: Proposta da Sociedade Paulista. Fórum Estadual em Defesa da Escola Pública, São Paulo, 14 de outubro, 2003. At: http://www.adunicamp.org.br/noticias/educacao/PEE.pdf Accessed: 01/04/2007.

Gama, Z.J. (s.d.). Nova Escola: Erros e acertos do programa: Folha Dirigida, 18/03/ 2004 At: http://www2.uerj.br/~clipping/marco04/d18/dirigida_nova_escola.htm Accessed: 8/10/2005.

Gillard, D. (2004). Education in England: a brief history. At http://www.dg.dial.pipex.com/history/index.shtml Accessed 12/01/07.

Hanushek, Eric A. y Raymond, Margaret E. (2003). Lessons about the Design of State Accountability Systems. In Peterson, Paul E. and West, Martin R. (editors), No Child Left Behind? The Politics and Practice of School Accountability, Washington, Brookings Institution Press.

Hess, Frederick M. (2003). Refining or Retreating? High-Stakes Accountability in the States. En: Peterson, Paul E. and West, Martin R. (editors), No Child Left Behind? The Politics and Practice of School AccountabilityWashington, Brookings Institution Press, 2003.

INEP (2001). PISA 2000. Relatório Nacional. Brasília: INEP.

INEP (2005).Vencendo o desafio da aprendizagem nas séries iniciais: a experiência de Sobral/CE. Brasília: Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira.

INEP (2006). Prêmio Inovação em Gestão Educacional 2006, A Política de Alfabetização como estratégia para elevação do desempenho escolar nas séries iniciais do Ensino Fundamental, Sobral / CE. Laboratório de Experiências Inovadoras em Gestão Educacional, Relatório de Avaliação em Loco, September 2006.

Jacob, Brian A. (2003). A Closer Look at Achievement Gains Under High-Stakes Testing in Chicago. In Peterson, Paul E. and West, Martin R. (editors), No Child Left Behind? The Politics and Practice of School Accountability, Washington, Brookings Institution Press, 2003.

Jones, Ken. (1989). Right Turn: The conservative revolution in education. London, Hutchinson Radius.
Lapointe, Archie E., Askew, Janice M. and Mead, Nancy A. (1992a). Learning Science. Report No. 22-CAEP-02, Educational Testing Service, Princeton, NJ, Feb.

Lapointe, Archie E., Askew, Janice M. and Mead, Nancy A. (1992b). Learning Mathematics. Report No. 22-CAEP-01, Educational Testing Service, Princeton, NJ, Feb.

Lawton, Dennis and Chitty, Clyde. (1988). The National Curriculum. Bedford Way Papers 33. Institute of Education, University of London.

Lima, Alessio C., Coelho, Sylvia M. and Soares, Lindomar da S., (2005). Sistema Permanente de Avaliação da Educação Básica do Ceará: uma leitura dos resultados. In Secretaria da Educação Básica do Ceará, Gestão para o sucesso escolar. Fortaleza, Edições SEDUC, 2005.

Linn, R. (2000).  Asessment and Accountability. Education Researcher. 25: 4-16

MEC (2003). Qualidade da Educação: uma nova leitura do desempenho dos estudantes da 4a série do Ensino Fundamental. Ministério da Educação/INEP, Brasília. Abril 2003.

MEC (2006). Tabelas do Censo Escolar 2006. Ministério da Educação/INEP, Brasília. At:  http://www.inep.gov.br/imprensa/noticias/censo/escolar/news07_02.htm

MEC (2007). SAEB – 2005 Primeiros Resultados: Médias de desempenho do SAEB 2005 em perspectiva comparada.

OECD (2004). Programme for International Student Assessment. Learning for Tomorrow’s World. First Results from PISA 2003. OECD, Paris, 2004.

Rodrigues, J.R.S. (2007). Responsabilização e Resultados Escolares no Rio de Janeiro. Master’s Degree Thesis. Pontifícia Universidade Católica, Rio de Janeiro.

Saliba, Alcyone V. (2003). Have You Heard About The Paraná (Brazil) School Report Cards? Mimeo.

Secretaria da Educação de São Paulo (1995). Coordenadoria de Estudos e Normas Pedagógicas. En: Rama, L. M. da S. et al. (org.). Legislação de ensino de 1º e 2º graus: compilação. São Paulo: Secretaría da Educação de São Paulo.

SMED (2006). Prefeitura Municipal de Belo Horizonte/Secretaria Municipal de Educação Monitoramento da Inclusão: Manual do Informativo de Contexto e Desempenho para as Escolas Municipais de Belo Horizonte. Belo Horizonte: SMED.

West, Martin R. y Peterson, Paul E.  (2003). The Politics and Practice of Accountability, (2003). In Peterson, Paul E. and West, Martin R. (editors), No Child Left Behind? The Politics and Practice of School Accountability, Washington: Brookings Institution Press.


{1} The National System for the Assessment of Basic Education, modelled on NAEP, took on its current configuration in 1995.

{2} The State System for the Assessment of Basic Education of São Paulo, created by the State Secretariat of Education in 1996, differs from other state systems insofar as the tests are constructed, applied and corrected with the help of the teachers. The grades tested have varied from year to year. In its first years, from 1996 to 1998, the sampled pupils were tested on the previous year’s syllabus using a longitudinal model. In more recent editions the system has applied Portuguese and maths tests simultaneously to the 90% of pupils in all elementary grades and 80% in all secondary grades who voluntarily sit the tests. In 2004 Saresp became universal and tested approximately 4.5 million pupils of which 4.1 million were from the 5.422 state schools. The remaining pupils came from the smaller number of municipal and private schools that chose to participate. 

{3} The Real (R$) is worth approximately half a US dollar (US$0.48) at current exchange rates. 

{4} The Permanent System for the Assessment of Education in Ceará, created in 1992, assesses performance in Portuguese and maths for a sample of pupils from the 4th and 8th grade of elementary school and the 3rd grade of secondary school. In 2001, SPAECE adopted computer aided testing.

{5} The Union of Education Workers of Paraná led strikes against the government’s “neo-liberal” policies in 2000 and 2001 and members undertook hunger strikes in 1998 and 2000.

{6} Resolução 14, Feb. 3, 2000.