High Stakes Accountability in Urban Elementary Schools

WP-02-22

Institute for Policy Research Working Paper

High Stakes Accountability in Urban Elementary Schools:

Challenging or Reproducing Inequality?

John B. Diamond

James P. Spillane

Northwestern University

Work on this paper was supported by the Distributed Leadership Project which is funded by research

grants from the National Science Foundation (REC-9873583) and the Spencer Foundation (200000039).

Northwestern University's School of Education and Social Policy and Institute for Policy Research also

supported work on this paper. All inquiries about this research project should be directed to the study’s

Principal Investigator, James Spillane at Northwestern University, 2115 North Campus Drive, Evanston, IL

60208-2615 or [email protected]. All opinions and conclusions expressed in this paper are

those of the authors and do not necessarily reflect the views of any funding agency or institution.

Abstract

In this paper, the authors use data from interviews and observations in four urban

elementary schools – two high and two low performing – to examine how schools

respond to high stakes accountability policies. The authors argue that school responses to

high stakes accountability depend on school context. In low-performing schools,

responses focus narrowly on complying with policy demands, focusing on improving the

performance of certain students, within benchmark grades, and in certain subject areas.

In contrast, higher performing schools emphasize enhancing the performance of all

students regardless of grade level and across all subject areas. Given the concentration of

poor students and students of color in the lowest performing schools, the authors

conclude that issues of educational equity need to be given careful consideration in the

implementation of high stakes accountability policies.

One of the most consistent findings in educational research is that family

background is linked to children's educational outcomes, attainment, and adult

occupational status (Blau and Duncan, 1967; Coleman et al., 1966; Jencks et al., 1972;

Orfield, 1993). While education is viewed by many as an important mechanism for social

mobility – tied to the “common school” belief that all children should have equal

educational access and opportunity (Cremin, 1951; Muller and Schiller, 2000) – many

scholars argue that schools reproduce social inequality. Over the past decade, policy

makers have mobilized an arsenal of policy instruments in an effort to ensure that all

children receive high quality education. One increasingly popular but controversial

strategy relies on external accountability mechanisms, including high stakes testing, to

transform instructional practices and make teachers and students more accountable for

their performance. Critics argue that these policies will exacerbate inequalities by leading

teachers to marginalize low-performing students (Clotfelter and Ladd, 1996; McDill,

Natriello, and Pallas, 1986) and causing teachers to teach these students only the material

covered on standardized tests. Others contend that such policies are misguided because

limited resources, unprepared teachers, and ineffective instructional practices rather than

incentives are the problem that needs to be addressed (Darling-Hammond, 1994).

Proponents argue that such assessments will reduce gatekeeping processes such as

tracking and low teacher expectations that disadvantage certain students. External

assessments, it is argued, provide objective information for school-based decision-

making and therefore work against more subjective judgements that contribute to

stratification (Muller and Schiller, 2000; Coleman, 1997). For example, supporters argue

that teachers’ assessments of students’ ability as well as decisions about course placement

and grouping arrangements inside classrooms could be based on more objective

information from standardized tests (Muller and Schiller, 2000).

Given the increasing emphasis on external assessment and accountability, and the

strong arguments on both sides of the issue, it is interesting that with a few notable

exceptions (Muller and Schiller, 2000; Roderick, Bryk, Jacob et al., 1999) research on

how these policies unfold in schools and the mechanisms through which they impact

student learning has been limited. Moreover, few studies have explored how the

implementation of these policies may be situated in certain school contexts and how this

may influence their impact on students.

The study that is reported in this paper examines the implementation of high-

stakes testing in high and low performing schools. We seek answers to two related

questions. First, is high stakes accountability policy perceived and implemented

differently in high and low-performing schools? Second, does the implementation of high

stakes testing in these schools suggest that it will reduce social stratification through the

mechanisms outlined by testing proponents?

High Stakes Accountability and Stratification

One aim of accountability policies is to insure that all students receive high

quality instruction and reach a certain level of competence in core subject areas (Muller

and Schiller, 2000). Some districts, like Chicago, have adopted a high stakes version of

these policies that link student performance on examinations to consequences for schools

and, in some instances, students themselves. Opponents of these policies argue that for

these approaches to be fair, instructional changes should precede consequences for

students (Heubert and Hauser, 1999) and that such policies create incentives for

marginalizing low performing students (Clotfelter and Ladd, 1996; McDill, Natriello, and

Pallas, 1986). Neutral observers caution that those implementing such policies must

insure the adequacy of educational resources for the tested students, attend to the

reliability and validity of the exams for their intended purposes, and avoid basing

decisions on one test (AERA 2000).

Proponents of these policies suggest that they can reduce inequality through

increasing student motivation, creating incentives for teachers to seek improvements in

student outcomes, providing more objective information about students’ performance for

school based decision-making, and increasing academic press in schools – particularly

those serving low-income and minority students (Coleman, 1997; Shouse, 1997; Muller

and Schiller, 2000).

Coleman (1997) advocates the development of output-driven schools, a key

component of which would be external assessment and accountability. He argues that

external assessments such as student performance tests would create new incentives for

school improvement, providing objective information for teachers to assess students and

make course placement decisions, thereby reducing the gatekeeping functions of schools

(Coleman, 1997; Muller and Schiller, 2000). Shouse (1997) argues that designing more

output-driven schools would also increase academic press and have particularly

beneficial consequences for students with lower socioeconomic status. Therefore,

proponents of these policies suggest that three key mechanisms – the creation of new

incentives, the provision of objective information for school decision making, and the

increase of academic press – will combine to reduce schools’ and teachers’ gatekeeping

practices and contribute to a reduction in stratification.

While arguments have been forwarded in support of and in opposition to these

policies, surprisingly little research closely examines how they play out in schools. The

data on outcomes that does exist presents a mixed picture. Data on the implementation of

high stakes testing in Chicago suggests that the percentage of students meeting minimum

competency requirements has increased since the introduction of the policy (Roderick et

al., 1999). However, the policy has differential impacts on students based on their family

background characteristics. For example, African American students were retained at a

much higher rate than their white and Latino/a counterparts because they tend to score

lower than whites and because of the higher proportion of Latino/a students who are in

bilingual programs and therefore exempt from the policy (Roderick et al., 1999). This

likely has a stratifying effect for African American students because grade retention may

be associated with negative long-term outcomes including reduction in self-esteem and

increased likelihood of high school drop-out (Roderick, 1994).

Muller and Schiller (2000) examine how state-level testing policy impacts high

school students’ graduation rates and mathematics course-taking. They show that these

policies equalized students’ academic attainment and reduced the impact of teachers’

gatekeeping through low-expectations, seeming to support the arguments of supporters of

these policies (Muller and Schiller, 2000: 210). However, they also find that when state

tests link students’ performance to consequences for schools it leads to stratification

based on SES, lending support to the arguments of opponents of testing policies.

Therefore, their findings do not strongly support the arguments of proponents of these

policies or their critics. They recommend more research using both qualitative and

quantitative methods to explore the mechanisms through which these policies influence

teachers’ practices and students’ outcomes. Research into the processes of accountability

policy implementation will help inform this discussion.

In this paper, we argue that in order to understand the implications of these

policies it is important to examine how they are understood and implemented in

particular school contexts. In the current paper, we examine schools’ responses to high

stakes accountability policy, paying particular attention to the implications of these

responses for issues of educational equity both within and across institutions. More

specifically, we examine how teachers and administrators in high and low performing

schools respond to high-stakes accountability policy focusing on their responses to

incentive structures, their interpretation and use of test score data, and their subsequent

instructional priorities. We argue that these responses to high stakes accountability are

situated in a school’s status with regard to accountability policy – probation versus high

performance – and argue that school status is correlated with students’ race and social

class. We conclude that differences in responses to accountability policy in different

types of schools may increase rather than reduce educational stratification.

The paper is organized as follows. First, we outline the theoretical tools used to

frame our discussion. Following this, we discuss the methodological approach that

guided the research. We then examine differences in schools’ responses to high stakes

accountability in four elementary schools – two high performing schools

and two

probation schools. In the final section we discuss these cases, paying particular attention

to the implications of school-level responses to high stakes testing for issues of

stratification. We argue that the picture emerging from these cases suggests that the ways

in which these policies are implemented in particular school context may exacerbate

rather than reduce educational stratification.

Theoretical Tools

Research on the role of family background and educational stratification

demonstrates consistent links between socioeconomic status and students’ outcomes.

Some explanations for this pattern focus on direct effects of family background such as

class-based disparities in parents’ beliefs and involvement patterns (Lareau, 1989; Sewell

and Shah, 1968 (a), Sewell and Shah, 1968 (b); Sewell and Hauser, 1980), family

structure (i.e. number of parents in the home or the number of siblings in the family), and

access to extra-familial resources through parents’ social networks and institutional

affiliations (Coleman, 1988; Coleman and Hoffer, 1987; Wong, 1998; Carbonaro, 1998;

Hao and Bonstead-Bruns, 1998; Hofferth et al., 1998; McNeal, 1999; Wilson, 1987).

Other scholars focus on the interaction between these background characteristics and

school practices. In these accounts, schools impact students though micro-political

processes such as low teacher expectations (Roscigno, 1998; Brophy and Good, 1973;

Rist, 1970, 1977; Rosenthal and Jacobson, 1968), tracking (Oakes, 1995), and cultural

reproduction processes (Bourdieu, 1979; Bourdieu and Passeron, 1977).

A third approach – institutional stratification– highlights the implications of inter-

organizational processes for social stratification (Roscigno, 2000). Because race and

social class shape school attendance patterns and contribute to the creation of highly

segregated school contexts (Orfield, Bachmeier, James, and Eitle, 1997), family

background can contribute to stratification through the distinctly different characteristics

of the schools students attend (Roscigno, 2000). Differences in schools’ monetary

resources (Elliot, 1998; Greenwald, Hedges, and Laine, 1994; Hedges and Greenwald,

1996; Kozol, 1991), instructional quality (Smith, Lee, & Newman, 2001), presentation of

valued knowledge (Anyon, 1991) course offerings (Ayalon, 1994), and social

organization (Bowles and Gintis, 1976), all contribute to the maintenance of social

stratification.

Roscigno’s institutional stratification perspective captures these inter-

organizational dynamics, emphasizing the “multi-level and inter-institutional nature of

racial educational disadvantage” (Roscigno, 2000:271). Roscigno writes:

Arguably, the most important of these inter-institutional linkages in relation to

race/class reproduction in education has to do with family background inequalities

and their consequences for achievement through the character and resources of

the schools one attends. What this means, more straightforwardly, is that family

background shapes residential options. Where one resides, in turn, has a large

impact on the school one attends and, consequently, achievement [emphasis in

original] (Roscigno, 2000:271).

He concludes that residential segregation leads to indirect effects of family background

through enrollment patterns in public versus private schools, the race and social class

composition of schools, monetary expenditures, and school climate (Roscigno, 2000).

Other work demonstrates that the concentration of low-income and African American

students in certain schools may have detrimental implications for student outcomes apart

from the individual characteristics of students (Bankston and Caldas, 1996). Taken

together, this work demonstrates that educational stratification based on race and social

class is at least partially maintained through race and class-linked institutional processes.

Building on this work, we argue that the implications of policies like high stakes

accountability are also shaped by institutional stratification processes. In contemporary

urban contexts there are several types of schools including private schools (both religious

and non-religious) which are often thought to be the highest quality, magnet schools

which are often considered the “elite” public schools, and neighborhood schools which

can be further divided into high and low quality categories. Social class and race are

important in patterning the schools that children attend, with the more highly valued

settings being most accessible to middle- and upper-class children. The different types of

public schools are likely to implement the policy differently. Therefore, if students are

concentrated in different types of schools based on race and social class, they will be

impacted by the policy in distinct ways.

Taking the Chicago Public Schools as an example, data

shows that schools on

academic probation have a higher percentage of African American and low-income

students, on average, than the typical Chicago Public School. African Americans make

up 52% of the district’s student population but 83% of students attending probation

schools. Likewise the district average of low-income students is 84% while the average

for probation schools has 92% low-income students. Perhaps most interesting, however,

is the fact that while white students make up 10% of the district student population, they

make up less than 1% of the students attending elementary schools on probation.

These figures are more striking when compared to data from Chicago magnet

schools which consistently rank among the district’s highest performing schools.

Elementary magnet schools contain 55% low-income students (compared to 92% for

probation schools) and 53% Black students (compared to 83% for probation schools).

White students make up 17% of elementary magnet school students (as compared to less

than 1% in probation schools).

These data demonstrate that structural processes related

to family background shape students’ access to schools of different quality. African

American and low-income students are more likely to be found in the lowest performing

schools while white and middle income students are more likely to be found in higher

performing magnet schools.

Table 1 presents the mean percentage of students by race

(black/white) and social class in the Chicago district as a whole as well as its magnet,

high performing, and probation elementary schools.

[insert table 1 here]

The multiple factors that contribute to this process are beyond the scope of this

paper, however, as we shall see, the observed patterns likely have important

consequences for students’ educational experiences.

In the discussion that follows, we

focus on school status in relation to high stakes accountability – probation versus high

performance – as an important factor that shapes schools’ responses to accountability

policy and students’ access to educational equity.

Research Methodology

This paper is based on data from the Distributed Leadership Project, a four-year

longitudinal study of elementary school leadership funded by the National Science

Foundation and the Spencer Foundation. The project began with a six-month pilot phase

during the Winter and Spring of 1999 involving seven Chicago elementary schools, four

interview only sites and three schools where we conducted interviews and extensive

fieldwork. The first full year of data collection (Phase 1) began in September 1999 and

focused on eight Chicago elementary schools, two of which were also part of the study’s

pilot phase. Year 01 data collection was completed in June 2000, and involved between

50 and 70 days of fieldwork in each of our eight sites. The four schools that are the focus

of this paper were selected because two were the highest performing in our sample and

the other two were probation schools and the lowest performing.

Site Selection. We used a theoretical sampling strategy (Strauss 1987; Glaser &

Strauss, 1967) to select study schools based on two primary dimensions. First, the schools

vary in terms of student demographics, including seven schools that are predominantly

African American, three that are predominantly Hispanic, and three that are mixed.

Second, while we are chiefly interested in schools that had shown signs of improving

mathematics, science, or literacy instruction (in terms of either process or outcome

measures), we also wanted to study some schools that had managed little change in

student outcome gains. We focus our analysis in this discussion on two high performing

schools and two schools that are on academic probation. In table 2 we outline schools by

racial composition and percentage of low-income students.

[insert table 2 here]

Data Collection. Research methodologies include observations and structured and

semi-structured interviews. In the schools reported on in this paper, researchers observed

school leadership events, meetings, and classroom instruction in grades 2 and 5 and

conducted interviews with teachers and school leaders. During Phase 1 of the study,

researchers spent the equivalent of three to four days per week per school over a ten-

week period in the Fall of 1999 and a 12 week period in the Spring of 2000. Leadership

events observed in these schools to date included grade level meetings, faculty meetings,

school improvement planning meetings, professional development workshops, and

supervisions of teaching practice. In addition, we observed a number of other events

including homeroom conversations between teachers, lunchroom conversations, grade

level meetings and subject specific workshops and meetings.

We completed interviews with teachers at the second and fifth grade levels and

school leaders (including lead teachers). Interview protocols focused on school leaders’

agenda and goals, their responsibilities, and the key tasks they perform as part of

promoting instructional change in mathematics, science and literacy.

We also selected

specific instances of school leaders’ practices to observe and then conducted post-

observation interviews with these leaders about the observed practice.

Using the protocols, researchers wrote detailed fieldnotes following each

observation. A total of 181 sets of fieldnotes detailing anything from 30 minute meeting

observations to three hour professional development workshops were compiled thus far.

All interviews were tape-recorded and transcribed.

Data Analysis. Data collection and data analysis are closely integrated, allowing

us to examine patterns and working hypotheses as they emerged from data analysis and

refine data collection strategies (Miles & Huberman, 1984). Coding categories were

developed based on the distributed leadership theoretical framework and initial analyses

of our observation and interview data. A commercial computer based qualitative coding

program – NUDIST - was used to code all project data. NUDIST allowed us to code the

emerging ideas and concepts from the data into free nodes that can be compared and

related to each other, forming larger “parent” nodes that can be stored in an index system

that brings the different components of the project together. Coders worked together to

code transcripts initially in order to develop a shared understanding of what each code

meant. Once coders had developed a “taken as shared” understanding of these codes,

they worked independently. We also used our field notes (which document the actual

observed practice of leadership) and interviews to construct our case studies for this

paper.

High Stakes Accountability: The Case of the Chicago Public Schools

A popular public policy strategy in recent efforts to improve America’s schools

involves holding school’s accountable for student achievement. Arguing for a K-12

curriculum that is grounded in more intellectually rigorous content, reformers propose to

use a variety of policy levers to hold schools accountable for students’ mastery of this

content. These policy efforts involve at least two components: specific student

performance outcomes and rewards and sanctions for schools (Clotfelter & Ladd, 1996;

Elmore, Abelmann, & Fuhrman, 1996). We identify each of these components below

and use the Chicago Public School’s accountability policy, an often referred to example

of a successful accountability policy in both policy and academic circles, to illuminate

each component.

To begin with, student performance outcomes as measured by tests, rather than

inputs (e.g., number of certified staff), are the primary mechanism that state and local

government agencies use to hold school’s accountable. While the 1988 Chicago School

Reform Act (P.A. 85-1418) included the decentralization of decision making to the

school site level and the formation of Local School Councils (LSC), the Chicago School

Reform Amendatory Act of 1995 gave much authority to the chief executive officer,

appointed by the mayor, who was able to place poorly performing schools in remediation

or on probation based on their performance on the Iowa Test of Basic Skills (ITBS).

Specifically, student performance on the ITBS at benchmark grades became the districts’

primary measure of school accountability and progress.

The second component of most accountability measures involve the creation of a

system of rewards and sanctions as well as intervention strategies designed to motivate

schools to improve student achievement. In Chicago Public Schools the key sanction is

the power of the Chief Executive Officer to place schools on probation because of low

performance as measured by standardized test scores. For example, in 1996 the CEO

placed twenty per cent of the elementary schools, 109 schools, on probation because

fewer than 15% of their students performed at or above national norms on the reading

and mathematics sections of the Iowa Test of Basic Skills (ITBS) (Hess, 2000; Wong &

Anagnostopoulos, 1998).

Schools on probation are required to develop a supplemental

school improvement plan that outlines specific strategies the school will take to improve

student achievement and defines criteria that will be used to judge the school’s progress

toward improvement. For technical assistance, schools use their discretionary funds to

purchase the services of an external partner whom they can select from a district-

approved list. If the district decides that a school has not made adequate progress, the

CEO can have the school reconstituted, ordering new LSC elections and replacing the

principal and faculty.

Some efforts to transform accountability arrangements also include rewards or

sanctions for students. This is important because, as some scholars note, teaching is co-

produced by teachers and students (Cohen and Ball 1996). Hence, an accountability

system that targets teachers and school administrators exclusively may place them in an

impossible position as they depend on their students to improve school performance. In

1996 the school district also ended social promotion, informing the students that

beginning with the 1996-97 school year if they failed to achieve at a certain level on the

ITBS they would have to attend summer school. Further, if by the end of the summer

students still fail to achieve at the required level on the ITBS they are not promoted to the

next grade level. These developments are important because the incentive structure

mobilized by the school district targets students in addition to teachers and

administrators.

General Impacts of High Stakes Testing

One of the core arguments of proponents of high stakes testing is that it provides

incentives for school improvement. In simple terms, when schools, teachers, and students

are made accountable for students’ outcomes, it is argued, they will seek to improve

them. Data from our schools indicate that school leaders do pay attention to high stakes

accountability. There were general patterns that cut across all of the schools in our

sample. Leaders at each school report paying some attention to the exams. Even in the

highest performing schools, schools where 50% or more of the students score at or above

national norms in core subject areas, school leaders report paying some attention to exam

results. In addition, leaders at both high and low-performing schools reported seeking

improvements in students’ outcomes. As the principal at one of the higher performing

schools reported:

When I look at the test results – and I happen to be one that believes that the test

results do tell you something about curriculum –Fifty percent are succeeding. I

look at it the other way, fifty percent of our children are not succeeding …

Hopefully our scores will go a notch up. You know, even one percent to show

that there is some effort and some results.

The push for improvement was consistent among school leaders across all of our schools.

None of the school leaders we interviewed was completely content with student

performance.

All of the schools also engage in some form of explicit test preparation activities.

There are differences, however, in the strategies used and the frequency of these

activities. At one school, which engages in the most extensive test preparation, students

are tested every Thursday. The stated goal at this school is to make the testing

environment comfortable for the students so that when they take the actual exam they

will not be overwhelmed or feel undue pressure. As the Assistant Principal at this school

explained:

We expose them to the testing situation at least once a week and we use the

bubble forms except for the little ones because their test is not like that. In other

words we try to simulate the testing even with the writing part of the test. I’ve

shown the teachers how to do booklets that simulate the real test. So we try to do

whatever we can so that our children are accustomed to taking tests so they’re test

smart kids and they’re not nervous.

Another school used a five-week assessment to structure instructional activities

for students and target specific problem areas. Finally at other schools these activities

were more sporadic, occurring increasingly often just prior to the testing period.

We also found that testing structured schools’ and teachers’ priorities with regard

to the content covered in classrooms and the attention paid to different subject areas.

With regard to instructional content teachers reported that testing was outmatched only

by other teachers and textbooks as an influence on the content they covered.

One

specific manifestation of this influence on content coverage was the lack of attention to

science instruction when compared to reading and mathematics. As a field, elementary

education is characterized by the differential valuation of mathematics, science, and

literacy instruction. Although most elementary teachers do not have a well-defined

subject matter specialization and do not work in situations where organizational

arrangements (e.g., departmental structures) directly support subject matter identities,

subject matter is an important context for teachers’ work (Stodolsky, 1988). A general

pattern at our schools was that science instruction was given lower priority than other

subject areas. Across our schools we found that when compared to mathematics and

language arts instruction, science had fewer formal and informal subject matter leaders,

less attention from school administrators, and fewer instructional specialists (Spillane,

Diamond, Walker et al. 2001). Further, our data suggest that accountability policy may

accentuate this undervaluing of science education. When asked how important testing

was on the content they covered in science, teachers ranked it below principals, other

teachers, standards, textbooks, and instructional specialists. Teachers in both high and

low-performing schools captured the situation:

So I go to my grade chairperson and she'll give me a list of the ten objectives in reading

and math that I must teach. Science and social studies are more flexible because the

students are not tested on the IOWA’s [Iowa Test of Basic Skills] in science and social

studies so that's more, you know, on the teacher's personal decision.

You know science isn’t one of your guides for whether a child is promoted or

graduates. So reading and math are what are stressed because those are what

everybody looks at. And to a certain degree, that’s what the teachers look at too. You

know I’ve got to get you on. I’ve got to get you out of this building. You’ve got to

get this in math, you’ve got to get this … in reading. So those two always come first.

We aren’t able to teach science as much as I would like to. Mainly because on the 3

grade level we aren’t tested on [science and social studies] we’re not tested on those

subjects we are tested on reading and math. … I just can’t fit it in. [There is] so much

math and so much reading that it’s hard to fit the science and social studies [in]. So

most of the time … I begin teaching science and social studies after the test.

Therefore, across all of our schools, and without regard to performance level, schools

paid attention to tests results and sought to improve students’ outcomes on them.

According to teachers’ reports, testing and accountability policy also influenced the

instructional content covered and the priority placed on different subject matter areas,

with science receiving less attention than reading and math. While general patterns

existed across the schools, school personnel at high and low performing schools

responded to these policies differently. The following section discusses some of these

differences, comparing responses to accountability policy at high and low performing

schools.

School Status and The Nature of Incentives

Proponents of high stakes accountability argue that it will create incentives for

instructional improvement. We argue that how schools respond to these incentives

depends at least in part on their status in relationship to accountability policy. In Chicago,

the incentives for probation schools are clear and direct, they are organized around the

threat of reconstitution. To avoid this schools must get off of probation and this becomes

the goal. In high performing schools, the incentives are structured around rewards more

than sanctions, with recognition for high performance likely being the primary motivator.

The following section discusses the ways in which school leaders (and schools as a whole

to a certain extent) at high and low performing schools orient themselves toward

accountability policy and the ways in which the incentive structures seem to shape these

orientations.

Incentives in Probation Schools

The two probation schools – Waxton and Field – emphasize getting off of

probation. As Waxton School’s principal stated when asked about the schools goals, “the

obvious goal is to get off probation! Now that’s it in a nutshell.” This emphasis on getting

off of probation in these two schools had an important impact on their responses to

accountability policy. While probation status allows school leaders to demand teachers

attention, the need to respond to immediate pressures from external actors such as

probation managers led to several related responses which tended to be cosmetic and

superficial with regard to classroom instruction.

The principal at Waxton School used the accountability policy as a “stick” to get

teachers’ attention and motivate them. She was in her first year at the school and used

accountability policy as a way to legitimize her push for teachers to improve. She

reminded teachers that she would replace those who were not pulling their weight,

suggesting that this was a “requirement” she would have to reluctantly enforce. In one

staff meeting, for example, she told teachers about how she had been asked by the

probation manager (and the central office) to begin the process of identifying teachers

who might potentially be replaced. She explained to the teachers that, although she

“could no longer ignore” these requests by the probation manager, she would drag the

process along to give each one of them “a chance” to improve. This was a tone she

consistently sounded, seeking to motivate teachers. In this sense, accountability policy

(and the set of actors associated with it) serves as a threat, a way for school leaders to get

the attention of teachers and push them to change their practices.

The external partner at Field School believes that teachers at this school are open to

change. As she has observed at this school and at others “teachers in probation schools

are more likely to be open [to change]. They know the next step is possibly

reconstitution.” Therefore, at both of the probation schools there is a sense that

accountability policy can be use to motivate teachers to be responsive to the efforts of

school leaders.

Another motivation strategy used at Waxton school was the pep-rally strategy. Here,

leaders sought to encourage teacher effort by expressing their confidence in their abilities

and promising that with hard work they could get off of probation. Consider the first

meeting of Waxton’s staff with the school’s new probation manager:

There was clapping of hands as Beatrice stood up to speak. She began by saying "it

is possible to get off probation" (another loud applause) and "you are going to get off

probation." (another loud applause). Beatrice then gave a little talk about how she

thinks the staff and children at Waxton are capable and can achieve. "Sometimes",

she continued, “it's only a question of knowing what to focus on, getting children to

be ready for the tests and getting the right tools.”

There was an air of a revival meeting atmosphere in this meeting, the emphasis being

liberation from the bondage of probation. This revival meeting approach was evident

again at another meeting:

Beatrice quickly read off the next activities on her list (noting that they were

running out of time) "Follow up in the classrooms after discussing each

strategy," "test-taking – we will do breakdown of test skills, so you focus on the

right skills that are asked in the tests" "We did this at [Maxwell] School and

found it very helpful" (“yessess” from teachers and interjections of "that's what

we need!"). Beatrice's colleague emphasized that Waxton will "get off probation"

which was followed by applause.

In a certain sense this strategy dovetails well with the use of probation as a stick to

motivate teachers. Being placed on probation is likely depressing. It labels the school as a

failure and places it in jeopardy of being “reconstituted.” In addition to this if school

leaders work too hard to challenge teachers without some sense of emotional support, it is

likely that it will be difficult to make change. This pep rally/revival meeting approach

may be well suited for creating social support in the push for academic improvement.

At the probation schools, accountability policy, and particularly probation status,

enhanced school leaders’ ability to get teachers’ attention. This is a potentially powerful

tool in the effort to implement instructional change, a tool that proponents of high stakes

accountability point to in support of these policies.

Managing the Impressions of External Actors

Another set of incentives that existed in probation schools but not high

performing schools was the need to impress external observers. Probation schools must

have an external partner and a probation manager. These actors play a role in determining

whether or not a school gets off of probation. Some probation school efforts were

designed to convince external observers that their schools were doing all they could to

improve students outcomes. The pressure to do this was captured by the principal who

relayed a conversation she had had from an office of accountability representative.

When Dr. Austin came out yesterday, he’s … from the office of accountability.

He said, “You know there are schools that are in worse condition, physical

condition than your school. Those schools are open because they are at 40

percentile.” And he said, “We don’t care if they run naked through the hallways

but they’re not on probation. Whatever it is that you need to do then that’s what

you will have to do to get your school off of probation. So there’s still something

wrong that has to be fixed . . . So whatever it takes, you have to fix it.”

At Field school, some of these responses emphasized superficial changes

designed to impress district officials and probation managers. For example, since being

placed on probation, Field school has adopted several programs in a rather haphazard

way in an attempt to demonstrate improvement efforts. This approach has been

unsuccessful in the eyes of one of the schools external partners, a consultant from a local

university, who argued that the problems of the school result from a lack of coherence in

their improvement efforts. Likewise, the following field note expresses a similar sense of

the lack of coherence of the schools program.

After sitting through a number of … meetings, I’m still struggling to see

coherence to the schools struggle to get off probation. While the components

seem to be in place … the quality of those components seems to be lacking.

Schools adopting several disconnected programs have been characterized as “Christmas

tree” schools. This pattern surfaced at many unsuccessful elementary schools

(Consortium 1993). In contrast, more recent work suggests that increasing instructional

program coherence is an important mechanism for instructional improvement (Newman,

Smith, Allensworth, and Bryk 2001).

Another strategy was to impress external observers through emphasizing the

appearance of student and teacher engagement in instructional activity and student

discipline. For example, at a Field School staff meeting the assistant principal warned

teachers that the school’s probation manager would be visiting and advised teachers on

what actions to take.

I don’t know if you saw Ms. Fox, our probation manager, here in the building

today. … Make sure your university organizers are visible, there should be

student work, your classroom should be attractive, they are going to be looking at

the decorum of our students so make sure you continue with working on good

behavior, most important they are looking for students on task and that teachers

are effectively teaching their students…there is no telling when someone might be

in to visit your class.

This reflects one of Field School’s responses to external pressures. The emphasis here is

on classroom management, décor, and the displaying of students work. The principal

does discuss the importance of demonstrating being “on task” but focuses less on

instruction issues and more on classroom management and appearance. In other words,

this strategy is cosmetic, emphasizing the trappings of instructional improvement while

the actual instructional practices are not emphasized.

Responses to high stakes accountability in the probation schools naturally emphasized

getting off of probation. The strategies used by school leaders included using the school’s

probationary status to motivate teachers – both through the threat of negative

consequences for low-performing teachers and through the promise of getting to the

probation free “promised land” and efforts to manage the impressions of external

stakeholders through adopting multiple programs and increasing the appearance of

instructional emphasis and innovation.

Incentives in high performing schools

Incentives in the high performing schools are less clear than incentives at

probation schools. One strategy used by school leaders in these schools revolved around

praise for accomplishment. In Kelly School professional development meetings the

principal regularly praised teachers for working hard to produce high student outcomes.

Teachers were encouraged to applaud their own and other accomplishments and test

results were prominently posted inside the school and shared with parents who visited the

school. All of this reinforced pride in past accomplishments and encouraged continued

improvement.

Given past success however, leaders were also forced to combat complacency on

the part of faculty. In the case of Baxter, the principal felt that teachers believed that the

school was performing better than it was. He used trend test-score data to challenge

teachers assumptions. He demonstrated that while the schools absolute outcomes were

excellent, when compared to other schools in their neighborhood their students were not

gaining as much year to year. As he explained:

The analysis made clear that out of the 12 schools [in the neighborhood], Baxter

was either at the bottom or really close to the bottom, in terms of the amount of

actual growth that students were making. Forget about where the growth started,

forget about the base. Forget about the end. Just, you know, how many months of

progress, on an average were sixth graders achieving.

School leaders used this data in school meetings to help create incentives for continued

instructional improvement. They took great pains to repackage standardized test data in

ways that captured teachers’ attention, transforming massive spreadsheets into relatively

easy to read charts that were color-coded by grade-level.

Likewise, at Kelly School, a monitoring system with a template that connects

teachers daily lesson plans to the material tested, district standards, and the level of

mastery of specific skills for each student was used to maintain a focus on constant

improvement. The assistant principal explained how this skill chart works.

It’s just an organizational tool. You look at this chart and you see that child

didn’t master that skill. That is something you can do in a small group. You can

assign your [teacher’s] aide to work with that particular child on that skill and

retest, cause we believe right away if the child didn’t master it … most kids only

miss it like a master is 80% maybe they got a 75 or a 72, so he just missed it by a

little bit, quickly review, go over it again and retest. The child masters it then

move him on.

At a professional development meeting the principal emphasized that teachers need to

insure that they are tracking the skill mastery of students.

“I noticed that the skill charts are not being filled out diligently enough. The

[university program] got us on the path to improvement through charting our

progress but we can’t get lax on this. We have completed 2 fifth week

assessments and if you have a lot of children not getting their skills you need to

re-teach. If a lot of your children are not getting the material it is not the children.

It is something to do with the way you taught it. You can’t teach the same way

every year. Its always the children. People make excuses. But that does not hold

up because we can take the same child in two different classes and they can do

well in one and have trouble in the other. But if you see students are having

trouble don’t go on. Its going to be evident that the students are not getting it so

think of another way to teach it.

Here school leaders help create a school-based incentive structure tied to the maintenance

of high performance. In this example, the teachers maintain primary responsibility for

student performance and excuses based on students’ limitations are discounted. In

another meeting (the following week) she again emphasizes the need to maintain focus

and work hard in order to keep test scores up.

[The principal] said that someone inquired about Kelly and asked if the school has

gifted students. She said that “our students are average … our instructional

program is what makes the difference… the only way we continue to improve is

through hard work. Just because we did well last year [on the ITBS] does not

mean anything. We have to continue to work hard and align our lesson plans.

In high performing schools leaders used praise for past performance and the need for

constant improvement to heighten teachers’ sense of accountability and motivate them.

In contrast to probation schools, the incentives in high performing schools were

based more on rewards than sanctions. In these schools, school leaders praised school

accomplishments in professional development meetings and proudly displayed students’

outcomes in the school and communicated these to parents. In addition, because the

accountability policy did not create sanctions for these schools, school leaders sought to

interject other forms of motivation through comparisons with other schools or through

building on the school’s past performance.

Therefore the nature of the incentives created by high stakes accountability are

different for high and low-performing schools. High performing schools are arguably in a

better position to focus on instructional improvement while probation schools must

respond directly to external pressures while they seek to enhance students’ outcomes.

Accountability Policy and “Objective” information

Proponents of accountability policy also argue that test results provide objective

information upon which to orient school decision making. The argument is that schools

with clear, objective information about student performance will be able to make better

informed decisions about instructional improvement (Coleman 1997). Once again our

data demonstrate distinct patterns in the interpretation and use of results in high

performing versus probation schools. We argue that the ways in which test results are

interpreted depends, at least in part, on the school context.

Interpretation and Use of Data at High Performing Schools

At both high performing schools, the full range of test score data was used to

inform strategies of instructional improvement. At both schools, leaders use the outcomes

for the entire school in addition to the item analysis to track overall school trends. For

example, Baxter school’s principal and a collection of school leaders

are involved in

data interpretation including longitudinal trend analysis and the analysis of movement

between quartiles. During an initial meeting with the principal, he shared a longitudinal

data analysis for both mathematics and reading which highlighted the movement of

students between quartiles for the entire school and for specific grade levels. In addition,

he reported on the development of a particular grade level, the class of 2003, in terms of

their performance in mathematics and reading.

At both Kelly and Baxter school leaders use the item analysis to identify specific

student needs within subject areas. For example, the Kelly principal explained how the

item analysis allowed school leaders to identify overall student needs:

We try to look at [test scores] in August if we have them back and we design our

program a lot looking at those skills that are measured on the ITBS [Iowa Test of

Basic Skills] and what the item analysis

indicate our weaknesses and our

strengths and so forth.

Kelly’s principal explained how the school uses test scores to track student performance.

When asked about how this information informed the instructional program in

mathematics the principal explained.

With the math I found that concepts, our children tend to do well in computation,

pencil and paper, figuring out the problem, 2 + 2, whatever they do well but when

it comes down to the concept and which __ looking hopefully they’re using – they

needed to use higher order thinking skills they tend to not do as well and we’re

working – we started last year we started focusing in on higher order thinking

skills because _____ moving more and more in that direction. The math problems

they have to explain how they got the answers not just get the answer. So what

we’ve been doing we have been working with this year a new initiative not all

teachers are doing this but next year every teacher will be doing this. The math

journal in which children they must explain – they must explain how it is that they

arrived at the answer that they got and as I said the total math is not – it’s pretty

good cause we’re at 61% but the math problem solving we tend to not do as well

and especially in the area of math concepts. So we’re working on that.

The principal’s interpretation is enhanced through her interactions with others in the

school who inform her interpretations and seek to implement instructional responses. The

school’s assistant principal, counselor, and technology coordinator all play active roles in

data interpretation. The Technology coordinator identified similar interpretation of the

test score data.

The biggest deficit in mathematics grade-wise, and we found it to be pretty much

the same in every grade, was the word problems. Them interpreting word

problems. Their computation skills are great. Just about in every grade. The

computation scores were good, average or above average. Data interpretation was

another area that we felt needed some work.

Baxter made similar use of the test score data. As the principal at this school explained:

One thing that helped us a lot in terms of being able to disaggregate our math data

is we were able to disaggregate it into three sections called problem solving and

data interpretation, concepts and estimation, and computation.

Therefore, both of the high performing schools used test results to identify macro-trends

across the school and focus attention on areas of specific needs. In both schools, the

“item analysis” was a tool that helped them identify where they should focus their

attention.

Interpretation and Use of Data in Probation Schools

At the probation schools the interpretation and use of the data was more general.

School leaders discussed the need to improve reading and mathematics but did not speak

in specific terms as they did in the high performing schools. There was limited discussion

of sub-dimensions of subject matter areas, rather the discussion revolved around reaching

the probation threshold in the two primary subject areas. In addition, the data was used at

these schools in pretty much the form in which it came from the district. There was less

re-packaging of the information and limited analysis of specific tends (with the exception

of the identification of specific students who were close to “passing” the exam).

The high performing schools use the test score data in ways that seem consistent

with the arguments of testing proponents. The test results are used to define students’

specific instructional needs and provide a basis for school level instructional decision

making. In contrast, the probation schools tend to focus more on the overall test results

and have a less systematic strategy for turning test results into useful information for

instructional change. This suggests that even the interpretation of “objective” data is

situated. Within higher performing schools, there is a more substantive interpretation of

the data. It suggests that the resources for data interpretation are greater in higher

performing schools. Therefore these schools are more likely to benefit from the

information in ways that lead to instructional improvement.

Accountability Policy and Academic Press

Academic press measures the normative emphasis on academic success and

reaching certain standards of achievement among both teachers and students (Lee, Smith,

Perry, and Smylie 1999). We argue that instructional focus is likely to increase in these

schools with high-stakes accountability. Schools will focus on academics more when the

incentives structure shifts, with rewards and sanctions being tied to students performance.

However, academic press as typically measured is content neutral. It suggests that high

standards exist but tells us little about how these standards are operationalized inside

schools. Therefore, what academic press means for different schools may vary.

Moreover, when schools are positioned differently within the accountability system their

“press” might manifest in different ways. We found that the probation schools increased

instruction focus, but in ways that were designed to respond to the policy demands of the

external environment – getting off of probation. Their efforts targeted certain students,

certain grade levels, and certain academic subjects. In contrast, the high performing

schools focused equally on mathematics and language arts instruction, emphasized

improvement for all grade levels, and worked to enhance the learning opportunities of all

students. The following section discusses these patterns.

Targeted Instructional Focus in Probation Schools

One of the critical issues that was continually raised at the probation schools was

getting off of probation. One way that the instructional focus manifested at these schools

is in an effort to increase the number of students at or above cutoff points at benchmark

grades. In these approaches, school leaders target certain students or certain grade levels

for extra assistance in an attempt to reach minimum acceptable performance levels. For

example, at Waxton school teachers and administrators focus on the benchmark grades in

order to reduce their retention rates. At Waxton, 50% of the eight professional

development meetings we observed over the year were largely or entirely focused on

some aspect of testing including topics such as skills tested in language arts; skills tested

in mathematics; constructing multiple test items; and preparing students for the ITBS.

Professional development and other efforts to improve testing, however, tended to be

targeted to particular grades (those that took the test) and subject areas. The external

partner focused its energy on teachers in the benchmark testing grades and provided

exam preparation books only for teachers at those grade levels.

At Field school, one approach was to identify those students who were close to

reaching national norms, and providing them with additional help. The school established

an after-school tutoring program for these students. This tutoring process and the student

selection process was discussed at a staff meeting by the assistant principal.

The after school program will start on Tuesday. All of you got the applications

for your children and they need to be returned on Monday … we have one class

for every grade level. The list of students may have seemed erratic. Ms.

Lawrence chose those students according to their ITBS scores. She chose those

students who she felt had the most potential to improve.

Later, a school administrator explained that the school targets those students who are

closest to the threshold in its effort to get off of probation.

The targeted assistance program is … for students who are … very close to

having the skills necessary to pass the test. … students in this program attend

three times per week from 2:30 (dismissal) until 4:30pm. Other students are

allowed to come but the students who are closest to passing the exam are targeted.

The Assistant Principal later explained this further stating that the school will use this

program to:

Work with selected students, taking a look at the IOWA scores and just really

focusing in on students that are at a median that we can work with and see if we

can get some growth spurts on them.

The school external partner, who attributed this focus to the external pressure being

applied by the school district, also discussed this strategy.

They [the school] leave behind [lowest performing students] and focus on [the

higher performing]. So many principals are under this pressure. It’s the name of

the game. When Vallas [the district CEO] comes and they have their region

meetings, they are told they have to get off probation. Even if your school shows

growth and doesn’t get off probation, they realize they will be looked at as not

doing the job.

Thus the lowest performing students in this school received limited assistance in

improving their scores. Instead, the instructional focus is on the students who were close

to making the cut-off for probation requirements. In this case the external accountability

mechanisms lead to a selective increase in instructional focus with limited implications

for the lowest performing students. In addition to this selective focus, the program

emphasizes content coverage (pacing) but not the teaching strategies that should be used.

Responses to testing in the probation schools were also structured by subject

matter. As we noted above all of the schools focused more on mathematics and reading

instruction than science instruction. However, in the probation schools, the instructional

focus emphasized one subject area – reading. For example, Waxton’s administration was

putting much of their efforts into language arts as opposed to other subject areas. The

Waxton principal explained the school emphasis on language arts instruction:

Being very honest, language arts specifically reading is one area that could impact

probation and since the school had been on probation for so long we felt a need to

address that curriculum area. And the mathematics scores were slightly higher than

the reading so that gave us the second reason.

While instructional emphasis is one of the outcomes predicted by proponents of testing

policy, the selective emphasis on certain students, grade levels, and subject matter areas

may limit the impact of the policy for all of the students in the school. Moreover, the

selective targeting of students seems to run counter to the intended impact of the policy.

Students who face challenges may in fact be marginalized by the responses of school

leaders in low performing schools.

Testing and Instructional Focus in High Performing Schools

In both high performing schools the exams are used to identify high and low-

performing groups of students. However, in contrast to the probation schools, the high

performing schools adopted interventions for all students, not just a sub-category. At

Baxter school, test score data was used by school leaders and teachers to diagnose the

effectiveness of certain teaching approaches. For example, when 5

grade teachers at

Baxter met to discus the prior years test results, they were pleased that the overall

percentage of students at or above national norms had increased. However, upon closer

examination, they discovered that the increases were among students moving from the

second to the third quartile. In discussing this, they determined that as a group it was

likely that they had been focusing instruction on students in the middle range, potentially

not addressing the needs of the lowest and highest performing students. As Baxter’s

principal explained, while the school advocates heterogeneous grouping the teachers had

identified a potential pitfall.

If you’re go and you look at the data that’s out there about how student grouping

effects achievement and you recognize that heterogeneous grouping with a

competent teacher is gonna be your best shot at being able to give everybody an

opportunity to succeed. What’s the biggest challenge you then have as a teacher?

Well the challenge is how do you deal with that? How do you manage that

enormous diversity of talent that’s in front of you? … How do you do that?

That is to say you know the moment that you allow yourself as a teacher to either

shoot at the middle you’re gonna be, and under-serve the youngsters who are

really more ready. Or even worse, that you out of mostly good intentions to drive

most of your teaching efforts by those youngsters who require the most

remediation and are most needy of your time and who you’re feeling most guilty

because you’re not serving.

Given their review of the test score data at the fifth grade level and similar analyses at

other grade levels, the school developed an approach that sought to address the needs of

all of the students while maintaining heterogeneous grouping.

Well, our assumption is that in order to be able to [address the needs of all

students] you need to teach high, if I can use that sort of piece of jargon, you need

to teach high and re-teach to the middle and lower. … the strategy at the school is

to continue to do the whole group instruction high. Teaching you know the mass

majority of that whole group instruction is to teach high and then to make

remedial provisions within the classroom as well as out of the classroom for

additional tutorial and other kind of remedial work to get to those youngsters who

are not getting it the first time. That’s the strategy.

At Baxter the analysis of movement within quartiles, and teachers’ and school leaders’

interpretation of that data, highlighted an instructional practice that needed to be revised.

Test results therefore informed teaching strategies in ways that targeted the instructional

focus toward all students rather than a sub-category.

The instructional focus of the high performing schools also extended beyond the

benchmark grades to include all grade levels. At Baxter, school leaders tracked test score

trends for all students and focused attention to all grade levels. For example, spearheaded

by the principal, Baxter’s leadership committee undertook careful study of school

standardized test data in math. They wanted to know how and whether high math scores

in grade 3 were being sustained or not through grade 5. Through longitudinal analysis,

the team determined that somewhere in grade 6, students’ scores started to slump.

Suspending school convention, the leadership committee convened a joint task force of

two groups that traditionally had little interaction: the third and fourth grade teachers and

the fifth and sixth grade teachers. This joint committee met for the good part of a year

and as a result of their work tried to build greater alignment in math topic coverage across

grade levels. Therefore test data provides an important resource for planning at this

school with the instructional focus extending to all grade levels rather than a subset.

Kelly school used the monitoring system discussed above to target instruction to all

students and all grade levels in ways similar to Baxter School.

Testing resulted in very different patterns of instructional focus at the two sets of

schools. In the probation schools, the instructional focus became targeted at certain grade

levels, certain students, and certain subject areas. Low-performing students and students

at non-benchmark grades were unlikely to be impacted by these strategies. In contrast,

the high performing schools targeted instruction at all grade levels and all students.

Though not reported in the data here, they also maintained a balance between

mathematics and literacy instruction.

Discussion and Conclusion

Having examined schools’ responses to high stakes accountability policy in high

performing schools and probation schools we argue that these contexts impact how the

policy is enacted. Proponents of these policies argue that they will create new incentives

for teachers, provide objective information for school decision making, increase

academic press, and that the combination of these mechanisms will reduce gatekeeping

practices. Our data suggest that, across each dimension, leaders at probation schools and

high performing schools structured their responses differently.

In probation schools, responses to high stakes accountability emphasize getting

off of probation, partially through managing the impressions of external stakeholders.

These efforts to convince outsiders that the school was making change efforts perverted

the intention of the policy in certain circumstances, prompting an emphasis on the

trappings of instructional innovation rather than substantive change. In addition, leaders

at these schools used test results to look at overall school and grade level outcomes but do

not connect these results explicitly to instructional decisions as supporters of these

policies suggest they would. Finally, with regard to increased instructional focus

(academic press), while school leaders were able to demand teachers attention through a

combination of threat and encouragement, they focused this attention in a process best

described as “selective press” which targeted specific grade-levels, students, and subject

matter areas all emphasizing reaching the goal of each school – getting off of probation.

In contrast, the incentive structure at high performing schools pushed school

leaders to reward and encourage teachers for their accomplishments while creatively

pushing for continued improvement. These schools use test data to track macro processes

of student performance and set the schools’ instructional agenda. This instructional

agenda, unlike at the probation schools, focused on all students, all grade levels, and

balanced attention to both of the core subject areas. Therefore, some of the key

mechanisms through which high stakes accountability are supposed to impact students’

educational opportunities are constructed very differently depending on the school’s

status in relation to the accountability system. This is very important considering that

students race and class often play a role in the type of school they attend.

Proponents of high stakes accountability suggest that these policies will reduce

schools’ gatekeeping processes and increase academic press. The data from these schools

suggests that the extent to which this is the case may depend on the status of the schools

in relationship to the high stakes accountability policy. The data from Field and Waxton

demonstrate that their response to accountability policy is situated in their probation

status which structures school-level responses to high stakes accountability in important

ways. Because probation status adds pressure to school leaders and teachers, it can lead to

practices that increase rather than reduce gatekeeping processes. For example, Field

School’s practice of focusing tutoring programs to serve only the students who are close

to national norms may have detrimental impacts on the lowest performing students, those

who the policy is designed to help. These students may be marginalized from

interventions that could increase their educational outcomes. Moreover, the targeted

responses of these schools, focusing on benchmark grades and subject areas with the

greatest chance of passing the probation threshold seems equally problematic. Focusing

on the benchmark grades for intervention suggests that the increase of academic press

may only impact a sub-set of students. Likewise, if the subject matter areas are

selectively focused upon this may ultimately limit students access to knowledge. The

responses of Waxton and Field schools do not seem to represent an increase in academic

press as much as a calculated, strategic effort to respond to the policy demands of the

external environment which may ultimately marginalize the lowest performing students.

Unlike the probation schools, the responses of school leaders at high performing

schools seem to more closely match the outcomes predicted by testing supporters. They

contribute to the development of clearer instructional foci and seem to reduce

gatekeeping processes within these schools by focusing on improving the learning

opportunities for all students across all grade levels.

The responses of these schools suggest that the implementation of accountability

policy may work against increased educational equality. If higher performing schools

construct the policies in ways that increase their academic press and reduce gatekeeping

while lower performing schools have the opposite effect, focusing primarily on

responding to external threats, then the policies could exacerbate rather than challenge

educational stratification. The situated nature of policy implementation should be an

important consideration for school reformers. Policy implementation is very much a local

process and understanding the variation in context even within districts appears to be

critical. Moreover, the fact that the likelihood of attending schools of different quality is

associated with social class, race, and residential segregation suggests that broader

structural factors impact school level processes in ways that should be attended to by

policy makers.

This paper is not meant to argue against high-stakes accountability. In reality,

these policies have led to increases in students test results across the Chicago district and

show promise for increasing schools’ focus on instruction, even if this happens in

imperfect ways. However, this paper is meant to sound a cautionary note with regard to

the reduction of race and class stratification. The findings reported here suggest that the

highest performing schools, those with higher percentages of middle-income and white

students, may benefit more from accountability policy than the probation schools that are

most in need of improvement. While this is clearly not the intent of the policy, attention

should be given to the potential implications of this process.

References

Anyon, J. (1981). Social class and school knowledge. Curriculum Inquiry, 11(1) 3-42.

Argyris, C. and D. A. Schon (1974). Theory in Practice: Increasing Personal

Effectiveness. San Francisco, Jossey-Bass.

Ayalon (1994) Monopolizing Knowledge? The Ethnic Composition and Curriculum of

Israeli High Schools. Sociology of Education. 67(4)

Bankston and Caldas (1996) Majority African American Schools and Social Injustice:

The Influence of De Facto Segregation on Academic Achievement. Social-Forces,

75(2):535-555.

Barr, R. and R. Dreben. (1983). How Schools Work. Chicago: University of Chicago

Press

Becker, H. J. and J. L. Epstein (1982). “Parent Involvement: A Study of Teacher

Practices.” Elementary Schools Journal 83: 85-102.

Blau, P. M. and O. T. Duncan. (1967). The American Occupational Structure. Chicago,

University of Chicago Press.

Bourdieu, P. (1977). Outline of a Theory of Practice. Cambridge: Cambridge University

Press.

Bourdieu, P. (1986). The Forms of Capital. Handbook of Theory and Research of the

Sociology of Education. J. G. Richardson. New York, Greenwood Press: 241-258.

Bourdieu, P. and J. C. Passeron (1990). Reproduction in Education, Society, and Society.

Beverly Hills, CA, Sage.

Bowles, Samuel. and Herbert. A. Gintis. 1976. Schooling in Capitalist

America. New York: Basic Books.

Bronfenbrenner, U. (1979) The Ecology of Human Development: Experiment By Nature

and Design. Cambridge MA: Cambridge University Press.

Brophy, J. and T. L. (1986) Good Teacher Behavior and Student Achievement: 328-375.

Burton, L. (1993). “Black Grandparents Rearing Children of Drug-Addicted Parents:

Stressors, Outcomes, and Social Service Needs.” The Gerontologist 32: 744-751.

Carbonaro, W. J. (1998). “A Little Help From My Friends Parents: Intergenerational

Closure and Educational Outcomes.” Sociology of Education 71: 295-313.

Clark, C. and Peterson, P. (1986). Research on teacher thinking. In M. C. Wittrock

(Ed.), Handbook of Research on Teaching, 255-296. New York: MacMillan.

Clotfelter and Ladd (1996)

Cohen, D.K. (1989). Teaching practice: Plus ça change..... In P.W. Jackson (Ed.),

Contributing to educational change: Perspectives on research and practice, (pp. 27-84).

Berkeley, CA: McCutchan.

Cohen, David K., and Deborah Lowenberg Ball (1998). Instruction, Capacity, and

Improvement. Philadelphia: University of Pennsylvania, CPRE: CPRE Research Report

Series, RR-42.

Cohen, D., Spillane, J., Jennings, N., & Grant, S. (1998). Reading Policy. Ann Arbor,

MI: University of Michigan.

Cole, Michael, and Yrjo Engestrom "A cultural-historical approach to

distributed cognition." In Distributed cognition: Psychological and

educational considerations, edited by G. Salomon, chapter 1. New

York: Cambridge University Press, 1993.

Coleman et al (1997) Redesigning American Education. Boulder: Westview Press

Coleman, J. (1988). “Social Capital in the Creation of Human Capital.” American Journal

of Sociology 94: S94-S120.

Coleman, J. S. and T. Hoffer (1987). Public and Private High Schools: The Impact of

Communities. New York, Basic Books.

Coleman, J., E. Q. Campbell, C. J. Hobson, J. McPartland, A. M. Mood, F. D. Weinfield,

and R. L. York, (Eds.) (1966). Equality of Educational Opportunity. Washington, D.C.:

U.S. Government Printing Office.

Comer, J. P. (1980). School Power: Implications of an Intervention Project. New York,

The Free Press.

Connell, R., D. J. Ashenden, et al. (1982). Making the Difference: Schools, Families, and

Social Division. Sydney, London, Boston, George Allen and Unwyn.

Cremin 1951. The American Common School: An Historical Conception: New York:

Teacher’s College Press.

Daring-Hammond, L. (1994) National Standards and Assessments: Will they improve

education? American Journal of Education, 102 (4).

Diamond, J. B. (2000). Beyond Social Class: Cultural Resources and Educational

Participation among Low-Income Black Parents. Berkeley Journal of Sociology, 44:15-

Diamond, J B. (2001) Multidimensional Family Capital and Low-Income Black Parents’

Educational Participation. Toward a Substitution Model of the Intergenerational

Conversion Process. In progress

Diamond, J. B., Randolph, A., and Spillane, J. (2000). Race, Class, and Teachers’ Beliefs

about Students in Urban Elementary Schools: Perception, Enactment, and the Duality of

Structure. Paper presentation at the Annual Meeting of the American Sociological

Association. Regular Education Session. Washington, D.C., August, 2000.

Doyle, W. (1983). Academic Work. Review of Educational Research, 53, 159-199

Educational Evaluation and Policy Analysis. (1990). 12(3).

Elloitt, M. 1998. School Finance and Opportunities to Learn: Does Money Well Spent

Enhance Students’ Achievement? Sociology of Education, 71:223-245.

Elmore, Richard F., Penelope L. Peterson, and Sarah J. McCarthey (1996). Restructuring

in the classroom: Teaching, learning, and school organization. San Francisco: Jossey-

Bass.

Epstein, J. L. (1987). “Parent Involvement: What Research Says to Administrators.”

Education and Urban Society.

Epstein, J. L. and S. Dauber (1991). “School Programs and Teacher Practices of Parent

Involvement in Inner-City Schools.” Elementary Schools Journal 91: 289-303.

Epstein, J. L. and M. Sanders. 2000. Connecting Home, School, and Community : New

Directions for Social Research, in Handbook of the Sociology of Education edited by

Maureen Hallinan pp. 285-306.

Ericson, F. (1986). Qualitative methods in research on teaching. In M.C. Wittrock (Ed.),

Handbook of research on teaching (pp. 119-161). New York: Macmillan.

Fordham, S. and J. U. Ogbu (1986). “Black Students' Success: Coping with the Burden of

Acting White.” Urban Review 18: 176-206.

Fuller, B., Elmore R. F., and Orfield, G. Who Chooses, Who Loses? Culture, Institutions,

and the Unequal Effects of School Choice. New York: Teachers College Press.

Fuson, K. C. (1996, April). Latino children's construction of arithmetic understanding in

urban classrooms that support thinking. Paper presented at the Annual Meeting of the

American Educational Research Association, New York.

Fuson, K. C., Smith, S. T., and Lo Cicero, A. (in press). Supporting Latino first graders'

ten-structured thinking in urban classrooms. Journal for Research in Mathematics

Education.

Galindo, R. and Escamilla, K. (1995) A Biographical Perspective on Chicano

Educational Success. Urban Review 27(1), 1-29.

Gamoran, A. (1986). Instructional and Institutional Effects of Ability Grouping.

Sociology of Education, 59(4), 185-198.

Gewirtz, Sharon, Stephen J. Ball, and Richard Bowe. 1997. Markets, Choice and Equity

in Education. Briston, PA: Open University Press.

Giddens, A. (1979). Central Problems in Social Theory: Action, Structure, and

Contradiction in Social Analysis. Berkeley & Los Angeles: University of California

Press.

Glaser, Barney G. and Anselm Strauss (1967). The Discovery of Grounded Theory.

Chicago: Aldine.

Grossman, P. (1987). A Tale of Two Teachers: The Role of Subject Matter orientation

in Teaching. Paper Presented at the Annual Meeting of the American Educational

Research Association, Washington, DC

Hedges, L. V. and R. Greenwood. 1996. Have Times Changed? The Relation between

School Resources and Student Performance. Pp. 74-92 in Does Money Matter? The

Effect of School Resources on Student Achievement and Adult Success, edited by G.

Burtless. Washington, D. C.: Brookings Institution.

Hedges, L. V. R. D. Laine, and R. Greenwald. 1994. Does Money Matter? A Meta-

Analysis of Studies of the Effects of Differential School Inputs on Student Outcomes.

Educational Researcher 23:5-14.

Hoa, L. and Bonstead-Bruns (1998). “Parent-Child Differences in Educational

Expectations and the Academic Achievement of Immigrant and Native Students.”

Sociology of Education 69: 126-141.

Hofferth, S. L., J. Boisjoly, et al. (1998). “Parent Extra Familial Resources and Children's

School Attainment.” Sociology of Education 71: 246-268.

Heubert, J. P. and R. M. Hauser (eds.) (1999). High stakes testing for tracking,

promotion, and graduation. Washingotn D.C. National Academy Press

Hutchins, E. (1995) Cognition in the Wild. Cambridge, MA: MIT Press.

Jencks, Christopher. S, S. M. Smith, H. Ackland, M. J. Bane, D. Cohen, H. Gintis, B.

Heyns, and S. Mickelson. (1972). Inequality: A Reassessment of Family and Schooling in

America. New York: Basic Books.

Kerbow, D. and A. Benrhardt (1993). Parent Intervention in the School: The Context of

Minority Involvement. Parents, Their Children and Schools. B. Schneider and J. E.

Coleman, Westview Press: 115-146.

Knapp, M. S. and Associates. (1995). Teaching for Meaning in High-Poverty

Classrooms. New York: Teachers College Press.

Kozol, J. (1991) Savage Inequalities

Lambert, Linda. et al.(1995). The constructivist leader. New York: Teachers College

Press.

Lareau, A. (1987). “Social Class and Family School Relationships: The Importance of

Cultural Capital.” Sociology of Education 56: 73-85.

Lareau, Annette (1989). Home Advantage: Social Class and Parent Intervention in

Elementary Education. New York: The Falmer Press.

Lave, Jean, and Etienne Wenger. Situated Learning: Legitimate

peripheral participation. New York NY: Cambridge University Press,

1991.

Lee, Valerie E., Julia B. Smith, Tamara E. Perry, and Mark A. Smylie. (1999). Social

Support, Academic Press, and Student Achievement: A View from the Middle Grades in

Chicago. Chicago: Consortium on Chicago School Research.

Leont'ev, Aleksei Nikolaevich. Problems of the development of the mind. Moscow:

Progress, 1981.

Liberman, A., Beverly Falk, and L. Alexander (1994). A culture in the making:

Leadership in learner-centered schools. New York: National Center for Restructuring

Education, Schools, and Teaching, Teachers College.

MacLeod, J. 1987 Aint No Makin’ It: Aspirations and Attainment in a Low-Income

Neighborhood, Boulder: Westview Press

McDill, Edward L., Gary Natriello, and Aaron M. Pallas. (1986). “A population at risk:

Potential consequences of tougher school standards for student dropouts.” American

Journal of Education, 94 (2): 135-181.

McLauglin, Milbrey., and Joan E. Talbert (1993). Contexts that matter for teaching and

learning. San Francisco: Jossey Bass.

McNeal, R. B. (1999). “Parent Involvement as Social Capital: Differential Effectiveness

on Science Achievement.” Social Forces 78: 117-144.

Meier, D. (1995). The Power of their ideas: Lessons for America from a small school in

Harlem. Boston: Beacon Press.

Miles, M. B. and A. M. Huberman (1994). Qualitative Data Analysis: An Expanded

Sourcebook. Thousand Oaks, Sage Publications.

Muller C. and Schiller, K. (2000). Leveling the Playing Field? Students’ Educational

Attainment and States’ Performance Testing. Sociology of Education. 73: 196-218.

Natriello, G. and McDill, E. L. (1990). Schooling Disadvantaged Children: Racing

Against Catastrophe. New York: Teachers College Press.

Newman, F. and G. G. Wehlage (1995). Successful School Restructuring. Alexandria,

VA, ASCD.

Newmann, Fred M., BetsAnn Smith, Elaine Allensworth, and Anthony Byrk. (2001).

School Instructional Program Coherence: Benefits and Challenges. Chicago: Consortium

on Chicago School Research.

Oakes, Jeanne (1985). Keeping Track: How Schools Structure Inequality. New Haven:

Yale University Press.

Orfield, Gary (1993). The Growth of Segregation in American Schools: Changing

Patterns of Separation and Poverty Since 1968. Harvard Project on School

Desegregation. National School Boards Association Council on Urban Boards of

Education.

Orfield, G., Bachmeier, M. D., James, D. R., and Eitle, T. 1997. Deepening Segregation

in American Public Schools. Cambridge, MA: Harvard Project on School Desegregation.

Pattillo-McCoy, Mary. (1998). “Church Culture as a Strategy of Action in the Black

Community.” American Sociological Review.

Pattillo-McCoy, Mary. (2000). Black Picket Fences: Privilege and Peril in a Black

Middle Class Neighborhood. Chicago: University of Chicago Press.

Pea, Roy D. "Practices of distributed intelligence and designs for education." In

Distributed cognition: Psychological and educational considerations edited by G.

Salomon, chapter 2. New York: Cambridge University Press, 1993.

Peshkin, A. (1993). The goodness of qualitative research. Educational Researcher, 22(2),

24-30.

Resnick, Lauren. (1991). 'Shared cognition: Thinking as social practice." In Perspectives

on socially shared cognition. Edited by L. Resnick, J. Levine, & S. Teasley. Washington,

DC: American Psychological Association (APA) 1991.

Rist, R. (1970). “Student Social Class and Teacher Expectation: The Self-fulfilling

Prophesy of Ghetto Education.” Harvard Education Review 40(3): 411-454.

Rist, R. C. (1977). On Understanding the Process of Schooling: The Contributions of

Labeling Theory. In Power and Ideology in Education edited by Karabel and Halsey.

Roderick, M. (1994). Grade Retension and School Dropout. American Educational

Research Journal, 31(4), 729-761

Roderick, . T. Bryk, B. Jacob, J. Q. Easton, E. Allensworth. (1999) Ending Social

Promotion: Results from the first two years. Chicago: Consortium on Chicago School

Research

Roscigno, V. J. (1998). “Race and the Reproduction of Educational Disadvantage.”

Social Forces 76(3): 1033-60.

Rosenthal, R. and L. Jacobson (1968). Pygmalion in the Classroom. New York, Holt,

Rinehart, and Winston.

Sampson, R. J., S. W. Raudenbush, et al. (1997). Neighborhoods and Violent Crime: A

Multilevel Study of Collective Efficacy. Science. 227: 918-924.

Sanders, M. G. (1998). The Effects of School, Family, and Community Support on the

Academic Achievement of African American Adolescents. Urban Education, 33, 385-

410.

Schon, D. A. (1983). The Reflective Practitioner. San Francisco, Jossey-Bass.

Sewell, William H. and R. M. Hauser 1980. "The Wisconsin Longitudinal

Study of Social and Psychological Factors in Aspirations and

Achievement" Pp. 59-100 in Research in Sociology of Education and

Socialization. Edited by A. C. Kerckhoff. Greenwhich: Ct: JAI Press.

Sewell, W. and V. P. Shah (1968). “Parents' Education and Childrens' Educational

Aspirations and Achievements.” American Sociological Review 33: 191-209.

Sewell, W. and V. P. Shah (1968). “Social Class, Parent Encouragement, and Educational

Aspirations.” American Journal of Sociology 73(5): 559-572.

Shouse, R. (1997) Academic Press, Sense of Community, and Student Achievement. In

Redesigning American Education Edited by James Coleman, Barbara Schneider, Stephen

Plank, Kathryn Schiller, Roger Shouse, and Huayin Wang. Westview Press

Smith, Julia B., Valarie E. Lee, Fred M. Newman 2001. Instruction and Achievement in

Chicago Elementary Schools. Consortium on Chicago School Research, University of

Chicago.

Spillane, James P., John B. Diamond, , Lisa Walker, & Rich Halverson, Loyiso Jita. (in

press) “Urban School Leadership and Elementary Science Instruction: Identifying,

Mobilizing, and Activating Resources in a Devalued Subject Area.” Journal of Research

in Science Teaching.

Spillane, James P., Richard Halverson, and John B. Diamond. (2001). “Investigating

school leadership practice: A distributed perspective.” Educational Researcher, 30 (3):

23-28.

Stake, R. E. (1995). The Art of Case Study Research. Thousand Oaks, CA: Sage.

Stodolsky, Susan. The subject matters. Chicago: University of

Chicago Press, 1988.

Strauss, A. (1987) Qualitative analysis for social scientists. Cambridge University Press

Wilson, William. J. 1987. The Truly Disadvantaged: The Inner City,

The Underclass and Social Policy. Chicago: University of Chicago

Press.

Wong, R. S. 1998. "Multidimensional Influences of Family Environment

in Education: The Case of Socialist Czechoslovakia." Sociology of

Education 71:1-22.

Table 1.

Mean Percentage of Students in CPS Probation, Magnet and High Performing Schools

by Race and Social Class

District Probation Magnet High Performing

% African

American

52% 83% 53% 27%

% White 10% .12% 17% 34%

% Low Income 84% 92% 55% 56%

Table 2.

School Demographics

School Name Racial Composition % Low-income

Baxter

(High Performing)

40% White

6% Black

26% Hispanic

26% Asian

63%

Field

(Probation)

100% African American 99%

Kelly

(High Performing)

100% African American 85%

Waxton

(Probation)

100% African American 97%

High Stakes Accountability in Urban Elementary Schools:

Challenging or Reproducing Inequality?

Appendix A

Mean Percentage of Students in Chicago High Performing

Elementary Schools by Race

African American

31%

Caucasian

35%

Latino/a

26%

Asian

Chicago Public Schools Student Population by Percent

Low-Income

Percent Low-

Income

84%

Percent Non-Low-

Income

16%

Percentage of Low-Income Students in Chicago

Probation Elementary Schools

Percent Non-Low-

Income

Percent Low-

Income

92%

Percentage of Low-Income Students

in Chicago Elementary Magnet Schools

Percent Low-

Income

55%

Percent Non-Low-

Income

45%

We define high performing schools as schools where 50% or more of the students perform at or above

national norms on the Iowa Test of Basic Skills. The Chicago Public Schools define probation schools as

those in which fewer than 15% of students perform at or above national norms on the ITBS. Appendix A

summarizes the data on student race and social class characteristics in the Chicago Public Schools as a

whole, elementary magnet schools, and probation elementary schools. While we focus on the data from

these four schools, our data base consists of the 13 schools that are part of the Distributed Leadership

Study.

The data reported here is taken from the Chicago Public Schools Office of Accountability public use data

base. This information can be accessed through this site at http://acct.multi1.cps.k12.il.us/

Appendix A contains a series of charts showing these patterns.

We do not suggest that family background is an absolute determinant of the type of schools children

attend. A substantial number of the students in magnet schools for example are African American and more

than half are low-income. However, the data demonstrates a pattern of students being sorted into different

types of schools in ways that correlate with race and class.

For those interest in school choice and parents decision making around school attendance please refer to

Fuller, Elmore and Orfield (1996) and Gewirtz, Ball, and Bowe (1997).

Our interview questions were designed to get at five core issues about the practice of leadership:

a. Getting the leaders to identify the key goals or macro functions they work on (e.g. building a school

vision, promoting teacher professional development, improving test scores, etc.)

b. Getting them to describe what day-to-day tasks they perform to attain these goals, i.e. the micro tasks

(e.g. observing classrooms, forming breakfast clubs, facilitating grade level meetings, etc.)

c. Getting them to describe how they enact the micro tasks; that is their practice as leaders.

d. Whether and how macro goals/functions and micro tasks are co-enacted; i.e. the extent to which their

functions are executed with the help from others in the school.

e. What tools and material resources (including designed artifacts, memos, protocols, organizational

structures) the interviewees identified as important in the execution of macro and micro tasks.

Observation protocols focused on:

1. The nature and substance of the task: what the leader(s) did and the goals of the activities including

the subject matter focus of the activity, if any.

2. How the task was enacted: including the artifacts/materials used and how they were used to enable

practice.

3. The timing and location of the task: the physical setting and context of the enactment, and the time

of the year, week, or day on which the task was enacted.

The patterns of involvement: including what the leaders/facilitators did during the enactment, whether

leadership was shared or not, and role of participants.

This percentage of students at or above national norms has increased incrementally since that time.

The categories of influence included other teachers, principals, assistant principals, the local school

council, parents, testing, standards, textbooks, and instructional specialists (i.e. mathematics teachers).

For example, in the case of a literacy committee report which drew extensively on test results, the

Russian Bilingual teacher, a reading teacher, the librarian, the reading specialist, the drama teacher, and a

local school council representative were all involved in the interpretation of test results and the

development of the report.

The item analysis is a document that shows classroom and student level test scores by the items correct

and incorrect.

Percentage of Low-Income Students in Chicago

Elementary Schools Scoring 50% or higher on

ITBS Reading

% low-Income

56%

% non-low-income

44%

The principal had a similar response for language arts. We limit the discussion to mathematics in order to

save space.

While they may not gain as much in terms of absolute test score results, this likely occurs because of the

fact that gains are more difficult at the top end because of ceiling effects.