Is There Value in Value-Added Testing?

The Milwaukee School Board is about to embark on an unprecedented expansion of standardized testing – despite warnings from experts on assessment; lack of input from parents, teachers, and principals; and a single public hearing held three business days after details of the expansion were released.

The proposal calls for almost tripling the number of standardized tests given to Milwaukee Public School (MPS) students. The overriding concern is that teachers will be pressured to focus on memorization and low-level skills rather than on encouraging students to think more critically, analytically, and in depth – the kind of learning not easily measured by a computerized, fill-in-the-bubble standardized test.

Milwaukee’s plan is part of a national trend to increase school accountability through reliance on standardized tests. In Milwaukee, a key argument is that the district needs better longitudinal data on student performance.

But the dilemma in MPS, as in other urban districts, is not the inability to produce data but rather how best to promote academic achievement. “We don’t need to have further documentation of failure,” notes Beverly Cross, Professor of Urban Education at University of Wisconsin-Milwaukee and a Rethinking Schools editor. “What we need are programs that push teachers to be more effective and creative in their teaching.”

Critics of the MPS proposal are particularly upset that the plan calls for standardized testing beginning in kindergarten – despite warnings that testing at such a young age is educationally inappropriate and may actually harm young children (see article Dangers of Early Childhood Testing).

MPS Superintendent Spence Korte was asked by Rethinking Schools if he could give an example of a respected early childhood educator or early childhood organization that supports standardized testing for children under nine years old. “You mean other than George W. Bush?” Korte replied.

Korte, a former elementary school principal, has in the past criticized reliance on standardized tests, but his administration developed the current proposal and Korte has played a low-key role in the current controversy.

The main criticism of the proposal, which is formally called a “Balanced Assessment System,” is the massive increase in standardized testing and its effect on teaching and learning, especially in early grades. Critics raise three other important problems.

• First, MPS is pushing the proposal with undue haste and a lack of democratic input. “This proposal touches the heart of the educational process,” Mary Diez, graduate dean at Milwaukee’s Alverno College and a national expert on assessment, wrote in a letter to the MPS School Board. “Shouldn’t parents, teachers, and others in the public who are concerned with the quality of public education be able to help shape the plan?” 
• Second, the plan calls for “replacing the district performance assessments.” MPS has been a national leader in districtwide performance assessments in which students are evaluated on concrete activities such as written essays, science experiments, or oral presentations. Under the proposal, performance assessments would exist instead in some undefined manner at the local school and classroom levels. “The enormous danger is that performance assessments will simply go away,” Cynthia Ellwood, principal of Hartford Avenue University School and former Director of Education Services in MPS, told Rethinking Schools. “The pressures will be so great within schools not to administer performance assessments in an era of decentralization that they will essentially disappear.” 
• Third, the plan does not even mention how the standardized testing will affect students with special needs, or students who do not speak English as their first language, or students in Milwaukee’s nationally acclaimed language immersion schools where young children receive virtually all their instruction in French, German, or Spanish.

The lone public hearing on the proposal was dominated by criticisms from parents, teachers, educators, and community activists. Critics, including staff people in central administration, worry that the public hearing was designed merely to meet legal requirements, not to solicit input. Money for the new testing program, for example, was in the MPS budget released in February, months before the hearing. Critics are also upset that the proposal is being rushed through at a time when parents and teachers are focused on dealing with the $32 million budget deficit facing MPS.

Despite the widespread dissatisfaction, the School Board’s Innovation and School Reform Committee approved the proposal May 9. The full board will vote on the proposal May 30 and supporters appear to have the necessary votes. However, one School Board member told Rethinking Schools that if Superintendent Korte actively opposes the proposal, it probably will not pass.

THE PROPOSAL’S ESSENTIALS

The MPS proposal relies on a concept of longitudinal testing known as “value-added assessments,” devised by William Sanders, a professor of statistics at the University of Tennessee. This approach is nominally designed to see how much “educational value” has been added to a student during a given time. It is especially popular in Tennessee, Texas, and North Carolina.

The heart of the MPS proposal is an increased emphasis on standardized testing and a decreased emphasis on performance-based assessments. Under the plan, an MPS student would take 52 standardized tests during their kindergarten through high school career, compared to the 19 standardized tests a student currently takes.

Presently, the first standardized test administered districtwide is the Wisconsin Reading Comprehension Test, (also known as Third Grade Reading Test) as part of a statewide mandate. MPS students also currently take the state-mandated standardized tests that are given in six academic subjects in fourth, eighth, and 10th grades. These tests, known as the Wisconsin Student Assessment System (WSAS) tests or Terra Nova tests, are developed by the publishing conglomerate McGraw-Hill. Beginning in 2004, state law requires that the district administer a high school graduation test.

The new MPS plan adds yearly standardized tests in four academic areas (math, writing, language arts, and reading) for students in first, second, third, fifth, sixth, seventh and ninth grades – the grades not covered by the statewide test. In kindergarten students will be given a reading test. Thus, students will take standardized tests in every grade. This expansion is not required by the state.

Indications are that the new series of MPS standardized tests will be bought from McGraw-Hill. The exact cost is unclear.

The new testing proposal also calls for “replacing the district performance assessments” as a measure of the district’s overall performance.

Currently, MPS has districtwide performance assessments that are given to all students. They were implemented in recent years in writing in fourth, fifth, eighth, and 10th grades. Eventually, performance assessments were added in science, math, oral language, and the arts.

While not without problems, performance assessments tend to be more educationally beneficial because they are tied to improving classroom teaching and learning, and to demonstrating that students can perform activities that are of use in the real world. Standardized tests, in contrast, focus on ranking students.

Three-fourths of the principals surveyed by an internal MPS audit said that “performance assessments in general positively focus teaching and learning in classrooms and provide useful information for program evaluation and improvement.”

Because the new testing proposal ends district-mandated performance assessments, they will no longer be used as district accountability measures. That means the performance assessments will have little or no weight when schools and principals are evaluated. Instead, the performance assessments are relegated to some yet-to-be defined “school-site classroom” role, as part of “everyday classroom instruction.”

Many educators fear that as a result, the performance assessments will disappear as a districtwide phenomenon.

The relationship between the “middle school proficiencies” and the downgrading of the performance assessments is unclear. In a highly publicized move three years ago, MPS mandated that beginning with this year’s eighth grade, students will have to demonstrate “proficiency” in six academic areas before moving into high school. Performance assessments played a significant role in meeting the proficiencies. It is not exactly clear how, under the new proposal, students will be expected to demonstrate their proficiency.

Monty Neill, executive director of the National Center for Fair & Open Testing (FairTest) in Cambridge, MA, has been involved in evaluating Milwaukee’s existing assessments. He calls the testing proposal a big step backward.

“Milwaukee has been a leader in developing more authentic forms of assessment,” he told Rethinking Schools. “It’s a pity that the district doesn’t build on what you have here. Once again, we are seeing a reduction of all the outcomes of schooling to what can be measured on standardized tests, the most simplistic form of assessment.”

IMPETUS FOR CHANGE

Major supporters of value-added standardized testing include several influential members of the Milwaukee School Board, the Milwaukee Journal Sentinel, and the Wisconsin Policy Research Institute, a conservative think tank.

School Board President Bruce Thompson said the initial impetus for the testing proposal came after a “realization on our part and on the part of [Superintendent] Spence [Korte] that we weren’t getting the kind of data we needed. Spence invited the Council of Great City Schools in and that gave us a devastating critique.” The Council, based in Washington, DC, is an advocacy group for the country’s major urban districts.

Citywide board member John Gardner is also a particularly vocal supporter of the testing proposal. At the committee meeting, board members Charlene Hardin and Don Werra opposed the testing plan, while Ken Johnson, Joe Dannecker, and Gardner backed the measure.

The Council of Great City Schools brought testing experts from other school systems to Milwaukee and issued a 20-page report in December calling for revamping the MPS research department, developing a “continuous, seamless process of norm-referenced testing,” and putting “a hold on further development of local assessments.”

The Council’s report indicates that the evaluation team that came to Milwaukee met almost exclusively with Central Office staff. It interviewed only five principals and no teachers or parents. Even though the team criticized the MPS performance assessments, it did not actually look at the performance assessments, according to the report’s appendix.

Diez of Alverno College was perplexed by the Council’s report. “Great City Schools encouraged them [the MPS administration] to go this route of increased standardized testing … but this route isn’t working in other big cities,” she said. “I don’t know why they don’t see that and work with what we’ve started [in MPS], which is working.”

Without notifying the public, the School Board apparently adopted the Council’s recommendations. In its Strategic Plan, developed this winter and approved March 28, the School Board calls for “a strategic assessment program by December of 2000,” specifically called a “measurement of value added.”

The School Board’s approval of its strategic plan coincided with a paper from the conservative Wisconsin Policy Research Institute extolling the virtues of “value-added” testing and calling for its adoption throughout the state.

In the Institute report, Standards-Based Education Reform in Wisconsin: What It Will Take to Make It Work, University of Wisconsin-Milwaukee professors Mark Schug and Richard Western recommend that Wisconsin “develop and implement a statewide system of ‘value-added’ assessment, to be incorporated in the WSAS [the current testing program], so that schools and school districts may assess their effectiveness by reference to their own starting points.” The report further advocates that such “value-added assessment becomes an indicator of district, school, or teacher effectiveness.”

Soon after the release of the Institute’s report, the Milwaukee Journal Sentinel endorsed the idea of value-added testing. “The state ought to look hard at the idea of yearly tests, so as to give public information on how well specific schools are performing – in other words, on how much learning they put into students’ heads,” the paper editorialized. The editorial did caution, however, that “testing does have pitfalls so the state should proceed cautiously, studying before acting, phasing in yearly testing and to keep the amount of tests manageable, sticking to core subjects.”

Whether MPS is heeding the advice to “proceed cautiously” is doubtful. Michael Czerwinski of the MPS Division of Research and Assessment said the changes were occurring with such haste, in part, so MPS can meet the state’s deadline for new requirements that must be implemented by 2002. The new law mandates that each district establish criteria by which fourth and eighth graders will be promoted and by which high schoolers will be given diplomas. Even though those criteria do not have to be in place until 2002, Czerwinski asserted that the district has a “legal and ethical” obligation to give parents and students “adequate notice of the new criteria and provide the students with the opportunity to learn.”

The new state law, however, does not call for value-added tests. Nor does it mandate standardized tests in first, second, third, fifth, sixth, seventh, or ninth grade. Other districts are under the same state mandates as Milwaukee, yet they are not rushing to impose “value-added” testing.

PROS AND CONS

The main arguments for the new testing program fall into three areas:

• The need to have better longitudinal data on students’ academic achievement. 
• The need to recognize that kids begin school at different places and grow at different rates. 
• The need to remove performance assessments as district-mandated school accountability measures.

Need for Longitudinal Data. The MPS proposal says that the “value-added” testing system will “inform students, parents, school sites, the district, and the general public as to how much educational value has been added through the learning experience during a given period of time.” It argues that even if all students do not perform the same, the schools should at least be able to show that some “value” has been added.

Kay Mantilla, Acting Director of the MPS Division of Research and Assessment, said the new tests would give “parents the knowledge of how their children are progressing outside the confines of a report card because our report cards are so diverse in Milwaukee.” She also said the data would be used mainly for “diagnostic purposes,” because teachers would get an “item analysis” of what students missed on the test.

At the School Board committee hearing, board member Gardner said he also wanted the data to be able to “distinguish [and] evaluate … the accomplishments of schools, programs, and most importantly teachers that are effective.”

Critics of the new testing program have several responses. First, the value-added standardized tests are very limited in what they measure. “The idea of seeing how kids do over time is reasonable, but you have to have good measures,” Neill of FairTest argues. “The standardized tests don’t tell you enough about what is really going on with a student’s learning.”

Second, the district already documents which schools and students are having difficulties, using standardized tests, districtwide performance assessments, and measures such as dropouts, attendance, and graduation rates.

Further, there are a number of classroom assessments that can be used to show student progress or the lack thereof. Milwaukee elementary school teachers, for example, are required to individually and formally assess students’ reading levels twice a year by having students read to the teacher and answer comprehension questions. Many teachers give end-of-chapter or end-of-unit tests in various subjects. Report cards are issued four or six times a year and parent conferences are held twice a year.

Given existing standardized tests, performance assessments, individual reading inventories, report cards, and parent conferences, there are adequate data for parents and teachers. In fact, those data are a much better reflection of a child’s academic strengths and weaknesses than a once-a-year test. As for schoolwide accountability, the district’s annual Accountability Report provides a great deal of data to indicate which schools are relatively more successful than others.

Students begin school at different levels. Advocates of value-added testing argue that because kids begin schools with different levels of knowledge and skills, it’s not fair to hold up one bar or standard for them to meet. Instead, advocates argue, each student should be expected to grow at different rates and be judged accordingly. Professor Sanders, credited with creating the complex statistical model to measure such “value-added” growth, explained this point in the May 2000 issue of Teacher magazine: “I believe we should visualize the curriculum not as a stair steps, but rather as a ramp. I want all kids to go up the ramp, but I recognize that not all kids are going to be at the same place at the same time. What I want to hold educators accountable for is the speed of the movement up the ramp, not the position on the ramp.”

Sanders’ argument is especially popular with those who believe that urban kids from low-income backgrounds should not be constantly compared to affluent suburban students. The view has a certain common-sense logic to it – but can cut different ways. Some worry that Sanders’ perspective can easily be turned into the view that “these kids aren’t going to make it anyhow. Let’s not expect high standards but just some growth from year to year.” The fear is that some teachers, schools or policymakers will be satisfied with mere “growth” for certain groups of kids, especially low-income students of color, and therefore won’t demand the resources and educational reforms necessary to ensure that all kids, regardless of background, attain high levels of achievement. (See article on the classroom ramifications of the Texas system, page 8.)

Sanders’ approach has also been criticized for its methodology. Douglas Reeves, director of the Denver-based Center for Performance Assessment and an assessment consultant to MPS in the past several years, criticizes Sanders for tying conclusions about achievement to the “single instrument” of a standardized test. Reeves argues that standardized tests should be “maybe one of ten things” used to evaluate performance. Otherwise, he said, there is “the possibility of misrepresenting … what the student really knows.”

NEGATIVE RAMIFICATIONS

Critics of the new MPS testing proposal fear that the advocates are too cavalier about the negative ramifications of an undue emphasis on standardized testing.

According to FairTest’s Neill, “Testing all students with norm-referenced, multiple-choice tests will substantially control the curriculum and teaching,” he said. “It is too high an educational price to pay for the amount of information that is gained.”

Lorrie Shepard, a professor at the University of Colorado at Boulder, has argued that researchers may gain “technically sophisticated analysis,” but only by “limiting the quality of the assessments used.”

At a Denver conference this February, Shepard raised cautions about Sanders’ value-added assessment. She noted that in Tennessee, which has been a national leader in value-added testing, “this has meant limiting assessment content to multiple-choice tests or to a commercially available test with only a limited range of test formats.” Such tests, especially when used for “high-stakes” decisions, “lead to both test-score inflation and curriculum distortion,” she said.

Many teachers, especially in elementary school, complain that their schools already dwell too much on the third- and fourth-grade tests. In some schools, recesses have been canceled, art and music schedules have been changed, and instruction in non-core academic areas has been curtailed in order to provide more time for test preparation.

Removing performance assessments as district mandates. A third rationale for the “Balanced Assessment System” is that performance assessments are not workable as a districtwide tool.

The MPS performance assessments have their roots in the early 1990s, during adoption of the district’s “K-12 Teaching and Learning Goals.” At that time, “there was widespread feeling in the district at every level – principals and teachers – that the kind of testing we were doing did not represent the goals we had for kids, what we actually wanted kids to be able to do,” said Hartford Principal Ellwood, then in charge of Curriculum and Instruction.

Reeves, a national expert on performance assessment, told Rethinking Schools “it would be a mistake” to get rid of the performance assessments as a district-wide requirement. “They give much richer data than standardized tests.” Reeves said he would “bet anyone a dinner at Karl Ratzsch’s” (a fancy German restaurant in downtown Milwaukee) that if the performance assessments become voluntary, many schools will “phase them out or make them simpler.”

“I am in favor of empowering schools but not empowering them to lower their expectations, which is what might happen if performance assessments become voluntary,” Reeves said.

Alverno’s Diez said performance assessments also do more to improve classroom learning than standardized tests, and are more classroom based. “Teachers are better able to know what the kids can do and how instruction needs to be changed to help them,” she said.

The positive impact of the MPS performance assessments has been dramatic in some schools, especially in science and writing.

In some elementary schools, science instruction had consisted of reading a chapter and answering the questions at the end. With the implementation of a science performance assessment in fifth grade, elementary schools pushed a more active approach – teaching kids the scientific process, conducting experiments, and demonstrating and reflecting on their work in science fairs.

Science Scores Rise

Interestingly, science achievement scores have risen the last two years in MPS. Ellwood says she’s “seen a real change in the attention to science in this district.” She attributes gains to the hard work of teachers who have been pushed by the performance assessments.

“Now we are moving away from the science performance assessment, with all its flaws, and we are not going to be testing science in many of the grades throughout the district,” Ellwood said. “So once again we will step away from our commitment to science.”

In writing, the performance assessments pushed schools where teachers had not spent much time teaching writing or encouraging students to write. “The record clearly shows that writing assessment had an enormous impact on increasing our students’ writing ability,” Ellwood said. “We went from well below the national average in writing to above the national average at the elementary school [level].”

The MPS 1998-1999 Accountability Report appears to confirm Ellwood’s view. “For many years, MPS has emphasized high student achievement in writing, which has been reflected in strong achievement levels on the WSAS [the state standardized tests],” the report says. “An important vehicle in maintaining this achievement is long-time use of the MPS Writing Assessment, administered to elementary students in grades four and five.”

Sandra Dickerson, the MPS curriculum specialist in charge of language arts, told Rethinking Schools that high schools had similar positive benefits as a result of the writing proficiency and performance assessment. It had, in fact, “helped close the achievement gap between ethnic groups and helped to improve achievement across all content areas,” she said.

“Why destroy an effective means of assessment and instruction?” she asked of the plan to eliminate the districtwide performance assessments.

Patrick Doyle, learning coordinator at Lincoln Center of the Arts, echoed such sentiments. The performance assessments at Lincoln, an MPS middle school, led to a “growth in the quality of teaching and learning in the classrooms,” he said.

“Performance assessments are a practical application of good pedagogy and they change your teaching,” Doyle said.

As an example, Doyle said that several years ago he brought together 28 Lincoln teachers from different disciplines to score the writing performance assessments. It was an “eye-opener for the staff” and as a result the school started a program of “writing across the curriculum.” Achievement scores in all subject areas have risen significantly on both the performance assessments and state-mandated standardized tests for Lincoln students, according to the MPS Accountability Report.

School Board member Gardner, on the other hand, said at the public hearing that the science and math performance assessments were ineffective in “90-95% of the schools.” He offered no data to substantiate his claim.

The performance assessments are not without controversy. Their quality has varied and staff development at times has been inadequate. Some teachers have also felt the assessments are too difficult. Despite problems, however, the performance assessments raised expectations of what students should be able to actually do, especially since they were non-negotiable and mandated by the district. “The bottom line is, teacher expectations rose because of the performance assessments,” Reeves said. He also said that Milwaukee has some of the best writing achievement in the nation and that this is a direct result of the performance assessments.

Interestingly, School Board President Thompson told Rethinking Schools that one reason he supports the value-added standardized tests is because the performance assessments were too hard. “We make ourselves look worse because we are more demanding than other Wisconsin districts,” he said. Thompson explained, for example, that in the WSAS science test “we do fairly well and then we do our own science assessment and we do miserable a year later.”

Performance assessments help in the professional development of teachers in two other ways as well – through involving a group of teachers in the actual development of the prompts and by developing a core of teachers who are trained in the scoring of the assessments. These activities have proven to be positive, although the lack of funds and some shortsightedness have led to problems. For instance, the quality of some of the assessments has been criticized. And this year, the central administration decided for financial reasons to score the performance assessments at the school level – rather than paying teachers out of central funds to meet on a citywide basis and score the assessments. In some cases, school-based scoring has lessened the consistency of the scores.

GOOD REFORM COSTS MONEY

Performance assessments generally cost more and require more staff development than standardized tests. One of the unanswered questions about the new testing proposal is whether it’s being driven by a school board that wants to reform the district on the cheap.

“I am convinced it’s a money problem,” Diez told Rethinking Schools in answering why she thought MPS was considering the new testing proposal. “It would be too expensive to bring the system up to what it should be. … They are not going to change what’s happening to kids if all they are doing is testing kids more often. There have to be ways to ratchet up the quality of teaching and learning.”

In explaining why relying on standardized testing will not improve education, Alverno’s Diez offers a metaphor she learned from her father, who worked in the livestock business. “He would say, ‘You don’t fatten a pig by weighing it more often,'” she wrote the MPS School Board. “The kind of standardized testing being proposed in the ‘Balanced Assessment System’ is, in effect, simply weighing. Where are we nurturing and feeding our students so that they will be able to do well on the tests? A balanced system needs to ensure the opportunity for students to learn and grow; if it does not, then testing becomes a cruel hoax.”