Why Poor Schools Can’t Win at Standardized Testing

The companies that create the most important exams also publish textbooks that contain many of the answers. Low-income school districts can’t afford to buy them.

First-grade students tear up their workbooks on the last day of classes at Smith Elementary, one of 23 Philadelphia schools closed by the state-run School Reform Commission in May 2013.
National Journal
Meredith Broussard, The Atlantic
Add to Briefcase
See more stories about...
Meredith Broussard, The Atlantic
July 15, 2014, 12:31 p.m.

You hear a lot nowadays about the ma­gic of big data. Get­ting hold of the right num­bers can in­crease rev­en­ue, im­prove de­cision-mak­ing, or help you find a mate — or so the think­ing goes. In 2009, U.S. Edu­ca­tion Sec­ret­ary Arne Duncan told a crowd of edu­ca­tion re­search­ers: “I am a deep be­liev­er in the power of data to drive our de­cisions. Data gives us the roadmap to re­form. It tells us where we are, where we need to go, and who is most at risk.”

This is a story about what happened when I tried to use big data to help re­pair my loc­al pub­lic schools. I failed. And the reas­ons why I failed have everything to do with why the Amer­ic­an sys­tem of stand­ard­ized test­ing will nev­er suc­ceed.

A few years ago, I star­ted hav­ing trouble help­ing my son with his first-grade home­work. I’m a data-journ­al­ism pro­fess­or at Temple Uni­versity, and when my son asked me for help on a work­sheet one day, I ran in­to an epi­stem­o­lo­gic­al di­lemma. My own gen­er­al know­ledge (and the In­ter­net) told me there were many pos­sible “cor­rect” an­swers. However, only one of these an­swers would get him full cred­it on the as­sign­ment.

“I need to write down nat­ur­al re­sources,” he told me.

“Air, wa­ter, oil, gas, coal,” I replied.

“I already put down air and wa­ter,” he said. “Oil and gas and coal aren’t nat­ur­al re­sources.”

“Of course they are,” I said. “They’re non-re­new­able nat­ur­al re­sources, but they’re still nat­ur­al re­sources.”

“But they wer­en’t on the list the teach­er gave in class.”

I knew my son would start tak­ing stand­ard­ized tests in third grade. If the first-grade home­work was this con­fus­ing, I was really wor­ried about how he — or any kid — was sup­posed to fig­ure out the tests. I had been spend­ing time with civic hack­ers, the kind of people who build soft­ware and crunch gov­ern­ment data for fun, and I de­cided to see if I could come up with a beat-the-test strategy de­rived from a pop­u­lar SAT prep course I used to teach.

In es­sence, I tried to game the third-grade Pennsylvania Sys­tem of School As­sess­ment (PSSA), the stand­ard­ized test for my state. Along with a team of pro­fes­sion­al de­velopers, I de­signed ar­ti­fi­cial-in­tel­li­gence soft­ware to crunch the avail­able data. I talked to teach­ers. I talked to stu­dents. I vis­ited schools and sat through School Re­form Com­mis­sion meet­ings.

After six months of this, I dis­covered that the test can be gamed. Not by us­ing a beat-the-test strategy, but by a shock­ingly low-tech strategy: read­ing the text­book that con­tains the an­swers.


Phil­adelphia is the eighth-largest school dis­trict in the coun­try, and its pub­lic stu­dents are over­whelm­ingly poor: 79 per­cent of them are eli­gible for free or re­duced-price lunch. The high-school gradu­ation rate is only 64 per­cent and few­er than half of stu­dents man­aged to score pro­fi­cient or above on the 2013 PSSA.

When a prob­lem ex­ists in Phil­adelphia schools, it gen­er­ally ex­ists in oth­er large urb­an schools across the na­tion. One of those prob­lems — shared by dis­tricts in New York, D.C., Chica­go, Los Angeles, and oth­er ma­jor cit­ies — is that many schools don’t have enough money to buy books. The School Dis­trict of Phil­adelphia re­cently tweeted a photo of May­or Mi­chael Nut­ter hand­ing out 200,000 donated books to K-3 stu­dents. Un­for­tu­nately, in­tro­du­cing chil­dren to clas­sic works of lit­er­at­ure won’t raise their abysmal test scores.

This is be­cause stand­ard­ized tests are not based on gen­er­al know­ledge. As I learned in the course of my in­vest­ig­a­tion, they are based on spe­cif­ic know­ledge con­tained in spe­cif­ic sets of books: the text­books cre­ated by the test makers.

All of this has to do with the eco­nom­ics of test­ing. Across the na­tion, stand­ard­ized tests come from one of three com­pan­ies: CTB Mc­Graw Hill, Houghton Miff­lin Har­court, or Pear­son. These cor­por­a­tions write the tests, grade the tests, and pub­lish the books that stu­dents use to pre­pare for the tests. Houghton Miff­lin has a 38 per­cent mar­ket share, ac­cord­ing to its press ma­ter­i­als. In 2013, the com­pany brought in $1.38 bil­lion in rev­en­ue.

Pennsylvania cur­rently has a multi-mil­lion-dol­lar con­tract with a com­pany called Data Re­cog­ni­tion Cor­por­a­tion (DRC) to grade the PS­SAs. DRC works with Mc­Graw-Hill as part of a con­sor­ti­um that has a $186 mil­lion fed­er­al con­tract to write and grade stand­ard­ized tests for the rest of the coun­try. Mc­Graw-Hill, mean­while, also writes the books and cur­ricula schools buy to pre­pare stu­dents for the tests. Every­day Math, the branded cur­riculum used by most Phil­adelphia pub­lic schools in grades K­—5, is pub­lished by Mc­Graw Hill.

Put simply, any teach­er who wants his or her stu­dents to pass the tests has to give out books from the Big Three pub­lish­ers. If you look at a text­book from one of these com­pan­ies and look at the stand­ard­ized tests writ­ten by the same com­pany, even a third grader can see that many of the ques­tions on the test are sim­il­ar to the ques­tions in the book. In fact, Pear­son came un­der fire last year for us­ing a pas­sage on a stand­ard­ized test that was taken ver­batim from a Pear­son text­book.

The is­sue of­ten has as much to do with word­ing as it does with facts or fig­ures. Con­sider this ques­tion from the 2009 PSSA, which asked third-grade stu­dents to write down an even num­ber with three di­gits and then ex­plain how they ar­rived at their an­swers. Here’s an ex­ample of a cor­rect an­swer, taken from a test­ing sup­ple­ment put out by the Pennsylvania De­part­ment of Edu­ca­tion:

Here’s an ex­ample of a par­tially cor­rect an­swer that earned the stu­dent just one point in­stead of two:

This second an­swer is cor­rect, but the third-grade stu­dent lacked the spe­cif­ic con­cep­tu­al un­der­pin­nings to ex­plain why it was cor­rect. The Every­day Math cur­riculum hap­pens to cov­er this ra­tionale in de­tail, and the third-grade study guide in­structs teach­ers to drill stu­dents on it: “What is one of the rules for odd and even factors and their products? How do you know that this rule is true?” A third-grader without a text­book can learn the dif­fer­ence between even and odd num­bers, but she will find it hard to guess how the test-maker wants to see that dif­fer­ence ex­plained.

Un­like col­lege pro­fess­ors, who simply as­sign books and leave it to the stu­dents to buy them, K—12 teach­ers have to provide stu­dents with books. But it’s not a simple mat­ter of or­der­ing one book per stu­dent per sub­ject. Based on the schools I vis­ited and the teach­ers I in­ter­viewed, each stu­dent needs at least one text­book and one work­book per class, plus a bunch of work­sheets and pro­jects the teach­er pulls from as­sor­ted web­sites (not to men­tion bind­er clips and con­struc­tion pa­per and scis­sors and oth­er pro­ject-based ma­ter­i­als). Books can be re­used year to year, but only if the state stand­ards haven’t changed — which they have every year for at least the past dec­ade.

Urb­an teach­ers have a kind of un­der­ground eco­nomy, Co­hen ex­plained. Some teach­ers hustle and ne­go­ti­ate to get books and pa­per and desks for their stu­dents.

Once I real­ized the dir­ect con­nec­tion between text­books and stand­ard­ized-test suc­cess, I tried to find out ex­actly how many Phil­adelphia schools were miss­ing books from the Big Three pub­lish­ers. I was also curi­ous how much money it would take to make up for the short­fall.

The first chal­lenge came when I asked the School Dis­trict of Phil­adelphia for a list of which cur­ricula were be­ing used at which schools. If you want to know which books should be in a school, you need to know the name of the cur­riculum the school uses. (Us­ing a branded cur­riculum like Every­day Math al­lows a school to place its or­ders more ef­fi­ciently and ne­go­ti­ate a bulk dis­count.)

“We don’t have that list,” an ad­min­is­trat­or at the Phil­adelphia Of­fice of Cur­riculum and De­vel­op­ment told me. “It doesn’t ex­ist.”

“How do you know what cur­riculum each school is us­ing?” I asked.

“We don’t.”

There was si­lence on the phone for a mo­ment.

“How do you know if the schools have all the books they need?”

“We don’t.”

Ac­cord­ing to dis­trict policy, every school is sup­posed to re­cord its book in­vent­ory in a cent­ral­ized data­base called the Text­book Stor­age Sys­tem. “If you give me that list of books in the Text­book Stor­age Sys­tem, I can re­verse-en­gin­eer it and make you a list of which cur­riculum each school uses,” I told the cur­riculum of­ficer.

“Really?” she said. “That would be great. I didn’t know you could do that!”

So I did what com­puter pro­gram­mers do in this kind of situ­ation: I cre­ated a work­around. I built a pro­gram to look at each Phil­adelphia pub­lic school and see wheth­er the num­ber of books at the school was equal to the num­ber of stu­dents. The res­ults of the ana­lys­is did not look good. The av­er­age school had only 27 per­cent of the books in the dis­trict’s re­com­men­ded cur­riculum. At least 10 schools had no books at all, ac­cord­ing to their own re­cords. Oth­ers had books that were hope­lessly out of date.

I vis­ited some of these schools and asked stu­dents how much ac­cess they had to text­books. “We had books at my high school, but they were from, like, the 1980s,” said Dav­id, a re­cent gradu­ate of Phil­adelphia pub­lic schools. A ju­ni­or at a pub­lic high school com­plained to me that her his­tory text­book had pic­tures of testicles drawn on each page.


When I vis­ited an al­gebra class at the Academy at Palumbo, a mag­net school in South Phil­adelphia, a math teach­er, Bri­an Co­hen, seemed sur­prised by the in­form­a­tion I presen­ted to him. Palumbo’s re­cords showed that the school used Fast Track to a 5: Pre­par­ing for the AB and BC Cal­cu­lus Ex­ams, a book pub­lished by Houghton Miff­lin. However, the quant­ity of books in the sys­tem read “0.”

“That’s strange,” said Co­hen after I sat in on his Al­gebra I class. “I’m not sure why it says we have zero cop­ies.” Had that branded cur­riculum had been se­lec­ted but nev­er ordered? Or had the books had been ordered but in­ter­cep­ted some­where along the way?

I asked if we could go look in the book closet and Co­hen took me down the hall. On the way, we stopped to chat with a col­league of his who taught cal­cu­lus. “Do you have enough books?” Co­hen asked.

“I do now,” she said. “Some school in West Phil­adelphia closed, and I man­aged to get all the text­books from there. I had a friend who hooked me up.” But she wasn’t us­ing Fast Track to 5; she had a dif­fer­ent cal­cu­lus book that wasn’t on my re­cord sheet.

Urb­an teach­ers have a kind of un­der­ground eco­nomy, Co­hen ex­plained. Some teach­ers hustle and ne­go­ti­ate to get books and pa­per and desks for their stu­dents. They spend their spare time run­ning cam­paigns on fun­drais­ing sites like DonorsChoose.org, and they keep an eye out for any ma­ter­i­als they can nab from oth­er schools. Phil­adelphia teach­ers spend an av­er­age of $300 to $1,000 of their own money each year to sup­ple­ment their $100 an­nu­al budget for classroom sup­plies, ac­cord­ing to a Phil­adelphia Fed­er­a­tion of Teach­ers sur­vey.

Co­hen and I ar­rived at the math de­part­ment “book closet,” which was ac­tu­ally just a corner in­side the locked and empty of­fice of the math de­part­ment chair­per­son. “Here’s where we keep the ex­tra books,” he said, ges­tur­ing to two short wooden book­shelves. A me­di­um-sized box with open flaps sat on the floor. Co­hen looked in­side. “Well, we found the AP Cal­cu­lus books,” he said. The box was filled with brand-new cop­ies of Fast Track to 5.

It would have been easy to blame this glitch on the lack of a cent­ral­ized com­puter sys­tem. The only prob­lem was, such a com­puter sys­tem did ex­ist, and I was look­ing at a prin­tout from it. The prin­tout said Palumbo had zero cop­ies of the book, but 24 books were sit­ting in front of me in a box on the floor of a locked of­fice.


The Phil­adelphia schools don’t just have a text­book prob­lem. They have a data prob­lem — which is ac­tu­ally a people prob­lem. We tend to think of data as im­mut­able truth. But we for­get that data and data-col­lec­tion sys­tems are cre­ated by people. Flesh-and-blood hu­mans need to count the books in a school and enter the num­bers in­to a data­base. Usu­ally, these hu­mans are ad­min­is­trat­ive as­sist­ants or teach­er’s aides. But severe state fund­ing cuts over the past sev­er­al years have meant cut­backs in the school dis­trict’s ad­min­is­trat­ive staff. Even the best data-col­lec­tion sys­tem is use­less if there are no people avail­able to man­age it.

Mi­chael Masch, the vice pres­id­ent of fin­ance and chief fin­an­cial of­ficer at Man­hat­tan Col­lege and the former chief fin­an­cial of­ficer of the School Dis­trict of Phil­adelphia, told me that he used to routinely send his staff in­to schools to do book­keep­ing and oth­er tasks that over­worked prin­cipals couldn’t handle. “Prin­cipals wer­en’t good at man­aging cash ac­counts or stu­dent ac­counts. They needed sup­port in per­form­ing ad­min­is­trat­ive func­tions be­cause they were un­der­staffed,” said Masch. “If the prin­cip­al doesn’t meet with every par­ent, deal with every crisis, they get cri­ti­cized. If they don’t do the in­vis­ible stuff, like the pa­per­work, they’re not go­ing to read about it in the news­pa­per. So they triage.”

When it comes to the book scarcity, Phil­adelphia prin­cipals re­act in pre­dict­able ways. “They are very pos­sess­ive of their text­books,” Re­becca Dhondt, the par­ent of a second grader and a fourth grader at Jenks Ele­ment­ary School, told me. “My daugh­ter is not al­lowed to bring her text­book home be­cause they don’t want it to get lost.” For the past two years, she has sur­veyed teach­ers to find out what’s on their wish lists (mostly trade books and ba­sic school sup­plies) and then col­lec­ted dona­tions from the com­munity. “When I first did it last year, the prin­cip­al said, ‘Oh, we have some of that stuff,’” said Dhondt. As with the AP cal­cu­lus books at Palumbo, the miss­ing items were sit­ting some­where in the school but hadn’t made it in­to the right hands. “There’s not enough sup­port to con­nect the sup­plies in the sup­ply closets or the lib­rar­ies with the teach­ers in the classroom,” Dhondt said. “They need to have enough money to con­nect the dots.”

Keep­ing track of sup­plies is one prob­lem; keep­ing track of the stu­dents who will use them is a whole oth­er chal­lenge. In Phil­adelphia schools, many stu­dents are in foster care or nav­ig­at­ing oth­er pre­cari­ous liv­ing situ­ations, which means they fre­quently switch schools. A re­cent re­port by the Chil­dren’s Hos­pit­al of Phil­adelphia showed that one in five Phil­adelphia pub­lic high school stu­dents has been in­volved with the child wel­fare or ju­ven­ile-justice sys­tem. One teach­er told me that when she taught in a West Philly high school, she gained or lost a stu­dent at least every two weeks.

“There is a set of lo­gist­ic­al is­sues in a dis­trict this big that most dis­tricts in the U.S. don’t face,” ex­plained Donna Cooper, the ex­ec­ut­ive dir­ect­or of an or­gan­iz­a­tion called the Pub­lic Cit­izens for Chil­dren and Youth. “Everything isn’t what it ap­pears.”


After I fin­ished the first round of data ana­lys­is in 2013, I went to the school dis­trict and asked to present my find­ings to Phil­adelphia Su­per­in­tend­ent Wil­li­am Hite. The dis­trict spokes­per­son told me Hite wasn’t avail­able and in­stead offered me a meet­ing with Steph­en Spence, the deputy for the of­fice of school-sup­port ser­vices.

Spence, a former gym teach­er in his early 60s, was in charge of school open­ings and clos­ings. His job used to be handled by a whole staff, but ever since the cut­backs, Spence had been single­han­dedly tak­ing care of everything from desks to car­pets.

I asked him how he veri­fied that schools had enough books at the be­gin­ning of each year. He ex­plained that every prin­cip­al was sup­posed to sub­mit a school open­ing check­list and a school-clos­ing check­list. On that check­list (a Mi­crosoft Word doc­u­ment that he emailed to all the prin­cipals), there was a box the prin­cip­al could tick to in­dic­ate that the school had all of the books it needed to op­er­ate.

“In­vent­ory is not mi­cro­man­aged at a cent­ral- of­fice level,” said Spence. “A prin­cip­al that has very good skills with tech­no­logy might de­vel­op an in­vent­ory sys­tem that they keep on­line. An­oth­er prin­cip­al who is not so good with tech­no­logy might have just a per­son who counts the books, car­ries them from one loc­a­tion to an­oth­er, puts them in the closet, and visu­ally checks that they’re there.”

I wondered about this, since a dis­trict-wide elec­tron­ic sys­tem had been cre­ated sev­er­al years back. In 2009, a stu­dent stood up at meet­ing of Phil­adelphia’s School Re­form Com­mis­sion and pro­claimed, “I don’t have a book.” After that, Su­per­in­tend­ent Ar­lene Ack­er­man had re­solved to com­pu­ter­ize the Dis­trict’s in­vent­ory. Chief In­form­a­tion Of­ficer Melanie Har­ris had told me that the sys­tem had been de­veloped us­ing in­tern­al re­sources.

“You’re say­ing that the on­line sys­tem is no longer in use?” I asked Spence.

The prin­cipals pre­ferred to use their own sys­tems, he said, and re­port their in­vent­ory to him. “I rely on the prin­cipals and, I’m go­ing to say, real-time data. It gets tracked through the doc­u­ments we talked about pre­vi­ously: the school-open­ing and the school-clos­ing check­lists.”

As Spence re­ceives the prin­cipals’ check­lists, he enters the in­form­a­tion in­to an Ex­cel spread­sheet on his com­puter.

“Does this Ex­cel doc­u­ment get shared with any­one?” I asked.

“It gets shared with as­sist­ant su­per­in­tend­ents,” said Spence. “We have meet­ings. We put the Ex­cel spread­sheet on a pro­ject­or on a large screen dur­ing our school-open­ing meet­ings.”

As a data-sci­ence pro­fes­sion­al, it was clear to me that Spence was in over his head. Mil­lions of books, hun­dreds of thou­sands of desks — it is im­possible to keep track of all of these ob­jects without tech­no­logy and suf­fi­cient people to track them. It’s just as dif­fi­cult to fig­ure out how to use the data cor­rectly.

The end res­ult is that Phil­adelphia’s num­bers simply don’t add up. Con­sider the eighth grade at Tilden Middle School in South­w­est Phil­adelphia. Ac­cord­ing to dis­trict re­cords, Tilden uses a read­ing cur­riculum called Ele­ments of Lit­er­at­ure, pub­lished by Houghton Miff­lin. In 2012—2013, Tilden had 117 stu­dents in its eighth grade, but it only had 42 of these eighth-grade read­ing text­books, ac­cord­ing to the (ad­mit­tedly flawed) dis­trict in­vent­ory sys­tem. Tilden’s eighth grade stu­dents largely failed the state stand­ard­ized test: Their av­er­age read­ing score was 29.4 per­cent, com­pared with 57.9 per­cent dis­trictwide.

One prob­lem is that no one is keep­ing track of what these stu­dents need and what they ac­tu­ally have. An­oth­er prob­lem is that there’s simply too little money in the edu­ca­tion budget. The Ele­ments of Lit­er­at­ure text­book costs $114.75. However, in 2012—2013, Tilden (like every oth­er middle school in Phil­adelphia) was only al­loc­ated $30.30 per stu­dent to buy books — and that amount, which was barely a quarter the price of one text­book, was sup­posed to cov­er every sub­ject, not just one. My own cal­cu­la­tions show that the av­er­age Phil­adelphia school had only 27 per­cent of the books re­quired to teach its cur­riculum in 2012-2013, and it would have cost $68 mil­lion to pay for all the books schools need. Be­cause the school dis­trict doesn’t col­lect com­pre­hens­ive data on its text­book use, this cal­cu­la­tion could be an over­es­tim­ate — but more likely, it’s a sig­ni­fic­ant un­der­es­tim­ate.

At the end of the 2012—2013 school year, the book budget was elim­in­ated al­to­geth­er. Last June, the state-run School Re­form Com­mis­sion — which re­placed Phil­adelphia’s school board in 2001 — passed a “dooms­day budget” that fell $300 mil­lion short of the dis­trict’s op­er­at­ing costs for the 2014 fisc­al year. (The gov­ernor of Pennsylvania had already cut al­most a bil­lion dol­lars from pub­lic edu­ca­tion fund­ing in 2011.) Phil­adelphia schools were al­lot­ted $0 per stu­dent for text­books. The 2015 budget like­wise fea­tures no fund­ing for books.

It may be many years un­til Phil­adelphia’s edu­ca­tion budget matches its cur­riculum re­quire­ments. In the mean­time, there are a few things the dis­trict — and oth­er flail­ing school dis­tricts in Amer­ica — can do. Stop giv­ing stand­ard­ized tests that are in­ex­tric­ably tied to spe­cif­ic sets of books. At the very least, stop us­ing test scores to eval­u­ate teach­er per­form­ance without provid­ing the items each teach­er needs to do his or her job. Most of all, avoid basing an en­tire edu­ca­tion sys­tem on ma­ter­i­als so costly that big, urb­an dis­tricts can’t af­ford to buy them. Un­til these things change, it will be im­possible to raise stand­ard­ized test scores — des­pite the best ef­forts of the teach­ers and stu­dents who will re­turn to school this fall and find no new books wait­ing for them.

What We're Following See More »
Sen. Cochran Back to Washington Tuesday
11 hours ago
Trump: Marino Withdrawing Nomination for Drug Czar
17 hours ago
Doesn’t Express Confidence in Marino
Trump to Declare Opioid Emergency Next Week
1 days ago

After initially promising it in August, "President Trump said Monday that he will declare a national emergency next week to address the opioid epidemic." When asked, he also "declined to express confidence in Rep. Tom Marino (R-Pa.), his nominee for drug czar, in the wake of revelations that the lawmaker helped steer legislation making it harder to act against giant drug companies."

Manchin Asks Trump to Drop Marino’s Nomination for Drug Czar
1 days ago
McCaskill Will Introduce Bill in Response to “60 Minutes” Scoop
1 days ago

In the wake of Sunday's blockbuster 60 Minutes/Washington Post report on opioid regulation and enforcement, Sen. Claire McCaskill (D-MO) has introduced legislation that "would repeal a 2016 law that hampered the Drug Enforcement Administration’s ability to regulate opioid distributors it suspects of misconduct." In a statement, McCaskill said: “Media reports indicate that this law has significantly affected the government’s ability to crack down on opioid distributors that are failing to meet their obligations and endangering our communities."


Welcome to National Journal!

You are currently accessing National Journal from IP access. Please login to access this feature. If you have any questions, please contact your Dedicated Advisor.