Why Poor Schools Can’t Win at Standardized Testing

The companies that create the most important exams also publish textbooks that contain many of the answers. Low-income school districts can’t afford to buy them.

First-grade students tear up their workbooks on the last day of classes at Smith Elementary, one of 23 Philadelphia schools closed by the state-run School Reform Commission in May 2013.
National Journal
Meredith Broussard, The Atlantic
See more stories about...
Meredith Broussard, The Atlantic
July 15, 2014, 12:31 p.m.

You hear a lot nowadays about the ma­gic of big data. Get­ting hold of the right num­bers can in­crease rev­en­ue, im­prove de­cision-mak­ing, or help you find a mate — or so the think­ing goes. In 2009, U.S. Edu­ca­tion Sec­ret­ary Arne Duncan told a crowd of edu­ca­tion re­search­ers: “I am a deep be­liev­er in the power of data to drive our de­cisions. Data gives us the roadmap to re­form. It tells us where we are, where we need to go, and who is most at risk.”

This is a story about what happened when I tried to use big data to help re­pair my loc­al pub­lic schools. I failed. And the reas­ons why I failed have everything to do with why the Amer­ic­an sys­tem of stand­ard­ized test­ing will nev­er suc­ceed.

A few years ago, I star­ted hav­ing trouble help­ing my son with his first-grade home­work. I’m a data-journ­al­ism pro­fess­or at Temple Uni­versity, and when my son asked me for help on a work­sheet one day, I ran in­to an epi­stem­o­lo­gic­al di­lemma. My own gen­er­al know­ledge (and the In­ter­net) told me there were many pos­sible “cor­rect” an­swers. However, only one of these an­swers would get him full cred­it on the as­sign­ment.

“I need to write down nat­ur­al re­sources,” he told me.

“Air, wa­ter, oil, gas, coal,” I replied.

“I already put down air and wa­ter,” he said. “Oil and gas and coal aren’t nat­ur­al re­sources.”

“Of course they are,” I said. “They’re non-re­new­able nat­ur­al re­sources, but they’re still nat­ur­al re­sources.”

“But they wer­en’t on the list the teach­er gave in class.”

I knew my son would start tak­ing stand­ard­ized tests in third grade. If the first-grade home­work was this con­fus­ing, I was really wor­ried about how he — or any kid — was sup­posed to fig­ure out the tests. I had been spend­ing time with civic hack­ers, the kind of people who build soft­ware and crunch gov­ern­ment data for fun, and I de­cided to see if I could come up with a beat-the-test strategy de­rived from a pop­u­lar SAT prep course I used to teach.

In es­sence, I tried to game the third-grade Pennsylvania Sys­tem of School As­sess­ment (PSSA), the stand­ard­ized test for my state. Along with a team of pro­fes­sion­al de­velopers, I de­signed ar­ti­fi­cial-in­tel­li­gence soft­ware to crunch the avail­able data. I talked to teach­ers. I talked to stu­dents. I vis­ited schools and sat through School Re­form Com­mis­sion meet­ings.

After six months of this, I dis­covered that the test can be gamed. Not by us­ing a beat-the-test strategy, but by a shock­ingly low-tech strategy: read­ing the text­book that con­tains the an­swers.


Phil­adelphia is the eighth-largest school dis­trict in the coun­try, and its pub­lic stu­dents are over­whelm­ingly poor: 79 per­cent of them are eli­gible for free or re­duced-price lunch. The high-school gradu­ation rate is only 64 per­cent and few­er than half of stu­dents man­aged to score pro­fi­cient or above on the 2013 PSSA.

When a prob­lem ex­ists in Phil­adelphia schools, it gen­er­ally ex­ists in oth­er large urb­an schools across the na­tion. One of those prob­lems — shared by dis­tricts in New York, D.C., Chica­go, Los Angeles, and oth­er ma­jor cit­ies — is that many schools don’t have enough money to buy books. The School Dis­trict of Phil­adelphia re­cently tweeted a photo of May­or Mi­chael Nut­ter hand­ing out 200,000 donated books to K-3 stu­dents. Un­for­tu­nately, in­tro­du­cing chil­dren to clas­sic works of lit­er­at­ure won’t raise their abysmal test scores.

This is be­cause stand­ard­ized tests are not based on gen­er­al know­ledge. As I learned in the course of my in­vest­ig­a­tion, they are based on spe­cif­ic know­ledge con­tained in spe­cif­ic sets of books: the text­books cre­ated by the test makers.

All of this has to do with the eco­nom­ics of test­ing. Across the na­tion, stand­ard­ized tests come from one of three com­pan­ies: CTB Mc­Graw Hill, Houghton Miff­lin Har­court, or Pear­son. These cor­por­a­tions write the tests, grade the tests, and pub­lish the books that stu­dents use to pre­pare for the tests. Houghton Miff­lin has a 38 per­cent mar­ket share, ac­cord­ing to its press ma­ter­i­als. In 2013, the com­pany brought in $1.38 bil­lion in rev­en­ue.

Pennsylvania cur­rently has a multi-mil­lion-dol­lar con­tract with a com­pany called Data Re­cog­ni­tion Cor­por­a­tion (DRC) to grade the PS­SAs. DRC works with Mc­Graw-Hill as part of a con­sor­ti­um that has a $186 mil­lion fed­er­al con­tract to write and grade stand­ard­ized tests for the rest of the coun­try. Mc­Graw-Hill, mean­while, also writes the books and cur­ricula schools buy to pre­pare stu­dents for the tests. Every­day Math, the branded cur­riculum used by most Phil­adelphia pub­lic schools in grades K­”“5, is pub­lished by Mc­Graw Hill.

Put simply, any teach­er who wants his or her stu­dents to pass the tests has to give out books from the Big Three pub­lish­ers. If you look at a text­book from one of these com­pan­ies and look at the stand­ard­ized tests writ­ten by the same com­pany, even a third grader can see that many of the ques­tions on the test are sim­il­ar to the ques­tions in the book. In fact, Pear­son came un­der fire last year for us­ing a pas­sage on a stand­ard­ized test that was taken ver­batim from a Pear­son text­book.

The is­sue of­ten has as much to do with word­ing as it does with facts or fig­ures. Con­sider this ques­tion from the 2009 PSSA, which asked third-grade stu­dents to write down an even num­ber with three di­gits and then ex­plain how they ar­rived at their an­swers. Here’s an ex­ample of a cor­rect an­swer, taken from a test­ing sup­ple­ment put out by the Pennsylvania De­part­ment of Edu­ca­tion:

Here’s an ex­ample of a par­tially cor­rect an­swer that earned the stu­dent just one point in­stead of two:

This second an­swer is cor­rect, but the third-grade stu­dent lacked the spe­cif­ic con­cep­tu­al un­der­pin­nings to ex­plain why it was cor­rect. The Every­day Math cur­riculum hap­pens to cov­er this ra­tionale in de­tail, and the third-grade study guide in­structs teach­ers to drill stu­dents on it: “What is one of the rules for odd and even factors and their products? How do you know that this rule is true?” A third-grader without a text­book can learn the dif­fer­ence between even and odd num­bers, but she will find it hard to guess how the test-maker wants to see that dif­fer­ence ex­plained.

Un­like col­lege pro­fess­ors, who simply as­sign books and leave it to the stu­dents to buy them, K”“12 teach­ers have to provide stu­dents with books. But it’s not a simple mat­ter of or­der­ing one book per stu­dent per sub­ject. Based on the schools I vis­ited and the teach­ers I in­ter­viewed, each stu­dent needs at least one text­book and one work­book per class, plus a bunch of work­sheets and pro­jects the teach­er pulls from as­sor­ted web­sites (not to men­tion bind­er clips and con­struc­tion pa­per and scis­sors and oth­er pro­ject-based ma­ter­i­als). Books can be re­used year to year, but only if the state stand­ards haven’t changed — which they have every year for at least the past dec­ade.

Urb­an teach­ers have a kind of un­der­ground eco­nomy, Co­hen ex­plained. Some teach­ers hustle and ne­go­ti­ate to get books and pa­per and desks for their stu­dents.

Once I real­ized the dir­ect con­nec­tion between text­books and stand­ard­ized-test suc­cess, I tried to find out ex­actly how many Phil­adelphia schools were miss­ing books from the Big Three pub­lish­ers. I was also curi­ous how much money it would take to make up for the short­fall.

The first chal­lenge came when I asked the School Dis­trict of Phil­adelphia for a list of which cur­ricula were be­ing used at which schools. If you want to know which books should be in a school, you need to know the name of the cur­riculum the school uses. (Us­ing a branded cur­riculum like Every­day Math al­lows a school to place its or­ders more ef­fi­ciently and ne­go­ti­ate a bulk dis­count.)

“We don’t have that list,” an ad­min­is­trat­or at the Phil­adelphia Of­fice of Cur­riculum and De­vel­op­ment told me. “It doesn’t ex­ist.”

“How do you know what cur­riculum each school is us­ing?” I asked.

“We don’t.”

There was si­lence on the phone for a mo­ment.

“How do you know if the schools have all the books they need?”

“We don’t.”

Ac­cord­ing to dis­trict policy, every school is sup­posed to re­cord its book in­vent­ory in a cent­ral­ized data­base called the Text­book Stor­age Sys­tem. “If you give me that list of books in the Text­book Stor­age Sys­tem, I can re­verse-en­gin­eer it and make you a list of which cur­riculum each school uses,” I told the cur­riculum of­ficer.

“Really?” she said. “That would be great. I didn’t know you could do that!”

So I did what com­puter pro­gram­mers do in this kind of situ­ation: I cre­ated a work­around. I built a pro­gram to look at each Phil­adelphia pub­lic school and see wheth­er the num­ber of books at the school was equal to the num­ber of stu­dents. The res­ults of the ana­lys­is did not look good. The av­er­age school had only 27 per­cent of the books in the dis­trict’s re­com­men­ded cur­riculum. At least 10 schools had no books at all, ac­cord­ing to their own re­cords. Oth­ers had books that were hope­lessly out of date.

I vis­ited some of these schools and asked stu­dents how much ac­cess they had to text­books. “We had books at my high school, but they were from, like, the 1980s,” said Dav­id, a re­cent gradu­ate of Phil­adelphia pub­lic schools. A ju­ni­or at a pub­lic high school com­plained to me that her his­tory text­book had pic­tures of testicles drawn on each page.


When I vis­ited an al­gebra class at the Academy at Palumbo, a mag­net school in South Phil­adelphia, a math teach­er, Bri­an Co­hen, seemed sur­prised by the in­form­a­tion I presen­ted to him. Palumbo’s re­cords showed that the school used Fast Track to a 5: Pre­par­ing for the AB and BC Cal­cu­lus Ex­ams, a book pub­lished by Houghton Miff­lin. However, the quant­ity of books in the sys­tem read “0.”

“That’s strange,” said Co­hen after I sat in on his Al­gebra I class. “I’m not sure why it says we have zero cop­ies.” Had that branded cur­riculum had been se­lec­ted but nev­er ordered? Or had the books had been ordered but in­ter­cep­ted some­where along the way?

I asked if we could go look in the book closet and Co­hen took me down the hall. On the way, we stopped to chat with a col­league of his who taught cal­cu­lus. “Do you have enough books?” Co­hen asked.

“I do now,” she said. “Some school in West Phil­adelphia closed, and I man­aged to get all the text­books from there. I had a friend who hooked me up.” But she wasn’t us­ing Fast Track to 5; she had a dif­fer­ent cal­cu­lus book that wasn’t on my re­cord sheet.

Urb­an teach­ers have a kind of un­der­ground eco­nomy, Co­hen ex­plained. Some teach­ers hustle and ne­go­ti­ate to get books and pa­per and desks for their stu­dents. They spend their spare time run­ning cam­paigns on fun­drais­ing sites like DonorsChoose.org, and they keep an eye out for any ma­ter­i­als they can nab from oth­er schools. Phil­adelphia teach­ers spend an av­er­age of $300 to $1,000 of their own money each year to sup­ple­ment their $100 an­nu­al budget for classroom sup­plies, ac­cord­ing to a Phil­adelphia Fed­er­a­tion of Teach­ers sur­vey.

Co­hen and I ar­rived at the math de­part­ment “book closet,” which was ac­tu­ally just a corner in­side the locked and empty of­fice of the math de­part­ment chair­per­son. “Here’s where we keep the ex­tra books,” he said, ges­tur­ing to two short wooden book­shelves. A me­di­um-sized box with open flaps sat on the floor. Co­hen looked in­side. “Well, we found the AP Cal­cu­lus books,” he said. The box was filled with brand-new cop­ies of Fast Track to 5.

It would have been easy to blame this glitch on the lack of a cent­ral­ized com­puter sys­tem. The only prob­lem was, such a com­puter sys­tem did ex­ist, and I was look­ing at a prin­tout from it. The prin­tout said Palumbo had zero cop­ies of the book, but 24 books were sit­ting in front of me in a box on the floor of a locked of­fice.


The Phil­adelphia schools don’t just have a text­book prob­lem. They have a data prob­lem — which is ac­tu­ally a people prob­lem. We tend to think of data as im­mut­able truth. But we for­get that data and data-col­lec­tion sys­tems are cre­ated by people. Flesh-and-blood hu­mans need to count the books in a school and enter the num­bers in­to a data­base. Usu­ally, these hu­mans are ad­min­is­trat­ive as­sist­ants or teach­er’s aides. But severe state fund­ing cuts over the past sev­er­al years have meant cut­backs in the school dis­trict’s ad­min­is­trat­ive staff. Even the best data-col­lec­tion sys­tem is use­less if there are no people avail­able to man­age it.

Mi­chael Masch, the vice pres­id­ent of fin­ance and chief fin­an­cial of­ficer at Man­hat­tan Col­lege and the former chief fin­an­cial of­ficer of the School Dis­trict of Phil­adelphia, told me that he used to routinely send his staff in­to schools to do book­keep­ing and oth­er tasks that over­worked prin­cipals couldn’t handle. “Prin­cipals wer­en’t good at man­aging cash ac­counts or stu­dent ac­counts. They needed sup­port in per­form­ing ad­min­is­trat­ive func­tions be­cause they were un­der­staffed,” said Masch. “If the prin­cip­al doesn’t meet with every par­ent, deal with every crisis, they get cri­ti­cized. If they don’t do the in­vis­ible stuff, like the pa­per­work, they’re not go­ing to read about it in the news­pa­per. So they triage.”

When it comes to the book scarcity, Phil­adelphia prin­cipals re­act in pre­dict­able ways. “They are very pos­sess­ive of their text­books,” Re­becca Dhondt, the par­ent of a second grader and a fourth grader at Jenks Ele­ment­ary School, told me. “My daugh­ter is not al­lowed to bring her text­book home be­cause they don’t want it to get lost.” For the past two years, she has sur­veyed teach­ers to find out what’s on their wish lists (mostly trade books and ba­sic school sup­plies) and then col­lec­ted dona­tions from the com­munity. “When I first did it last year, the prin­cip­al said, ‘Oh, we have some of that stuff,’” said Dhondt. As with the AP cal­cu­lus books at Palumbo, the miss­ing items were sit­ting some­where in the school but hadn’t made it in­to the right hands. “There’s not enough sup­port to con­nect the sup­plies in the sup­ply closets or the lib­rar­ies with the teach­ers in the classroom,” Dhondt said. “They need to have enough money to con­nect the dots.”

Keep­ing track of sup­plies is one prob­lem; keep­ing track of the stu­dents who will use them is a whole oth­er chal­lenge. In Phil­adelphia schools, many stu­dents are in foster care or nav­ig­at­ing oth­er pre­cari­ous liv­ing situ­ations, which means they fre­quently switch schools. A re­cent re­port by the Chil­dren’s Hos­pit­al of Phil­adelphia showed that one in five Phil­adelphia pub­lic high school stu­dents has been in­volved with the child wel­fare or ju­ven­ile-justice sys­tem. One teach­er told me that when she taught in a West Philly high school, she gained or lost a stu­dent at least every two weeks.

“There is a set of lo­gist­ic­al is­sues in a dis­trict this big that most dis­tricts in the U.S. don’t face,” ex­plained Donna Cooper, the ex­ec­ut­ive dir­ect­or of an or­gan­iz­a­tion called the Pub­lic Cit­izens for Chil­dren and Youth. “Everything isn’t what it ap­pears.”


After I fin­ished the first round of data ana­lys­is in 2013, I went to the school dis­trict and asked to present my find­ings to Phil­adelphia Su­per­in­tend­ent Wil­li­am Hite. The dis­trict spokes­per­son told me Hite wasn’t avail­able and in­stead offered me a meet­ing with Steph­en Spence, the deputy for the of­fice of school-sup­port ser­vices.

Spence, a former gym teach­er in his early 60s, was in charge of school open­ings and clos­ings. His job used to be handled by a whole staff, but ever since the cut­backs, Spence had been single­han­dedly tak­ing care of everything from desks to car­pets.

I asked him how he veri­fied that schools had enough books at the be­gin­ning of each year. He ex­plained that every prin­cip­al was sup­posed to sub­mit a school open­ing check­list and a school-clos­ing check­list. On that check­list (a Mi­crosoft Word doc­u­ment that he emailed to all the prin­cipals), there was a box the prin­cip­al could tick to in­dic­ate that the school had all of the books it needed to op­er­ate.

“In­vent­ory is not mi­cro­man­aged at a cent­ral- of­fice level,” said Spence. “A prin­cip­al that has very good skills with tech­no­logy might de­vel­op an in­vent­ory sys­tem that they keep on­line. An­oth­er prin­cip­al who is not so good with tech­no­logy might have just a per­son who counts the books, car­ries them from one loc­a­tion to an­oth­er, puts them in the closet, and visu­ally checks that they’re there.”

I wondered about this, since a dis­trict-wide elec­tron­ic sys­tem had been cre­ated sev­er­al years back. In 2009, a stu­dent stood up at meet­ing of Phil­adelphia’s School Re­form Com­mis­sion and pro­claimed, “I don’t have a book.” After that, Su­per­in­tend­ent Ar­lene Ack­er­man had re­solved to com­pu­ter­ize the Dis­trict’s in­vent­ory. Chief In­form­a­tion Of­ficer Melanie Har­ris had told me that the sys­tem had been de­veloped us­ing in­tern­al re­sources.

“You’re say­ing that the on­line sys­tem is no longer in use?” I asked Spence.

The prin­cipals pre­ferred to use their own sys­tems, he said, and re­port their in­vent­ory to him. “I rely on the prin­cipals and, I’m go­ing to say, real-time data. It gets tracked through the doc­u­ments we talked about pre­vi­ously: the school-open­ing and the school-clos­ing check­lists.”

As Spence re­ceives the prin­cipals’ check­lists, he enters the in­form­a­tion in­to an Ex­cel spread­sheet on his com­puter.

“Does this Ex­cel doc­u­ment get shared with any­one?” I asked.

“It gets shared with as­sist­ant su­per­in­tend­ents,” said Spence. “We have meet­ings. We put the Ex­cel spread­sheet on a pro­ject­or on a large screen dur­ing our school-open­ing meet­ings.”

As a data-sci­ence pro­fes­sion­al, it was clear to me that Spence was in over his head. Mil­lions of books, hun­dreds of thou­sands of desks — it is im­possible to keep track of all of these ob­jects without tech­no­logy and suf­fi­cient people to track them. It’s just as dif­fi­cult to fig­ure out how to use the data cor­rectly.

The end res­ult is that Phil­adelphia’s num­bers simply don’t add up. Con­sider the eighth grade at Tilden Middle School in South­w­est Phil­adelphia. Ac­cord­ing to dis­trict re­cords, Tilden uses a read­ing cur­riculum called Ele­ments of Lit­er­at­ure, pub­lished by Houghton Miff­lin. In 2012”“2013, Tilden had 117 stu­dents in its eighth grade, but it only had 42 of these eighth-grade read­ing text­books, ac­cord­ing to the (ad­mit­tedly flawed) dis­trict in­vent­ory sys­tem. Tilden’s eighth grade stu­dents largely failed the state stand­ard­ized test: Their av­er­age read­ing score was 29.4 per­cent, com­pared with 57.9 per­cent dis­trictwide.

One prob­lem is that no one is keep­ing track of what these stu­dents need and what they ac­tu­ally have. An­oth­er prob­lem is that there’s simply too little money in the edu­ca­tion budget. The Ele­ments of Lit­er­at­ure text­book costs $114.75. However, in 2012”“2013, Tilden (like every oth­er middle school in Phil­adelphia) was only al­loc­ated $30.30 per stu­dent to buy books — and that amount, which was barely a quarter the price of one text­book, was sup­posed to cov­er every sub­ject, not just one. My own cal­cu­la­tions show that the av­er­age Phil­adelphia school had only 27 per­cent of the books re­quired to teach its cur­riculum in 2012-2013, and it would have cost $68 mil­lion to pay for all the books schools need. Be­cause the school dis­trict doesn’t col­lect com­pre­hens­ive data on its text­book use, this cal­cu­la­tion could be an over­es­tim­ate — but more likely, it’s a sig­ni­fic­ant un­der­es­tim­ate.

At the end of the 2012”“2013 school year, the book budget was elim­in­ated al­to­geth­er. Last June, the state-run School Re­form Com­mis­sion — which re­placed Phil­adelphia’s school board in 2001 — passed a “dooms­day budget” that fell $300 mil­lion short of the dis­trict’s op­er­at­ing costs for the 2014 fisc­al year. (The gov­ernor of Pennsylvania had already cut al­most a bil­lion dol­lars from pub­lic edu­ca­tion fund­ing in 2011.) Phil­adelphia schools were al­lot­ted $0 per stu­dent for text­books. The 2015 budget like­wise fea­tures no fund­ing for books.

It may be many years un­til Phil­adelphia’s edu­ca­tion budget matches its cur­riculum re­quire­ments. In the mean­time, there are a few things the dis­trict — and oth­er flail­ing school dis­tricts in Amer­ica — can do. Stop giv­ing stand­ard­ized tests that are in­ex­tric­ably tied to spe­cif­ic sets of books. At the very least, stop us­ing test scores to eval­u­ate teach­er per­form­ance without provid­ing the items each teach­er needs to do his or her job. Most of all, avoid basing an en­tire edu­ca­tion sys­tem on ma­ter­i­als so costly that big, urb­an dis­tricts can’t af­ford to buy them. Un­til these things change, it will be im­possible to raise stand­ard­ized test scores — des­pite the best ef­forts of the teach­ers and stu­dents who will re­turn to school this fall and find no new books wait­ing for them.

What We're Following See More »
Trump Leads Tightly Packed Group Vying for Second
38 minutes ago

In one of the last surveys before New Hampshirites actually vote, a Monmouth poll has Donald Trump with a big edge on the Republican field. His 30% leads a cluster of rivals in the low-to-mid teens, including John Kasich (14%), Jeb Bush and Marco Rubio (13% each) and Ted Cruz (12%). On the Democratic side, Bernie Sanders leads Hillary Clinton 52%-42%.