The Treasure Hunter

Most of today’s post is an op-ed written by Dr.Robert M. Wachter, a professor at the University of California, San Francisco, and published in the New York Times on January 16, 2016. The reason it has taken me so long to post it here is that Wachter described a better way to evaluate physicians, but stopped short of suggesting how teachers can be more fairly evaluated.  Although I had planned to fill in that missing part from the time I first read the piece, I haven’t gotten around to  it until now.

Two of our most vital industries, health care and education, have become increasingly subjected to metrics and measurements. Of course, we need to hold professionals accountable. But the focus on numbers has gone too far. We’re hitting the targets, but missing the point.

Through the 20th century, we adopted a hands-off approach, assuming that the pros knew best. Most experts believed that the ideal “products”— healthy patients and well-educated kids — were too strongly influenced by uncontrollable variables (the sickness of the patient, the intellectual capacity of the student) and were too complex to be judged by the measures we use for other industries.

By the early 2000s, as evidence mounted that both fields were producing mediocre outcomes at unsustainable costs, the pressure for measurement became irresistible. In health care, we saw hundreds of thousands of deaths from medical errors, poor coordination of care and backbreaking costs. In education, it became clear that our schools were lagging behind those in other countries.

So in came the consultants and out came the yardsticks. In health care, we applied metrics to outcomes and processes. Did the doctor document that she gave the patient a flu shot? That she counseled the patient about smoking? In education, of course, the preoccupation became student test scores.

All of this began innocently enough. But the measurement fad has spun out of control. There are so many different hospital ratings that more than 1,600 medical centers can now lay claim to being included on a “top 100,” “honor roll,” grade “A” or “best” hospitals list. Burnout rates for doctors top 50 percent, far higher than other professions. A 2013 study found that the electronic health record was a dominant culprit. Another 2013 study found that emergency room doctors clicked a mouse 4,000 times during a 10-hour shift. The computer systems have become the dark force behind quality measures.

Education is experiencing its own version of measurement fatigue. Educators complain that the focus on student test performance comes at the expense of learning. Art, music and physical education have withered, because, really, why bother if they’re not on the test?

At first, the pushback from doctors and teachers was dismissed as whining from entitled and entrenched guilds spoiled by generations of unfettered autonomy. It was natural, went the thinking, that these professionals would resist the scrutiny and discipline of performance assessment. Of course, this interpretation was partly right.

But the objections became harder to dismiss as evidence mounted that even superb and motivated professionals had come to believe that the boatloads of measures, and the incentives to “look good,” had led them to turn away from the essence of their work. In medicine, doctors no longer made eye contact with patients as they clicked away. In education, even parents who favored more testing around Common Core standards worried about the damaging influence of all the exams.

Even some of the measurement behemoths are now voicing second thoughts. Last fall, the Joint Commission, the major accreditor of American hospitals, announced that it was suspending its annual rating of hospitals. At the same time, alarmed by the amount of time that testing robbed from instruction, the Obama administration called for new limits on student testing. Last week, Andy Slavitt, Medicare’s acting administrator, announced the end of a program that tied Medicare payments to a long list of measures related to the use of electronic health records. “We have to get the hearts and minds of physicians back,” said Mr. Slavitt. “I think we’ve lost them.”

Thoughtful and limited assessment can be effective in motivating improvements and innovations, and in weeding out the rare but disproportionately destructive bad apples.

But in creating a measurement and accountability system, we need to tone down the fervor and think harder about the unanticipated consequences.

Measurement cannot go away, but it needs to be scaled back and allowed to mature.          We need more targeted measures, ones that have been vetted to ensure that they really matter. In medicine, for example, measuring the rates of certain hospital-acquired infections has led to a greater emphasis on prevention and has most likely saved lives.     On the other hand, measuring whether doctors documented that they provided discharge instructions to heart failure or asthma patients at the end of their hospital stay sounds good, but turns out to be an exercise in futile box-checking, and should be jettisoned.

We also need more research on quality measurement and comparing different patient populations. The only way to understand whether a high mortality rate, or dropout rate, represents poor performance is to adequately appreciate all of the factors that contribute to these outcomes — physical and mental, social and environmental — and adjust for them. It’s like adjusting for the degree of difficulty when judging an Olympic diver. We’re getting better at this, but we’re not good enough.

Most important, we need to fully appreciate the burden that measurement places on professionals, and minimize it. In health care, some of this will come through advances in natural language processing, which may ultimately allow us to assess the quality of care by having computers “read” the doctor’s note, obviating the need for all the box-checking. In both fields, simulation, video review and peer coaching hold promise.

Whatever we do, we have to ask our clinicians and teachers whether measurement is working, and truly listen when they tell us that it isn’t. Today, that is precisely what they’re saying.

Avedis Donabedian, a professor at the University of Michigan’s School of Public Health, was a towering figure in the field of quality measurement. He developed what is known as Donabedian’s triad, which states that quality can be measured by looking at outcomes (how the subjects fared), processes (what was done) and structures (how the work was organized). In 2000, shortly before he died, he was asked about his view of quality. What this hard-nosed scientist answered is shocking at first, then somehow seems obvious.

“The secret of quality is love,” he said.

Our businesslike efforts to measure and improve quality are now blocking the altruism, indeed the love, that motivates people to enter the helping professions. While we’re figuring out how to get better, we need to tread more lightly in assessing the work of the professionals who practice in our most human and sacred fields.

****************************************************

In education the “secret of quality” is also love.  When teachers not only care about their own performance but also about their students’ engagement with learning, the quality of education soars. Great teachers do not spend a lot of time delivering lectures, setting student goals, or handing out worksheets. Instead they open the doors to fascinating topics, invite their students to walk through those doors, and offer themselves as guides.  It doesn’t matter whether a topic is in English, history, science, or math as long as there is space within it for students to get an understanding of the whole and then find their greatest interest in a particular part of it.  Ultimately, students should work in small groups or with a partner to produce something of value: a chart, a demonstration, artwork, or a written essay to be shared with the rest of the class and perhaps others outside the classroom.

Although not all teachers– especially those new to the profession–will be able to carry out the processes and achieve the results described, the signs of progress will be apparent to any observer.  Certainly, principals are the ones best qualified and most available to observe classrooms at the most crucial points of student learning and over time. But parents should be encouraged to visit their children’s classrooms too.  Great teaching can’t be faked, and anyone –however inexperienced– will know it when they see it.

Today’s post is something I wrote long ago–1994 to be exact.  It came to my mind again because an old friend referred to it.  I have changed a few words to modernize the piece, and I want young readers to know that “The Catcher in the Rye” was a top best seller in its time and often taught in high school English classes.

When I first read J.D. Salinger’s novel, “The Catcher in the Rye“, the catcher metaphor struck me as silly, a clumsy device invented to justify a meaningless book title. Would any real teenager see himself — as the novel’s protagonist, Holden Caulfield, does–as a rescuer of children, and why were children in need of rescue anyway?

Although I am still cynical about Salinger’s creative motivation, I find the “catcher” image far more poignant and real in today’s world. In contrast to the world of Holden Caulfield’s rye field near the cliff, our world today has so many children who are physically, economically, socially, or psychologically in danger. Statistics don’t tell the story of many children’s tragic lives, but we as educators see the evidence day after day in their anger, apathy, self-destructiveness, and resistance to learning.

Because we are where children are, because they will drive us crazy if we do nothing, and because we care, teachers must be today’s catchers in the rye.

I have lost faith in any and all large-scale solutions to educational problems. They just put more paperwork, regulations, and job titles between children and the help they need. Where schools are failing, it is not because they don’t have enough programs and consultants, but because they have lost the human touch. Children mired in the morass of family and community decay can’t benefit from higher standards, instructional technology, or remedial programs; they need caring adults to pull them out of the muck and set them on solid ground–one at a time. Only then can each child, in his or her own way, begin the adventure of learning.

I have no magic formula for child catching. Each rescue must be worked out in personal terms that fit the catcher and the child. It probably doesn’t matter if the means are sophisticated or crude, gentle or tough, as long as at least one sensible adult is looking after the welfare of each child. I do believe, however, that there are some conditions that are essential for child-catching to work. The framework of operation must be small, physically close, and flexible. Forget any plan for recruiting 500 teachers as catchers, training them, and setting up a schedule for patrolling the rye. We need small schools or schools that are divided into small community units; classroom time, space, and organization that allow personal relationships to flourish; legitimacy for play and conversation in school; authority in the hands of front-line practitioners; and educational visions unclouded by political pressure to cover academic ground, raise test scores, or produce workers for industry.

Within such a framework, teachers are able to catch children who stray too close to the edge. They know each child as an individual and see most of the things that are happening to him or her. Kids hang around and tell them what they cannot see. Teachers also find time to talk to each other about classroom problems and to work with their classes to make changes in rules or processes without having to implement any special programs or bring in any outside consultants.

Although permanent rescue is a slow process and an imperfect one, catching often shows quick, dramatic results. I credit those results to what I call the “wart theory of education”. In essence, that theory asserts that children’s problems are like warts: If you can destroy just a few of them, the rest will get the message and go away. Children who are carrying intolerable burdens of poverty, family dysfunction, bad learning habits, and social ineptitude may shake them off in the space of a few weeks when a caring teacher takes time to talk through a single problem with them or tutor them in one small skill.

I have seen schools that do an impressive job of rescuing large numbers of children over time. Ironically, they are not the same schools that produce the highest test scores, send the most students on to college, or attract the attention of the media.  Mostly, such schools don’t even worry about whether the data on achievement and behavior makes them look good. Catching children is its own reward when you’re out there in the rye. 

One thing I have been advocating for a long time is learning for real life. In my opinion too much of what is taught in schools from kindergarten through college has little or no use in the world where we live full-time and have continuing needs to operate successfully.  Today’s post refers to two articles I have read recently.  The first one, “States Move to Issue High School Diplomas Retroactively” appeared in Education Week on January 26th, and the other, “The Wrong Way to Teach Math” was published in the New York Times on February 28th.

Apparently, the popular saying “Use it or lose it” has been around all over the world for centuries.  We know, and our ancestors knew, that knowledge and work skills not used regularly soon slip away from our grasp and are difficult to re-capture.  That’s why long before written language was available to the masses, it was common for ordinary people to chant or sing pieces that represented the basic principles and beliefs of their culture.

Although educators today accept the truth of the “Use it” saying as much as tennis players and linguists do, we have failed to honor it in our schools.  The grand assumption among educational policy makers is that a good education is the accumulation of a vast store of information and skills in order to cover all the possibilities of future lives. That’s why all students are required to study math and American history well into high school and most of them must also study a foreign language. The sad, but inevitable, result is that few of us remember much that was taught later on in our lives unless we need to use it regularly.

In an article I read recently the author, Mathew Andrew Hacker, argues that high school and college math courses for the majority of students should be more practical, teaching the kind of math we all need in our everyday lives. He says, “Ours has become a quantitative century, and we must master its language.  Decimals and ratios are now as crucial as nouns and verbs.”  To make his point he describes the elitism of the math courses he has seen being taught in high schools and community colleges today; they are still focusing on the math used only by mathematicians, architects, and scientists.

Hacker goes on to describe how he teaches math at Queens College in New York, which he recommends for the mass of students not aiming for technical careers.  When he gives a few examples of the problems he presents to his students, even I was able to solve them and appreciate their usefulness for ordinary citizens in their everyday lives. He ends his article by explaining why our schools should be teaching differently because traditional mathematics does not line up with real world needs.  For example, the “base 10” emphasized in all areas of mathematics does not fit with the way we measure time in seconds, minutes, hours and weeks, and thus makes it very difficult for us to figure out percentages of time, a common need in our lives.

I agree with Hacker completely. If mathematics were taught as an every-day skill through K-12–and in college for those not majoring in it–I believe that almost everyone would remember and use it effectively in their adult lives. And that belief drives me further to think about the content of other classes in the regular education sequence that does not stay with us very long or serve us well.  Those classes that stand out in my mind are American history and Science. I did well in both as a student but forgot almost everything soon afterward. I can’t remember the names or accomplishments of most of the American presidents before my time, what the French and Indian War was all about, or which elements the symbols in the Periodic Table stand for.  Can you?  Fortunately, I can still do everyday math–much of it in my head–but I can no longer solve any algebra or geometry problems in the ways I was taught. If such skills and knowledge are so quickly lost by most of us, is it not time to re-think and re-construct the K-12 curriculum? Make it real; make it relevant; make it memorable.

Today’s post, as promised, is another reference to research that shows the importance of  human feelings about themselves in any situation where people are being asked to do a particular task.  My argument, as it has been in my posts over the past week–and earlier in “Let Teachers Reinvent the Wheel”–is that you can’t formally “teach” students grit, perseverance, or resilience, but you can motivate them to develop those behaviors on their own by consistently treating them as important, powerful people.  

When the National Reading Panel began its work in 1998, one of its goals was to bring reading research up to the “Gold Standard” of medical research. Unfortunately, reaching this goal was –and still is–impossible because reading researchers, unlike medical researchers, cannot separate treatment effects from effects caused by other factors, such as classroom distractions, student hunger or illness, home culture, work habits, and personal feelings.

Medical researchers have the advantage of being able to remove or minimize outside factors by assembling very large groups of potential subjects and then forming randomly selected treatment and control groups. Just as important, they are able to use a “double blind” procedure in which neither the subjects nor the people applying the treatments know which doses are the experimental ones and which are placebos.

Rather than trying to explain how all the factors present in reading research may skew results, I have chosen to focus on the one I think is the most troublesome: the “Hawthorne Effect.” This term derives from a series of studies done in the Hawthorne Works Plant of Western Electric near Chicago from 1924 to 1932. There, researchers manipulated various physical conditions, such as lighting and the number of rest breaks, to see which ones most affected worker output. What they found was that almost any change increased production, and what they concluded was that being a part of a scientific experiment created feelings of importance and belonging that were a more powerful determinant of productivity than any change in conditions.

It is hard to believe that the outcomes of much reading research are not similarly affected. In many studies both the teachers and their students know when they are part of a research plan and whether they are in the treatment group or the control group. There is no way to “double-blind” such a study when the teachers are the people applying the treatment, only one group of students has new materials, and the type of instruction given to that group is different from what is going on in the rest of the school.

Under such conditions, the feelings of the experimental teachers and students are pretty certain to be positive. After all, being chosen to implement a scientific experiment is an honor. It implies that the researchers and the school’s administrators think you are intelligent, competent, and trustworthy enough to do it well. Add to this the special training for teachers and the frequent classroom visits of researchers monitoring implementation.

In contrast, neither the teachers nor the students in a control group get any extra attention, new materials, or special training. They may also feel dishonored by not being the chosen ones. Their feelings could be called a “Negative Hawthorne Effect”

Just as production increased in the Hawthorne studies, we can expect that the results will be better for the experimental group in the type of reading study just described, whether or not the new program or strategy is any better than the old ones. This explains why so many studies report positive results for the treatments applied, and it should make us cautious about accepting those conclusions.

But, hold on! There is a way to do reading research that eradicates the influence of the Hawthorne Effect: have two or more treatment groups, with each one trying out a different program and receiving the benefits of being a part of an experiment. Ideally, there should be a control group also, using the old program. Actially, a few research studies do follow this two-treatment format, but not enough of them. It is more expensive and more complicated to carry out, and most researchers—being human—believe ahead of time that one type of program is superior to all others and want to prove that when they design their studies.

The lesson for us as followers of education  research is to consider not only results but also how carefully those studies were carried out, especially how likely it was that the “Hawthorne Effect” came into play.

Today and on Sunday I will post essays I wrote a few years ago about research on human motivation that were originally published in Education Week.  I have chosen to post them again here because they are related to the pieces I posted on Sunday and Wednesday.  Today’s essay was originally titled, “Rediscovering the ‘Pygmalion Effect’ in American Schools.”

“High expectations” is the mantra of today’s school reformers, who are convinced that the trouble with public education is that students have been allowed to slide by with little effort. Their version of high expectations is requiring college-preparatory courses, advanced subject matter, more-difficult assignments, and a longer school day and year for all students. They believe that research and the records of selected schools show that demanding more of students brings the desired results.

But do they understand the research, or know what successful schools really do?

The original research on teacher expectations tells a far different story from what today’s reformers are calling for. More than 40 years ago, Robert Rosenthal and Lenore Jacobson conducted an experiment in a California elementary school that produced what they called “The Pygmalion Effect”, in a reference to a Greek myth and George Bernard Shaw’s famous play,“Pygmalion.” All three highlight the amazing transformation of an ordinary person into someone special. In their book Pygmalion in the Classroom, they described the study in detail and interpreted its lessons for education and other human interactions.

 The experiment consisted of giving false information to teachers about their students and then sitting back to see what happened. On the pretext of testing the reliability of a newly developed test to predict future student achievement, the researchers administered a traditional IQ test to all students at the beginning of the school year. Afterward, they reported to teachers, based supposedly on the tests, the names of students who were about to have a spurt in academic performance.

In reality, these students were a randomly selected percentage of the student body, and their scores showed nothing but their current IQs. At the end of the year, and again two years later, all students were retested, and the results showed that a significant number of the identified “spurters” had in fact made unusual intellectual and performance gains and maintained them over time. Teachers’ grades and written reports also recorded marked improvements in learning and behavior for most of those students.

Although the researchers did not examine what happened in classrooms that year, teachers’ written reports were clear about what did not happen: no extra time, no advanced curriculum, no individual tutoring, no differentiated instruction or assignments.

Rosenthal and Jacobson speculated that what teachers gave their spurters—but not their other students—were unmistakable signals of their faith in them: smiles, nods of approval, more opportunities to ask and answer questions, and a kindly tone of voice. Teachers’ expectations of student success, and their unconscious communication of those expectations, made all the difference.

In its time, this study, along with its replications in three other schools and similar behavioral studies, garnered widespread and authoritative attention. Although there was some criticism of methodology and score interpretation, critics did not contest the researchers’ conclusion that the expectations in teachers’ minds were the determining factor in the success of the identified children.

Now, 40 years later, the reality of the “Pygmalion effect” stands unrefuted by further research, while it is supported by considerable evidence from classrooms where poor and minority children have made great strides in their learning because their teachers believed they would. It is also supported by countless stories of successful people who were struggling in school and life until some adult—a teacher, a boss, a family friend—saw something special in them and encouraged them to make the most of it.

The discrepancy between the Pygmalion researchers’ concept of high expectations and that of today’s reformers stems from the multiple meanings of the word “expectation.” To the researchers, it meant the power of belief to influence the behavior of others. To the reformers, it means the power of authority to exact compliance from underlings.

As a lifelong educator, I am not so starry-eyed as to think that believing in students is all that teachers and schools have to do to enable them to succeed. Every school needs a strong curriculum, high-quality materials, well-planned instruction, extra-help options, and meaningful assessments. But all those components should be calibrated to the ages, interests, prior learning, and physical and emotional capacity of the students at hand, not to the illusions held by so many pundits, business leaders, and politicians.

Quality education and student self-esteem are not mutually exclusive. We can have both, but not until we understand the essential nature of human behavior and learning, recognizing that schools must appeal to and support the strengths of students, not play on their fears and weaknesses.

Schools are meant to be wellsprings of vigor, interest, exploration, growth, and illumination. Rigor, the word so often used by reformers to describe what schools should emphasize, is more properly the companion of harshness, inflexibility, and oppression. It is time to change the current conception of high expectations back to its original meaning.

P.S. If you wish to respond to this post or others with descriptions of your own experience that you’d like me to post, send your piece to <jyatvin@comcast.net.>