Click. Connect. Learn.

All posts in Assessment

Cheating Is Not A Test Security Issue

Posted March 25th, 2012 by Bill

The Atlanta Journal-Constitution has done a review of cheating on tests in school districts across the country. The results aren't pretty.

A tainted and largely unpoliced universe of untrustworthy test results underlies bold changes in education policy, the findings show. The tougher teacher evaluations many states are rolling out, for instance, place more weight than ever on tests.

And, cue the folks who miss the point entirely:

Daria Hall, director of k-12 policy with the nonprofit The Education Trust, said education officials should take steps to ensure the validity of test results because of the critical role they play in policy and practice.

“If we are going to make important decisions based on test results — and we ought to be doing that — we have to make important decisions about how we are going to ensure their trustworthiness,” she said. “That means districts and states taking ownership of the test security issue in a way that they haven’t to date.”

Cheater Pen

No. This is not about test security (read: another unfunded mandate for schools to enforce a system of assessment that has never worked all that well, even before it was pushed front and center into education policy. Read: another "growth opportunity" for "educational services providers" who will supply a that guarantees your tests are completely, totally secure).

This is about designing assessment that can't be cheated, and about not tying pay - for both teachers and administrators - to performance on flawed, oversimplified assessments. Portfolios come to mind as an option that would reflect the experience of learners within their class, provide a clear and accurate representation of growth and learning.

However, the argument against portfolios would have us believe that they are just too expensive and time consuming.

But what's more expensive? Running a good portfolio system that works, or paying for tests that are imprecise measures of a small subset of what people actually learn.

What's more time consuming? Running a good portfolio system that works, or dedicating class time to teaching the test, taking the test, and trying to catch the cheaters after the fact.

To all the people who get a lot of attention for saying that our educational system is broken: please stop, and consider that our assessment system is broken, and is getting in the way of student learning.

You're not saving money or time when what you buy is broken. You're not assessing more efficiently when people can sidestep your efficiency measures. You're not measuring good performance when people cheat their way to the top.

Image Credit: "What the hell is a cheater pen anyway?" taken by DigitalCellulose, published under an Attribution Non-Commercial No Derivatives license.

What Nicholas Kristof Leaves Out: Discussing the Value of Teachers

Posted January 12th, 2012 by Bill

Nicholas Kristof has a piece in today's NY Times titled The Value of Teachers. In this piece he points to a recent comprehensive study that looks at the earning gains for students who have "good" teachers.

The money quote comes in the third paragraph:

That’s right: A great teacher is worth hundreds of thousands of dollars to each year’s students, just in the extra income they will earn.

Kristof buries the fact that the study is based on value-added methodology and conflates student performance on test scores with good teaching. He alludes to value-added in the 11th paragraph, but never actually addresses the fact that test scores and value added analysis aren't infallible.

Ms. Powaga - Fifth Grade, an Oasis in the Desert

The study authors (and this piece shouldn't detract from the worth and value of the study, which merits a read) are clear on this, even though Kristof is not. The executive summary (pdf download) of the study leads with a discussion of value added analysis:

Researchers have not reached a consensus about the accuracy and long-term impacts of VA because of data and methodological limitations.

The researchers conclude that, for their study, value-added analysis is a valid tool, as they look at over a million students from 4th grade to adulthood. As I said earlier, the study is a good read.

However, in his article on the study, Kristof uses false equivalencies:

Conversely, a very poor teacher has the same effect as a pupil missing 40 percent of the school year. We don’t allow that kind of truancy, so it’s not clear why we should put up with such poor teaching.

Truancy and the quality of a teacher are two very separate things. Conflating them here serves a rhetorical purpose (truancy = bad; bad teaching = truancy) but aside from being an interesting rhetorical gimmick, it just doesn't make sense.

The piece also commits one of the standard mistakes made in many pieces about teacher quality: it assumes that there is an objectively "good" teacher that will work for every kid in every class. The reality is (and people who have worked in school with kids can attest to this) different teachers connect with different kids. Sally's great teacher will be Jimmy's average teacher. We're dealing with human beings here, and human experiences differ.

However, the main (intentional?) oversight in the piece is the complete inattention to the elephant in the room in the school reform debate. If a kid comes from an upper middle class or higher in the socioeconomic ladder, they will attend one of the best schools in the world, in the United States Public School System. The "crisis" in public education is not present in high-rent zip codes. So, when we talk about the problems facing public education, let's situate them honestly. They are connected to issues of poverty, and issues of health, and in many cases, to problems surrounding food insecurity.

Kristof alludes to the importance of poverty, but then dismisses the importance of the issue as something that can be undone by good teachers:

we need an array of other antipoverty measures as well, especially early childhood programs. But the evidence is now overwhelming that even in a grim high-poverty school, some teachers have far more impact on their students than those in the classroom next door.

In the piece, Kristof declares that the problems facing education have an "obvious" solution:

The obvious policy solution is more pay for good teachers, more dismissals for weak teachers.

Wouldn't it be awesome if it was that simple? Unfortunately, the realist in me has a hard time believing that poverty and inequal access to quality education can be solved just by giving good teachers a raise. Until we start talking about education, poverty, and health together, as three related issues, the "obvious" solutions will obscure our vision of the hard challenges we need to overcome.

And part of that discussion needs to include what happens to education when good teachers are forced to work under the limits of bad policy.

Image Credit: "Ms. Powaga - Fifth Grade, an Oasis in the Desert" taken by Michael 1952, published under an Attribution license.

Social Learning and The Freedom To Change Your Mind

Posted December 15th, 2010 by Bill

Last week, Fred Bartels posted that, over the holiday break, he was going to start doing some brainstorming about an online progressive school. In response to some initial questions, Fred started to flesh out his vision.

I responded within the thread; this post is an attempt to extend and clarify some of those thoughts.

Cats in a Bowl

Part of the puzzle in defining and "building" an online school requires that we address issues related to reuse, redistribution (of both lessons and completed projects), and possibly recontextualization/remixing of lessons and materials created as part of the learning process.

Lessons, in this context, are really semi-structured exercises that can support a broad array of research-based, project-based experiences.

Assessment shifts from teachers determining what a student needs to know to a student articulating what they learned and considered valuable from the process.

The role of the teacher (and really, every other learner in the system) is to help people spot the gems that arise from their experiences.

Portfolio-based assessment is more readily suited toward documenting this type of experience than multiple choice tests, but whatever form the assessment takes, the assessment should highlight the learner's understanding of their experience as the starting point for determining what has been learned. Toward that end, assessment should include reflection back on how a learner has progressed, and part of schooling would need to include methods to support students as they identify where they have grown, and where they need work.

When it comes to developing a system/web site/web application to support this type of learning, there are many systems that already do this (and a bunch that don't - as a general rule, any system predicated on a hierarchy where the teacher controls a class-like space will be less than satisfying). Rather than getting too deep into the mechanics of designing another one, it might be more instructive to look at common elements/habits of mind that support this type of learning.

The communities, and their output, are endlessly iterative. They support a never ending stream of questions, responses, conversations, outside inputs, search, recontextualization of existing sources, original research, publishing, revision, and so on.

Learners can choose to dip into the stream and highlight what they consider important or valuable; over time these highlighted/curated/researched/freshly articulated/endlessly revised objects become what some people might call "finished." Personally, I think it is more accurate to call them snapshots, as we should all reserve the right to change our minds as we discover more.

But the key to any system like this is the underlying expectation that learning never ends, can always be revised, and should always be subject to new input from various sources. A system that supports this type of learning should simplify the discovery of these new sources of information, and the publication and revision of snapshots of learning in progress.

Image Credit: Photo "Baby Cats" taken by randomix, published under a Non-Commercial/No Derivatives/Attribution license.

Assessment

Posted August 15th, 2010 by Bill

One way of changing education is to change how we assess learning.

This isn't going to be a post about standards, but we need to start with them to get into the center of the discussion (this is not to say that standards are not a subject worthy of close consideration; rather, they are just not the main focus here, today).

  • Standards define curricular goals and objectives.
  • Textbook companies prepare packaged materials that are "aligned to the standards." These textbooks, in theory, are designed to address the curricular goals and objectives as defined by the standards (and for fun, ask a textbook rep to demonstrate how their texts "align to the standards." Ask them to define the process by which the texts are "aligned to standards." Then, get out the boots, and enjoy the hijinks that will ensue).
  • Student learning is measured by a standardized test that claims to assess a student's base of knowledge as measured against the standard.
  • The "quality" of a school is determined (in part or in whole) by how students have done on the test. Test results can be a key factor in closing down schools.
  • The "quality" of a teacher is determined (and in many of the merit pay schemes, teachers are rewarded or punished) based on student scores on these tests.

So, let's take an enormous, completely unjustifiable leap of faith and assume that the standards actually define something meaningful, for one reason and one reason only: this post is not about standards, it's about assessment.

When a curriculum is defined by a pre-packaged text, teachers and students are relegated to content consumers. Teachers get the text; they deliver the text; they test on the text, and teacher effectiveness is tied to how students perform on the test that purportedly measures how well students "know" the content that has been delivered to them. Any process used to "learn" the material is overshadowed by the means of assessment that defines the experience, and defines one's success or failure within that experience.

It's also worth noting that in lower performing schools, there is more motivation to stick with the "proven" or "traditional" route of using a standards-aligned text, as this provides a level of cover and plausible deniability should a school not meet growth goals. In an environment where sanctions accompany low test scores, using alternative means of working with kids is equated with gambling with kid's futures - unless, of course it's happening under the auspices of TFA, KIPP, or a charter school. Higher performing schools - where socioeconomic level appears to play a role - tend to have more freedom to experiment, largely because the threat of sanctions for "failure" is missing.

This is why serious discussions about assessment are a necessary part of the dialogue around improving education. What would an educational environment look like where, in addition to or instead of a standardized test, students had the opportunity to show their mastery via two portfolios: one defined by the school, and the second defined by the student?

The process of building a portfolio (ie, of crafting the assessment) is also a learning process. Selecting and justifying elements in a portfolio requires a level of critical, reflective thought that is not present in either preparing for or taking current standardized tests. It's a more efficient means of mastering both material and life skills than the assessments that currently claim to measure those skills.

What would teacher professional development look like if a teacher was assessed on how they provided feedback on student work? What if teachers developed professional portfolios that included curriculum they developed, modified, collaborated on, and/or shared? Most teachers create curriculum on a regular basis as workarounds for sections of the text that are weak or not suited for their classroom; what if creating and sharing these units was made an explicit requirement for growth and development as a teacher? What if this ongoing creativity and collaboration was a factor in assessing an educator's professional growth?

These shifts are possible now; they require a change in how we look at assessment, which potentially could inform changes in what and how we teach.

Changing assessment is hard. Generally, more individualized assessment takes more time. From a business place, it's hard to plan a "disruptive" business around this because you can't really streamline the time required for good feedback. The challenge (and therefore the opportunity here) is to make tools that simplify and streamline creating portfolios of work that demonstrate learning. The benefit - especially when compared to other forms of evaluation, and certainly to standardized testing - is that the process of creating and justifying the artifacts that demonstrate learning is also a process that supports and reinforces learning.

But this is a subtle point, and one that is often buried beneath the time required to assess portfolio-based projects versus the time required to process a standardized test. Ironically, the quest for efficiency in assessment has occurred at the expense of efficiency in learning.

Syndicate content