Tuesday, November 17, 2009

You Get What You Measure

Two watch words often thrown about regarding the future of museums are “transparency” and “accountability.” The former is getting easier and easier to achieve—what excuse is there, really, for not having your museum’s audited financial statements, institutional plan and key policies available on the web? Heck, you can put the institutional plan up as a wiki document while you are writing it, and call for comment. You can even put your budget up as you are writing it, ditto.

“Accountability” is thornier because it entails some common understanding of “accountable for what and to whom.” On the face of it, it seems quite reasonable to demand accountability from nonprofits. If a museum is asking for public support, it should be able to report that it is doing some public good.

The problem is that the very act of measuring and reporting on a person or an organization changes their behavior. You get what you measure, and this can have unintended consequences. Just look at the No Child Left Behind act, which many feel have gutted art education, sports, music and many other “extras,” not to mention diverting attention from fostering reasoning or creativity, in order to “teach to the test.” Even in schools that don’t game the system outright by lowering standards, or trying to transfer underperforming students to someone’s classroom or school, or providing students with test results (i.e., cheating) this has a stifling effect on a well-rounded liberal education.


So what are some of the measures proposed for museum accountability, and what effects might they have on museum behavior?


• The classic measure leading the annual report is attendance. This is not as easy as it sounds to measure, but gives some intuitive indication of how many people are exposed (and potentially benefit from) the museum’s content. On the other hand, as a measure of success this has being accused of fueling growth for growth's sake and “pandering to the lowest common denominator.” Indeed, some museum people dream of limiting museum attendance to provide the kind of private experience usually only enjoyed by staff. (Or at poorly attended museums.) I seem to remember that when Emily Rauh Pulitzer, the founder of The Pulitzer Foundation for the Arts was planning her museum in St. Louis, she dreamt of limiting visitation to 50 people per day.


This article suggests we give weight to dwell time. Traditionally measured by exhibit evaluators or docents through laborious observation, it is now technologically possible to track a visitor by a GPS signal or RFID tag using their phone or other mobile device. The classic citation is that the average museum visitor spends 3 seconds in front of a work of art (does anyone even know where that statistic comes from?) Is longer always better? Might it encourage long label copy, requiring people to puzzle through reams of wall text? And how do you weight intensity? Being blown away by Michelangelo’s David for 10 minutes may form a life-long memory, while playing with a museum interactive for an hour may be quickly forgotten.


• Some museums brag on the number of exhibits (temporary and permanent) mounted in a give year. This may be a measure of staff creativity (and energy) but does it necessarily benefit the audience? Staff often presumption that changing exhibits fuel repeat visitation, but I know from many, many years of reading MAP and Accreditation Program self-studies and reports that this is not always true. And if most visitors only comes once or twice a year (no matter what the exhibit schedule) how do they benefit from the resources used to create a more ambitious schedule? Could valuing number of exhibits in and of itself burn resources without leading to more public benefit, while leading to staff exhaustion and burnout? (Not to mention tapped-out sponsors.)


• Another popular measure for the annual report is collections acquisitions. As you may know, I have long argued with respect to collections plans that more is not always better. Collecting more stuff may gratify curators and donors without necessarily benefiting the public commensurate with the resources (space, supplies, time) eaten up by new acquisitions. But using acquisitions as a measure of success inherently values more stuff over more nuanced values of the good derived from collections.


• Many people feel that the gold standard for accountability is measuring outcomes in some way, shape or form. Surely this is the ultimate solution to answering the question “how is the world different because your museum exists?” IMLS promoted this approach to measuring success of museum projects when it adopted Outcomes Based Evaluation (OBE) about a decade ago. This approach, since quietly abandoned, required museums to track changes in knowledge, attitudes, behaviors or life conditions of the audience. While it sounds great at first, such measurement quickly becomes both challenging and problematic. Challenging because many of the potential outcomes would only emerge in the long term, and are hard to track and measure. Problematic because it values concrete measurables and (by implication) devalues experiences that fuel open-ended imagination and inspiration.


• Here’s one that I haven’t seen actually used, but is certainly reasonable given the current economic climate. With museums (and other cultural amenities) more and more often being touted as economic drivers, isn’t it reasonable to track the actual effect? Financial Return on Investment would report on much revenue is being generated for the community for the money invested in the museum. Put aside for the moment how tricky this is to track, and how easy to fudge. If this becomes a major measure of accountability, won’t museums be driven to value visitation by non-local audiences over their own communities? (And wouldn’t museums then be vulnerable to public support being diverted from museums to other organizations that can show they bring a larger economic ROI?)


In a future shaped by an expectation of greater accountability, it behooves museum practitioners to choose measures that we feel are appropriate and can be implemented without disproportionate investment of resources. So weigh in—if your museum had two choose three measures to report on each year, in order to be accountable to the public, what would they be?

4 comments:

Center for the Future of Museums said...

Also see this blog post "Weighing Outputs: Measuring Social Impact in Museums and Nonprofits" by Colleen Dilenschneider http://tinyurl.com/ylrnaag

Kate said...

As non-profits we are not just asking the public for support we are actively receiving it in the form of tax money. The museum belongs, in trust, to the public and as such they are answerable to their community. Attendance can act as a tool for measuring the success of not only the program and exhibits, but also the marketing. Museums should not pander to the lowest common denominator but look to accommodate multiple generations. A dynamic exhibit that offers something for each generation should be the goal of every exhibit. If we start to claim indignation over “pandering to the lowest common denominator” then we start to use academia as an excuse rather than an impetus to look outside the box in finding a way to appeal to our communities.

Geoff said...

I think that most of the criteria listed in this post are important to consider in determining the effectiveness of your museum. Things like dwell time can be a useful way to gauge the depth of a visitor's experience and attendance numbers can show how many people are being reached by the museum. Externally speaking, return on investment gives the community who supports us an incentive to keep on supporting. I do not argue the importance of these numbers and figures; what I would like to argue against is the centrality of numbers to our role in the community. Things like this should be used to determine what a museum can improve, but it is a far cry from determining an institution's worth. When considering the value of a museum, how can numbers like attendance even be considered? The question we should ask is not how high is attendance. Rather we should be asking, are we about some thing? Or are we for some one?

cjn212 said...

My thesis "Administrative issues of Visitor Studies in US Art Museums" covered a lot of the basic history of VS & proposed some common-sense fixes to common problems (confirmed by my original study of 145 museums / 40.73% of sample). Anyone thinking about this issue is welcome to read it: http://bit.ly/c4Tg7

Hope it helps!