Measuring the value of culture

Justine Roberts, Principal

There is a lot of discussion about the role of evaluation in our work - as museums, what is it we really can evaluate?  What is our sweet spot as learning organizations - content knowledge, skills, attitudes?  Is it reasonable to attribute certain kinds of impacts (behavioral change eg) to museums?  What set of instruments should we use to capture impact?  Who should do evaluation and when?  How should the results shape our work? Which comes first, priorities or results?  And does an evaluation framework negatively impact the ways we develop public programs, or even staff professional development opportunities - does it make an exhibit or program more linear, more focused on content transfer, and less wonder-ful?  There are others, these just come to mind.


So it was interesting to come across a project from the Museums, Libraries and Archives Council (MLA) in the UK that attempts to put a comprehensive framework in place for thinking about the role of evaluation for self-choice learning social-service organizations. They start with a definition of learning and then systematically define outcomes for both visitors of MLAs (what they call Generic Learning Outcomes) and for organizations themselves (which they call Generic Social Outcomes).  Using that as a foundation, this framework then shows how to create evaluation tools that help measure impact.  

So it links content to metrics.  It is an actionable framework, and because it measures success, it is also designed to tell a coherent, fundable, story about the work of a public service organization. 

This tight relationship between what it means to “learn”, the types of learning expected from the audience and the wider impact of the organization’s work, and the evaluation tool designed to measure learning is very strategic.  

This is a systematic framework and calls for a kind of on-going self-evaluation policy or attitude within an organization.  Individual programs, initiatives, exhibits, or workshops are evaluated, but there are no one-off, stand-alone evaluations.  Every time something is measured (professional development, summer camps, a new stage production, etc) it uses the same goals. There is no need to reinvent WHAT to evaluate for each event, and evaluation results are automatically consistent with the big, visionary impacts of the MLA (Whether they show success or not).


This project steps out of the stream of day-to-day work and program delivery and thinks about evaluation as part of the overall trajectory of an organization.  That makes it an interesting way to think about the importance of evaluation and how it functions.


Inspiring Learning

The MLA describes “Inspiring Learning” as “a self-help improvement framework” and encourages MLAs to use it to:       
       *Assess your strengths and plan improvements
       *Provide evidence of the impact of your activities through the generic learning and generic social outcomes
       *Improve your strategic and operational performance

What is in it?

As the saying goes - you get what you measure.  So it matters what framework you use and you have to start with a clear agreement on key terms.  MLA does this with a definition of learning as active engagement with experience: 

They expand that as follows:

•            It is what people do when they want to make sense of the world
•            It may involve the development or deepening of skills, knowledge, understanding, values, ideas and feelings
•            Effective learning leads to change, development and the desire to learn more

So they are starting by positioning the work of MLAs as something of a fusion of learning frameworks to include cognition, social/emotional learning, skills development, and personal dimensions.

The MLA believes organizations should model the kinds of learning that they hope to foster.  Because of that, self-assessment is the first step in the bigger strategic process of identifying areas where MLAs can have an impact. The Council has created a kind of SWOT worksheet using their scale of impacts get you started.

From that point, the framework turns outward and looks at how to evaluate and measure impacts on individuals (GLO), and communities (GSO).

Generic Learning Outcomes

I want to talk about what the GLOs are because I think this way of looking at the opportunities for learning in MLAs is really nice.  MLA has identified 5 ways that audiences grow and develop through their relationships with organizations:

1.      I think the yellow circle is in some ways the most interesting because it is calling for lasting behavioral change as an outcome of a visit to a library, museum, gallery, or archive.  The difficulty of measuring this is one issue, and I am not sure that Inspiring Learning has solved that question.  But it is in many ways the ideal of many museums working on exhibits and programs related to climate change, wellness, and human rights.  So it is something I think the field is very interested in figuring out.  With that exception, I think the GLOs are familiar and part of how we think about museum outcomes here in the States.  So this diagram offers a handy tool that is easy to use - and when you go to their website each of the circles is clickable and pops up a more detailed description.

Generic Social Outcomes are intended to help measure the benefits, or social value, of these types of organizations.

So if that is the framework then you can use it to create research tools, questionnaires, focus group guides and other materials that support evaluation of programs and offerings against the GLO.

This is fundamentally political

In a way, Inspiring Learning is quietly pushing MLAs to recognize their potential contributions to their local and professional community, and also to policy discussions that impact their work in arts education, cultural funding, out of school time learning, healthy communities and more.

To that end, the MLA council has taken care to ensure Inspiring Learning aligns with other existing UK policies and initiatives.  Those include:

• Every Child Matters - a chart setting out everything from nutrition to educational attainment to public play space – quality of life, basic health and wellbeing, opportunity, and 2020 Children’s Plan rolled together.
• UK primary school curriculum framework
• The Department for Innovation, Universities and Skills (DIUS) - which is working on a vision of informal learning for the 21st century and ties the GLOs to adult informal learning.
• The RARPA framework – a National Institute for Adult Continuing Education (NIACE) framework that tracks progress for adult learners.

So is the framework helpful or just another hurdle?

The Museums, Libraries and Archives Council wants to make their membership stronger, more productive, and more sustainable, and to improve their social standing.  They are thinking about how to demonstrate value by creating ways to measure impacts. Fair enough but why an evaluation framework in particular?

Inspiring Learning was developed specifically as a response to the 2007 Local Government and Public Involvement in Health Act which shifted assessment from a service–based model to an outcome-based model. As MLA explains it, this shift meant no longer assessing the quality of programs but instead looking at how to demonstrate value by showing improved quality of life for individuals and communities.  That left the question: how do you identify the contributions of MLAs? 

Clearly MLA is operating in a very different environment than we have here in the States.  Museums here certainly respond to funder priorities and opportunities, and some of those can shape the experience of many museums across the country as they position themselves to be more competative.  But we don’t have the same pressure (safety net?!) of national funding and we don’t have national mandates or policies like this one in the UK that push a sector-wide response.

Even so, The Wallace Foundation, which studies Out of School Time (OST) organizations has come to the similar conclusion that we need to better “link and align learning opportunities within a community” if we hope to create high performing/high impact non profits. Although working in a different country, and coming at the question from a different point of view, they have arrived at a surprisingly similar formulation of the issue:

“Being results-oriented means synching the nonprofit’s mission and vision with its programmatic activities and then setting goals that best reflect the intended outcomes of the activities.  In a mission- driven/results-oriented nonprofit, operations decisions are cross-walked with the mission to ensure that the services provided are aligned with the overall strategy.”

The Wallace Foundation clearly sees program evaluation as part of a larger process of learning and strategic thinking that will drive organizations to be more effective.  However, they do diverge from the MLA framework in some important respects.  They are not in favor of holding OSTs accountable for community-wide measures of success, and they do not try to map out the criteria that individual organizations would use to evaluate learning outcomes.

At the end of the day 

I think there is a lot to take away from Inspiring Learning.  For one thing, its comprehensive and consistent focus on strategy.  The tag line is “placing learning at the heart of the museum.”  But it applies that to policy as well as to people, physical space, and partners.  It is a chance to align an organization top to bottom along a consistent strategic axis.  It recognizes that exhibits succeed not just when they are coherent with the style of the museum, or when they are able to meet their fundraising targets, but when they improve the museum’s capacity, make it more competitive, raise awareness and profile, and contribute to the social capital of their communities.

For another, it asks some important high level questions and encourages social service organizations to be explicit about them.  These include: what is our public value? How do we demonstrate our contributions? What are our priorities as a cultural organization? What is our own culture as a learning organization? It can seem like an organization is struggling just to deliver service and keep the doors open these days.  But it is also important to be able to say what difference you make in the world, and why you matter.  That seems to me one of the messages of Inspiring Learning.  

Although it takes seriously the idea that MLAs have to prove, demonstrate and perhaps even justify their work – and that can be very uncomfortable – the demand for evidence of need, outcome and performance data, and improved efficiency and effectiveness are all growing steeply here in the States as well as in the UK.  If this is our reality, a framework like GLO seems very handy.

Engagement at the Urban Waterfront

 By Maria Mortati

Recently, I was invited to participate on a panel in SF about creating engaging public spaces at the urban waterfront.

Yes, I know, the waterfront is already a draw– it's a giant confrontation between ourselves and the great unknown. In urban areas, however, it's pretty challenging to simply get to the water (on the east side of San Francisco that is particularly true), let alone engage with it.

Planning & Design

Dan Hodapp, Senior Waterfront Planner at the Port of San Francisco helped the audience understand how the Port looks at the watefront on a larger scale: the diversity of usages it needs to support, and the ways the city needs to meet it. This includes how far people must walk amongst access points, industry, and entertainment. 

Challenging  Design Environment

I talked about 'Impermanence as a Design Strategy' for creating engaging exhibits at the water's edge. It's hard to do permanent work there: environmental impact on maintenance, urban impact on exhibits, politics of civic art & design. 

I based this on case studies of art;  Ledia Carroll's project through Southern Exposure on tracing the edge of the built-over Mission Lake in SF, informal shared research; the Hidden Ecologies Project, and my work designing and building prototypes along the waterfront in SF for the Outdoor Exploratorium. What these things have in common is that they were/are fairly achievable on a variety of scales, and yet had a lot of impact.

Large Scale Public Involvement
Kate Bickert, Director of Trails Forever for the Golden Gate National Parks Conservancy talked about the involvement of the community in the redevelopment of Crissy Field. That was and is a fantastic, long-term model of civic engagement in a local resource. 

She also showed discussed some cools projects such as the Learning Barge, being developed a the UVA School of Architecture: 

"The Learning Barge will address this deficit as it traverses the highly polluted and urbanized Elizabeth River. The educational spaces and interactive exhibits will provide meaningful, hands-on education... The Learning Barge will serve residents of the Elizabeth River watershed and Hampton Roads population of 1.6 million. Over 19,000 formal and informal learners are anticipated each year, including K-12 students and teachers, diverse community groups, families, children, adults and elders of all backgrounds." -Learning Barge Web Site

BTW, there's also the Science Barge in Yonkers, NY. It's a floating sustainable demonstration garden that is a great draw for the waterfront and for informal science education.

Public Art at the Waterfront

Ned Kahn (do I have to tell you who he is?) shared his fascination with (and I should say mastery of) intersections between the built world and the nature of fluid dynamics. 

It was beautiful.

Image: Lars Ploughman

Crowdsourcing & Exhibit Development

Justine Roberts, Principal

View of MOS second floor exhibit hall.

Wednesday next week (Oct. 27 2010) The Museum of Science Challenge being run by InnoCentive will officially close.  I am talking about an experiment in crowdsourcing exhibit ideas that MOS intends to include in a future blockbuster traveling exhibit about STEM.  

Paul Fontaine, VP of Education at MOS, is running the project and said the goal for MOS was to experiment with more collaborative methods of gathering ideas, prototyping exhibits, and really open the doors to the creative work of the museum:

"Museums in general are reinventing themselves in lots of ways – its a Cambrian explosion.  There are new methods of using technology, more input from community partners, venues for all types of experiences, and new methods of creative input is on that continuum of experimentation."

I agree with Paul.  I am following along with the Liberty Science Centers' Exhibit Chefs project - one of the many who have signed up to stay in the loop although I haven't really contributed to the effort.  And of course there was the Tech Museum's experiment in opening exhibit development up through second life.  And I think Science Buzz at SMM has proven its success in identifying ideas its audience is interested in, and using the web as a testing ground before incorporating topics into exhibits. In the children's museum community, The Building for Kids in Appleton WI is promoting itself as "designed by kids for kids". 

So many museums are thinking about this question and testing the waters.  It will be fascinating to see the profile of the person who wins the challenge.  But even more interesting to see if this is just a one-off or whether there are long-term implications within MOS for broad collaboration with the public.

Who is InnoCentive?

InnoCentive is a crowdsourcing company focused on solving problems.  They see crowdsourcing as an opportunity to bring new insight and fresh perspective to bear on a tough issue that professionals have been wrestling with for many years. This is often highly technical – finding new stem cell lines for instance. 

InnoCentive can be the public face of the challenge and their clients can remain anonymous. In many cases the people who win the challenge aren’t in the field that the seekers (companies sponsoring the challenge) are in.  Instead they work in an adjacent field and arrive at their proposals by taking technologies they use and adapting them to a new discipline.  As Kathryn Plazak, Manager of the Public Good Campaign at InnoCentive says: “That’s where leaps of innovation happen and breakthroughs happen.”

About the Challenge

InnoCentive approached MOS as part of an effort to build their non profit and public service portfolio.  When they heard about the opportunity to run a Challenge, the team at MOS immediately knew what they wanted to focus on.  For years at ASTC, internally, and in focus group feedback, the museum had been discussing how to solicit original ideas for large scale exhibits. 

So the MOS/InnoCentive Challenge is an opportunity to get lots of creative ideas, fresh thinking and new perspectives.  Although they won't see the results for another few days Paul is already excited about the outcome.  

"Like opening a present when you don’t know what's inside." 

Paul also said part of the fun of this project has also been in learning something new.  

"We encourage people to experiment and learn and have open-ended experiences and museums can too!" he said.  

Does it Work?

InnoCentive has worked with government and mission driven organizations before. Recently they worked with the NESTA Foundation (National Endowment for Science, Technology and the Arts, UK) to explore ways of increasing parental involvement in children’s education.  NESTA was particularly interested in urban disadvantaged communities.  This is an issue that the professionals closest to the issue, who work with the target populations and key institutions on a daily basis, have struggled to make the tools and strategies they were using effective. 

The InnoCentive partnership allowed NESTA to cast a wider net and bring new voices into the conversation.  Most than just a survey of attitudes and ideas, the format of the challenge acknowledged the untapped expertise of the educators, parents, and other community members to propose actionable solutions.  This just wrapped up so its not clear yet how any of the new ideas will play out.  But the challenge succeeded in raising awareness of the issue, stimulated a broad conversation, brought new partners and stakeholders to the table, and netted some new interesting proposals. 

For MOS it will be a big step to translate winning ideas into exhibits.  They see the submissions as the gem around which they will have to construct a visitor experience.  And that process can take years - it needs to be put into an educational framework, there is fundraising and prototyping to do, and then fabrication.  Plus there are other projects already in progress in the pipeline. But Paul expects the team to be inspired by what comes out of this process.  He is not expecting submissions to be fully formed exhibit interactives but does expect ideas for incredible vehicles for some aspect of STEM, an innovative math exhibit, or original science concepts.

Do you need a partner like InnoCentive to use crowdsourcing?

When I heard about this project I wondered how hard crowdsourcing really is, and what a company like InnoCentive contributes - why not just train your own staff and build this capacity as part of a general shift toward greater dialogue with visitors and more participatory processes?

InnoCentive clearly has some advantages - they have a base of 200,000 registered “solvers” who regularly check their site to see what’s come up.  In addition for each challenge they produce a marketing campaign including press releases, blogs and targeted outreach to particular communities that should be aware of any given challenge.  They frame the challenge to make sure its actionable and use their website to post and receive submissions.  They also handle legal agreements for intellectual property. Once a challenge is posted their staff answer questions, screen out inappropriate submissions, and package the ones that do meet requirements and send them to the seeker.  The seeker does the analysis, evaluates and picks the winner. InnoCentive will vett to make sure solvers are who they say they are.

For many organizations it does make sense to use a service like this.  As a one-off it’s a lot of infrastructure to put in place.  But if an organization is embedding crowdsourcing into its culture this kind of challenge is one way to get started down a new road.


Thanks to Paul Fontaine and Kathryn Plazak for talking to me about this fabulous project.

On the Edge of our Seats regarding The Arts and the Economy in the United Kingdom

 Janet Petitpas, Senior Associate, London 

On October 20th, the government in the UK will be announcing its comprehensive spending review and it is anticipated that government spending cuts will be savage. Like most countries in our current economy, these cuts are necessary to make up for a deep structural deficit that was accumulated during the years leading up to the crash.

The cuts are expected to hit the cultural sector very hard, although the arts are not alone. Cuts at the levels projected are along the lines of no spending on roads and the closing of the majority of the courts – in other words, unprecedented reductions that far exceed even those of the Thatcher administration. Anticipated cuts include 25% reduction in expenses for Home office, Climate and Energy Change, Community and Local Government, and 25% reductions in Culture, Media and Sports. Both Education and Defence expect a 10-20% cut and cuts to Health are likely to include 40% cuts to social care.

The current estimate for public sector job losses to come is 600,000 in addition to the 750,000 private sector workers who have already lost their jobs. Already, families are seeing reductions to their childcare benefits. Bleak all around, but back to the arts.

The arts community in the UK has gotten together to campaign against cuts that are seen as so extreme that they would sabotage the country’s status as a global leader in arts and culture. More than 100 of the UK’s leading artists have formed Save the Arts and have sent an open letter to the culture secretary Jeremy Hunt warning that cuts to arts and museum funding could obliterate decades of work that have gone into establishing a rich British cultural environment. The letter emphasized that art:

“enriches the lives of millions of Britons and attracts millions more visitors from other countries. It does all this at a cost that is no more than a tiny fraction of the national budget.” 

Included in the Save the Arts website is an animation created by artist David Shrigley illustrating the effects of such drastic cuts. Most at risk will be smaller regional museums and art galleries, which provide inspiration to aspiring artists. Artist Anish Kapoor told the Guardian newspaper: 


“the arts bring in a huge revenue to the UK and they will be vital to any recovery. It hasn’t got the emotive impact of cuts to, say, hospitals or schools, but it is just as important. The arts are a central way in which we gain a sense of community, of something collective in our society; it gives us our sense of place.” 

Arts and Business statistics show that private investment in culture has decreased 4.5% from 2008-2009 and will likely continue to decrease as the businesses and individuals have fewer funds to spread. This means that private sources of income will not be able to be raised to make up for public funding shortfalls.

So we wait for the shoe to drop. Already there are many repercussions from the current economy including:

       • The Museums, Libraries and Archives Council (MLA) has been abolished and there is hope that some of the programs they ran will be picked up by Arts Council England (ACE). 

       • The National Trusts is undergoing a re-structuring to result in a savings of £10 million from a range of sources over the next three years. There will likely be approximately 60 positions eliminated as part of the spending reductions. 

       • The Arts Council announced £19 million in cuts on October 4, 2010. While this is a large number, arts organizations were braced for bigger impacts. Of the £19 million in cuts, only £1.8 million or .5% of funding will be lost for all 880 organizations currently funded. 

      • The Natural History Museum in London is preparing for public spending cuts by planning for £2.3 million reductions in their expense base. Up to 40 positions may be eliminated and there are fears that an entire research unit will be shut. 

      • Eureka! The National Children’s Museum in Halifax has dropped plans to open a London branch. The decision follows a feasibility study that concluded that funds would be extremely difficult to secure in the current economy. 

While most economists agree that cuts are necessary, some argue that the depth of the cuts make it harder for the economy to begin growing its way out of the recession. At the June meeting of the G20, a general commitment was made to halve all deficits by major industrial nations by 2013. This means that each of these 20 governments is simultaneously trying to tighten their spending and with everyone cutting costs, it isn’t clear who will be left to stimulate the economy.

We’ll hold our breath until October 20th but it is clear that spending cuts are coming and we are all hoping that the arts in the United Kingdom are able to weather the storm.

Sources:
Museums Journal, October 2010, Museums Association, London.
Turner Prize Winners Lead Cuts Protest in Open Letter, Peter Walker, The Guardian, Saturday, October 2, 2010
The Knives are Out, John Lanchester, The Guardian Weekend, October 2, 2010. 
Millions Lose Out in Children Benefit Cut, Isabel Oakeshott, The Sunday Times, October 3, 2010. http://www.fundraising.co.uk/news/2010/09/21/what-25-cut-arts-funding-looks http://www.artscouncil.org.uk/news/arts-council-england-implements-cuts-201011-budget/ http://www.guardian.co.uk/books/2010/jul/26/jeremy-hunt-to-close-mla http://www.guardian.co.uk/news/datablog/2010/jun/18/arts-council-cuts-list-spreadsheet http://www.guardian.co.uk/culture/2010/jun/13/natural-history-museum-cut-jobs

The Test of Time

Ron Davis, Principal

It was with both excitement and quite honestly some trepidation that I attended the ASTC 2010 conference hosted by the Bishop Museum in Honolulu, Hawaii. My trepidation stemmed from the fact that Gyroscope had worked with the Bishop Museum as consultants to help lead the creation of the Bishop’s Science Adventure Center from 2001 until the opening in late 2005, from strategic planning, all design phases from concept to final design documents, through installation and opening day. I was the Lead Designer for the project –nearly five years of inspiration, perspiration and a lot of collaboration. 

Now, almost five years had passed and our peers in the industry were coming to have a look.

It isn't enough to have a great exhibit on opening day - it's the test of time that counts. A great exhibit has to live, breathe, and grow, as this one has. The museum has added several new elements and made improvements to others, while keeping the core experiences–like the erupting volcano, lava-melting furnace and deep water ROV tank- in top condition.

It is with great pride in the original vision for this project, and in our relationship with the Bishop Museum staff, that I can say that it looks fantastic! The Science Adventure Center was sparkling for the audience of museum professionals who came by this past weekend and it is sparkling for all the visitors who will visit the Science Adventure Center in the weeks and years to come.

Congratulations, Bishop Museum!

Here are some images from the conference.  More can be found on our website including some from the opening!

View from the entry ramp. You can see the mezzanine and the large volcano farther back.

This is taken past the front desk.  On the wall the panels showcase the materials that have shaped the island including types of stone and the wind itself.

Ripple Islands interactive.

From the balcony visitors control the volcanic eruptions.

Inside the volcano the exhibit introduces visitors to how they work. These cool snails were added by staff after opening.

Looking down into the Ripple tank.  Visitors can drive ROVs in the deep tank. There is a passage below which is washed with the play of light and shadow from the moving water in the skylight.