Interpreting the future

This post is a chapter excerpt by Art Processors co-founder and Creative Director Tony Holzner from the forthcoming Routledge International Handbook of New Digital Practices in Galleries, Libraries, Archives, Museums and Heritage Sites (eds H. Lewi; W Smith; S Cooke; D vom Lehn). The new publication presents a fascinating picture of the ways in which today’s cultural institutions are undergoing a transformation through innovative applications of digital technology. More information on the publication can be found here.

As cultural institutions continue to expand beyond a historical focus and ‘look into the future’ (Antonelli, 2014), what are the opportunities and how can we best capitalise on these to increase public engagement? This chapter focuses on technological change in relation to new digital interfaces and their role in enhancing visitor engagement in the GLAM sector. 

The approach adopted for the digital interpretation of exhibitions at the Museum of Old and New Art (Mona) in Hobart, Tasmania, Australia is used as a reference point for exploring how more natural interactions between people, place and content can be created. Consideration will be given to new human–computer interfaces—resultant from mixed reality (MR) technology—that promise to deliver increased access to collections and superior visitor experiences. 

When the subject of technological innovation is raised, it is often coupled with talk about ‘disruptive innovations’. Much of this is, admittedly, hype and coopting of the latest new term to sell products that are decidedly not disruptive. For an innovation to be disruptive, it must substantially disrupt an existing paradigm and displace the market leaders. It is interesting to explore the application of disruptive innovation to the GLAM sector. While directly associating GLAM organisations with private enterprise notions relating to growth and disruption can be fraught, I would argue that GLAM organisations fundamentally have a purpose of value creation just like every other business, in that they work to increase a resource’s usefulness to other humans. Because disruptive innovations are typically produced by an outsider, are not immediately applied by the predominant status quo, and involve considerable risk from a financial and reputation perspective, they tend to have a higher degree of impact once realised. Brown and Wyatt (2010) state: 

Truly innovative ideas challenge the status quo and stand out from the crowd—they’re creatively disruptive. They provide a wholly new solution to a problem many people didn’t know they had.

Mona, in Tasmania, founded by David Walsh, has many of these qualities.

A central element of Mona’s disruption to the dominant museum paradigm has been the removal of traditional wall labels that provide information on the works of art. Walsh was clear on the interpretive approach required for Mona from the outset. His rationale stemmed from a realisation that conventional approaches were less than ideal; inhibiting both exhibition design and the free thoughts of museum audiences. Walsh (2014) writes: 

And here’s the thing I realised. When museums became adornments to the state they needed to communicate the profundity of their collections. They needed labels on their walls. And they needed you and me to be able to read them. As scribes throughout history had demonstrated, and Gutenberg’s press had highlighted, the best way to make them legible is black text on a white background. Thus museum design the world over. (p. 183)

The digital substitute to traditional artwork wall labels that has been created specifically for Mona is ‘The O’. At its simplest, The O is a personal touchscreen interface that provides every Mona visitor with a list of artworks ordered by physical proximity, with the work physically closest to you shown at the top. Each artwork on The O can have any amount and type of interpretive content associated to it, ranging from Walsh’s personal insight, candid audio interviews with the artist, research-based material and, on occasion, popular music. A centralised content management and publishing system allows Mona’s team to regularly update The O’s interpretation throughout the lifecycle of an exhibition. This reduced publishing friction frees curators, researchers and writers to help produce the kind of raw, unedited, emotive and sometimes frivolous responses to the artworks that the audience overwhelmingly prefers. On any given day, fresh content might be added, like this: 

I was sitting in a bar yesterday (the wine and beer bar here, back up the hill. Not this one, in the museum. No decent names yet. Any suggestions? ‘Bar Stard’ just came to mind. Don’t worry, I’m not serious). I was playing with this thing you have in your hand, ‘The O’ (perhaps for ‘Orgasm’ or ‘Oracle’) and I looked up this work, by Christoph Ruckhaberle. To my surprise there was no ‘Gonzo’ entry for it so I thought I’d better write one. Trouble is, I don’t have anything to say, and since this is the very first work in the gallery I wouldn’t want to give you the false impression these entries are frivolous. 

When it launched in 2011, Mona’s new approach to museum interpretation had an immediate impact on audience engagement. A survey commissioned in 2012 across a sample of 1,100 visitors found that over 80 per cent of visitors considered that The O enhanced their experience of the museum, and over 70 per cent of visitors preferred The O to traditional wall plaques and signage (Mona Museum Hobart, 2012). 

The absence of traditional museum wall labels removes artwork metadata from the equation, making the visitor’s natural curiosity the sole driver for engagement. The museum never prompts you via a sign or label to ‘go here’ or ‘look there’. Any bias created by attaching the celebrity (or lack of celebrity) of the artist to every artwork is removed. It is an inherently visitor-centric approach, as it is fundamentally about democratising the museum experience. 

You can see the outcome of this explorative, curiosity-driven approach in full effect in exhibitions that juxtapose the works of major artists beside comparatively lesser-known names. Similarly, distinct styles will also be mashed together—an abstract expressionist work placed beside a kinetic sculpture for example. Such was the case in Mona’s inaugural exhibition, ‘Monanisms’, in which a painting by Vasily Kandinsky, with its mix of curves, colours and lines, was hung nearby Conrad Shawcross’s intense light sculpture Loop System Quintet 

Figure 33.1 Loop System Quintet by Conrad Shawcross.
Source: Fraser Mummery, https://flic.kr/p/b3uczt Creative commons licence. 

(Figures 33.1 and 33.2). Visitors stood transfixed by the kinetic sculpture for long periods, paying only cursory attention to the Kandinsky—sometimes missing it entirely—their interest not piqued. The O lends a helping hand in these kinds of curatorial arrangements in the form of a personalised post-visit experience that records and helpfully distinguishes between the works that were seen and unseen via Mona’s website. 

The agency of the visitor is further amplified by Mona’s artwork rating system. The O displays a prominent ‘love or hate?’ option atop each artwork. As distinct from the echo chamber of social media ‘likes’, designed primarily to funnel ever more targeted advertising towards us, Mona’s love–hate binary exists to reinforce the visitor-centric approach by alluding to every visitor, whether consciously or unconsciously, that their thoughts about the museum and the artworks contained within it matter. There is no immediate outcome from tapping ‘love or hate’ in The O, but visitors still enjoy sending their thoughts to the museum. (On average, each visitor votes 4.4 times per visit. Interestingly, the ratio of love to hate averages a constant of 70 per cent in favour of love.) However, tangible effects of the love–hate provocation can be observed in the interaction between visitors. One example is Mona’s librarian Mary Lijnzaad’s observation of a Tasmanian outlaw motorcycle gang roaming through Mona’s exhibits laughing and joking, and generally enjoying themselves. Leather-clad and intimidating, the diligent Mona invigilators probably had some nervousness about what they were up to. On closer inspection though, each bikie had The O in hand and was jamming in ‘hate’ on every single artwork. Call it engagement if you like, or better, call it having a great time! In effect, Mona is shouting out to every visitor: tell us what you really think! So they do. 

Figure 33.2 
Mona ‘Monanisms’ exhibition design, 2011. Source: Leigh Carmichael, Mona 

The absence of wall labels also achieves significantly greater flexibility in exhibition design; dramatic lighting amplifies the visual connection to each work, at all times accompanied by The O’s interpretive content. A second visitor survey conducted in 2014 across a sample of 6,411 participants found that Mona’s approach increased the enjoyment of art by 77 per cent, with 25 per cent of the audience describing the museum visit as a ‘life-changing experience’ (Franklin & Papastergiadis, 2017). These types of extraordinary responses have resulted in Mona quickly rising to become one of the world’s top museums to visit, with the Lonely Planet’s Ultimate Travelist (Hope, 2015) ranking it 20th, ahead of international institutions including New York’s MoMA and London’s Tate Modern. 

These examples span approximately 10 years of research and development, and while not exhaustive, they provide an overview of the benefits created by a unified approach to museum exhibition design and user-experience design. But what happens next? And what might the interpretive interfaces of the future look like?

We are at an interesting juncture in terms of human–computer interfaces. There is a clear trend towards more natural and subtle interface design. The field of MR is shaping up to be the engine for this imminent wave of technological change. MR describes the ‘merging of real and virtual worlds to produce new environments and visualisations where physical and digital objects co-exist and interact in real time.’1 MR represents a mixing of reality spanning the real, to the augmented, to the virtual environment.

Relevant to this is the growing capability of smart devices and the move away from traditional two-dimensional interfaces and towards three-dimensional (3D) environments. From a GLAM sector perspective, one of the more practical benefits of the move to 3D will be the ability for the end-user to get closer to artefacts and artworks. The combination of more natural, gesture-based interfaces, coupled with high-resolution 3D scanning and display systems, means that objects locked away behind glass display cases can be examined effortlessly, in greater detail and from any angle. The MR version of this interface can then transport the 3D object to its original location, returning it to its natural resting place and re-establishing the important context so often lost with traditional object display approaches. 

Until now, digital interfaces (described above) would typically rely upon our personal smart devices or larger format displays like projections. But as with the old rotary dial telephones of the past, the pervasiveness of the ‘little black mirror’ shall also come to pass. 

Much of the big money in consumer technology investment is currently following the path to the visual cortex (MR start-up Magic Leap has raised over $2.3 billion in funding to date [Matney, 2018]). With vision accounting for two-thirds of the electrical activity in our brain, or about 2 billion of the 3 billion firings per second (Fixott & Sell, 1957), it is safe to say Silicon Valley is not betting on Smell-a-vision as the next big thing.

So-called ‘smart glasses’ might just be the ‘singularity’ for natural interfaces. Just like normal corrective lens and sunglasses have pride of place into the window of our soul and enhance our view of the world, the electronic version will achieve this and a great deal more. By intercepting the light that falls on a glass lens, Silicon Valley, for better or worse, will have control of humanity’s vision. Obviously, this is a big deal. No longer is light simply going to fall into our eyeballs haphazardly. Instead, miniaturised cameras will be the first in line, sending it off to the Cloud for processing, and special optical ‘waveguides’ will then throw it back into our visual cortex all in the blink of an eye. And, sitting in direct contact with our temples, the arms of our smart glasses may even use electroencephalography (EEG) to read our minds and serve as a thought-control interface (Taylor, 2017).

If this sounds alarming, you are not the first to think so. However, as this is a discussion about future museum interfaces, let us examine how the smart glasses of the not-so-distant future could be used for good rather than evil. Let us assume we now have direct access to the visual cortex and our wearable smart devices are working hard to process the world around us. Leaving the deep philosophical, moral and privacy issues aside, what might happen when we explore our favourite museum?

Knowing exactly what you are observing, and to a certain degree, even what you are thinking, coupled with the ability to seamlessly overlay content across your field of vision, theoretically enables an entirely natural digital interface. We will be able to skip many of the typical selection-based interface approaches of today, instead using computer vision to immediately detect the artwork or artefact you are focusing on and then automatically provide relevant and personalised content. Aided by EEG and thought control, the digital interface will also be aware of your sentiment towards the object you are looking at. Shortly after focusing your eyes on any object in the exhibition space, your emotional state can be accurately assessed, and the interface can respond accordingly. For example, the absence of any strong emotional response might mean we take no action at all, since you are not interested in that work! However, if your emotional response does exceed a set threshold, we can then assume your interest is piqued, and automatically overlay an interpretive interface (or inversely, a lack of interest could be used to identify an opportunity to convince the viewer otherwise?). This too can be, by several orders of magnitude, subtler. Digital overlays across our vision do not need to be slammed across our field of view to obscure what drove our interest in the first place. Instead, more natural design approaches could overlay in the form of a virtual book across an open palm. Looking down at our hand is all that is needed to display the interpretation for the artwork of interest. The typeface would always be set to our ideal reading size. If we prefer to listen rather than read, natural language processing will perfectly vocalise the content in whatever language we desire. 

Aside from the immediacy and intuitive nature of these type of vision and thought control-based interfaces, there is another potentially interesting upside. To make it all work seamlessly, we may have to train our minds to concentrate a little better. If we are at the exhibition, and our mind is a little scattered, the thought-based interface might struggle to accurately intuit our interest level. In other words, the system will work optimally when we are focused on the exhibition. In this manner, the exhibition experience would bias towards a state of increased mindfulness, requiring greater focus on the physical works; this could end up being the best thing to happen to museums. 

Note 

• ¹ https://en.wikipedia.org/wiki/Mixed_reality. 

References 

• Antonelli, P. (2014). The journey is the destination. Retrieved from www.moma.org/explore/inside_out/2014/06/26/the-innovation-route-the-journey-is-the-destination/. Brown, T., & Wyatt, J. (2010) 
• Design thinking for social innovation. In Standford Social Innovation Review, Winter. Retrieved from https://ssir.org/articles/entry/design_thinking_for_social_innovation. 
• Fixott, R. S., & Sell, S. B. (1957). Evaluation of research on effects of visual training on visual functions. American Journal of Ophthalmology, 44(2), 230–236. 
• Franklin, A., & Papastergiadis, N. (2017). Engaging with the anti-museum? Visitors to the museum of old and new art. Journal of Sociology, 53(3), 670–686. 
• Hope, E. (2015, 18 August). MONA tops international modern art galleries as ‘world’s best’. The Mercury. Retrieved from www.themercury.com.au/lifestyle/travel/Mona-tops-international- modern-art-galleries-as-worlds-best/news-story/3a86ee2408b9ba22ba97597d325d7e10. 
• Matney, L. (2018, 8 March). Magic Leap raises $461 million in fresh funding from the Kingdom of Saudi Arabia. Tech Crunch. Retrieved from https://techcrunch.com/2018/03/07/magic-leap-raises-461-mil lion-in-fresh-funding-from-the-kingdom-of-saudi-arabia/
• Mona Museum Hobart. (2012, March). The O Device 2012 Survey Report.
• Taylor, T. (2017). Not your standard shades. Retrieved from www.si.com/edge/2017/01/06/safilo-inter axon-brain-sensing-eyewear-sunglasses-athletes-mental-fatigue-focus. Walsh, D. (2014). A bone of fact. London, UK: Picador.