Fast Forward: Q&A With Dolby's Chief Scientist, Poppy Crum

Virtual and augmented reality were hot topics at SXSW this year, but the conversations extended beyond headsets. To be truly immersive, experiences need to incorporate all five senses. Sight, sound, touch, smell, and even taste.

No company knows more about technology and human senses than Dolby, which has pioneered everything from surround sound to HDR imaging. I was fortunate enough to sit down with Dolby Laboratories' Chief Scientist, Poppy Crum, at the show.

Crum is also an adjunct professor at Stanford University in the Center for Computer Research in Music and Acoustics and the Program in Symbolic Systems. Crum was at SXSW as part of the IEEE's Tech for Humanity Series. She understands sound and a whole lot more.

Costa: Poppy, thanks so much for joining us today.

Crum: Thank you for having me. It's great to be here.

So we're going to talk about hearables, we're going to talk about augmented reality, we're going to talk about maybe a little virtual reality, maybe we'll talk about those painless migraines that the two of us experience from time to time. First of all, your role at Dolby. What does a work day look like for you? What do you do when you get to the office?

We have a large team of computational neuroscientists and people that are experts in sensory perception. If you look back at the history of Dolby and go back, even 50 years, at the core of the company, it has always been an understanding of human experience. I think it helped differentiate how we think about building technology.

So on a daily basis, the people on my team and the people I work with, we look across technologies. We're not just sound anymore, it's really about a holistic sense. We have labs, and there are a lot of experiments that go on throughout the day.

Our new building has up to 100 labs, but we've got some amazing biophysical labs. My background is as a neuro physiologist—the same with a lot of people on our teams. And there's human physiology that's happening on a daily basis to think about new technologies, and there's some very seminal work happening on thinking about how we experience information that's multisensory and really looking towards how are we going to consume content in the future that is so rich and what that's going to mean for how it affects our bodies, how it affects how we engage with others and our senses.

One of the things that you've talked about at the show and that we've covered a lot more on PCMag is this hearables segment. Hearables is not a word that I think a lot of our audience is familiar with. When you hear the word hearables, what does that encompass to you? What does it mean?

Right now, I think it's a term that's still being defined. I like to take it as a very large subset of products and possibilities. It's a wireless device or in some cases, even wired. Because Amazon Echo I will call a hearable and that's plugged in, but it's a device that has microphones, or sensors, but it's capturing data from the environment, using that data in some way to enhance your experience. Also there are a lot of companies right now that are thinking in the wearable space, a device that's wireless, worn on your body, but there are also static hearables if we look at Google Home ($99.00 at Target) and Amazon Echo, and those are transformational.

A hearable doesn't necessarily mean that it's about just enhancing the sounds around you. It might be taking information about the sounds around you and using that to somehow give you an enhanced experience in the world. You could think about it's capturing ... the ear, it turns out, is a great place to pick up biophysical information so that you can capture a lot of information there. You can imagine, obviously, analytics capturing information just about the sounds around you, your conversations, using that as a way to enhance your day or optimize you.

There's also a lot of concerns with this. One thing we see that I think is worth calling out, and I think it's what's going to transform and help define what this space are changes in regulations.

Right now, hearing aids are hearables. Hearing aids are an augmented reality device, but you have this class of consumer devices that have the capability to help mitigate hearing loss, have the capability to be an augmented reality device for someone with normal hearing, have the ability to even be a gaming device. You're getting the crossover of these fields the medical device that's falling into this larger class and you're going to have a consumer device class that is clearly crossing these boundaries and doing a lot of similar processing for us.

In terms of hearing aids, people think of hearing aids as once you start to lose your hearing, you can then get a hearing aid that will then give you some of your hearing back, but there are lots of interesting things that could happen if people started augmenting their hearing with ... they have normal hearing, but they want to have something more than normal hearing.

Absolutely. I'm always a big believer in not creating this arbitrary boundary where we say, "Now I have hearing loss." Our hearing, because of many elements in the world and sound, even aspirin is an ototoxic that you have to be-

Is that right?

Absolutely. The combination of elements that you put in our hearing starts degrading in our early 20s and probably even earlier with some of the loud sounds that people listen to.

Especially South by Southwest.

Yes. Hearables have so many different capabilities you can have. Whether it's having control over streaming your content directly to your device - wirelessly - control of elements, the spatialization of information. Things we talk about --. there's been a big push in taking information, augmenting our visual sense with Google Glass or some of these devices from other companies, and really - what we want -- the sonic component of that [information] can be very critical to allowing us to get past what we might call a capacity limitation. And to take information from our world and really represent it as a sound.

It seems to me that the thing that's made people think differently about sound and voice controls and voice interactions, but also privacy, are devices like Google Home and Amazon Echo, which are really the first mainstream voice interfaces that we've had for this digital world and it comes with all these different consequences, which I think we're just starting to sort out. Where do you see that market heading?

That's a great question. I have to say, I think these devices are transformational in so many ways, and I am a strong user and a strong sharer of them, partly to also understand and look at where they're going to evolve to and how I use them in my life on a daily basis. The idea that people are willing to have microphones always on, that's a big leap. What can we do with that?

Right now,voice is a wonderful thing. It's giving people control; it's getting them onboard to have this device as an assistant in their lives. Ten years down the road, I don't want to have to control more things in my life. I want my devices to ... I'm going to trust my data more than I'm going to trust me to know what I need in some cases and I want the devices to be...

Anticipatory.

Absolutely. I want them to be proactive and capturing a lot of information about the sounds around me, whether I've been coughing and modulate my temperature. Or just to make my appointments for me when I need to or to be able also just to make our lives moving forward without having to control all of the devices.

You could imagine that the Echo could detect whether you've been sniffling. It could detect whether or not you sound like you have a sore throat, or if you've been coughing, and then lets you know that it sounds like you have a cold.

To go ahead and schedule your doctor's appointment for you before you [know you need it]. That sounds like a little far reaching to want that, but at the same time I think we're going to get there. I think the step of having voice control, integrating these into our lives will make us comfortable with it.

In the future, the only time we'll be uncomfortable is when things don't work and when it's gone and when it's absent.

I had an experience with an Echo that let me realize how much it had transcended the interactions that I was having and could help people across different demographics. But yes, I think people have thought about these devices a lot for accessibility, which is wonderful. Like the access that it can provide to both very young children or different age ranges or different people that...

People that have disabilities or shortcomings or gaps that could be filled in using technology.

Absolutely. In my case, I had a relative that was in the hospital and he passed away a few weeks ago. I had bought him, right when Echo came out; I bought him an Echo as an accessibility device. Here's the transformation. I took it to the hospital, and devices like this are phenomenal for a hospital setting. The privacy issues do become important to consider, but we use it predominately for music playing in that setting.

But at this point and time, my relative was not very vocal, had not been speaking, and we'd been playing music on the Echo that we thought was what he would want to hear, Bach and very calming music and honestly, some of the last words he said, and I'm not kidding, which I remember, were, "Alexa, play Al Green." He wanted to her Al Green, Sly and The Family Stone and this device gave him that accessibility. It was empowering and at a time where it was very powerful.

There are a lot of technologies at play there. You've got the fact that Al Green is available and that there's this vast music library that's a voice command away and then you've got the voice command itself, which makes it possible for him to request it personally. So there's a lot going on there.

I think when you bring up the privacy issue, that's another question is that before Alexa is making our doctor's appointments, I suspect that some pharmaceutical company will be offering to give us a cold medicine table or our allergies are acting up and offer a Zyrtec, and I think that's almost an intermediate step we have to get through where who's going to control all that data that is being captured and that we're giving away in this audio format.

We have to embrace it. If we don't think about the regulation side and we don't think about how we're going to make people comfortable with providing you, even more, data than we currently are. I think Amazon right now says, "We are only listening when you say Alexa," but these devices to do what they really can do you have to be listening all the time.

There's a big trend right now of companies, insurance companies, whether it's car insurance or health insurance offering consumers a deal or a way of having lower rates if they allow their data to be tracked--if they give away their data. I think it's very powerful. I think it will be part of our future, there's no question about that, but that future of what the ramifications of sharing that data, that hasn't even been defined yet, and it's hard to predict. So, we really have to think about what that future looks like.

Also, I think there's a permanence issue where people are, "I don't mind if a give my healthcare company or my insurance provider the number of steps I do every week," but that information doesn't go anywhere, and the steps you do this week will be searchable and indexed 30 years from now, and that idea of digital permanence which we live with today, which really, in human history, never really happened before. When you add the fact that you could have a microphone on in your kitchen all the time, all that data will not go anywhere. Amazon will always have it.

That future is unwritten, and we don't know the ramifications. We haven't defined these regulations, but those can even change in the future because many things can change.

And here's another thing. I think culturally, the EU has tried to enact much more consumer protective regulatory legislation right now, but it's not clear what it means because the data exists. We need to ensure interoperability when you're traveling. We need assure security for small IoT devices. That's a really critical thing. I think groups like NIST are very active to solve this.

When you look at what Dolby does and all the technologies that they're working on, you start to see common themes, and one of them is that the company is really trying to give humans super sensory perception, superhuman powers. It sounds over the top. It sounds hyperbolic, but there's a bunch of examples of people getting superhuman powers using technology. Can we just talk about a couple of those?

Absolutely. So I come from a background as a neurophysiologist, who's thinking about how we integrate these things to technology and there are a couple of things that are important. When we think about what augmented reality is or what technology can do for us today ... when I first joined Dolby, maybe we were in the process of working on Dolby Vision and Dolby Vision is a high dynamic range and wider color gamut imaging technology. Just to get an idea, the typical display you would have bought about three years ago would have been 300 to 400 candelas per square meter. The moon, the natural moon is about one to two thousand, sunlight on black pavement, you're getting up to 15,000 candelas per square meter.

So display technology was very far away from what our actual sensory system could handle. And in the process of development, we were working with some content and devices that allowed us to get up to 20,000 candelas per square meter in the brightness that was produced-

And this isn't necessarily contrast, this isn't resolution, this is the brightness.

Yes, from a perception and sensory perspective side, for a lot of viewing distances, we're pretty maxed out in what our sensory systems, your perceptual system can handle regarding resolution. But regarding brightness and color gamut, we were no where near. So it's pretty exciting to be, "okay there's this area that we can enhance the experience, and we can lead in this field." That's pretty exciting.

So in doing these studies, I think what made me realize how powerful this could be, and also redefine what I thought immersive technology should be about was this. We were watching some fire content and the candelas. I think we were playing at about four to six thousand candelas per square meter, and I was watching it, and I felt my face react to just the fire. This man was whipping a flame at me, so there's this big specular, really bright flame and I thought, "Oh." The display itself must be producing the heat. So I talked to one of the key developers and asked him. He was like, "It should be constant." So I get a thermal camera and used thermal imaging to track the screen. It was totally constant. But we were able to show consistent changes on people's faces that were content dependent.

So just based on the luminance that was reaching the retina, my brain says, "Okay, I've never experienced flame that was this bright that wasn't real, so probabilistically my brain is going to behave with that fire as if it is real."

So now when we think about immersive technologies and creating superpowers and creating experiences, we can be thinking about them engaging our physiological system in a natural way, an authentic way, and then we can enhance them. We have the ability to go beyond what you might experience in the natural world, and that becomes pretty powerful.

Yeah, it's a great example. We've also talked a little bit about how in PCMag we've tested a bunch of ER systems. There are very promising technologies in the works there. We always focus on what it looks like on the screen, on the resolution, but the audio component to virtual reality and augmented reality is just as important as the visual component and helps sell the experience. Can you talk a little bit about what you're finding there and how much that adds to the experience?

Recommended by Our Editors

Fast Forward With Pandora VP of Product, Chris Becherer

A great trend across, not just across our company, but others, is that recognition that you can't study one sense without the other. If we're looking towards future devices and holistic devices in AR and VR. Right now, all of the main VR devices are visual, but a hearing aid, as far as I'm concerned, even though they might not want to be called this, is an augmented reality device. It's trying to augment from the base physiological system, the state of it. Cochlear implants, are one of the first implanted biophysical devices.

If you think about what do we want in augmented reality? We want to enhance our experience of the world. We want it not to be something that interferes and gives us too much data that we can't do the things we used to do. Right now, we're in the trend of really cool experiences, but we haven't enhanced and touched in the way that I'm looking forward to making the mundane activities in our lives better. And allowing us to be more engaged with people because those things are enhanced. I'll give an example of that in a sec, but really, we want to be connected. We want to be engaged with our lives. We want to be able to take in more information. We want to have heightened experiences and we want to be able to have control over how rich these experiences are.

So, if we take all the data that we can capture or ... more data is not always better, but if we are trying to enhance our visual ... I've seen this happen in the military, I've seen it happen with technology where I want to create an AR, and I take information that I'm trying to enhance, and I put it all in my visual system. I put it all in glasses, or I put it in places where I have to look at it. Well, our brain can't process that much information. We can't take it in at the same time.

So that's where sound ... we have a capacity limitation in one sense, but guess what? We have other senses. And suddenly we open up this whole store that we can go and really use to enhance our experiences, and guess what? This is cliché, but regarding creating a physiological experience, what do I do when I'm watching a scary movie and scared? I turn the sound off. So not only is that a really visceral part, it's a whole opportunity to enhance and enrich our lifestyle in terms of how much information we can take in, and how much information we can process. Simply having voices separated or sound elements spatially disparate, we can process them at the same time instead of having our brains and the information be completely useless. It prevents us from having all of our brain and cognitive capacity taken< up by doing that one thing. It really opens up our situational awareness and what we can do.

The audio component and all these VR things, as we start testing them, that's something we're putting into every review, is that it looks like this, it sounds like this, and then the two usually should add up to something greater than the sum of the parts.

There are so many different things we can do with sound in AR or VR. Obviously creating a spatialized experience is important, but it is a key part of it. Dolby Atmos is an example of an object-based sound delivery and rendering. So when things are authored in Atmos and the sound elements live with a data package, a data stream attached to it. So, you can imagine, that can be very powerful. That can tell you where a sound should be. Regardless of how you're recreating that sound, it has that location. It has a width, it has a loudness, but it can also GPS coordinates. It can also have all sorts of information. We can have emotional content in the future about how it's supposed to affect you if we know something about the interaction that's happening.

These are the types of future things that could happen. If you think about an AR device, it can provide a lot of information. We currently use maps to look down at our phone or in a car. These things can be done in a lab very easily to create a sonified version that's giving you tracking information and telling you, not verbally, but giving you pings that tell you where sounds are, where you should be going. The problem is that we have to make sure it's robust and consistent across users and that's a more complicated problem.

Mappings a great example that it is still screen based by and large and when Google Glass came out, you could see all this information, this little tiny little window in your visual display. What I really wanted was for it to tell me in my ear whether I should turn right or left.

Yep.

And you're right there, right next to the ear, you could just give me the voice command. I thought it was a better audio system and sound system than the visual system regarding wearable solutions.

Absolutely. And again, well, not again, I think you've hit on the main point, which is these are all tools, and right now, today, we have tools that we can use in amazing ways, including neuro control and things, but the question is not let's ... it's picking the right tools for the data, for the technology, for the user experience that you're trying to target. You need to contextually optimize for and personalize for your individual and for what they're trying to achieve, and that's been a big gap in what we're building right now. Amazing technology and it's going to just get richer. But it's going to move around and I think we have to embrace what we're trying to get to and think about how, as a human system, we can best experience that.

One of my favorite things to do is to look at other species. You can look at whether it's a frog or a fly, or a bat, species that have these amazing superpowers to do things that go way beyond what their physical bodies and a coarse look at their brain might tell you. Understanding how they solve that problem and more, why their evolution solved that problem for them, gives us insights. We have to think that way. We have to think more about what are the environmental pressures on our systems, and how is technology going to best help us solve that?

And delivering more and more data probably isn't always the right solution.

Rarely, rarely. I mean, what makes us successful a the human is what our brains throw away. I'm always talking about illusions, I love illusions. They're fun, but they're also this beautiful insight to when our brains get things wrong. If we experience data exactly as it is in the world, you actually end up with a pathological condition that you're not functional in the world. And what our brain is constantly doing is helping us weight certain information and not even notice different information. If we noticed all the luminance changes created by light, we would never see holistic objects. We need to function, and we need to know what to action on, and that's what's powerful.

Let me ask you some questions I ask all my guests. Number one, what is the thing that is keeping you up at night? What technology trend are you most concerned about going forward?

Two things. One I think we've already touched on, there's so much power in creating these sort of superpowers, and what we can do with AR and VR. My hope is that we look at all the technologies we're building as tools, and we think about using the right tools in the simplest and the most, I say robust, but it's consistent across all our users to get to the experience that's intended rather than the one that's the most sexy and shiny.

The other one though is--it's really about the data. There's so much power to having microphones always listening that will move our technology forward, move our lives forward. But we have to be comfortable with what's happening to that data. We have to be comfortable with regulations whether it's self-driving cars or hearables, or just the security in IOT devices. If we don't have strong regulations and standards and understanding and, in many cases, transparency to the algorithms driving some of these probabilistic decisions and the AI behind some of these devices, I think it will delay and hold back innovation. It could delay the capabilities of the technology to have major impact in a positive direction for so many people. I want to see that happen. So I think it's going to be about standards and regulations to help make that come to life.

On the positive side, what are you most excited about? What do you think is going to change the world and gets you excited about going to work every day?

There is a trend and a shift that we don't think about one sense. We're thinking about technology holistically. We're thinking about the impact it has on sight and sound together We think of our entire physiological system and, I think, how we feel, our emotions. The things people care about now and the way we describe the impact of technology is much richer regarding what it means for human connection, and what it means for just the power it can have on our experiences. That's something that I definitely think is a new direction.

So, regarding a product, a service, a technology gadget that you love, that changed your life. Is there anything that stands out that you're like, "This is the thing that makes my life better every day?"

Static hearable devices. I use an Echo every day, and I've seen it used across such a broad generation gap and use cases. I'm really excited where technologies like that are going and right now, it's really in its early stages of the impact it's going to have.

So if people are listening to this talk and they're interested in your work, how can they find you online, interact with you?

You can find me online at LinkedIn, Poppy Crum. You can also find me at on Twitter @poppycrum.

For more Fast Forward with Dan Costa, subscribe to the podcast. On iOS, download Apple's Podcasts app, search for "Fast Forward" and subscribe. On Android, download the Stitcher Radio for Podcasts app via Google Play.