Emotion and the Internet of Things by Scott Smith

Thingscon Amsterdam at November 7, 2014.

Last month I was invited to speak at Thingscon's inaugural Dutch event in Amsterdam while on a swing through the country (I'll be speaking at the main Thingscon in Berlin in April, stay tuned for details). My topic of choice focused on the emerging use of "emotion" in the Internet of Things—a topic that's been on my mind, and one that an increasing array of startups are beginning to build their pitches around. The idea that technologies that sense and report, and will be present both on our bodies and in our immediate environments is one that can be both exciting and fraught with problems. In 20 minutes, I tried to scratch the surface of some of these issues, illustrated by the slides embedded above.

I've been tracking so-called emotional technology for some time, but a recent product introduction from French company Withings—a wireless camera for the home that features, among other selling points, something it calls "cry recognition"—prompted me to think about how we got here, why companies are pitching products they claim can detect mood, stress, sentiment and distress and act on them, and what might come next. 

Assessing a user's state of mind isn't a new thing in human-computer interaction. Joseph Weizenbaum's ELIZA program, designed at MIT in the mid 1960s as a test of natural language processing, mimicked a psychotherapist that asked a sequence of questions to assess a user's state of mind. While it matched patterns in responses to generate the next question in a fairly flat, mechanical fashion, some users were taken in and felt they were engaging with something that understood them. 

Flash forward almost 50 years, and technologists and researchers are trying to do the same thing, but with more sophisticated kit and better pattern matching. Google's Glass has been used by groups like the Fraunhofer Institute to match facial geometry of that cataloged as exhibits of particular emotional states. Even some entry-level camera tech now detects the geometry of a smile as a trigger to take a picture—though it's the shape of a smile and not happiness as a state. So we can fake a machine into thinking we're happy for the camera, just as ELIZA faked interviewees into believing it was concerned with their wellbeing.  

The kind of multi-sensor devices that make up the IoT are a boon to companies seeking to identify emotional states. Companies like Neumitra and Affectiva correlate physiological responses such as changes in skin temperature with increased motion, as one example, to flag states of increasing stress, which in cases of PTSD or certain forms of autism can be helpful for both individual and/or caregiver. Like earthquake detection, small signals detected early on can both make someone with PTSD or prone to emotional outbursts aware of what's coming, as a precursor perhaps to behavioral change. These are, however, conditions studied mostly in clinical or therapeutic settings, based on peer-reviewed research. And yet, such skin conductance or motion detection can be fairly cheaply built into a consumer device where ongoing testing doesn't happen past the R&D stage. A racing pulse, sweat, and increased motion can mean many things, not just a stress attack.

Voice is another hot area for the emotional IoT, as evidenced by the cry detection from Withings, and tech like Beyond Verbal's, which again reads patterns, here in the sound of a person's voice, to match against patterns that suggest certain emotions. Beyond Verbal has developed what it calls a "wellness API" that makes this capability available to other developers' services, making it easier to build emotion detection into a range of hardware and software—both clinical and consumer. Think about a car not wanting to start because you sound stressed, or your mobile flagging what sounds like excitement to an app or, say, an advertiser. As Anthony Townsend recently wrote, the Internet is a two-way street here, and we've seen emotional manipulation happen already in social networks, so why not the IoT?

Given that we are pushing into the era of always listening devices, or in the case of in-home cameras, always watching lenses, this creates an interesting situation where your personal devices are monitoring your "emotions" pretty much constantly, looking for behavioral and physical patterns that match preset assumptions about state of mind. And given how many consumer digital services fund themselves, by selling data collected about you to advertisers or other third parties, this sets up some interesting and undoubtedly problematic near futures. Similar APIs are being developed for facial expression recognition, such as one from a company called Emotient, so faces are game too. 

These are early days, and we can learn some interesting things in the consumer sphere from collective "emotion" as extrapolated from sensor data. The recent data from Jawbone from wearers near the Napa earthquake that suggests some people couldn't get back to sleep after the early morning quake suggests something interesting about group responses to such disturbing events. What, we don't yet know. We certainly wouldn't want anti-anxiety meds sprayed on the community based on it—sample sizes are too small, and, more importantly, we know only that some people kept moving and didn't settle back to sleep right away. It only broadly matches the patterns of anxiety, but doesn't prove much of anything. Likewise we can see some interesting correlations between movement, maybe stress levels, and group activity from big social events, as shown here in this recent look at wearable data from a Burning Man attendee. But we don't see much more than time and motion—not states of happiness or depression.

Calibrating what we expect technology to do is important in any discussion, but many times more so in discussions about emotion, so critical are they to human social interaction and wellbeing. The distance between calling something sadness or depression, for example, is as wide as the distance between an app store and a hospital. And too often technology developers and marketers ignore the difference—sometimes with innocent intentions, and sometimes for expedience. The recent uproar over British charity Samaritans' creation of Samaritans Radar, a service that it hoped would identify people on Twitter at risk of suicide based on key words in their tweets shows how problematic good intentions can be. 

I suppose my point here is that the trivialization of emotion that can occur in the rush to commercialize new technologies and market based on their appeal to wellness, happiness or safety, for example, is something those involved in creating these products and services have to be extremely careful to avoid. Many emotional issues require proper research, clinical experience, testing, approval and serious support services. Hoping to "move fast and break things" in the rush to create "disruptions" and ship product doesn't work here. In a world of indiscriminate data collection, sharing and leakage, promiscuous APIs, and falling cost of development and deployment of technologies, using emotion as an appeal requires careful, thoughtful approach.

Hopefully as the IoT matures, its proponents and participants will be willing to slow down and trade a quick hit now for long-term consumer trust. Otherwise, we're stepping into a world where emotions, not just data, are manipulated by both code and coders alike.  

Getting There Together by Scott Smith

Sunset in Helsinki. 

Sunset in Helsinki. 

A few weeks ago I had the privilege of giving the closing thoughts at Flow Talks, held at the fantastic Flow Festival in Helsinki under brilliantly sunny skies. At these talks, organized by the good people of Demos Helsinki, I was preceded by Ville Niinistö, Finland's Minister of the Environment, the great Marko Ahtisaari, Tero Ojanperä, Jufo Peltomaa, Tarja Teppo, and other brilliant folks from around the world working on cracking problems of sustainable growth in a market economy increasingly shaped by technology.

I was asked to sum up what was, in the end, a very diverse set of talks and topics. I had some thoughts on paper going in, but as I listened, I sketched a summation not of everyone's individual presentations, but of the broader set of challenges they were pointing to. Below are my cleaned up notes from that closing. Thanks to all who invited me, hosted, provided input, and commented afterwards.


In 2005, Bruce Sterling wrote a non-fiction book called “Shaping Things,” which considered how to think about made objects—and some unmade, digital ones—in a networked future. One of the early chapters in the book, nearly a pamphlet, ends with the thought “Tomorrow composts today.” Bruce used that concise but powerful idea to suggest that the futures, in particular designed and made futures, don’t appear out of thin air, but are made up of the fragments of a technosocial today, some decomposing, some not, but much of it functioning as sedimentary layers. It’s layers, and legacy, all the way down.

What we do today, and the tools, infrastructure, models and rules we lay down today, or decide to lay down tomorrow, will become a nutrient, or a poison, to what comes next and next and next.

And since we are talking about sustainable futures, he added this cheery framing: “The quest for a sustainable world may succeed or it may fail. If it fails, the world will become unthinkable. If it works, the world will become unimaginable.

In practice, people will experience mixed success. So, tomorrow’s world will be partially unthinkable and partially unimaginable. Effective actors will be poised between these two conditions with plans and crowbars.”

Plans and crowbars. That’s what we’re here to discuss, to be blunt. Which plans, and crowbars held by whom? This is what I think is critical for us at this crossroads in the sustainability debate.


Whatever path we take, we largely know how to do the end technology, whether its dealing with energy or food systems or transportation or one of the half dozen other critical systems we need to address to achieve some level of future sustainability. We may not have it yet, but we have a good idea what it takes,. Though we have a lot of progress still to make on efficiency and cost, we’re already knee deep in researching and refining approaches to many of the component parts of sustainable systems, in everything from PV technology to protein substitutes to new materials and on and on. Whatever it is, chances are someone somewhere is on it, even if it’s just a notional idea.

We have the capability to build out infrastructure, if not the funds and willpower. We already have plenty of networks, bandwidth and data storage capacity, and it’s getting cheaper, so adding more is at least possible.

But we are at a critical juncture, economically and in terms of how markets see sustainability. What we call the technologies of sustainability are, for better or worse, starting to be embraced as part of what comes after internet and mobile as a revenue generator—the source of the next boom markets. It’s taken long enough, but as communication and media reach for thinner and thinner returns, the funders of new technologies, the sage VCs and investment banks and the like are eyeballing the types of technologies I mentioned, for energy, food, transport, etc—as their next golden egg. Silicon Valley is latching on to resource constraints in these areas which is both good and also potentially problematic.  

It’s showing in the business models. Sustainability sells when it’s chic. Some iconic products like Nest and Tesla, are positioned as a premium product, which can have the shortsighted impact of further linking sustainability to luxury. An electric car is attractive when it’s wrapped in the styling of a sports car, the ultimate paradigm of conspicuous consumption.

And this does sell. Nest supposedly moved 40 to 50,000 units per month last year in US at $250 per unit. And it’s having an impact. Supposedly 20-50% reduction in energy loads during summer peak and saved hundreds of millions of kilowatt hours of usage through its intelligence. Bravo. Tesla targets selling 35,000 cars aimed for in 2014, even at a luxury price point. That’s 35,000 fewer potential gas-guzzling luxury rides on the road. Great. I’ll take it.

In the case of hybrid vehicles in the US, as much focus has gone into creating SUVs—and positioning them as efficient but powerful vehicles, a case of Jevon’s Paradox, perhaps, where greater efficiency is used not for resource savings, but for more power.

But this isn’t the way forward, however popular and sexy these new brands are. Sustainability can't be cast as a lifestyle choice.

Behavioral change is the key, and we've done little about that beyond a good deal of greenwashing.

There is limited amount of pressure that can be exerted on companies and governments—and less incentive for investors to back the technologies of sustainability without the reality of broad-based markets generating by behavior change.

So we risk having sustainability, renewable energy, new food systems, new approaches to waste management, and new forms of transportation only being framed as more "disruptive" innovations in the market.

And that, to me, is dangerous. This can't be a conversation about Uber or self driving cars without meaningfully rethinking mobility at a systems level. This can't be a conversation only about Airbnb as a low cost travel option without also having a conversation about lowering consumption and resource sharing and material reduction, not just reallocation.

We're not going to get far thinking about doing more with less in food production and distribution if the headlines are about Soylent.


How do we drive broad-based behavioral change? One important part is giving people the ability to build new narratives around what smart sustainable futures look like for them and their community. This is a form of agency building. I've talked about flatpack futures in technology, where big companies ship glossy concept videos about seamless futures where all problems are solved. They are *for* us, but we are missing from them. We haven't written the story.

People, communities, networks have to make the conscious change. This won't happen if we only ask them to accept or reject a particular price point or lifestyle choice. And it can't just be the state. Most states lack money, will, influence, or all three. And alongside the narrative vacuum is not unrelated governance vacuum. That's why people need to fill them.

Technologists and designers and business analysts need to work with people before and during, not just after, a great idea is developed

And there's never been a better time to do it. Since the beginning of industrialization, the tools for this have never been cheaper, simpler or more accessible.

Three quick, broad examples.

We’ve been waiting for solar for 40 years or more. Finally the economics are right for the solar citizen, but at the moment they appear wrong for much of industry. Spain, for example, has seen a rollback of significant progress to expand and deeply root solar technology there. But, when government and industry pushed back on subsidies, some communities have found ways around it and taken their panels elsewhere.

Similarly, as telcos and cablecos tighten their grip on networks, communities are working out ways to create strong local networks, mesh networks, community broadband. It’s a long game, like solar, but in the end, the small will prevail against the large.

And then there’s food. A century of industrial-scale food systems are starting to crack. Communities are doing amazing things with local food systems, and here we’ve had several companies talking about how to smarten local logistical systems to get industrial efficiencies to local scale. This brings the power to feed communities home.


Stewart Brand opened his seminal work, 1968’s Whole Earth Catalog with this line: “We are as gods, we might as well get good at it.”

I respectfully disagree with Stewart Brand. We are humans. And the people many of us are working with, for and on behalf of are as well. Humans with hopes, dreams, desires, successes and humans who suffer failures. We aren’t, despite how an increasingly vocal group of new technology entrepreneurs style themselves, superhuman, with the power to terraform society.

We’re here talking about “smartups,” which you’ll hear more about: smart startups that find leverage points in infrastructure and systems where they can change how we use resources, and enable more viable futures. The smartest smartups forgo practicing the sympathetic magic of startup culture—growth hacking, rapid scaling, pivots and/or exits, and weave themselves into the communities they serve — become part of and co-design futures within those communities. They enable, instead of control or own.

There’s been a lot of talk about huge valuations of the new breed of startups, particularly in the sharing economy lately—and I guess massive VC valuations are fine and dandy. But is that what really matters to the world at large, the people you’ll ultimately serve? Set aside your market valuation and ask, what is your community valuation? How would the people you serve value you?

Let’s not forget where we are, and the scale of the problems we face. There isn’t a hack—or even a hackathon—for the existential threat of climate change, for example. There isn’t a killer app, a winning plan, a single rockstar entrepreneur who is going to solve the epic problems of the Anthropocene—adjusting a civilizational course toward sustainability. The market can help generate and support the many small solutions that will be needed, but it can’t be counted on to provide a solution or solutions for everyone. And that’s what’s required. Not winners and losers. Not a neighborhood, or state or country. We need approaches that scale for everyone.

Tomorrow does indeed compost today. What are we going to do to fertilize a flourishing future for all of us?