Communications and augmented reality: envisioning

Computers have a problem: they are, quite literally, a small window into a large on-line world and a small window in the real-world. Moreover, the on-line world bears little resemblance to the real world. So, we’re forced to constantly reconcile and interpolate the two. This divergence has also extracted human cost: the quality of communications have weakened, social interactions have twisted, and our sense of the chasm between the on-line and real worlds has strengthened. I am tired of these mental gymnastics and, if you think about it, I think you are too.

What follows is not a Luddite-technology-is-bad-we-should-go-back argument. Quite the opposite. Technology has evolved far enough where we can begin to return to the social interactions and relationships that we, as a species, have evolved. We can merge these two worlds using computers to modify our view of reality.

On-line is not off-line

The on-line world has different social rules than the real world. That’s bad because our brains are optimized for the real world. Until evolution catches up (I’m looking forward to jacking my brain directly into a network. Seriously.) we’re stuck with multiple personalities. The on-line personality flames when in person they would behave more normally; trolls argue because they can be anonymous. In the real world anti-social behavior has a clear and rich feedback channel. For example, a meeting troll will clearly see people’s reactions starting with frowns moving to that person being shut down or asked to leave.  A troll in the real-world would be socially ostracized. But on-line the social interaction channel can be  twisted and in some ways it becomes an outlet to play out socially unacceptable behavior as an outlet for socially challenged individuals.

Electronic communications have become so severely truncated to the point that it has created a new communication language. While it’s easy to pick on Twitter or SMS-length messages it misses the point. Research shows on-line communication has become shorter. I believe this has two sources. One, people prefer face-to-face interaction. Recently I sat down with groups of college students to discuss their communication patterns. One top theme was that they preferred sending a text and avoided calling because “it’s annoying.” “I’d rather talk to them face to face. It’s just easier.”

Reason two: electronic communication has been reduced to a very simple set and has discarded or degraded critical communication components such as emotion, tone, body language, and non-verbal conversation cues. As a result, we frequently misinterpret the emotional content of  messages.  “Sure” can be interpreted so many ways. A terse statement like this can connote anger or frustration or a relaxed personal style. Low quality audio or video experiences can introduce similar problems with loss of tone, intonation and other subtle vocal queues.  Another example is that eye contact is critical for conversational turn taking. Even today’s high quality video and conferencing experiences don’t provide a good solution for natural turn taking.

Users have admirably tried to overcome this missing information. For emotion we use emoticons :) and acronyms (LMAO) to prop up our messages. For conversation control my Dad has a habit of saying “go ahead” to control the conversation when we start talking over one another because we can’t see each other or hear an inhale indicating one of us is going to talk.

“Social experiences have become somewhat abstract and disconnected; the rules we developed growing up don’t apply in the same way. The Internet multiple personality has evolved.” — April Spence, Community Manager, F5 Networks

Buffer overload

Our brains have developed filters/capacity limiters in our brain help us to function in the face of attention overload. For example, computers, because of their efficiency, can overflow my communication capacity leaving me feeling conflicted: frustrated, overwhelmed, empowered, accessible, distracted, interrupted, appreciated, responsive, stressed. My filter is challenged by 300 new e-mails per day, four to five instant message sessions, at least one on-line conference session, and ten people dropping in. How do I know what’s important? Why am I included on certain messages? This also brings up a related problem: interruption management. My train of thought is constantly interrupted causing pain and frustration.  There is a clear and well documented cost to interruptions that is too rich for this short post(helpful framework).

The chasm problem

I use my iPhone as my GPS device. It’s useful — I can see where I am in real time as I follow the route. But I’m constantly translating the information on the screen to the real world. Not only do I glance up and down (distracting and dangerous), but I’m also zooming to get to the level of detail I need. The issue is that the information on these little screens doesn’t easily translate to the physical world. I am, you are the translator. We stand astride a chasm– translating the physical world for use in the on-line world and vice versa.

When you think about it, my monitor, smartphone and slate are laughably small viewing portals to a massive on-line universe. Ironically, the on-line universe spends most of its time describing the physical world!

I can point to the historical factors of why this keyhole exists. But it doesn’t need to. Let’s merge the on-line world into the real-world.

It’s time to focus on people, make technology transparent and bridge the chasm between the on-line and off-line worlds. 

Augmented reality and mediated reality aren’t new ideas. Here’s the basic concept. One one side, we have full reality. On the other is complete virtual reality (reality is fully replaced by a computer construct).

Computer modification of reality spectrum.

The first stop after reality augmented reality is the computer adding to my view of reality. Imagine GPS driving directions superimposed on your view of the road. Mediated reality takes it one step farther by modifying your view of reality.  Imagine driving on a foggy night — the computer dims the reality view and overlays a computer-generated view of the road enhancing the lanes of the road and identifying obstacles.

A couple of examples: General Motors, BMW.

I don’t look at computing as merely an object in the world, nor as a replacement for the world (virtual reality has significant problems — it’s time has not yet arrived). Integration is simply the next progression of technology becoming an integral part of how we sense and interact with reality.

I’ve shared some great example above of how GM and BMW are working with these concepts. Others in the marketplace are product-izing augmented reality such as Layar, and Word Lens.

How might augmented and mediated reality be combined with communications and collaboration? Let’s take a look.


Tom walks into the oval office to join the Business Leadership Council meeting with the President. Upon entering the office, he is automatically joined to an on-line conference for that meeting. The President begins sharing a presentation which Tom sees superimposed into the middle of the room. When he look across at the President, he sees his own key messages he wants to deliver to the President superimposed above his head. As he looks to his right to his industry colleague, he sees a list of recent e-mails from him, including one reiterating his key points. Getting what he needed, Tom turns down the detail on his view so he can focus on the people. A bit later, an alert pops up in his peripheral vision. He looks at it which brings it into view. It’s a message from his assistant with key data that is relevant to a point being made.

Kara enters the conference room and approaches the table. Looking down, she considers a projection on the table that shows the PowerPoint presentation for her meeting, and the participants currently joined. She taps to join the meeting. Knowing that their will likely be conflict in the meeting, she chooses a meeting landscape. This scene is projected onto the meeting room walls making it appear as if the meeting is happening in the middle of a green field in the Alps. The presentation is suspended in view as if on glass. Remote participants join and their video is added to the panels in the field. Kara adds a white board for a planned brainstorming session. Kara is a more data-centered person so she dials the collaboration emphasis to bring the presentations and supporting data to the fore and the video of the other participants fades into the background. Walking to the wall she uses her hands to draw a process they are considering. Carlos, working remotely from a slate device adds to the process diagram. Kara sees Carlos’ panel move to the whiteboard so she knows he’s focused on it.

Tam is traveling for business. While looking out her hotel window she sees a meeting reminder appear. She taps and joins the meeting. Her glasses dim the hotel room and she sees a virtual conference room spread out in front of her in 3d. Tam has dialed up the personal aspect of her collaboration as seeing the reaction of her team is very important to her. Video feeds in very crisp detail from her team are arrayed in front of her. As the computer is sensing the room, she sees a presentation virtually projected onto the left wall and on right wall is a whiteboard for brainstorming. Walking to the brainstorm wall, she uses touch to draw onto the wall (virtually of course).


John is an adult living with autism. He has a difficult time recognizing people’s faces and often feels overwhelmed in public due to the amount of visual stimulation. Putting on his glasses, he heads to breakfast at a local diner with some friends. As he walks, his street view is somewhat dimmed and instead he sees clean wire frames of the street. Concert bills on the telephone post that would have previously distracted him are dimmed so they don’t cause such a visual disruption.  As he arrives at the diner his glasses recognize his friends and superimpose their names on his view. With subtle modifications to his view of reality John is able to go through the day more comfortable and relaxed.

Jane has visual dyslexia which has made reading more difficult. After she was diagnosed, she received a special set of glasses. These glasses connect to a small ear bud. Now, when she opens a book or is accessing web sites she can also hear the computer reading the text which helps her comprehension of the text and learning.

I believe that computers separate us from the real world where we are better equipped to operate. The challenges I’ve highlighted above are some of the symptoms of this disconnection. I envision a future where computers are significantly more integrated into the physical world so that we can return to the patterns embedded in our social mores. I see augmented and mediated reality as very promising tools to help us integrate our two worlds.

In my next post I’ll explore the mechanics of integrating communication and modified reality.

Disclaimer: the opinions and views expressed in this blog are mine and do not necessarily state or reflect those of Microsoft.


About ideabrdg

I'm a Senior Program Manager at Microsoft in Unified Communications. My areas of expertise are commmunication clients, platform, contextual communications, communications-enabled business processes and authoring communications convergence. This blog is a space to ask big questions. And ponder some answers. Among the diverse themes I'll consider communications, context, productivity, innovation, idealism and other grand and minute topics. Please join in the conversation.
This entry was posted in Augmented reality, Collaboration, Communication, Mediated Reality. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s