Communications and augmented reality: envisioning

Computers have a problem: they are, quite literally, a small window into a large on-line world and a small window in the real-world. Moreover, the on-line world bears little resemblance to the real world. So, we’re forced to constantly reconcile and interpolate the two. This divergence has also extracted human cost: the quality of communications have weakened, social interactions have twisted, and our sense of the chasm between the on-line and real worlds has strengthened. I am tired of these mental gymnastics and, if you think about it, I think you are too.

What follows is not a Luddite-technology-is-bad-we-should-go-back argument. Quite the opposite. Technology has evolved far enough where we can begin to return to the social interactions and relationships that we, as a species, have evolved. We can merge these two worlds using computers to modify our view of reality.

On-line is not off-line

The on-line world has different social rules than the real world. That’s bad because our brains are optimized for the real world. Until evolution catches up (I’m looking forward to jacking my brain directly into a network. Seriously.) we’re stuck with multiple personalities. The on-line personality flames when in person they would behave more normally; trolls argue because they can be anonymous. In the real world anti-social behavior has a clear and rich feedback channel. For example, a meeting troll will clearly see people’s reactions starting with frowns moving to that person being shut down or asked to leave.  A troll in the real-world would be socially ostracized. But on-line the social interaction channel can be  twisted and in some ways it becomes an outlet to play out socially unacceptable behavior as an outlet for socially challenged individuals.

Electronic communications have become so severely truncated to the point that it has created a new communication language. While it’s easy to pick on Twitter or SMS-length messages it misses the point. Research shows on-line communication has become shorter. I believe this has two sources. One, people prefer face-to-face interaction. Recently I sat down with groups of college students to discuss their communication patterns. One top theme was that they preferred sending a text and avoided calling because “it’s annoying.” “I’d rather talk to them face to face. It’s just easier.”

Reason two: electronic communication has been reduced to a very simple set and has discarded or degraded critical communication components such as emotion, tone, body language, and non-verbal conversation cues. As a result, we frequently misinterpret the emotional content of  messages.  “Sure” can be interpreted so many ways. A terse statement like this can connote anger or frustration or a relaxed personal style. Low quality audio or video experiences can introduce similar problems with loss of tone, intonation and other subtle vocal queues.  Another example is that eye contact is critical for conversational turn taking. Even today’s high quality video and conferencing experiences don’t provide a good solution for natural turn taking.

Users have admirably tried to overcome this missing information. For emotion we use emoticons :) and acronyms (LMAO) to prop up our messages. For conversation control my Dad has a habit of saying “go ahead” to control the conversation when we start talking over one another because we can’t see each other or hear an inhale indicating one of us is going to talk.

“Social experiences have become somewhat abstract and disconnected; the rules we developed growing up don’t apply in the same way. The Internet multiple personality has evolved.” — April Spence, Community Manager, F5 Networks

Buffer overload

Our brains have developed filters/capacity limiters in our brain help us to function in the face of attention overload. For example, computers, because of their efficiency, can overflow my communication capacity leaving me feeling conflicted: frustrated, overwhelmed, empowered, accessible, distracted, interrupted, appreciated, responsive, stressed. My filter is challenged by 300 new e-mails per day, four to five instant message sessions, at least one on-line conference session, and ten people dropping in. How do I know what’s important? Why am I included on certain messages? This also brings up a related problem: interruption management. My train of thought is constantly interrupted causing pain and frustration.  There is a clear and well documented cost to interruptions that is too rich for this short post(helpful framework).

The chasm problem

I use my iPhone as my GPS device. It’s useful — I can see where I am in real time as I follow the route. But I’m constantly translating the information on the screen to the real world. Not only do I glance up and down (distracting and dangerous), but I’m also zooming to get to the level of detail I need. The issue is that the information on these little screens doesn’t easily translate to the physical world. I am, you are the translator. We stand astride a chasm– translating the physical world for use in the on-line world and vice versa.

When you think about it, my monitor, smartphone and slate are laughably small viewing portals to a massive on-line universe. Ironically, the on-line universe spends most of its time describing the physical world!

I can point to the historical factors of why this keyhole exists. But it doesn’t need to. Let’s merge the on-line world into the real-world.

It’s time to focus on people, make technology transparent and bridge the chasm between the on-line and off-line worlds. 

Augmented reality and mediated reality aren’t new ideas. Here’s the basic concept. One one side, we have full reality. On the other is complete virtual reality (reality is fully replaced by a computer construct).

Computer modification of reality spectrum.

The first stop after reality augmented reality is the computer adding to my view of reality. Imagine GPS driving directions superimposed on your view of the road. Mediated reality takes it one step farther by modifying your view of reality.  Imagine driving on a foggy night — the computer dims the reality view and overlays a computer-generated view of the road enhancing the lanes of the road and identifying obstacles.

A couple of examples: General Motors, BMW.

I don’t look at computing as merely an object in the world, nor as a replacement for the world (virtual reality has significant problems — it’s time has not yet arrived). Integration is simply the next progression of technology becoming an integral part of how we sense and interact with reality.

I’ve shared some great example above of how GM and BMW are working with these concepts. Others in the marketplace are product-izing augmented reality such as Layar, and Word Lens.

How might augmented and mediated reality be combined with communications and collaboration? Let’s take a look.


Tom walks into the oval office to join the Business Leadership Council meeting with the President. Upon entering the office, he is automatically joined to an on-line conference for that meeting. The President begins sharing a presentation which Tom sees superimposed into the middle of the room. When he look across at the President, he sees his own key messages he wants to deliver to the President superimposed above his head. As he looks to his right to his industry colleague, he sees a list of recent e-mails from him, including one reiterating his key points. Getting what he needed, Tom turns down the detail on his view so he can focus on the people. A bit later, an alert pops up in his peripheral vision. He looks at it which brings it into view. It’s a message from his assistant with key data that is relevant to a point being made.

Kara enters the conference room and approaches the table. Looking down, she considers a projection on the table that shows the PowerPoint presentation for her meeting, and the participants currently joined. She taps to join the meeting. Knowing that their will likely be conflict in the meeting, she chooses a meeting landscape. This scene is projected onto the meeting room walls making it appear as if the meeting is happening in the middle of a green field in the Alps. The presentation is suspended in view as if on glass. Remote participants join and their video is added to the panels in the field. Kara adds a white board for a planned brainstorming session. Kara is a more data-centered person so she dials the collaboration emphasis to bring the presentations and supporting data to the fore and the video of the other participants fades into the background. Walking to the wall she uses her hands to draw a process they are considering. Carlos, working remotely from a slate device adds to the process diagram. Kara sees Carlos’ panel move to the whiteboard so she knows he’s focused on it.

Tam is traveling for business. While looking out her hotel window she sees a meeting reminder appear. She taps and joins the meeting. Her glasses dim the hotel room and she sees a virtual conference room spread out in front of her in 3d. Tam has dialed up the personal aspect of her collaboration as seeing the reaction of her team is very important to her. Video feeds in very crisp detail from her team are arrayed in front of her. As the computer is sensing the room, she sees a presentation virtually projected onto the left wall and on right wall is a whiteboard for brainstorming. Walking to the brainstorm wall, she uses touch to draw onto the wall (virtually of course).


John is an adult living with autism. He has a difficult time recognizing people’s faces and often feels overwhelmed in public due to the amount of visual stimulation. Putting on his glasses, he heads to breakfast at a local diner with some friends. As he walks, his street view is somewhat dimmed and instead he sees clean wire frames of the street. Concert bills on the telephone post that would have previously distracted him are dimmed so they don’t cause such a visual disruption.  As he arrives at the diner his glasses recognize his friends and superimpose their names on his view. With subtle modifications to his view of reality John is able to go through the day more comfortable and relaxed.

Jane has visual dyslexia which has made reading more difficult. After she was diagnosed, she received a special set of glasses. These glasses connect to a small ear bud. Now, when she opens a book or is accessing web sites she can also hear the computer reading the text which helps her comprehension of the text and learning.

I believe that computers separate us from the real world where we are better equipped to operate. The challenges I’ve highlighted above are some of the symptoms of this disconnection. I envision a future where computers are significantly more integrated into the physical world so that we can return to the patterns embedded in our social mores. I see augmented and mediated reality as very promising tools to help us integrate our two worlds.

In my next post I’ll explore the mechanics of integrating communication and modified reality.

Disclaimer: the opinions and views expressed in this blog are mine and do not necessarily state or reflect those of Microsoft.

Posted in Augmented reality, Collaboration, Communication, Mediated Reality | Leave a comment

Why “Color” matters and why you should pay attention

Two weeks ago Color ( launched a social photo sharing experience. It’s an interesting experience, but this is why you should be paying attention: Color, using interesting social features, launched a large-scale context engine. This is a huge move. Context has a strong positive effect on productivity. This means greater social connection opportunities, greater productivity for your company and fascinating new innovation opportunities. If you’re leading, you should be thinking about context in your computing life.

Why is context interesting? When communicating (the life blood of organizations and social interactions), context can significantly reduce the amount of time required to answer: who, what, why, where, when and how. If the answers to those questions are provided automatically as a natural, effortless part of the conversation then communication is just easier.

Intuitively, you already know this.  When I call my local pizza place they greet me by name and ask if I’d like the same pie as last time. That feels good. When someone introduces me to someone else at a party and shares a common connection the conversation is easier.

Let’s expand on a scenario. Imagine you are reading a spreadsheet and have a question about some sales figures. Next to those figures you see who owns that information and you start a phone call. When they pick up that call, they get your entire context. They know what file you’re looking at, the view you see, that it’s the middle of the night where you are, your role, how you’re connected to them in the organization and a mutual colleague. You, in turn, also see similar information. When your call is connected their cognitive context switching has already occurred; you’re operating in the same space.

In that example the value of context is 1) knowing what the person is calling about which reduces the amount of time needed for query/response to understand what the call is about, 2) the information is provided to the recipient; they don’t have to spend cycles remembering the figure or taking time to look it up and 3) the call initiator didn’t have to expend energy to add data to the conversation. Software did it automatically.

As you know, productivity is the measure of production efficiency. Let’s assume that the product of this example is a decision. Productivity is determined by evaluating how much input is required to arrive at the decision. As I noted, the costs for information gathering and topic query/response are lower than a conversation not supported with data. With the same output the input is lower and therefore more productive.

Why do I categorize Color as  a context engine?

Color uses this context to build a shared experience communication: “Look who else nearby loves this concert!” That shared experience enables an easy conversation (social productivity).

When a photo is taken the Color application and server (I presume) uses a variety of sensors and information to build a real time context profile of the person and the image. This could include (I don’t actually know their system) Where are they (GPS + image interpretation)? What are they looking at (the photo they took)? What are they participating in? What other devices are nearby (via Bluetooth)? What time is it? What are the environmental conditions (taken from the photo)? Who else nearby is participating?

Context is the data that can describe the situation surrounding a person, place, thing over time. Color has built an engine that can capture, transmit and represent that data.

Simple beginnings

The beginning components of context are (1) compiling data to describe the situation and (2) displaying that context in a format that is useful to the participants. The context capture points are straightforward: where did a conversation/sharing start from? What is the current task/activity?

Color’s implementation of context is the photos combined with sophisticated sensors and algorithms that they use to build your context profile. Their display, of course, is a photo sharing application that makes that contextual connection for you. Color has already announced another natural application of context: advertising. Imagine you’re at Safeco Field in Seattle. Pyramid Brewing across the street might love to advertise a post-game special to you.

Facebook has begun to venture down the context path. Their advertising engine uses context to serve more relevant advertisements and their cross-site “like” shows friend connections on non-Facebook sites.


As context usage begins to mature I expect several trends.

One, context will become an elastic, prismatic experience. For example, imagine you’re in a meeting with 10 other people. In that meeting space we have a shared context that all parties negotiate and describe. In the course of the meeting you reference an e-mail to review a past decision, that e-mail changes the shared context and adds richness with additional people connections, topic swarms, and context that is saved with the meeting. Also in the course of that meeting you have a sidebar conversation. In this simple model, you would have two context profiles. Your focus would influence (but not determine!) your primary context.

Two, context will push privacy boundaries. Context can carry a large amount of information that people may or may not be aware is being shared. Simple examples are location, permission-based information, and personal information. People will need to be able to manage this context and be aware of it. Additionally, these context engines will need to respect the regulatory and national laws.

Three, context will drive innovations in relevancy. One of the key challenges in computing is the feeling of being overwhelmed. With massive amounts of communications and information people need a way to manage the river of input. Context is one excellent method for solving the overload problem. A relevancy framework can help people keep focused and productive. Obvious solution examples are ad-hoc group formation, notification management, priority management, and managing topic pivots.

Four, your context will interpret other context using social rules. Critical to communication are social rules. Context will also be required to understand and honor the pertinent rules. For example, in some cultures hierarchy is critical in the communication process. It’s frowned upon if someone lower in the hierarchy makes direct, unsolicited contact with someone higher. Context must take these rules into account.

Five, context will begin to negotiate on your behalf. Imagine my context profile knows I’m currently using a full screen application, working on a high priority topic for a nearby deadline. If someone contacts me with a matching topic context I will likely want to communicate. It’s in-line with my task and wouldn’t require a context switch. Therefore the interruption cost would be lower and I wouldn’t have to expend energy returning to my original focus. My context profile would negotiate with a request to determine whether or not that interruption would be productive or relevant. Context will be able to negotiate to manage intrusiveness, visibility, importance and relevance.


Without much effort it’s easy to see any number of innovations in healthcare, finance, service, manufacturing, and consumer spaces. They key will be finding the right applications at the right times and proving the value propositions.

Simplistically, I think the value propositions can be broken down like this:

  • personal: social interaction, life management;
  • business: productivity increase and customer satisfaction.

Equally compelling is a technical discussion of how context is formed, communicated, interpreted, shared, stored and displayed. I’ll post on that in the future. Another interesting  discussion is the different market components of context. Who provides the engines? How do developers leverage context? How do businesses target the right context enablement scenarios for the right gain? How are people educated about context and its use? What is the commercial value of context?

Context will play a key role in future computing and I believe it to be one of the key innovations that meaningfully moves the productivity needle  in computing.

Disclaimer: I don’t know Color’s plans or their thinking. This is my interpretation of a potentially important new addition to the context space and of course my thoughts on context. And finally: the opinions and views expressed in this blog are mine and do not necessarily state or reflect those of Microsoft.

Posted in Context | Leave a comment

Getting Started

As introduction,  my profile:

Hi, my name is Chris Schindler. Currently, I’m a Senior Program Manager at Microsoft in Unified Communications. My areas of expertise are communication clients, platform, contextual communications, communications-enabled business processes and authoring + communications convergence. This blog is a space to ask big questions. And ponder answers. Among the diverse themes I’ll consider are communications, context, productivity, innovation, strategy and other grand and minute topics. Please join in the conversation. I will curate responses. Flame wars are unproductive and anathema to the discussion I’m interested in. Please bring mutual respect and a willingness to rationally discuss viewpoints.

Finally, this is my personal blog. The opinions and views expressed in this blog are mine and do not necessarily state or reflect those of Microsoft.

I look forward to our discussion.


Posted in Uncategorized | Leave a comment