New technology, amazing or crazy?

Our generation faces the fastest phase of the technology development. 10 years ago, mobile phones were not popular yet. Only big mobile phones existed among business people. Back then, they held the phone on their ears talking loud in front of people and surely they caught people’s attention because phones were so rare and they can show off. But after a while there were just more and more people shouting loud in the streets, and normal people just shook their heads and mumbling ”Crazy people”.

Then later, we have wireless device like the bluetooth in-ear head phone, it turned out that people then talked to themselves in the streets, no surprise called “freaks” by normal people. Interestingly some people might put one hand on the ear even if they had that head phone, transfering ”I am talking on a phone, I am not crazy” message.

So I can’t help thinking, what will 3D interaction brings us? The same it is something that we never saw before like talk to a thing while you are walking. Will we be called crazy when we are gesturing around something? We would be lucky and amazing if there is something happening like a light on after the gesture. What if our crazy movement does not work? Will people be regarded as crazy when using a new technology?

Physical interfacing & social interaction

One of the aspects we wanted to research is what happens to social interaction when people are offered an interface to interact with computers in a physical way. Once again we reflect research done with touch screens onto 3D interaction.

As an experiment a big touchscreen was placed in a zoo to provide the visitors of more information. This screen enabled multiple users to work on it at the same time, but what happened? Because they were using a single big interface, some interactions had to take place near or in the other users’ operating area. Interfering with the ongoing interaction of another user results in a conflict (I cannot interact, or I will bother the other user).

When interfacing via a 3D interface, the borders of personal space are harder to detect. Estimating how far you have to physically move to reach a virtual object is harder when the object (and you) are scaled to fit on the screen. Especially during intense interactions (games), where people are watching at the screen and not at each other, conflicts could arise. It is the challenge of the software that is receiving the data to use that conflict and transform it into a positive experience. You could compare it to a game of twister, where the conflict is the biggest part of the game: it makes it more interesting.

Can new technologies replace old ones?

In our another post One step forward, two steps back we alreasy talked about the development of the computer system from command line to GUI to touch screen. It seems that the command line was almost replaced by GUI interface in the past. In the same interview with Mr. Derboven, we asked him if he thinks that 3D interaction would be the next step after touch screen.

The answer is no.

Starting from touch screen, it could not replace all the applications we have now because it lacks the feeling of the physical feedback. As we already experienced, we can type much faster on the traditional keyboard rather than on an ipad, because we can feel where the key is. Also in a traditional keyboard mobile phone, we can type texts much faster and we can even type without looking at the screen. Although touch screens replaced most of the mobile phones despite this disadvantage, there are some devices that cannot be replaced by touch screen. For example in sports news or sport games. Normally the performance of the player will be played multiple times with different speed after the live play. By adjusting a knob on the special video machine, one can change the speed of the video. Some people wanted to make a piece of software to realize this, but they failed. The reason is when a person operates the device, he has to look at the big screen and adjust the speed of the knob in his hand to get the best play back speed, the lack of the physical contact like in software cannot do this. Or the person should look at the play screen and also the ipad at the same time, which is not possible either.

In 3D interaction, the lack of physical contact is more severe, when we are doing gestures in front of a screen, we are doing some things in the air, which is already strange to the user. Another problem is without the buttons or keys, we don’t know the gestures we do is good or not, we do not know what will happen next. The fear of using new technology is more obvious in 3D interaction.

3D interaction might be used in some specific application to enhance the physical interact or visual effect. But cannot replace anything.

 

 

The possibility of 3D applications

Recently we had an interview with Jan Derboven, who majors in touch interfaces and gestural interaction and one of the researchers at the Centre for User Experience Research  in K.U.Leuven. We talked about the future of 3D interface application and we got some interesting conclusions.

First, let me introduce some 3D interaction applications we talked about in the interview:

• Do we still remember the April Fool’s Gmail Motion from gmail in 2011? https://mail.google.com/mail/help/motion.html
• 3D shopping based on the 2D dressing game http://www.popgals.com/fashion-shopping-girl.html

Although Gmail Motion does not exist, we asked Mr Derboven how he thinks about this application using 3D camera and gestures to operate email. The answer is absolutely no-go. 3D interaction may have a big future, but imagine you are in front of a computer, you simply want to reply to an email. Instead of typing, you have to do all kinds of weird gestures (as seen below), which is awkward and not understandable. Next to the social problem, it’s technically not feasible to distinguish all the gestures.

Talking about a girl standing in front of a screen, and a 3D camera captures her body shape, and shows it on the screen. Then the girl can just wave her hands, chose an outfit and dress on her “virtual body”. Mr Derboven said this is possible and cool, because this is not 3D interaction on its own because the basic technology that we are used to is a mirror. Traditionally girls dress up and stand in front of the mirror to see if something fits. Introducing 3D shopping into the story is not a replacement of the mirror, but an enhancement. It makes shopping easier and possibly more fun because you can try on new outfits a lot faster. Especially for the guys out there, that might be just great!

According to Mr Derboven the conclusion is: 3D interaction applications in an abstract context like Gmail Motion are surrealistic in the future. There is absolutely no link between moving around and writing an email, while 3D technology based on an existing technology that already uses physical interaction (standing in front of a mirror) is quite feasible.

3D interaction in teleconference

Nowadays the technology is better and better developed, people can have conference just online instead of troubling go a long way  to another city and go back to the place you work which saves us a lot of time.

But, somebody might ask why use 3D teconferencing?

Well there are a few reasons: first, using 3D technology gives the people who attends the meeting the feeling  of being there, just like in a room talking across a table instead of staring at the screen. Thus people can communicate like face to face. Second , the ability to “look around” the scene helps preserve information that is lost during normal 2D video conferencing. For example, imagine that you are seating in a meeting room and someone’s head is blocking your view of a whiteboard. In 3D system, as in real life, you would naturally move your head around for an unobstructed view. In traditional video conferencing, you must interrupt the meeting and ask that the remote user move his or her head.

Source:  http://ispr.info/2011/06/08/kinect-hack-creates-3d-teleconferencing/

The Holodeck

Every science fiction fan has seen it already in Star Trek: The holodeck. This can really simulate any situation that can be programmed into a real life hologram projection in a room. For those who have never seen it before, here is a short video:

Two major aspects have to be taken into account: how do you interact with such an environment, and how can it be displayed? 3D camera’s can already give an answer to the first question and that is exactly what Microsoft has tried to accomplish in a proof of concept

Maybe 3D interaction will not play a key role in your living room, but it will surely be the most important component of physical and virtual interaction.The gap between virtual and physical world is becoming increasingly smaller: we move into the virtual world, and the virtual world moves into the physical world. The former is already available as 3D games for the kinect, and also the latter is under research.

Sources

Real-life holodecks? Microsoft Kinect augmented reality room is the closest thing yet

http://research.microsoft.com/apps/video/dl.aspx?id=154571
http://gizmodo.com/holodeck/

Natural user interfaces are not natural

In 3D interaction applications like Kinect xbox 360， the core technique lies in gestural recognition and interfacing. Gestural interfaces are not new. It has been part of the interfaces scene for some years. Gestural control sounds natural, as stated by Kinect from Microsoft, when play games you just have to move your body and the system will respond to you.

However in fact, most gestures are neither natural nor easy to learn. When develop a gesture, normally the developer has to ask many people to do experiments and at last define the tolerance or boundaries and then design further. While the hard thing is that  gestures are individual and also cultural dependent. For example, when westerners travel to India, they have trouble in interpreting the Indian headshake, which appears to be a diagonal blend while in west people shake vertical for yes and horizontal for no. Moreover, research shows children from America and from Britain have different throw gestures too, because the different typical sports in both countries: American football and British soccer.

Figure 1 American throw football

 

Figure 2 British throw soccer

And a lot of interesting gestures from different countries can be found here:

http://www.youtube.com/watch?v=BM9Iu4OQXAw&feature=player_embedded

As a result, if develop a game using gestural interface, some users have to “learn” gestures, which appears to be absolute unnatural.

 

Source: Natural User Interfaces Are not Natural  by Donald A.Norman

How realistic is Minority Report

You might know the movie Minority Report in which Tom Cruise has a very futuristic computer system. But how realistic is it?

We do not consider any technical limitations for the moment, this review is solely about the interaction with the system. After inspecting all of the gestures that are performed in this fragment, the ‘copy from oversized USB stick’ action is quite clear to investigate:

Move files:

• Point to files
• Rotate arm to lock
• Circle to group
• Pinch and throw
• Stop pinching

Could you imagine learning that many steps to simply move files from one location to another? I would prefer CTRL+C and CTRL+V! Also you can notice that the majority of the actions toggle something (meaning that there is no continuous feedback visible while moving the hands). These type of actions (so called gestures) are highly discouraged (see softkinetic page in sources) because they require very strict and predetermined interactions. Another reason is that even though the procedure might be well defines, cultural or other influences result in different outcomes. Imagine in how many ways you could draw a circle in the air. The biggest problem with this fragment is that all gestures are performed after each other, and that those transfers from one gesture to another will result in unwanted triggers of other actions.

We can conclude that such a system might be realised in the future, but only for a single and a very well trained person to operate it.

Source: Softkinetic’s IISU middleware

One step forward, two steps back.

In less than 20 years we went from command line inputs, to a GUI with mouse control to touch interfaces. Today we go one step further: we wave in the air, as if we were some crazy person.

Command line required the user to know all of the commands and its parameters. Though there was a command help to give the user a starting point.

Command line in DOS

The next big revolution was the GUI of which the first popular one was introduced by Apple. This GUI removed the need to know all of the commands and grouped the functionality in nice menus.

Apple's Macintosh

And then there was the ipod/phone, something that has a single button and a whole bunch of strange gestures you have to perform on the screen. Somehow it seems that we went back to the command line era or even worse: you don’t even have a help command!

Iphone gestures

And finally we arrive at 3D interaction with computer systems. You have absolutely no idea what to do, or how to get an action done. You don’t even feel if the device is ‘listening’ to you.

3D gestural control

Somewhere in Human Computer Interaction (HCI) it went seriously wrong. This is the time to improve all of that!

Source: “Gestural Interfaces: A Step Backwards In Usability” by Donald A. Norman and Jakob Nielsen

Physical gaming

In 2006 the Nintendo Wii was available to all customers, the same year as the Playstation 3 was released but only a year after the release of the Xbox 360. Even though the Wii did not have the best performance or even the most realistic graphics, it did became the best selling console on the market thanks to a revolutionary controller.

In 2010 both Microsoft and Sony realised that they were doing it all wrong: both of them released their own interpretation of physical gaming interfaces. The Playstation Move is a hybrid between the WiiMote and a lightsaber that does not bring that much novelty. The Xbox Kinect gets rid of all extra gear needed to play games and uses the entire body to interact with the system.

But the questions is: is this just a hype or is physical gaming really the future?

Research shows that we should make a distinction between play and game: the former is a form of relaxation or entertainment while the latter is about competing and improving. One of the test subjects reported that

“When I am playing to relax and I play baseball, I swing like I would with a real baseball bat. But if I am playing to beat somebody else then I do what I need to do the movements.”

which shows that because of technical limitations nowadays, the difference between them cannot be made. But wasn’t that the case with an ordinary game controller too? Because we interact with the system in a more natural way, we also expect the system to understand what we take for granted. Non verbal communication is mostly visible in the details and those are not detectable yet, preventing the user from being fully immersed into the game.

From our own experience we see the same happening with the Kinect camera: only strong and clear moves are detected by the camera which does not give the feeling of full immersion (a feeling that you are actually there). Microsoft claims that their next camera will be greatly improved and will be capable of detecting even your fingers. An increase in resolution is not the only aspect that should be fixed! To prevent noise – picked up by all sensors – from interfering with the game, values are filtered and averaged resulting in a more artificial experience. Very often, the ways of interaction are predetermined and mapped to what the user does (I never managed to throw the bowling ball in Wii Sports away from the lane).

To summarize this post, we found that there are two major aspects in physical gaming nowadays: technical limitations and limited interpretations. The latter we illustrate once more by this quote

“As in strategy games where the artificial intelligence is adapting to the gamer’s mental and strategic skills, a movement-based game should adapt to the physical skills and mood of the gamer.”

Physical gaming has the full potential of becoming a more immersive way to interact with computer systems, but it will take some time until we are there.

Source: http://www.sciencedirect.com/science/article/pii/S187595210900007X