Sixth Sense technology
Sixthsense is a wearable gestural interface device developed by Pranav Mistry,
a PhD student from MIT that augments the physical world around us with
digital information and lets us use natural hand gestures to interact
with that information.
Watch the following video of Pranav Mistry presenting sixth-sense device to get a clear idea about sixth sense device.
SixthSense prototypes cost approximately $350 to build (not including the computer), the main cost being the micro-projector. Don't feel disappointed if you can't spend $350. You can still build the software and play with it for free.
Sixth sense device's software is now open source. I appreciate this move from Pranav Mistry, but the source code is full of bugs and exceptions as it is yet in beta version. I have edited the code to make sure it is compatible with both Visual Studio 2008 and Visual Studio 2010.
When you click on ‘Add Markers’, the screen starts blinking and after 4-5 secs it stops with a still picture so that you can add those markers on your hand as new markers. You can simply drag-select the colored marker on the screen by mouse cursor. Once you have all four markers (O, P, M, and N) added the software will start tracking the four markers. You can set the thresholds to better the tracking. You can save those markers and can reuse in subsequent executions, or can add new markers every time. Check this video.
Test: You can see whether the four markers are being tracked and moving while you are moving those particular markers that is on the fingers.
Draw: The red marker (M) draws when the yellow marker (N) is hidden.When both the markers are visible you can move the mouse pointer to select different colours or eraser etc.
Photo: Make a rectangle with your pointer fingers and thumbs and when the gesture is recognized,the countdown starts Remove you hands from the camera frame to capture the desired photo. (The countdown lasts for 3 secs)
Gallery: Displays the photos that you have taken.
Mail: Displays a list of email from outlook (required that MS Outlook be running for this to work)
Weather: Displays a weather app that can be localised to your area.
Stock: Displays 16 stocks values real time, use the slider at the bottom of the projected display to set the time scale.
Effects: Play with beautiful wavy lines and explore your imagination.
Newspaper: It projects newspaper stories and related videos from Youtube or other web sites,directly onto the surface of the paper.
Book: For book-recognition,the program gets activated with a gesture, and the system snaps a photo of the book, compares it with book-cover images it finds online, computes a match, and retrieves and projects the ratings.
Clock: An Analog clock will be projected by doing a circle gesture.
Suggested Product: Logitech QuickCam Pro for notebooks ($23)
Note:If you are just checking the software, your laptop's in-built web-camera will work.
Suggested Products: You can buy either laser (AAXA - $110, Microvision - $200) or L.E.D (3M MPro110 - $270) projectors.
Suggested Product: Any 1”X1” first surface mirror
Suggested Product: Microphone of Logitech QuickCam pro for notebooks ($23)
Suggested Product: Any Windows computer
Now that you have all these pieces, you need a way to combine them. I recommend using Lego strips to form the base. The projector, camera, and mirror assembly can be directly put onto this base. You can also use Velcro to combine the products.
How to make your own sixth sense device?
You must be desperate to get your own sixth sense device. It is not available for sale so you cannot buy it from anywhere. The only option you have is to make it yourself. Yes, You can make it yourself!SixthSense prototypes cost approximately $350 to build (not including the computer), the main cost being the micro-projector. Don't feel disappointed if you can't spend $350. You can still build the software and play with it for free.
Sixth sense device's software is now open source. I appreciate this move from Pranav Mistry, but the source code is full of bugs and exceptions as it is yet in beta version. I have edited the code to make sure it is compatible with both Visual Studio 2008 and Visual Studio 2010.
Compiling and running the software component
The sixth sense device runs on windows platform and majority of the code is written in C++ and C#. Make sure you have Visual Studio, Direct X, Adobe Flash player (active x) and MS Outlook installed on your computer.-
You first need to download the following source code(zip file) and
extract it. If you don't want to compile the software and save time then
you can directly download the already compiled Sixth Sense software binary(.exe). If you download the exe file then skip steps 2 and 3.
Download Sixth Sense device source code
-
Open WUW01.sln (which is present in the folder "code"). If you are using
VS 2008 then instead of opening WUW.sln, open
SixthSense\code\WUW01\WUW01.csproj
-
Press F5 to debug it. If you debug it successfully, a black screen
appears with a little square box on the right just hover your mouse
around it and you will find Option tabs such as Camera, Tokens and Apps.
Note: If you do not have the DIRECTX installed on your machine ( as stated in the requirements ) , it doesn't make any difference. Just comment out the following line nos. 41, 42 and 74 in the WUW.cs file. This works beacuse only a matrix class of the DIRECTX namespace is declared in the code but never used. So you can safely comment out these lines and remove the DIRECTX references.
I have solved various bugs and exceptions including the black box error. Also I have solved XamlParseException which gave following error:
Cannot create instance of 'WeatherReaderUI' defined in assembly 'WeatherReaderMVC, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.I have commented the line no. 111 and 1863(shown below) to solve the problem. Check if the program compiles and runs successfully in your system after removing the comment.
111: WPFControl_Weather01.UserControl1 Control_weather = new WPFControl_Weather01.UserControl1();1863: elementHostWeather.Child = Control_weather;
Camera:
- Here you can select the camera you want to use from the drop down options.
- If you are having multiple cameras you can select from those option or if you have single camera it will automatically start capturing.
- You can also adjust your camera settings (Brightness,contrast,..etc) by clicking on "Adjust Camera Properties". (Note: Sometimes an exception occurs while clicking on Adjust Camera Properties button)
Tokens:
This version of code uses colored fingertips as markers. One need 4 different colors in order to run most of the Apps. I recommend to use Red, Yellow, Blue and Green (or colors that are different enough). Matt colored tape/objects works better as they have less reflective changes.When you click on ‘Add Markers’, the screen starts blinking and after 4-5 secs it stops with a still picture so that you can add those markers on your hand as new markers. You can simply drag-select the colored marker on the screen by mouse cursor. Once you have all four markers (O, P, M, and N) added the software will start tracking the four markers. You can set the thresholds to better the tracking. You can save those markers and can reuse in subsequent executions, or can add new markers every time. Check this video.
Apps:
There are three ways to launch any application. Apps tab can let you choose the apps you want to load and run. One can also load apps from ‘Menu’. ‘Menu’ is triggered using Namaste gesture. Once on Menu screen you can choose the App by hand-click (Hand Click gesture works by moving the O marker and hiding P marker.Test: You can see whether the four markers are being tracked and moving while you are moving those particular markers that is on the fingers.
Draw: The red marker (M) draws when the yellow marker (N) is hidden.When both the markers are visible you can move the mouse pointer to select different colours or eraser etc.
Photo: Make a rectangle with your pointer fingers and thumbs and when the gesture is recognized,the countdown starts Remove you hands from the camera frame to capture the desired photo. (The countdown lasts for 3 secs)
Gallery: Displays the photos that you have taken.
Mail: Displays a list of email from outlook (required that MS Outlook be running for this to work)
Weather: Displays a weather app that can be localised to your area.
Stock: Displays 16 stocks values real time, use the slider at the bottom of the projected display to set the time scale.
Effects: Play with beautiful wavy lines and explore your imagination.
Newspaper: It projects newspaper stories and related videos from Youtube or other web sites,directly onto the surface of the paper.
Book: For book-recognition,the program gets activated with a gesture, and the system snaps a photo of the book, compares it with book-cover images it finds online, computes a match, and retrieves and projects the ratings.
Clock: An Analog clock will be projected by doing a circle gesture.
Hardware Components
Now you have the needed software. Let's check out the various hardware components.Camera
The camera is the main input device of the SixthSense system. The camera acts as a digital eye of the system. It basically captures the scene the user is looking at. The video stream captured by the camera is passed to computer which does the appropriate computer vision computation.Suggested Product: Logitech QuickCam Pro for notebooks ($23)
Note:If you are just checking the software, your laptop's in-built web-camera will work.
Projector
The projector is the main output device of the SixthSense system. The projector unit used in prototype runs on a rechargeable battery. The major functions of the projector is to project graphical user interface of the selected application onto surfaces or walls in front.Suggested Products: You can buy either laser (AAXA - $110, Microvision - $200) or L.E.D (3M MPro110 - $270) projectors.
Mirror
The mirror reflects the projection coming out from the projector and thus helps in projecting onto the desired locations on walls or surfaces. The user manually can change the tilt of the mirror to change the location of the projection. For example in application where we want the projection to go on the ground instead of the surface in front, we can change the tilt of the mirror to change the projection.Suggested Product: Any 1”X1” first surface mirror
Microphone
The microphone is an optional component of the SixthSense. It is required when using a paper as a computing interface. When the user wants to use a sheet of paper as an interactive surface, he or she clips the microphone to the paper. The microphone attached this way captures the sound signals of user’s touching the paper. This data is passed to computing device for processing. Later, combined with the tracking information about user’s finger, the system is able to identify precise touch events on the paper. Here, the sound signal captured by the microphone provides time information whereas the camera performs tracking.Suggested Product: Microphone of Logitech QuickCam pro for notebooks ($23)
Computer
Of course, you will need a computing device (i.e laptop/netbook). The camera, the projector and the microphone are connected to this device using wired or wireless connection.Suggested Product: Any Windows computer
Now that you have all these pieces, you need a way to combine them. I recommend using Lego strips to form the base. The projector, camera, and mirror assembly can be directly put onto this base. You can also use Velcro to combine the products.
