A hot, boring, lazy Friday just turned exciting. As we reported earlier, Microsoft has been sued by Ohio based Impulse Technology Limited. Impulse claims that the Xbox Kinect infringes 7 patents the company owns. I have been closely following Microsoft’s patent portfolio for quite a while. Microsoft has been filing for several (and I mean) several patent relating to body movements, gestures and Kinect. I have done a quick preliminary search to find some of the patents. There are more, the search on USPTO is more broken than their patent system.
Going through the patent applications and Impulse’s statement, the case seems more against how Kinect Fitness leverages body scanning, room environment for health stuff. The patent claims are broad and also cover some of Kinect’s key operations. Here’s what Impulse has sued Microsoft for:
The first claim in 3 of the 7 patents (6,430,997 6,876,496 & 6,308,565) is the same. Way to go patent lawyers.
A testing and training system comprising a continuous tracking system for determining changes in an overall physical location of a player, in a defined physical space; a computer operatively coupled to the tracking system for updating in real time a player virtual location in a virtual space corresponding to the physical location of the player in the physical space, for updating a view of the virtual space, and for providing at least one indicia of performance of the player moving in the physical space, wherein the at least one indicia is or is derived from a measure of a movement parameter of the player.
Patent 6,430,997 has 30 claims, some key ones:
A reactive power training system comprising: a reactive training device which provides cues to elicit responsive changes in an overall physical location of a subject in at least two dimensions; and a resistive training device; wherein the reactive training device and the resistive training device are used in a training sequence.
A method of reactive power training for a user using a reactive training device and one or more resistive training devices, comprising: performing a training sequence, wherein the performing the training sequence includes: exercising utilizing the reactive training device, to perform reactive training bouts; and performing resistive strength enhancing activities on the one or more resistive training devices, to perform resistive training bouts; wherein the reactive training bouts are alternated with the resistive training bouts.
marker worn by the subject, wherein the reactive training device tracks movement of the marker.
a transmitter worn by the subject which transmits information to the reactive training device without being physically connected to the reactive training device.
wherein the information includes an identification of the subject.
wherein the information includes the subject’s heart rate.
wherein the transmitter acts as a marker, allowing the reactive training device to track location of the subject.
Claim #2 and #3 are Kinect Fitness. The rest of the claims I’ve listed also relate to Kinect Fitness. However, Kinect does not use any such transmitter device, sensor does all the magic. Kinect Fitness however does receive feedback about the user’s body. The Kinect sensor can be considered as the reactive device in this case.
Patent 6,765,726 has 16 claims:
A multiuser physical assessment system comprising: a first tracking system for determining changes in a first overall physical location of a first user within a first physical space; a second tracking system for determining changes in a second overall physical location of a second user within a second physical space; and at least one computer operatively coupled to the tracking systems for updating user virtual locations in a virtual space corresponding to the physical locations of the users; wherein performance of at least one of the users in the virtual space is modified so as to handicap one of the users relative to the other user.
[â€¦] first and second displays operatively coupled to the at least one computer, wherein the displays display respective views of the virtual space; wherein the displays display different views of the virtual space; wherein the tracking systems are both continuous three-dimensional tracking systems; and wherein the at least one computer updates in real time the user virtual locations in the virtual space.
[â€¦] wherein the first view includes first virtual representations of at least parts of both of the users; and wherein the second view includes second virtual representations of at least parts of both of the users.
All claims in this application are related to how Kinect works.
Patent 6,876,496 has 20 claims:
the computer provides at least one indicium of performance of the player moving in the physical space, wherein the at least one indicium is or is derived from a measure of a movement parameter of the player.
the view of the virtual space includes a player icon located at the player virtual location.
the updating of the protagonist virtual location is made in response to the changes in the physical location of the player, such that the virtual protagonist and the player engage in an interactive task.
These claims, again, go on to cover how Kinect tracks the users and displays it. Instead of an icon, Kinect shows a translucent 2D figure.
Patent 7,359,121 has 27 claims. Key claims:
A method for prompting a user to engage in a physical activity, the method comprising: causing displaying of a representation of the user on a monitor, wherein the displaying of the representation includes moving the representation of the user to reflect movement of the user; causing displaying of one or more virtual objects on the monitor, wherein the displaying of the one or more virtual objects includes displaying the one or more virtual objects to prompt physical motion of the user [â€¦]
[â€¦] displaying the virtual objects of the method includes prompting the user to perform an interception task that includes the representation of the user moving to have the representation of the user intercept at least some of the targets.
Prompting user to engage in a physical activity, is more of what Microsoft & Kinect game advertisers are doing. Having said that, this patent is about Kinect, a moving representation of the user is the 2D figure we see in games.
Patent 7,791,808 with 20 claims:
the tracking system continuously tracks an overall physical location of the user in a defined physical space corresponding to the virtual space; wherein the computer updates in real time a user virtual location in the virtual space corresponding to the physical location of the player in the physical space; and wherein the viewpoint of the view is from the user virtual location.
the view is a first person view
he first person view is a first person perspective view
the tracking system tracks the translations in three dimensions
the tracking system includes a beacon worn by the user
the tracking system includes a camera
All claims, except user wearing a beacon are how Kinect works.
Patent 6,749,432 with 18 claims has a funny title: Education system challenging a subject’s physiologic and kinesthetic systems to synergistically enhance cognitive function
A method of educating comprising: prompting a person to engage in body core movement which elevates the person’s metabolic rate; and prompting the person to engage in a cognitive academic learning task while the person’s metabolic rate is elevated, wherein the prompting to engage in a cognitive academic learning task includes displaying a view of a virtual space, and wherein the promoting to engage in body core movement and the prompting to engage in a cognitive academic learning task both include promoting the person to engage in a body core movement cognitive academic learning task.
he updating a view includes updating a first person perspective view of the virtual space from the virtual location
prompting to engage in a body core movement cognitive academic learning task includes prompting the person to engage in interactive challenges that involve manipulation of objects in a virtual world by body core movement of the person in a defined physical space
further including ascertaining the degree of metabolic rate elevation
including adjusting the prompting to engage in body core movement to maintain the metabolic rate elevation at a desired level
ascertaining the degree of metabolic rate elevation includes monitoring the person’s heart rate
These claims would be akin to Kinect Fitness including a freaky 3D doctor telling you why you need to do certain exercises. This patent application deals with Kinect Fitness and maybe broadly apply to other games.
Patent 6,208,565 has a 100 claims. Most of these claims are slight modifications of claims in the other patents or exactly the same. Once again, way to go patent lawyers.
These are the 7 patents Impulse has sued Microsoft. Now let’s take a look at a fraction of the Kinectgesture patents Microsoft has applied for.
Patents 20110032336 20110109724 20100194872 deal with body scanning:
A device for capturing depth information of a scene, the device comprising: a camera component, wherein the camera component receives a depth image of the scene; and a processor, wherein the processor executes computer executable instructions, and wherein the computer executable instructions comprise instructions for: receiving the depth image of the scene from the camera component; determining whether the depth image includes a human target; scanning the human target to locate at least a first joint, if the depth image includes a human target [â€¦]
A 3D user scanning device, using cameras.
Patent 20110035666 is about displaying the position of a user on the screen. (Something Impulse has claimed.) The abstract of the patent says:
A capture device may capture a user’s motion and a display device may display a model that maps to the user’s motion, including gestures that are applicable for control.
A claim from the patent:
A method for determining whether to provide feedback, the method comprising: receiving image data for a scene, wherein the image data comprises depth data representative of a target’s motion or position in a physical space; comparing the depth data representative of the target’s motion or position in the physical space [â€¦]
Patent 20110085705 is about scanning props (or objects as Impulse has mentioned in their applications):
acquiring one or more color images from one or more color sensing image sensors, the step of performing prop tracking includes using color information from the one or more color images to help track the second target.
the step of performing skeletal tracking includes receiving position information regarding the second target, the position information is taken into consideration when determining the first position of the first target.
the step of performing prop tracking includes receiving position information regarding the first target, the position information is taken into consideration when determining the second position of the second target.
receiving orientation information from the first prop, the step of performing prop tracking uses the orientation information to help track the first prop.
Kinect allows you to scan your own skateboard and use it in a game.
Patent 20100195867 is about tracking the user as he moves. Impulse has this in their patent claims. Microsoft’s claims include focusing on the user by ignoring non-target items from the background and some pretty smart algorithms to track the user’s movements:
[â€¦] analyzing the observed depth image with a hand-identifying algorithm configured to identify hands on the human target; and increasing relative confidence of the exemplar pose if the exemplar pose more closely places hands in a same location as the hand-identifying algorithm [â€¦]
removing non-target background information from the observed depth image includes removing depth image information outside of a sphere surrounding the target
the source includes a depth camera
the source includes stereo cameras
Patent 20100278384 complements the above patent and is about estimating the human body’s pose.
Patent 20100195869, 20100197395, 20100197400, 20100197391, 20100197392 & 20100197399 are titled Visual Target Tracking and are related to tracking the human body movements. Claim from 20100195869:
A method of tracking a human target, the method comprising: representing the human target with a machine-readable model configured for adjustment into a plurality of different poses
One of the newer patents I came across yesterday is about multi-user tracking. This is also there somewhere in Impulse’s claims. Patent 20110175809
A processor-implemented method for tracking multiple people in a motion capture system, comprising the processor-implemented steps of: tracking at least one group which includes a first group, the first group comprises peoples’ bodies in a field of view of the motion capture system, the tracking includes detecting the peoples’ bodies in the field of view
Patent 20100306685 is having the user’s avatars on screen as feedback about the user’s physical movements. Claim from the application:
method for providing feedback to a user about a computing environment, the method comprising: recognizing, using an image-based capture device, a presence of a first user in a capture area; associating a first avatar with the first user and displaying the first avatar on a display screen; recognizing aspects of the first user within the capture area; and modifying an appearance of the first avatar to provide feedback to the first user about at least one of capabilities, features, rights or permissions of the first user in the computing environment.
Most of the Microsoft patent I’ve listed here are yet to approved. Whereas Impulse has approved patents. How that plays out will have to be seen. The patents I’ve listed are only those I could find, there are more. Kinect is a product developed using 3D camera trcking technology developed by PrimeSense. I have not looked into patent held by them or whether the patents were transferred to Microsoft.