Nimble Sense
More actions
| Nimble Sense | |
|---|---|
| |
| Basic Info | |
| Type | Input Device, Motion Tracker |
| Subtype | Hands/Fingers Tracking |
| Platform | Oculus Rift DK1, Oculus Rift DK2 (headset mounts); desk mount |
| Developer | Nimble VR |
| Announcement Date | October 28, 2014 (Kickstarter launch) |
| Release Date | Canceled |
| Price | US$99 (Kickstarter early-bird pledge for the first 500 backers, camera plus DK2 mount; never sold at retail) |
| Website | https://web.archive.org/web/20141030234321/http://www.nimblevr.com/ (archived) |
| Image | |
| Field of View | 110° |
| Audio | |
| Camera | ToF Camera |
| Device | |
| Sensors | Time-of-flight depth camera with a pmd photonic mixer device and an eye-safe infrared laser |
Nimble Sense is a 3D camera mounted on Oculus Rift that tracks the motion of the user's hands. By creating a point cloud in front of the user, Nimble Sense camera along with its software enables the skeletal tracking of hands tailored for Virtual Reality.[1][2] The device was developed by Nimble VR, a San Francisco startup previously known as 3Gear Systems, and was offered through a Kickstarter campaign that opened on October 28, 2014, with the camera offered to early backers for US$99.[3][2]
On December 11, 2014, Nimble VR announced that they were acquired by Oculus VR, the creator of Oculus Rift (Platform).[4] The Kickstarter for Nimble Sense was canceled. The Nimble VR team joined Oculus to work on VR full-time, and the camera was not released.[5] The campaign had raised $135,511 from 1,076 backers against a goal of $62,500 when it was canceled.[6][2]
History
3Gear Systems
Nimble VR began as 3Gear Systems, founded in 2012 by Rob Wang, Chris Twigg and Kenrick Kin.[4] In a 2012 introduction on the company blog, 3Gear described Wang as an MIT PhD in computer vision who had created hand-tracking algorithms and a color patchwork glove, Twigg as a Carnegie Mellon PhD who had worked in visual effects research and development at Industrial Light and Magic, and Kin as a UC Berkeley PhD who had developed multitouch tools for creating 3D environments at Pixar.[7] Wang's glove research, written with Jovan Popović at MIT, used an ordinary webcam and a cloth glove printed with a distinctive color pattern to track hand pose in real time; the project page states that 3Gear Systems was started "based in part on this technology."[8][9]
3Gear's first product was a software development kit for finger-level gesture recognition. It ran on two Microsoft Kinect cameras mounted on an aluminum frame above a desk, a setup that TechCrunch reported cost about $330 in off-the-shelf parts. The APIs were available in C++ and Java, and the beta SDK was free for hobbyists and researchers, with commercial licenses negotiated separately.[10][7] Early backers included K9 Ventures and Uj Ventures, and the company also received National Science Foundation research funding.[7] In April 2014 TechCrunch reported that 3Gear had raised $1.9 million from K9 Ventures, Intel Capital and CrunchFund and was working on a USB camera device for gesture control of PCs.[11] A biography of Wang published by the Seattle chapter of the Chinese Institute of Engineers states that the company won "Best Startup" at Augmented World Expo in 2013.[12]
In its farewell note, the team wrote that its work "started off with color gloves, evolved into markerless tracking with multiple Kinect cameras, and eventually led to the development of our own tiny 3D camera, the Nimble Sense, that could be mounted on an Oculus Rift."[13] According to GamesBeat, the company had spent two years building skeletal hand-tracking software for every depth-sensing camera on the market before it rebranded from 3Gear to Nimble VR and designed a camera specifically for VR.[3]
Kickstarter campaign
Nimble VR revealed Nimble Sense and launched its Kickstarter campaign on October 28, 2014, with a funding goal of $62,500.[3][2] "The Oculus Rift brought your eyes and your head into VR," Wang said at the launch. "We're bringing your hands."[3] The first 500 backers could get the camera and a DK2 mount for $99, and the company expected to ship in June 2015.[2] Display Daily reported that the team had obtained quotes from three experienced camera module makers in the United States and Asia before starting the campaign.[14]
By the time of the Oculus announcement the campaign had more than doubled its target.[15] The final total was $135,511 from 1,076 backers.[6]
Acquisition by Oculus and cancellation
On December 11, 2014, Oculus announced that Nimble VR, the Stockholm-based 3D mapping company 13th Lab and motion capture researcher Chris Bregler were joining the company. Oculus described Nimble VR's focus as "high-quality, low-latency skeletal hand tracking, which has the potential to be part of a great VR user experience," and said that "Nimble VR, 13th Lab, and Chris will all be winding down their existing projects to focus on VR full-time at Oculus as part of both product engineering and Oculus Research."[4][5] Financial terms were not disclosed.[15]
Nimble VR canceled the Kickstarter shortly before it was due to end, and Road to VR reported that backers' pledges were refunded.[5] In their announcement, Wang, Twigg and Kin wrote: "We're excited not only to continue to push at the boundaries of input and user experience in VR, but to do so with the resources and means to make a bigger impact on a larger audience."[13] Engadget noted that the deal followed Oculus's earlier hiring of John Carmack and Michael Abrash and its acquisition of controller designer Carbon Design.[16]
Design and hardware
Nimble Sense is a time-of-flight depth camera. An infrared laser emits pulses of invisible light, and an infrared camera with a 110 degree field of view senses the light reflected from the surroundings, using the known speed of light to calculate the distance to points in the scene.[2] The sensor is a photonic mixer device from the German company pmdtechnologies, paired with an eye-safe laser.[14][17] Display Daily described the camera as about the size of a pack of gum.[14]
Wang said the camera "captures a dense 3D point cloud every 20 milliseconds that can be used to bring not only hands, but arms, legs, and even your desk into VR."[2] Nimble VR said its hand tracking worked best for objects 10 to 70 cm in front of the user, which the company called the natural zone of interaction for the hands.[2] The company pitched it as a device for a headset, but also said it could be mounted above a monitor or placed on a desk.[1] The Kickstarter version was to ship with a DK2 mount that replaced the headset's cable cover, with a DK1 mount and a desk mount available at other pledge levels.[2]
| Specification | Nimble Sense (Kickstarter prototype, 2014) |
|---|---|
| Sensing method | Time-of-flight depth sensing with an infrared laser |
| Sensor | pmdtechnologies photonic mixer device |
| Field of view | 110 degrees |
| Capture rate | One 3D point cloud every 20 milliseconds |
| Optimal range | 10 cm to 70 cm from the user |
| Mounts | Oculus Rift DK2 (replaces cable cover), Oculus Rift DK1, desk |
| Kickstarter price | $99 for the first 500 backers, camera and DK2 mount |
| Planned shipping | June 2015 (canceled) |
The table lists figures published by Nimble VR during the Kickstarter campaign, as reported by Road to VR and Display Daily.[2][14]
Software and features
The camera was paired with Nimble VR's skeletal hand tracking software, which the company said delivered low-latency, accurate hand input.[1] The company's website listed uses such as picking up and turning over virtual objects, operating a user interface without removing the headset to reach a keyboard or mouse, and casting spells in games.[1] Developers were offered an API with the skeletal tracking software and code examples, and Nimble VR planned to release four open-source Oculus Rift demos.[14][2] SlashGear reported that the tracking software itself was not open source, but the example programs were.[18]
Because the camera captured a point cloud of everything in front of it, it could also show the wearer a rough view of the real surroundings. Engadget described this as "a blurry version of your own body and the world around you," clear enough to drink from a cup while wearing the headset.[17]
Reception
In a November 2014 hands-on, Engadget's Ben Gilbert reported that the arm and hand tracking worked without perceptible lag in the demos. In one demo, he raised a hand to aim and opened it to fire lightning at enemies. In another, Wang picked up virtual dice, spun them, placed them in a cup and poured them back onto a table. Gilbert noted that nothing could be felt when grabbing a virtual object, because the system had no haptic feedback; Wang told him the team was not against a glove that approximated feedback, but that a glove was another peripheral to put on.[17] SlashGear made a similar point about the lack of tactile feedback, contrasting it with devices such as Dexmo that provide it.[18]
Road to VR's Ben Lang wrote that latency figures had not been published but that the demonstration video looked passable on a first look. The article repeated an endorsement from the Kickstarter page by developer Blair Renaud of Iris VR: "I've tried just about every VR control scheme out there, and I have to say that the Nimble Sense is by far the most promising I've tried so far."[2]
Legacy
Oculus did not release Nimble Sense after the acquisition.[5] Hand tracking reached Oculus hardware five years later: in December 2019 the Oculus Quest received controller-free hand tracking as an experimental feature in system software version 12. In its report, UploadVR wrote that "the first indication of Facebook's interest in finger tracking for VR was revealed in late 2014," when the company acquired Nimble VR, and noted that Nimble's technology was not just software but a dedicated depth sensor. The same report quoted Robert Wang, then an engineering manager, at Oculus Connect 6, where he said the Quest feature "only scratched the surface of what we can do with hand tracking".[19] Wang's biography states that his team shipped hand tracking on the Oculus Quest platform in 2019.[12]
See also
References
- ↑ 1.0 1.1 1.2 1.3 "Nimble VR: Bring your hands into virtual reality". Nimble VR (archived by the Internet Archive). Nimble VR. 2014. https://web.archive.org/web/20141030234321/http://www.nimblevr.com/. Retrieved 2026-09-27.
- ↑ 2.00 2.01 2.02 2.03 2.04 2.05 2.06 2.07 2.08 2.09 2.10 2.11 Ben Lang (2014-10-28). "Nimble Sense Kickstarter Aims to Bring Time-of-Flight Depth-sensing Tech to VR on the Cheap". Road to VR. https://www.roadtovr.com/nimble-sense-kickstarter-virtual-reality-oculus-rift-time-of-flight-input/. Retrieved 2026-09-27.
- ↑ 3.0 3.1 3.2 3.3 Dean Takahashi (2014-10-28). "Nimble VR aims to let you use your hands in virtual reality". GamesBeat. https://gamesbeat.com/nimble-vr-aims-to-let-you-use-your-hands-in-virtual-reality/. Retrieved 2026-09-27.
- ↑ 4.0 4.1 4.2 "Nimble VR, 13th Lab, and Chris Bregler join Oculus". Oculus Blog. Oculus VR. 2014-12-11. https://www.meta.com/blog/quest/nimble-vr-13th-lab-and-chris-bregler-join-oculus/. Retrieved 2026-09-27.
- ↑ 5.0 5.1 5.2 5.3 Scott Hayden (2014-12-12). "Oculus Acquires Nimble VR and Two Others for Computer Vision Expertise". Road to VR. https://www.roadtovr.com/oculus-acquires-nimble-vr-13th-lab-chris-bregler-computer-vision/. Retrieved 2026-09-27.
- ↑ 6.0 6.1 Nicole Arce (2014-12-13). "Oculus VR Buys Hand-tracking Tech Expert Nimble VR: Why We're So Excited". Tech Times. https://www.techtimes.com/articles/22038/20141213/oculus-vr-buys-hand-tracking-tech-expert-nimble-vr-why-were-so-excited.htm. Retrieved 2026-09-27.
- ↑ 7.0 7.1 7.2 "Hello from 3Gear Systems". 3Gear Systems Blog. 3Gear Systems. 2012-10-02. http://threegearsystems.blogspot.com/2012/09/hello-from-3gear-systems.html. Retrieved 2026-09-27.
- ↑ Robert Y. Wang, Jovan Popović. "Real-Time Hand-Tracking with a Color Glove". MIT CSAIL. http://people.csail.mit.edu/rywang/hand/. Retrieved 2026-09-27.
- ↑ Robert Y. Wang, Jovan Popović (2009). "Real-time hand-tracking with a color glove". ACM Transactions on Graphics, vol. 28, no. 3. https://doi.org/10.1145/1531326.1531369. Retrieved 2026-09-27.
- ↑ Kim-Mai Cutler (2012-10-05). "3Gear Systems Hacks Kinects To Create The Future of Gestural Computing". TechCrunch. https://techcrunch.com/2012/10/05/3gears-gestural-computing. Retrieved 2026-09-27.
- ↑ Ryan Lawler, Colleen Taylor (2014-04-02). "With $1.9 Million In Funding, 3Gear Systems Is Building Gesture Control That Actually Works". TechCrunch. https://techcrunch.com/2014/04/02/3gear/. Retrieved 2026-09-27.
- ↑ 12.0 12.1 "Dr. Robert Y. Wang Bio". Chinese Institute of Engineers, Seattle Chapter. https://www.cie-sea.org/mr-robert-y-wang-bio/. Retrieved 2026-09-27.
- ↑ 13.0 13.1 "Nimble VR is joining Oculus". Nimble VR (archived by the Internet Archive). Nimble VR. 2014-12-11. https://web.archive.org/web/20141212014105/http://nimblevr.com/. Retrieved 2026-09-27.
- ↑ 14.0 14.1 14.2 14.3 14.4 "Nimble Sense Hopes to Bring Hand Gestures into Virtual Reality". Display Daily. 2014-11-12. https://displaydaily.com/nimble-sense-hopes-to-bring-hand-gestures-into-virtual-reality/. Retrieved 2026-09-27.
- ↑ 15.0 15.1 Greg Kumparak (2014-12-11). "Oculus Acquires A Hand-Tracking Company And An Augmented Reality Company". TechCrunch. https://techcrunch.com/2014/12/11/oculus-acquires-a-hand-tracking-company-and-an-augmented-reality-company. Retrieved 2026-09-27.
- ↑ Ben Gilbert (2014-12-11). "Oculus is buying a company that brings hands into virtual reality". Engadget. https://www.engadget.com/2014-12-11-oculus-nimble-vr.html. Retrieved 2026-09-27.
- ↑ 17.0 17.1 17.2 Ben Gilbert (2014-11-04). "Touching virtual reality with Nimble VR's Sense camera". Engadget. https://www.engadget.com/2014-11-04-nimble-sense-hands-on.html. Retrieved 2026-09-27.
- ↑ 18.0 18.1 JC Torres (2014-11-04). "Nimble Sense Brings Your Hands Into VR Without Gloves". SlashGear. https://www.slashgear.com/nimble-sense-brings-your-hands-into-vr-without-gloves-04353787/. Retrieved 2026-09-27.
- ↑ David Heaney (2019-12-10). "Oculus Quest Built-In Apps Get Controller-Free Hand Tracking, SDK Out Next Week". UploadVR. https://www.uploadvr.com/oculus-quest-finger-hand-tracking/. Retrieved 2026-09-27.