Jun Rekimoto

Jun Rekimoto (暦本 純一) is a Japanese computer scientist who works in human-computer interaction, augmented reality and human augmentation. He has been at Sony Computer Science Laboratories (Sony CSL) since 1994, where he founded the Interaction Laboratory in 1999, became deputy director in 2011, chief science officer in 2023 and senior fellow in 2026. From 2007 to March 2026 he was a professor in the Interfaculty Initiative in Information Studies at the University of Tokyo.

His early work at Sony CSL produced NaviCam (1994-1995), a palmtop computer with a video camera that recognized color-coded ID tags and overlaid information on a video see-through screen, and CyberCode (1996), a printed 2D visual tag that a cheap camera can read to obtain both an ID number and the tag's 3D position. CyberCode was later used in Sony products including the VAIO C1 laptop and the PlayStation 3 card game The Eye of Judgment (2007). His other systems include Pick-and-Drop, Augmented Surfaces, HoloWall and SmartSkin. He was elected to the ACM SIGCHI Academy in 2007 and received ACM UIST Lasting Impact Awards in 2013 and 2017.

Reviewed 13 October 2026. Career dates, education, awards, NaviCam, CyberCode, Traxion, JackIn and other paper details checked against the Sony CSL profile and CV, IEEE Spectrum, Asian Scientist, University of Tokyo pages, the original papers and Crossref DOI records. About review dates.

Early life and education

Rekimoto grew up in Tokyo. According to a 2014 IEEE Spectrum profile, he bought a programming textbook in 1971, at age 10, and, having no access to a computer, wrote programs in a notebook and imagined how they would run. He studied information science at the Tokyo Institute of Technology, receiving a B.A.Sc. in 1984, an M.Sc. in 1986 and a Ph.D. in 1996. The Spectrum profile says an article by Alan Kay on graphical interfaces drew him to human-computer interaction while he was a student there.

Career

NEC and the University of Alberta

After his master's degree Rekimoto joined NEC, working in its research and development division on interfaces for Unix. His CV lists him at NEC's Software Engineering Laboratories from 1986 and as a visiting scientist at the University of Alberta, Canada, in 1992-1993. In Canada he experimented with virtual reality but, according to IEEE Spectrum, was disappointed by its unrealistic graphics and began proposing instead to layer information over the real world, the approach now called augmented reality. With Mark Green he published "The Information Cube: Using Transparency in 3D Information Visualization" at the Third Annual Workshop on Information Technologies and Systems (WITS '93) in 1993.

Sony Computer Science Laboratories

Rekimoto joined Sony CSL in Tokyo in 1994. In the IEEE Spectrum profile, Rekimoto recalls that the lab's founder, Mario Tokoro, told him the freedom he would have there also meant responsibility, with "no excuses". His CV lists him as a senior researcher from 1998, and in 1999 he founded and became director of Sony CSL's Interaction Laboratory. He became deputy director of Sony CSL in 2011, chief science officer in 2023 and senior fellow in 2026. Sony CSL has also described him as research director of its Kyoto research laboratory. He is a co-founder of Koozyt, Inc.

University of Tokyo

In 2007 he became a professor in the Interfaculty Initiative in Information Studies at the University of Tokyo, a post Sony CSL lists as ending in March 2026. A 2018 Asian Scientist profile described him as director of the human augmentation lab at the Interfaculty Initiative; Rekimoto told the magazine that the lab uses augmented and virtual reality, "like head-mounted displays, telepresence robots and drones", and the profile called him a proponent of using technology to augment human capacity, "be it cognitive or physical". He gave his final lecture as a University of Tokyo professor on 28 February 2026 at the Ito Hall of the Ito International Research Center, under the title "Homo Cyberneticus: Human-AI Integrationの時代へ" ("Homo Cyberneticus: toward the era of human-AI integration"). Sony CSL announced on 2 July 2026 that he had been named Professor Emeritus of the University of Tokyo.

Research

Head tracking for fish tank VR

At the 1995 IEEE Virtual Reality Annual International Symposium (VRAIS '95), Rekimoto presented a head tracker that used only computer vision on an ordinary workstation to report the position of the user's head in real time. It was intended for desktop fish tank virtual reality, so that a user could use 3D systems without putting on any equipment. According to the paper, an experiment by the author suggested that the tracker can improve users' understanding of complex 3D structures shown on the display.

NaviCam and handheld augmented reality

NaviCam was a Sony CSL prototype and an early step toward what Rekimoto called "augmented interaction": a palmtop computer with a small video camera that recognized the user's situation from the real-world environment. A first technical report and a video were produced in 1994. In the UIST '95 paper "The World through the Computer", written with Katashi Nagao, the device detected color-code IDs placed in the environment and superimposed situation-specific messages on its video see-through screen. The authors argued that combining ID recognition with a portable see-through display addressed problems of both ubiquitous computing and earlier AR systems.

In a second 1995 paper Rekimoto called this configuration the "magnifying glass approach": instead of a head-mounted or head-up display, the user holds a palmtop-sized video see-through device and moves it around to compare real and augmented views. That version of NaviCam used a miniature gyro sensor for orientation and vision-based ID recognition to find the device's rough position and the objects in front of it. In his experiment the palmtop display was faster to use than the head-up configurations and received higher subjective scores. A journal version appeared in Presence: Teleoperators and Virtual Environments in 1997. IEEE Spectrum described a demonstration in which scanning a code at the lab's front door made NaviCam show arrows on the floor pointing to meeting rooms and offices, and noted the device's resemblance to later smartphones. In 2013 the UIST '95 paper received the ACM UIST Lasting Impact Award.

Rekimoto applied the same handheld approach to shared AR in TransVision (1996), in which two or more people each hold a palmtop-sized see-through display and look at the same computer-generated 3D model on a real tabletop. Because no head-mounted displays are worn, participants can see each other and use gestures while they collaborate.

Matrix codes and CyberCode

At APCHI '98 Rekimoto described "Matrix", a method in which a printed square 2D matrix code both identifies a real-world object and serves as a landmark for registering graphics on the camera image. He noted that such codes cost almost nothing to print and attach, whereas the magnetic or ultrasonic trackers then used in AR were often inaccurate and had a limited tracking volume.

Sony CSL dates CyberCode, the related visual tag system, to 1996. The system binarizes the camera image, searches for candidate codes, checks the cell pattern, and estimates the camera's 3D position from the shape of the code in the image, so that graphics can be overlaid on the real scene with only a camera and printed fiducial markers. In the DARE 2000 paper written with Yuji Ayatsuka, CyberCode encodes 24 or 48 bits (excluding error-correction bits), fewer than other 2D barcodes, so that it can be read by low-resolution CCD or CMOS cameras of 100,000 pixels or fewer; the paper notes that some of the applications had been released as bundled software on a Sony notebook PC with a built-in camera. Sony CSL lists the uses as the VAIO C1 camera laptop (1998), an AR kiosk installed in the Sony Building in Ginza from 2006, and The Eye of Judgment on PlayStation 3 in 2007. His speaker biography in the ISMAR 2013 program described CyberCode as "the world's first marker-based AR system".

Interactive surfaces and multi-computer interaction

Much of Rekimoto's work in the late 1990s and early 2000s moved interaction off the single screen. Pick-and-Drop (UIST '97) let a user pick up an object on one display with a pen and drop it on another display; the pen stores nothing itself, but a pen ID and a network pen manager give the illusion that it carries the object. Augmented Surfaces (CHI '99, with Masanori Saitoh) used camera-based object recognition so that laptops could treat table and wall projections as a spatially continuous extension of their screens; a technique called "hyperdragging" moved data between computers, and digital data could be attached to physical objects such as a videotape. HoloWall (UIST '97, with Nobuyuki Matsushita) was a wall display sensitive to fingers, hands, bodies and objects. SmartSkin (CHI 2002) used capacitive sensing with a mesh-shaped antenna to recognize several hand positions and shapes and the distance between the hand and the surface, with all sensing built into the surface; table-size and tablet-size prototypes were built.

Haptics

"Ambient Touch: Designing Tactile Interfaces for Handheld Devices", written with Ivan Poupyrev and Shigeaki Maruyama for UIST 2002, received the ACM UIST Lasting Impact Award in 2017.

Traxion (UIST 2013) produces a virtual force sensation from an illusion. An actuator made of an electromagnetic coil, a metal weight and a spring is driven by an asymmetric signal, and the user feels the device pull or push in one direction even though it has no mechanical link to the ground. The device measures 35.0 by 5.0 by 7.5 mm and weighs 5.2 g; suggested uses included a finger-mounted pedestrian navigation aid and an untethered input device with virtual force. Rekimoto told MIT Technology Review that testers said "it feels as if there's some invisible string pulling or pushing the device"; the device was shown in the SIGGRAPH 2014 Emerging Technologies program.

Human augmentation and telepresence

In his keynote at ISMAR 2013 in Adelaide, "From Augmented Reality to Augmented Human", given on 3 October 2013, Rekimoto used the term "Augmented Human" for research that enhances people's physical as well as intellectual abilities, and called for a shift "from human-computer-interaction to human-computer-integration". The talk covered wearable eye sensing, flying cameras used as external eyes, and hand control through functional electrical stimulation.

Flying Head, developed with Keita Higuchi, is a headset that lets the wearer see through the camera of a small unmanned aerial vehicle and control it with head movements; it was shown at SIGGRAPH Asia 2012 Emerging Technologies and published in the CHI 2013 extended abstracts.

The JackIn project, with Shunichi Kasahara, is a form of human-to-human telepresence in which a remote user "jacks in" to another person's first-person view; according to Asian Scientist, the name was inspired by William Gibson's novel Neuromancer, in which a hacker "jacks in" to cyberspace. JackIn Head adds a head-worn omnidirectional camera that captures 360-degree spherical video, stabilizes it to reduce the shakiness that can cause dizziness, and streams it so that remote viewers can look around independently of the wearer's head direction. It was presented at SIGGRAPH Asia 2015 Emerging Technologies and at ACM VRST 2015, and a journal version with Shohei Nagai appeared in IEEE Transactions on Visualization and Computer Graphics in 2017.

Professional service

His CV lists him as program co-chair of UbiComp 2005, program chair of the Japanese WISS workshop in 2000 and 2001, and associate editor of the International Journal of Human-Computer Studies from 2003 to 2005. He served on program committees including UIST 2000, CSCW 2000, IEEE VR in 2000 and 2001, and the International Symposium on Augmented Reality (ISAR) in 2001. From 2017 to 2023 he was a research supervisor for the Japan Science and Technology Agency PRESTO program "The Future of Humans and Interactions".

Awards and recognition

Year	Award
1998	Multi-Media Grand Prix Technology Award, Multimedia Contents Association of Japan
2000	iF Interaction Design Award
2003	Japan Inter-Design Award
2005	iF Communication Design Award
2007	Elected to the ACM SIGCHI Academy
2012	Good Design Best 100 Award
2013	ACM UIST Lasting Impact Award, for "The World through the Computer" (UIST '95)
2017	ACM UIST Lasting Impact Award, for "Ambient Touch" (UIST '02)
2018	Asahi Shimbun Award, National Invention Awards

Selected publications

Year	Title	Venue
1995	"A Vision-Based Head Tracker for Fish Tank Virtual Reality"	VRAIS '95
1995	"The World through the Computer: Computer Augmented Interaction with Real World Environments" (with Katashi Nagao)	UIST '95
1996	"TransVision: A Hand-held Augmented Reality System for Collaborative Design"	VSMM '96
1997	"NaviCam: A Magnifying Glass Approach to Augmented Reality"	Presence, vol. 6, no. 4
1997	"Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments"	UIST '97
1998	"Matrix: A Realtime Object Identification and Registration Method for Augmented Reality"	APCHI '98
1999	"Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments" (with Masanori Saitoh)	CHI '99
2000	"CyberCode: Designing Augmented Reality Environments with Visual Tags" (with Yuji Ayatsuka)	DARE 2000
2002	"SmartSkin: An Infrastructure for Freehand Manipulation on Interactive Surfaces"	CHI 2002
2013	"Traxion: A Tactile Interaction Device with Virtual Force Sensation"	UIST '13
2017	"JackIn Head: Immersive Visual Telepresence System with Omnidirectional Wearable Camera" (with Shunichi Kasahara and Shohei Nagai)	IEEE TVCG, vol. 23, no. 3