Jarrell Pair

During the late 1990’s I became intrigued by the concept of linking digital information to the physical world to create a “real world wide web.” I was in part introduced to this concept by J.C Spohrer’s paper “Information In Places” (Spohrer, J. C. 1999. Information in places. IBM Syst. J. 38, 4 (Dec. 1999), 602-628). At the time, pursuing this idea was challenging due to a lack of internet connected mobile platforms. However, over the next several years, this idea became more viable as camera equipped 3G mobile phones and Bluetooth GPS units became available.

In 2006, I received seed funding to prototype systems for annotating real world locations with images and text gathered by mobile device users. These annotations were tagged with GPS data and could be viewed as “Geo Post It” notes providing recent, contextually relevant photos and text regarding these locations.

A system was designed in which the notes could be viewed in online digital maps or overlaid on the real world through augmented reality (AR) binoculars or on a physical map using a webcam or AR handheld device.

I evaluated a number of mobile phones and PDA’s for use in this project.  We primarily used one of the most feature rich smartphones at the time, the I-mate JasJar manufactured by HTC. Its relatively high quality camera and ability to pair with a Bluetooth GPS, made it ideal for our purposes. I worked with Stephan Themis to implement a Windows Mobile 5 application which was used to capture GPS registered photos and text notes.

Front view of I-mate JasJar/HTC Universal running Geo Post It application

Rear view of I-mate JasJar/HTC Universal showing integrated camera

Bluetooth GPS unit

Stephan Themis also developed a web accessible map for displaying Geo Post It notes. Links to online maps and screenshots showing notes made with GPS linked smartphones are below.

Geo Post It in Barcelona, click image to view full size

Geo Post It in Barcelona: Click here to view all notes in Barcelona on an online digital map.

Geo Post It in Venice, California: Click image to view full size

Geo Post It in Venice, California: Click here to view all notes in Venice, California on an online digital map.

A proof of concept prototype was developed to view Geo Post It’s overlaid on their real world locations through augmented reality binoculars. The system consisted of 5DT Virtual Binoculars connected to a tablet PC running AR software developed by Anton Treskunov. Video see through AR capabilities were added to the binoculars through the use of a front mounted camera. To simplify tracking and registration issues, the system was designed to be used on a stationary tripod. A Bluetooth GPS unit and an Intersense Intertiacube were mounted to the binoculars’ case. These sensors were intended to provide position and orientation data though they were not functionally integrated into the proof of concept prototype.

AR binoculars rear view	Front view showing AR video see through camera
View of Geo Post It's seen through AR binoculars	AR binoculars connected to tablet PC running AR software
GPS and inertial sensors mounted on AR binoculars

An AR system was also prototyped which combined the familiarity of a physical map with the versatility of a digital version.  In this proof of concept , a camera was mounted above a physical map which incorporated ARTag tracking fiducials. These AR tracking markers facilitated the viewing of Geo Post It’s and other 3D graphics overlaid onto the physical map when viewed on an adjacent LCD monitor. This alternate viewing and interaction modality would be useful in supporting collaborative work and discussions among multiple users. Geo Post It’s could also be seen when the physical map was viewed with a handheld AR device.  For this task, we utilized a Sony Ultra Mobile PC. When its rear mounted camera detected an ARTag fiducial, Geo Post It’s and 3D graphics would appear in the handheld’s video see through AR display.  This proof of concept also utilized AR software developed by Anton Treskunov.

AR map view

Geo Post It viewed on map with handheld device

In the years following this project, mobile phone chipsets became available which incorporated accelerometers, GPS receivers, and compasses.  Consequently, smartphones with these sensors emerged as a compelling platform for advancing the concepts prototyped in this project.  Work in this area advanced rapidly and has been brought to the mass market through AR world browser apps such as Layar, Wikitude, and Junaio.