Black Friday: 50% OFF from 27 NovemberBlack Friday: 50% OFF now

00:00:00 Get the deal
Sketchar
A pale wooden desk with a blank sheet of paper and a pencil

Sketchar long read · 10 minutes · touch everything

The history of augmented reality

From a prism over a sheet of paper in 1806 to the phone in your hand. Scroll it, switch the layers, draw on it.

Start with the short answer

Move the phone over the sheet: this is how Sketchar shows the sketch

The short answer

What is augmented reality?

The definition first, then two centuries of history you can touch.

Augmented reality (AR) is a technology that adds computer-generated images to your view of the real world, live and locked in place, so the virtual layer looks like part of the scene.

Three signs of AR

This three-part test comes from Ronald Azuma's 1997 survey and is still the standard definition. Virtual reality replaces the world; augmented reality keeps it and adds to it. As a working technology AR dates to 1968, when Ivan Sutherland's head-mounted display first drew computer graphics over a real room. The name is younger: Tom Caudell coined “augmented reality” at Boeing in 1990.

  1. Real and virtual togetherYou keep seeing the real world. The virtual layer is added on top of it.
  2. Interactive in real timeThe picture reacts as you move, with no noticeable delay.
  3. Registered in 3DVirtual objects stay fixed to real places, surfaces and things.
An artist's desk seen from their seat: a still life in the distance, the brass-framed prism of a camera lucida hanging low over a blank sheet of paper In front of the artist In the prism: the scene over the paper On the paper: only your drawing

1806London

The first overlay was made of glass

Long before computers, artists wanted what AR gives us today: a picture laid over the real world.

Look into the prism: there the still life lies over the paper, and the hand in it repeats yours. Trace it. On the sheet itself there is only what you draw. Illustrations made for this page.

In 1806 the English scientist William Hyde Wollaston patented the camera lucida: a small four-sided prism on a rod. Look down through its edge and you see two things at once, the scene in front of you and the sheet of paper under your hand. All that is left to do is trace.

Nothing is projected. The image exists only in the artist's eye, which is why the device needs no dark room and fits in a pocket. The astronomer John Herschel sketched landscapes with one. Henry Fox Talbot was so unhappy with his own camera lucida drawings that he set out to make nature draw itself, and ended up inventing photography.

A Victorian theatre stage with a pale reflected figure standing among the furniture

1862London

A ghost walks on a real stage

A sheet of glass, a hidden actor and a lamp: the first augmented scene shown to a paying audience.

Lamp 70%
Turn the lamp up and down: the ghost is only the reflection of a lit actor. Illustration made for this page.

On 24 December 1862 the Royal Polytechnic Institution in London staged a scene from Charles Dickens's “The Haunted Man”. A pale figure appeared among the living actors, walked through the furniture and faded away. The effect became known as Pepper's ghost.

The engineer Henry Dircks came up with the idea; the lecturer John Henry Pepper made it work on a stage and got the name. A large sheet of glass stood at an angle between the actors and the audience. A performer hidden below the stage was lit by a strong lamp, and the glass reflected that figure into the scene. Turn the lamp down and the ghost is gone.

1900Dublin

A mark that floats over the target

The next overlay came from optics for aiming: a glowing mark that stays on the target when your head moves.

The view from a fighter cockpit: blue sky, clouds below and an aircraft ahead, seen through the glass of a gunsight
The reticle stays on the target
Move the pointer or drag to shift your head. A mark painted on the glass slides off the aircraft; the reflected one stays on it. Illustration made for this page.

In 1900 the Dublin telescope maker Howard Grubb patented the reflector sight. An illuminated reticle is reflected in a piece of angled glass and focused at infinity, so it seems to hang in the distance over whatever you are aiming at.

By the Second World War reflector sights were standard in fighter aircraft. Engineers then added speed, altitude and the horizon to the same glass, and the gunsight became the head-up display. The British Blackburn Buccaneer, first flown in 1958, was one of the first aircraft fitted with one. It was information drawn over the real world, decades before anyone called it AR.

A 1960s computer laboratory with a mechanical arm hanging from the ceiling

1968Cambridge, Massachusetts

The first headset draws a cube in the room

This is the moment most historians mean when they say augmented reality was invented.

Move the pointer or drag: the cube stays fixed in the room while your view turns. The wireframe is drawn live, as it was in 1968. Background illustration made for this page.

In 1968 Ivan Sutherland and his student Bob Sproull showed a head-mounted display at Harvard. Two small cathode-ray tubes sat by the wearer's temples, and half-silvered mirrors laid their picture over the real laboratory: a wireframe cube floating in the air.

The key was tracking. A mechanical arm hung from the ceiling, measured where the head was pointing and redrew the cube to match, so it stayed put as the wearer looked around. That arm, hanging over the user's head, earned the nickname the Sword of Damocles. Three years earlier Sutherland had described the goal in an essay called “The Ultimate Display”.

1990Seattle

The idea gets its name

For twenty years the technology had no common name. An aircraft factory gave it one.

A plywood assembly board on a workshop table
Press the button: the headset shows where the next wire goes. Diagram over an illustration made for this page.

In 1990 the Boeing researcher Tom Caudell, working with David Mizell, was looking for a better way to build wire bundles for the 777. Workers laid out every wire on large boards, following paper diagrams. Caudell proposed a see-through headset that would draw the path of the next wire directly on the board, and he called the approach augmented reality. The term first appeared in print in their 1992 paper.

The decade that followed turned the name into a field. In 1992 Louis Rosenberg built Virtual Fixtures, an early interactive AR system, at a US Air Force laboratory. In 1993 a Columbia University team showed KARMA, a headset that guided its wearer through laser printer maintenance. In 1997 Ronald Azuma published the three-part definition quoted at the top of this page.

An American football field seen from a broadcast camera, with players lined up

1998Baltimore

A yellow line that isn't on the field

Millions of people saw augmented reality for the first time without knowing it had a name.

The line moves as you scroll; drag the slider to place it yourself. Players stay in front of it because only the colours of the grass are repainted. Illustration made for this page.

On 27 September 1998 ESPN aired a game between the Cincinnati Bengals and the Baltimore Ravens with something new: a yellow first-down line painted across the field. It was not there. Sportvision's 1st & Ten system drew it into the live picture.

The trick is that players run over the line, not under it. The system keeps two colour palettes: the greens of the turf, which get repainted yellow, and the colours of uniforms and skin, which never do. At launch it took a truck of equipment and a crew of four, and cost $25,000 to $30,000 a game.

1999Seattle

A printed square becomes an anchor

To place a virtual object, a computer has to know exactly where the camera is. A black square solved that.

A wooden table by a window, seen at an angle
Drag the card across the table, turn it with the slider, then cover it. Illustration made for this page.

In 1999 Hirokazu Kato, working with Mark Billinghurst at the University of Washington's HIT Lab, presented ARToolKit. It let an ordinary camera find a printed black-square marker and work out its own position relative to it, many times a second.

Once the software knows where the marker is, it can draw a 3D object standing on it. Move or turn the card and the object follows. Cover the marker and the object vanishes. ARToolKit was later released as open source and became the starting point for a generation of AR projects, among them ARQuake, an outdoor version of the game Quake built in 2000.

A European city street in daylight

2008Salzburg · Amsterdam

AR moves into the pocket

A phone with a camera, GPS and a compass could finally do what the headsets did: show information where it belongs.

Move the phone along the street: the labels exist only inside its screen. Illustration made for this page.

In October 2008 the Austrian company Mobilizy released Wikitude for the T-Mobile G1, the first Android phone. Point the camera at a street and labels for nearby places appeared over the picture. Layar followed from Amsterdam in June 2009 with an “AR browser” built on the same idea.

The next years were full of experiments. Esquire printed an AR issue in December 2009. Google revealed Project Glass in April 2012 and sold the Explorer Edition for $1,500 from 2013 until the programme closed in January 2015. In 2013 IKEA let people place furniture from its catalogue in their own rooms.

2016Redmond

A headset pins holograms to the room

The first self-contained headset that mapped the room by itself and kept virtual things exactly where you left them.

A bright living room with a sofa and a large window
Holograms
Switch the holograms on and off in the menu and drag them around the room: each one stays where you leave it. Illustration made for this page.

Microsoft showed HoloLens on 21 January 2015 and began shipping its Development Edition on 30 March 2016 for $3,000. It needs no phone, no computer and no markers. Its cameras and depth sensor scan the room, build a map of it and pin holograms to real places: leave a window on a wall, walk away, come back, and it is still there. Microsoft called it the first fully untethered holographic computer.

In July 2017 Sketchar came to HoloLens. The sketch hangs on the wall or on the sheet in front of you, and both hands stay free to draw. Headsets have followed the same path since: Apple Vision Pro went on sale on 2 February 2024 for $3,499, and in September 2025 Meta began selling Ray-Ban Display glasses for $799. The direction is the one Sutherland set in 1968: graphics that live in the world, not on a rectangle.

Sketchar on HoloLens: the story on our blog

2016Worldwide

AR becomes something everyone has tried

Two apps turned a laboratory technology into an everyday habit.

130M

downloads of Pokémon GO in its first month, July 2016

500M

downloads by September 2016, two months after launch

1B

downloads by July 2019

On 15 September 2015 Snapchat launched Lenses: face filters that track your features and redraw them live. Putting on a virtual mask became something people did every day.

On 6 July 2016 Niantic released Pokémon GO. Players walked real streets to find creatures shown on a map and through the camera. It was downloaded 130 million times in its first month, earned $206.5 million in that month, and made “augmented reality” a household phrase.

A wooden table with a sheet of paper on it

2016–2017Vilnius

The sheet itself becomes the anchor

Before ARKit existed, Sketchar taught a camera to hold on to a blank sheet of paper.

Drag the sheet and turn it, then switch between the two ways of tracking. Illustration made for this page.

A blank white sheet is the hardest thing for a camera to hold on to. It has no texture, and the hand that draws on it covers most of it. Sketchar's team took on this problem in 2016, a year before ARKit was announced, and in 2016–2017 built markerless tracking. No printed markers, only a white sheet: the camera works out its perspective and how it is turned towards the phone.

The idea is not the one ARKit would bring later. ARKit fixes a picture in the room; Sketchar glues the virtual sketch to the paper itself. Pick the sheet up, turn it, move the phone at the same time: the lines stay where they belong, even when your hand hides most of the sheet. It reached users in two steps: the first public version, in April 2017, still asked for a few hand-drawn circles as a base, and Sketchar 5.0 in May 2019 needs nothing but the sheet. The method is protected by US patent 11,030,754 B2. At the end of 2018 Huawei announced a strategic partnership with Sketchar.

How Sketchar's tracking works

2017Cupertino · Mountain View

Phones learn to see surfaces

Markers were no longer needed. The phone itself began to understand the room.

The same wooden table, ready to be scanned
Move over the table to scan it. When the surface is found, tap to place objects on it. Illustration made for this page.

Apple announced ARKit on 5 June 2017 and shipped it with iOS 11 on 19 September. Google previewed ARCore on 29 August 2017 and released version 1.0 in February 2018 for about 100 million Android phones.

Both do the same job. The phone watches how distinctive points in the picture shift as you move, combines that with its motion sensors and works out where it is. Then it finds flat surfaces, a floor or a table, and can stand virtual objects on them at the right size. This is the technology behind most AR apps today, drawing on paper included.

A desk seen from above with a blank sheet of paper A hand holding a pencil
A phone on a holder above the sheet
On the phone screen: blue guide lines of a simple drawing of a cat

2012San Francisco

AR drawing: the overlay comes back to paper

The template exists only on the phone's screen. Watch the phone and guide the hand over the paper, or draw right on the phone. Illustrations made for this page.

Sketchar

The story began with artists tracing through a prism. It comes back to the same sheet of paper, now through a phone.

AR drawing works like a camera lucida with a computer inside. You pick a picture, point your phone at a sheet of paper, and the lines appear on the sheet on your screen. Sketchar tracks the paper itself, so the lines stay in place when you move your hand, the phone or the sheet. You trace them with a real pencil and end up with a real drawing.

Try it above: watch the phone, guide the hand, then switch the template off and look at what is yours.

Sketchar's team in blue T-shirts at the AWE exhibition in Dongguan, 2017

Sketchar's founder thought up AR drawing in 2012 and launched the first AR drawing app in 2016

In June 2012 Andrey Drobitko, who went on to found Sketchar, was walking around San Francisco with a friend when the idea came to him: follow the virtual lines of a drawing on a real surface, seeing them through a phone camera. The first prototype was tested on 22 June 2012.

Phones of that time could not yet run AR well enough, so development of the app began in August 2016. On 30 December 2016 Sketchar, the first AR drawing app, launched on Product Hunt, and more than 5,000 people signed up for early access. The first iOS version was released on 17 April 2017.

Read the story on the Sketchar blog

Keep reading

Questions about augmented reality

Augmented reality adds computer-generated images to your view of the real world, live and locked in place. You still see the room, the street or the sheet of paper in front of you, with a virtual layer on top.

Ivan Sutherland is most often credited. In 1968 he built the first head-mounted display that overlaid computer graphics on a real room. The term “augmented reality” came later: Tom Caudell coined it at Boeing in 1990.

As a working technology, in 1968. The name dates from 1990 and the standard definition from 1997. Optical forerunners such as the camera lucida (1806) and Pepper's ghost (1862) are much older.

Ivan Sutherland's head-mounted display of 1968, nicknamed the Sword of Damocles after the tracking arm that hung over the user's head.

Virtual reality replaces what you see with a computer-generated world. Augmented reality keeps the real world in view and adds to it.

The phone works out its own position from the camera picture and its motion sensors, finds flat surfaces, and draws virtual objects so that they stay fixed to those surfaces as you move.

AR drawing shows the lines of a picture over your paper through a phone camera, so you can trace them with a real pencil. It is the modern version of the camera lucida.

The idea behind Sketchar came to its founder Andrey Drobitko in San Francisco in June 2012, and the first prototype was tested on 22 June 2012.

Sketchar. It launched on Product Hunt on 30 December 2016, and its first iOS version was released on 17 April 2017.

It is tracking that needs no printed or drawn markers: the camera recognises the blank sheet of paper itself and keeps the virtual drawing glued to it, even when the sheet and the phone both move. Sketchar's team built it in 2016–2017, before ARKit was announced, brought it to every user in version 5.0 in May 2019, and holds US patent 11,030,754 B2 for the method.

Share this long read

Black Friday: 50% OFF from 27 NovemberBlack Friday: 50% OFF now

00:00:00 Get the deal