What is Augmented Reality?

Imagine if it was possible to browse life the way we browse the Web. Imagine envisaging the world through your screen in a brand new way. Imagine augmenting characters in the monotony of life. Well, gone are the days where we prefixed the above statements with ‘imagine’, now it’s time to replace it with ‘experience’. Welcome to the world of Augmented Reality. Augmented Reality mesmerized the world by viewing the world from an eye of a computer.

Augmented Reality is the ability to insert and overlay digital and virtual information in the real world. As revolutionary as it sounds the technology commenced in 1962, when Morton Heilig patented Sensorama a simulator with visuals, sound, vibrators and smell. Later in 1990 Tom Caudell coined the term Augmented Reality. It combines Virtual Reality with the real world in real time.

The use of AR is much more than enhancing gaming graphics. It is also used by the following

  • Military- HUDs (Heads Up Display) and HMD (Head Mounted Display) are two types of ARs used by the military. HUDs are more often used by fighter pilot and HMDs are used by troops on the ground.
  • Medicine- AR is used by students to practice surgery. Complex medical conditions are visualised and solved using AR. 3-Dimensional screening of various objects is done by AR.
  • Navigation- GPS is enhanced by AR using the camera of our smartphone to give us a better route
  • Archaeology- Used in museums to replicate the structure and to display information. Reconstruct how the model looked like ages ago.
  • Training- Simulations are used to train employees to improve their dexterity.

The drawbacks of Augmented Reality

  • Invasion of Privacy: With the rise in the usage of AR devices invasion of privacy will have steep rise. Imagine you looking at an anonymous person and suddenly the biodata of the person appears, just by pointing your smartphone at them. Pokemon Go is another example of how a stranger can invade your privacy wherein the proxemics of people were invaded massively.
  • Motion Sickness: If the viewing system lacks proper spatial cues then the person can suffer from motion sickness.

What is Heads Up Display (HUD) and can it be augmented with Real-time Data?

Heads Up Display (HUDs as commonly referred) are display units which allow the user to monitor important information on a display without the distraction of putting his or her head down. The origin of HUD Technology is actually derived from Jet Fighters. Back then the technology was quite orthodox as compared to the current scenario. They were roughly based on a machine gun reflector sight which illuminated the target in the site when properly aligned. This technique was eventually adopted by the Royal Air Force back in the day.

HUDs are classified into four generations as follows

  • First Generation: They used a Cathode Ray Tube and a Phosphorous coated screen. It allowed night pilots to monitor targets directly via HUD radar.
  • Second Generation: They used solid lights such as LEDs to project images onto a pilot’s windscreen from an LCD. They are commonly found on commercial aircraft, due to their high cost then they were unlikely to be found in cars.
  • Third Generation: They don’t require projection at all anymore but instead utilise a specially coated viewing surface called an optical waveguide which manipulates visual light spectrum waves allowing automakers to display readouts like speed, GPS navigation, music controls and even web content directly on their proprietary HUD systems on the windshield
  • Fourth Generation: They use lasers to display high definition images directly on to clear surfaces making them very compact, portable and versatile.

Heads Up Display isn’t only useful while operating a vehicle or solving crimes. In fact, there is a humungous list of programs that transforms a person’s smartphone camera into a miniature HUD.

The sole purpose of the Augmented Reality based heads-up display is to reduce the distractions of the driver by providing all the relevant information prevalent in the field of view which the driver is supposed to know while driving.

With ever increasing traffic density in large cities, even a single moment of distraction while you are behind the wheel can cause something dangerous to happen. Therefore it is mandatory to drive scrupulously. The advanced prototype for safety where the aim is to make sure that the driver’s eye is always on the road is given by Augmented Reality Heads up Display.

This HUD analyses the traffic conditions with augmentations for the driver. It differs from the usual windshield HUD as the information is more user-friendly as it interacts with the surroundings. It expands the field of view as compared to normal HUD, AR-HUD can present information up to 7 meters from the current position of the driver. It inserts coloured graphics and covers a width of 130cm with a section height of 60cm in the driver’s field of view.

Another prospect of AR-HUD is the Digital Mirror Device (DMD) which is a movie projector industry based product which is now integrated into a vehicle. It enables animated sequences to be displayed. Useful data is projected into the real traffic scenes on the basis of radar data and camera images. This is particularly helpful in freeways when the navigation indicates a change of lane. The symbols move from the current lane to the exit lane and issue the message directly. The colour code for all the navigational systems is blue. The AR-HUD supports the driver if the vehicle is in danger of unintentionally drifting out of a lane.

Some of the features for which augmentation can be created are:

Adaptive Cruise Control (ACC)

Blind Spot Detection (BSD)

Navigation Routing (NavR).

Lane Departure Warning (LDW)

Vehicle Proximity detection/Headway control.

Blind Spot Detection.

Pedestrian Detection and Alerts.


When ACC is enabled, an orange coloured crescent-shaped marking in the AR-HUD highlights that the system is taking its bearings from the vehicle at front. Thus the driver knows that the vehicle’s assistance system is operational which creates trust between the system and the driver. Thus AR-HUD creates acceptance for these assistance systems. There are red coloured poles which indicate the blind spots while driving. The AR-HUD reflects the navigation information in the real exterior view denoting the driver which turn they have to take without looking back and forth between the navigational screen and the real world. AR-HUD can also help illumination areas in the windshield for possible pedestrian detection in low light conditions where vehicle sensors are very much able to perceive environment than the human eye.

For developing an AR-HUD you need versatility among HUD-Technology, Data Handling and Driver Assistance Systems to create augmentation which aptly suits your current driving situation. Taking the dynamics of the vehicle into account the control unit of the system calculates a model of the real exterior view from the driver’s perspective and can thus generate the augmentations at the correct visual point.

Architecture of an Augmented Reality System:

An Augmentation of the Real Word needs sensor’s and Integration with existing ADAS systems to enable higher interpretation of environment around the car. A sample architecture looks like this:

Augmented Reality architecture

The entire Augmented Reality System can be divided into:

  • Sensors.

  • Control System.

  • Graphics.

  • Mechanical Construction.

Sensors:  The sensors used can be Camera or can be data(Perception Model) shared by other ADAS systems in the Car. This allows the Control system to detect and perceive environment so that important features of the environment can be obtained. Some systems can also utilize existing infrastructure of Lidar, Radar and Camera.

Control System: This system receives all the data from the sensors and stores all the graphics and Calibration matrices of the Lens and camera to interpret and locate the features in world space of the car. This system does feature extraction, object recognition, data fusion, tracking, overlay generation, State Machine.

Construction: This system takes care of the construction of the projector, integration and projection on the glass for effective Augmentation. This often involves study on the lens, Projector, Angles of Projection and Anti Glare. Check out this article to know more on the construction of HUD System.

Graphics: This is usually a part of the Control system, but often needs a lot of research on the colour combination, text instructions, integration of sound and Graphics so as to avoid the distraction of driver.

Common Challenges in Designing Augmented Reality based HUD system in Cars:

  • Overcoming of Delay in Projection. The detection of Objects, processing and projection takes some computation time, this computation time is often visible when we try to mark the objects in real time. This needs the algorithms to be very efficient, optimized so as to have delays under 200ms.
  • Correct Projection of Markers: Projection of Markers becomes often difficult due to the position of eyes, for short/tall people the projection can change and is often difficult to compensate for the changes in the projection.
  • Detection of Objects, lanes and Traffic signs.  All signs related to traffic signs, lane colours, objects change with different countries so its often difficult to make a universal product that can suit all countries, that means a large dataset and can be difficult to get.
  • Universal Acceptance.  With different cars the Front windshield and integration with existing infrastructure can be a challenge.
  • Integration with other ADAS Features. It’s often necessary to integrate many functionalities to cut costs, this can sometimes lead to a generic system/calibration that can be difficult to work with.
  • Compensation during jerks/Speed Bumps. During the jerks, the system needs to take care of the markers, which need to be compensated for the changing pitch.

For Construction and working of HUD check this article. LINK

Visit Homepage for latest updates and Technical articles: Automotive Electronics

Visit  our forum to discuss : Automotive Electronics Forum

This article was written by Aditya  and Eeshan for any correction or guest article mail us at contact@automotivelectronics.com



Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.