Augmented reality document management

Published: 8 April 2024
Contributors: Molly Hayes, Amanda Downie

Augmented reality (AR) refers to the real-time integration of digital information into a user’s environment. AR technology overlays content onto the real world, enriching a user’s perception of reality rather than replacing it.

AR devices are equipped with cameras, sensors and displays. This can include smartphones and tablets creating mobile AR experiences or ‘wearables’ like smart glasses and headsets. These devices capture the physical world and then integrate digital content (for example, 3D models, images or videos) into the scene, blending digital and virtual worlds.

Augmented reality technologies have a range of applications in commerce, manufacturing and entertainment.

Explore the IBM Institute for Business Value's report on how generative AI can give companies the tools to meet rising consumer expectations.

Subscribe to the IBM newsletter

Augmented reality works through the deployment of camera-equipped hardware such as smart glasses or heads-up displays.

Mobile devices like iPads or iPhones, which are already built with technologies such as GPS, accelerometers and sensors, are particularly compatible with augmented reality applications and can make the technology more accessible to the average consumer. In recent years, several technology companies have released APIs such as Apple’s ARKit and Google’s Arcore, which facilitate the development of mobile AR applications for Android and iOS.

Though the types of data and sensors a particular AR software might draw on vary, the basic steps of the AR process are:

An AR device receives a stream of video content from a user’s field of vision, sensing the environment and tracking physical objects in view. This might include the collection of data from accelerometers, gyroscopes, GPS or lasers alongside a video stream to track the user’s position and orientation.

AR software scans and processes this environment—this might mean connecting to an object’s digital twin, a 3-D copy of the object stored in the cloud. It might also mean using artificial intelligence (AI) to recognize the physical object. During this process, AR software processes the information it has received, identifying objects and environmental features that can be augmented. This might involve sensors on the physical object sending data to a digital twin, or combining tracking data with other information such as a product’s price or equipment lifecycle data.

Information streamed from the AR software is displayed on the AR device, superimposing computer-generated content onto the user’s field of vision. Digital information is rendered in the correct perspective and orientation, appearing to the user as if the object is physically present. The user follows interaction instructions, sending commands through a touchscreen, with physical gestures, or by voice. These commands are received by the software and sent to the superimposed digital object so it can be manipulated by the user.

Augmented reality, virtual reality and mixed reality are all technologies that change the user’s perception of the physical world with computer-generated content.

AR overlaps digital content onto the real world. Using augmented reality, users still see and interact with their physical environments while experiencing supplementary digital information overlaid onto their field of vision. Augmented reality can be accessed on a mobile device through augmented reality apps, or with AR glasses. Examples include:

• Mobile mapping apps that provide an augmented reality experience with directional arrow and street name overlays, helping users navigate the real world on foot
• Ikea Place, an app that allows users to virtually place furniture in their homes by using AR
• Social media filters such as Snapchat filters that allow users to alter the images seen through their phone cameras in real-time

Virtual reality immerses users in a digital environment, typically by using a VR headset, head-mounted display or VR goggles. Unlike AR, VR completely replaces the physical world, surrounding users with a 360-degree view of computer-generated environments. Examples include:

• The metaverse, an immersive virtual reality digital world developed by the company Meta
• The VR Museum of Fine Art, which allows users to virtually visit famous museum exhibits from across the world

Mixed reality, which can be considered an umbrella term for some advanced forms of AR, creates experiences where digital content interacts with the real world, blending virtual objects with the physical environment. In mixed reality, a user can interact simultaneously with physical and virtual items. Examples include:

• AR video games or platforms such as Microsoft’s HoloLens that allow users to use physical objects (such as a water bottle or a piece of furniture)
• A prototyping and design application that can project a holographic image of an object in the space where it will be placed

There are two fundamental types of augmented reality, marker-based and marker-less. While the former is less cost-intensive and more accessible, the latter provides a more immersive experience.

Marker-based AR applications overlay digital content onto a physical trigger in a real-world environment. This trigger might be a QR code, an image or another target marker. When a device’s camera detects this marker, it triggers the display of the associated AR experience. As this kind of AR can be accessed at any time from a range of devices, it is the most flexible AR model.

Marker-less AR, by contrast, doesn’t require a specific trigger. This kind of AR relies on device sensors, such as GPS, accelerometers and cameras, to understand and map a user’s environment in real time. By analyzing the user’s physical environment, often by using algorithms and computer vision, these AR systems determine where to place digital content, allowing for a more spontaneous and dynamic experience.

AR can provide immersive learning experiences for students. This might include exploring interacting 3D models and scientific simulations overlaid onto their physical surroundings.

AR games like Pokemon GO blend virtual creatures or objects into real-world locations, creating engaging and interactive experiences.

In healthcare, AR can be used for medical training, surgical planning and patient education. Surgeons can overlay patient data, anatomy models and surgical guidance onto their field during procedures, improving precision and patient outcomes.

AR can be used in industrial settings to train workers, offer step-by-step instructions during the manufacturing process (link resides outside ibm.com), and create digital twins of products or equipment.

Wayfinding AR apps can overlap directions, points of interest, and contextual information onto a user’s real-life experience, surfacing information in an intuitive way.

AR has had a significant impact on that way consumers interact with products and brands. By blending digital content with the physical shopping environment, AR technology can enhance engagement and personalize the shopping experience, creating a truly omnichannel customer experience.

Through AR-enabled apps for mobile devices or in-store displays, shoppers can virtually try on clothing, accessories or cosmetics. They can also visualize home decor in their living spaces—the furniture retailer Ikea has had particular success with its AR application. These immersive experiences blend online and offline shopping, allowing consumers to make more confident purchasing decisions.

AR seamlessly integrates with various shopping channels, including brick-and-mortar stores, ecommerce platforms and mobile apps to deliver consistent and unified shopping experiences across multiple touchpoints. For instance, a customer might use an AR-enabled mobile app to try on makeup before visiting a store to make a final purchase.

AR can enhance the user experience by providing interactive virtual environments such as maps, product locaters and contextual information overlaid onto the customer’s real-world view. Retailers might use AI to guide customers to specific products or promotions within the store or display offers based on a shopper’s location, encouraging exploration within the store.

IBM iX helps design and deliver human-centered experiences across the customer lifecycle spanning sales, service, marketing and commerce. Help your business define its strategy and accelerate time-to-value with insights and intelligent workflows that use data and AI.

Harness the power of data and AI with the IBM iX Experience Orchestrator—our proven, open source framework that accelerates composable commerce and modernization efforts. Use our insight-led, outcome-driven customer experiences to build trust and profitability, expanding your reach and relevance.

Retailers often need to innovate within traditional frameworks. From customer prediction to supply chain efficiency and personalized marketing, our retail technology solutions can revolutionize your efficiency and productivity.

Learn what steps an extended-reality expert recommends businesses take to create next-generation customer experiences that evolve alongside the metaverse itself.

Read the blog Resource Taking a design-led, data-driven approach to experience transformation

Learn how businesses can create competitive advantage with more sustainable, resilient supply chains that power innovation, a better customer experience, and growth.

Explore the solution Report Experience is everying

Value increases when technology meets design. Learn how top CEOs are staying on top of ever-shifting changes with personalization and AI.

Read the report Case study Tearing down silos to build better delivery of city services

Learn how the City of Helsinki and IBM Consulting co-created faster, more flexible customer experiences with a digital assistant network.

Read the study Blog 6 ecommerce trends to watch

Explore the six trends, including augmented reality, that are projected to heavily impact the global market.