Immersive experiences are reshaping business
Surveys suggest that a large majority of consumers view AR as important in their shopping journey, especially in luxury retail. Many science fiction films have explored how technology will assist people with shopping, but what was once fiction is now a reality. Buying goods online was once a gamble as to how something would look on your body, or how furniture would fit within your space; with screen-based AR, anybody with a mobile device no longer needs to rely on this prediction.
Using AR for activations can also benefit brands looking to engage with their target audience. But beyond product visualisation or retail enhancement, AR has evolved into a creative medium for storytelling and experience design in a way that can extend a brand’s identity into new, participatory dimensions.
What is screen-based augmented reality?
Screen-based augmented reality is a slight marketing misnomer, as all AR requires the use of a digital screen, whether that be the screen of your mobile phone or the smaller screens found within head-mounted devices. For this article, we will be focusing on the former.
This form factor of augmented reality is the most widely adopted form of AR today, purely down the market saturation of handheld screens. It works by using the device’s camera to show a live feed of what is within view and then, through machine learning algorithms colloquially known as ‘computer vision’, overlays 3D content to situate the virtual on top of the real world in a believable manner.
How screen-based AR works
Computer vision
As briefly mentioned above, AR software uses computer vision to process raw data in real-time. Loosely speaking, it is a broad field of artificial intelligence that utilises machine learning algorithms to interpret different data streams from a number of different sensors (camera for visual, microphone for audio, etc).
Specifically for AR applications, this usually falls within a few categories:
Spatial tracking
For an AR app to work, it needs to determine the position, orientation and movement of both the device it is running on, but also potentially moving objects within a three-dimensional physical space.
This happens with Inertial Measurement Units (IMU) (accelerometer, gyroscope, magnetometer) for motion and orientation tracking, in combination with the camera for providing visual context.
More modern devices are often now equipped with LiDAR (Light Detection and Ranging), which works by emitting laser pulses and measuring the time it takes for light to reflect back, accurately allowing the device to determine distance.
- Why this matters for brands: It enables AR experiences to live naturally within real-world environments, whether that’s a virtual product placement perfectly positioned in a store, an immersive installation that reacts to a visitor’s movement, or a campaign that transforms a city street into an interactive brand world.
Image recognition
Triggering specific digital information along a user’s journey is done through a process known as image recognition. By training an IR model on a brand image, packaging or logo, the application can be configured to do a number of things, ranging from showing a block of text relating to the product, or instantiating a 3D object that becomes anchored in space to the image.
- Why this matters for brands: It turns physical touchpoints into entry points for storytelling, transforming a static label, poster, or object into a dynamic brand experience. This not only extends campaigns beyond traditional media but also invites audiences to discover and interact with the brand in unexpected, memorable ways.
Popular AR platforms and tools
A number of companies have invested heavily in creating a suite of tools that make it possible for businesses to create engaging augmented reality apps and experiences:
ARKit
ARKit is the AR suite offering from Apple, and enables you to develop AR applications for iOS with support for filters, games and product visualisations. More recently, this has been integrated with VisionOS, Apple’s latest operating system for the new Vision Pro device.
ARCore
Google’s SDK, ARCore, allows cross-platform development, which means you only have to create a single application to release on Android, iOS and Web. The offering is essentially the same as ARKit, but has slightly less support for iOS due to it not being the native toolkit.
WebAR
There are a wide number of platforms that support WebAR development, with the main benefit being that your customers do not need to download an application (which can often be a barrier to prevent adoption) in order to access the experience.
Each platform has unique strengths, and it is important to consider what your goals are when choosing, but they all enable augmented reality work across a wide range of industries.
Business benefits of screen-based AR
Interactive product demonstrations
Across a wide breadth of industries, from cosmetics to furniture, screen-based AR lets users preview virtual objects in their own physical world. This reduces purchase hesitation and boosts conversion rates. In 2020, Gucci partnered with Snapchat to launch the first global AR shoe try-on campaign, allowing users to virtually try on their new sneaker collection through Snapchat’s AR platform.
This initiative wasn’t just about providing a digital mirror; it was a strategic move to deepen emotional connections with the brand. By integrating AR technology, Gucci blended the digital and physical worlds to foster a sense of personal connection with the brand, as users could visualise themselves wearing Gucci products in their everyday lives.
Enhanced customer experiences
Brand activations powered by AR, whether it be scanning packaging, in-store filters, or immersive campaigns, create memorable moments that drive interest and loyalty. For instance, Pepsi’s AR bus shelter campaign in London overlaid otherworldly 3D elements onto live camera feeds, generating viral excitement and a 35% uplift in brand recall.
Streamlined training and onboarding
In industrial settings, AR can project digital twins or step-by-step digital information over machinery, helping new employees interact with processes more efficiently.
Scalable marketing
Because it only requires an AR-enabled device, screen-based AR is more scalable than smart glasses or head-mounted displays, making it ideal for B2B marketing leaders seeking high-population reach. Snap’s AR Lenses and WebAR platforms like 8th Wall have powered interactive launches for brands such as Nike, Adidas, and Netflix, reaching millions globally without the need for dedicated hardware.
Summary
Screen-based augmented reality should no longer be considered an optional addition to a campaign, you should see it as a competitive advantage. For brand managers and marketing leaders, the opportunity is clear: leverage AR not just for entertainment, but to transform brand activations, training, and customer experiences.
Behind the piece
More like this
