The use of Augmented Reality (AR) in mobile app development is gaining momentum very quickly. Some apps enable placement of a 3D object (i.e. a piece of furniture) in order to inspect how it might look in a room through the live view of the phone camera. For shopping, that is a great use case that other teams are exploring at eBay. The eBay Mobile team decided to introduce a utility-focused feature for a different real-world, augmented-reality use case. We wanted to provide a fun and innovative way to expedite and simplify selling and shipping on eBay. Many teams' collaborative ideas, prototypes, and hackathon projects inspired this initial beta feature.
Rather than looking at how a 3D object looks in a room through augmented reality, how about flipping that concept around? It might be useful for our customers to see whether a real object in a room fits inside a transparent 3D box sized to real-world, real-shipping box dimensions.
Finding the correct size box is an essential step when preparing a product to sell. Our previous research clearly indicated an opportunity for mobile app innovation to reduce the barriers to selling online, while hopefully adding both fun and utility to the selling experience.
“Which box?” (WB) is an Augmented Reality feature that allows users to quickly determine whether a USPS shipping box will fit around an item in order to ship it safely and cost-effectively. This is the first new Augmented Reality (AR) feature integrated into the eBay Core mobile app on the Android platform. It utilizes the device’s camera and the pre-installed ARCore APK toolkit.
The UI employs basic gestures and interactions to ensure a simple and successful user experience. Simply place the box on the item, rotate it if needed, and see if it fits. Rapid prototyping, iterative design and development sprints, as well as research with our users was incorporated directly into developing the initial version of this feature. Simple “on the glass” user-guidance and box-picker widgets were developed to guide the user through simple placement of the box in the space. Those interactions will evolve as more standards are formed for UX in AR. We learned from showing this feature to eBay users who were interacting with AR for the first time.
When development began, we used developer preview versions of the ARCore APK, prior to its official 1.0 release. Engineering research spikes were completed to investigate the potential of utilizing the ARCore Unity plugin and its development environment. Unity enables developing highly polished 3D and game experiences using a separate, mature IDE, as well as excellent 3D asset creation, manipulation, and integration capabilities. The other alternative considered was native 3D graphics development done using OpenGL ES. Unity-based development of AR will almost certainly be used for other features, but we decided to implement the WB feature as a standard Java-based Android Activity and GLSurfaceView.Renderer.
OpenGL ES on Android requires a little more heavy lifting with regard to assets and interactions, but a native Java implementation enabled expedient integration with the existing Android eBay Core codebase. One of the primary goals for the team was to iterate quickly, delivering incremental value to the customer via small experiences integrated within the eBay Core app. This enabled the team to adapt and adjust as needed to better ensure user success. Native Java and OpenGL also allowed utilization of 2D UI widgets that had been developed for reuse within the eBay Core codebase. Much of the OpenGL code is also reusable for other features within eBay.
The architecture of this feature also follows the Model-View-ViewModel (MVVM) pattern, including the use of Android Data-Binding. The box picker carousel is a UI component seen on multiple views within eBay Core and is initialized with Box ViewModel objects. Each box item, and the widgets that control the experience, are updated at runtime through data-binding. For example, when the box orientation toggle is clicked, it’s bound to a handler in the activity presenter, which changes the bound orientation field of each box in the picker carousel. The bound Boolean state of that field helps manage the state of the toggle as well as the orientation of the boxes in the picker carousel. Vector assets were used for all of the UI elements.
android:onCheckedChanged="@{(view, isChecked) -> presenter.onOrientationCheckChanged(isChecked)}"
android:src="@{ContextCompat.getDrawable(context, uxContent.imageSource)}"
Design of the 3D assets is critical to the implementation of most AR applications. WB utilizes the 3D asset format recommended in the ARCore documentation. Predefined Wavefront .obj and image-based texture/material assets were used for the translucent box assets with dimensions and eBay packing tape materials built-in. One critical step during asset design was making sure they were the correct size and scale in real-world space. Also, when working with transparent assets, one key step in OpenGL ES is to utilize blending correctly as below:
The new feature was enabled in eBay Core v5.18.1 simultaneously with Google’s release of ARCore 1.0. It is available on these supported devices: https://developers.google.com/ar/discover/.
The eBay Core app is an “AR Optional” application, because obviously it does not require ARCore for the majority of functionality. ARCore is specified in the AndroidManifest:
<meta-dataandroid:name="com.google.ar.core"android:value="optional"/>
Ultimately, augmented reality will be able to combine its power with other technologies such as artificial intelligence and object-detection. Experiences like Which box? will be able to do much more than simply allow the viewing of 3D objects in the room. They will be able to automatically detect, make recommendations, and guide the user through the shipping process, as well as other fun and innovative eBay buying and selling use cases.