Tiktok in AR/VR

Introduction

Welcome to the “Reimagine” series. In this series, I’ll pick from the Top 100 apps in the app stores and reimagine them as AR/VR apps in three-dimensional space.

This article will explore TikTok. The simplest approach would be to project the 2D TikTok interface into a big screen in AR glasses for the user. This kind of application might appear in the next few years. However, I hope to break away from the two-dimensional space and re-examine TikTok from a three-dimensional perspective.

For this imaginative exercise, let’s assume we have the perfect pair of AR glasses at our disposal. Our exploration will delve into three parts:

• What problem is it solving?
• Why can AR/VR solve this problem better?
• What would the 3D user experience be like?

What problem is it solving?

We can broadly categorize TikTok users into content creators and content viewers.

Viewers

Problem: Alleviating boredom

Solution: By providing short video content, users can easily discover and filter content they are interested in for instant entertainment and enjoyment.

Depending on the use case, we can further divide viewers into fragmentary time users and focused time users.

Fragmentary time users: These users browse TikTok during short pockets of time in their life (like waiting for a bus, queuing, etc.). Notably, in these scenarios, users need to stay aware of their surroundings, for example, noticing when the bus arrives, whether they need to move forward, etc.

Focused time users: These users browse TikTok during large blocks of free time (like before sleep, leisurely weekends, etc.). In these scenarios, users are typically in a comfortable environment, without the need to pay attention to changes around them. They can fully invest in consuming TikTok content.

Creators

Problem: High cost of making long videos and the content’s low likelihood of being discovered by users

Solution: By adopting the short video format, the cost of producing high-quality videos is reduced, and a special ranking algorithm increases the chances of videos becoming viral.

Creators can be further divided into UGC (User Generated Content) creators and PGC (Professionally Generated Content) creators.

UGC creators: These creators typically use the built-in camera on their phones to record everyday life, shoot dances, capture moments, or create interesting content with filters. Their main need is to be able to record life instantaneously and conveniently and to edit videos simply and quickly.

PGC creators: They typically use professional microphones and cameras and may rely on external professional software for video editing, script design, and graphic design. Their main need is to produce high-quality content more conveniently and hope this content can be efficiently distributed to users through the platform.

While TikTok has many other uses, such as live streaming e-commerce, brand promotion, AI-generated content, etc., due to space limitations, I’ll mainly focus on the four use cases mentioned above. One of TikTok’s strengths is its accurate recommendation algorithm, but since it is “invisible” backend work and unrelated to the user’s direct experience, this article will focus on aspects other than the recommendation algorithm.

Why can AR/VR solve these problems better?

Meeting the demand for immersive experiences

In the current 2D social media, users can only consume content through text, images, and videos. While these methods can effectively convey information, it can’t compete with 3D experiences in terms of immersion and authenticity.

In the era of spatial computing, we can display three-dimensional models, 3D images and videos with depth perception, and even 3D scenes that can completely replace a user’s reality. These new forms of media won’t replace the old ones but will coexist with them, enriching our media ecosystem.

Breaking the limitations of interactivity

The interactivity of 2D social media mainly takes the form of likes, repost, comments, etc. However, in a 3D AR/VR environment, users can interact with content and other users in more intuitive and rich ways, such as moving directly in the 3D environment, manipulating objects, gesturing, etc.

Unlocking the space for creativity

In a 2D environment, creators are limited to expressing themselves within a two-dimensional plane. However, in a 3D environment, they can create richer, more dimensional content. This means that TikTok could transition from a “short video” platform to a “short-form experiences” platform.

Creators will no longer be restricted to a fixed-size screen but will be able to create in a three-dimensional space. They can choose to only display 2D content in this space, mix 2D and 3D content, or completely showcase pure 3D scenes. In pure 3D scenes, creators can also choose whether to completely replace the user’s reality with virtual content or intertwine virtual content into the user’s reality.

In conclusion

For content consumers, the existing TikTok already offers instant fun by combining video and music. However, in the future, TikTok can transport users into “short experiences” through various virtual scenes, providing unprecedented dopamine stimulation.

For content creators, this three-dimensional creative space will attract more game developers and modelers to participate. The cost of creating “short experiences” will be much lower than developing a full game, undoubtedly sparking more innovation and creativity.

What would the user experience be like?

Riker sat alone in the airport terminal, like an island in a sea of bustling travelers. He casually put on his AR glasses, selected the TikTok app, plunging into its mixed reality mode. He could see his surrandings as before but he just stepped into a new universe.

In the narrow terminal aisle in front of Riker, two vivid virtual hip-hop dancers appeared, performing the latest dancing challenge.They may have been captured as 3D video thousands of miles away, yet their movements and smiles were rendered as vividly as if they were dancing in front of him. In this modern age, creating 3D videos and photos with AR glasses was as straightforward as shooting 2D videos with a smartphone. As the dancers grooved, 3D heart emojis were floating upwards behind them. Suddenly, one dancer slipped, faltering towards a passenger sitting opposite Riker. But the passenger remained oblivious, for the spectacle unfolded solely within Riker’s AR glasses. Riker let out a slight chuckle. Although he wasn’t a dance enthusiast, this novel experience still intrigued him. He formed his hand to a thumbs-up gesture. It was instantly recognized by the AR glasses, and one more heart emoji rose slowly behind the dancers.

With a light swipe on his thigh, the next experience appeared before Riker’s eyes. It was a financial news segment. The news anchor, dressed in a blue suit, seemed to be standing in real life in front of Riker’s left side. To Riker’s right, a 2D stock price chart presented the falling shares of a sports shoe company, which later animated into a 3D chart that introduced a new parameter -— user complaint numbers. As the anchor narrated the company’s spiking complaint numbers, Riker was interacting with the 3D chart, exploring it from various angles. At the end of the video, the chart morphed into the company’s logo, and a virtual basket of fresh tomatoes and a basket of roses appeared before Riker. The anchor encouraged viewers to vote whether they would continue to purchase the company’s sport shoes. Watching other tomatoes flying towards the company’s logo, Riker picked up a virtual tomato and threw it too. In this era, creating such 3D interactive experiences has become incredibly simple. A slew of entry-level 3D scene design software sprang up, allowing many creators to effortlessly make 3D interactive content.

The boarding announcement echoed through the terminal. With a swift gesture, Riker confined the TikTok app to a space in his immediate right before rising to join the boarding queue. Standing in line, he could enjoy TikTok on his right while keeping an eye on the movement of the queue ahead.

Having found his seat on the plane and fastened his seatbelt, Riker switched TikTok to immersive mode. In this mode, his surroundings were completely replaced by virtual environment, as if he had been teleported to another space. The first thing he saw was a cute golden retriever rolling on a sofa in front of him. This was a 3D video, and Riker felt as if he were sitting in that family’s living room, as if he could reach out and touch the fluffy belly of the dog. If this was in mixed reality mode, TikTok would automatically remove the living room background and only place the golden retriever in Riker’s real environment. But in immersive mode, Riker felt as if he had been transported directly into that family’s living room, totally immersed in their happy moments with the dog.

In the next experience, a huge pair of sports shoes appeared before Riker. Against the deep black background, the shoes were spinning elegantly like a prized exhibit. Accompanied by a powerful narrative voice, Riker was guided to look at his feet and try the shoes in AR – it was a 3D advertisement. Noticing Riker didn’t try the AR fitting, virtual gestures appeared on both sides of the shoes indicating it could be interacted with. Curiously, Riker pinched and drag the shoes to magnify them, revealing their stunningly detailed textures. Riker thought the shoe looked familiar, it seemed to be the same pair the dancers in the previous video was wearing when they slipped. Without thinking deeper, Riker swiped his finger lightly, moving on to the next experience.

…

Final thoughts

It’s worth mentioning that in today’s internet environment, due to bandwidth limitations, real-time downloading of large 3D scene files to create a smooth feed still presents challenges. But with the advent of the 5G era, everything could change. Under the speedy 5G network, real-time downloads of 10-20MB or even larger 3D scenes will become a breeze, just as it was once unimaginable for TikTok to play 5-6MB short videos smoothly in a 3G network environment.

At the same time, 3D models and content will become as common and easy to obtain as pictures and videos today. In the future, creating a 3D model will be as simple as taking a photo - in fact, we can now create 3D models by scanning objects with a phone, a technology that will only become more convenient and popular in the future.

Similarly, creating 3D experiences will be as easy to get started as video editing but with a high ceiling. Users can easily layout 3D items, animations, 3D videos, 2D pictures or videos with an editor, and control the order of play, spatial position, and size changes by adjusting the timeline.

Going one step further, recording 3D videos or taking 3D photos will be as easy as creating 2D content. Once AR glasses become widespread, anyone can easily record life’s wonderful moments in 3D. It’s a new world filled with infinite possibilities and imagination, and we look forward to its arrival.

Closing

I want to reiterate that all the views I share here are my own. These thoughts do not represent the positions of my company or employer, nor do they contain any confidential information. My sole aim is to stimulate imagination and enthusiasm for potential AR/VR applications, helping everyone better understand this new field full of unknowns and possibilities through my personal insights and observations.

I am fully aware that although I’ve made thoughtful explorations, with technological advancements and the passage of time, my depiction of the future might prove incorrect. Looking back from the future, many of my views may seem naive. But that won’t dampen my passion and interest in this exploration. I believe that only through trial and error can we discover new possibilities and innovate better solutions.

Finally, I hope this series will inspire you, and I also look forward to your sharing of ideas and observations.