The Next Generation Software

Table of Contents

• Intro
• I. The Form Shift — From Screens to Surroundings
  • 1. Immersion is a Gradient, Not Binary
  • 2. The 3D Interface is Inevitable
• II. The Interaction Shift — From Commands to Context
  • 1. The Great Gap: Input vs Output
  • 2. Closing the Gap: UI + AI is the Next Interface
  • 3. The Next Leap: Contextual Understanding
  • 4. The Output Shift: Generative Spatial UI
  • 5. A Story: Food, Fitness, and Feedback
• III. The Agency Shift — From Tools to Companions
  • 1. The Capability–Hardware Gap
  • 2. A Framework: Intent × Latency
  • 3. From Apps to Agents
  • 4. Trust become the new currency
• Conclusion

Intro

The future of software is spatial, contextual, and alive.

I believe there are three big shifts in the future of consumer software

• Form: how interfaces evolve from flat screens to immersive surroundings.
• Interaction: how AI shifts from command-based tools to context-aware systems.
• Agency : how software moves beyond being functional to becoming truly collaborative.

In this blog, we will examine them together. As always, these reflections are my own. They don’t represent the views of my employer, nor do they draw on any confidential information. My aim is simply to explore possibilities and share personal perspectives on where AR/VR and AI might take us.

---

I. The Form Shift — From Screens to Surroundings

When people hear “AR” or “VR,” they often picture full-on Matrix mode: everyone jacked into goggles, lost in the metaverse, and never touching grass again.

Honestly, I get it. That version does sound dystopian.

But that’s not the real story. Reality is subtler—and far more useful.

1. Immersion is a Gradient, Not Binary

Think of immersion as a slider, not a switch.

On one end, you have reality as it is. But sometimes you want to remix it just a little. Maybe replace the ads on the street with photos of your dog? That’s 10% of your field of view becoming virtual. Waiting in line, you might want entertainment floating in front of you while also keeping track of when the line moves — that’s 60% immersed. And at night, if you want to hang out with friends remotely, you can crank it all the way up to 100% in virtual worlds.

The key point: immersion isn’t 0 or 1. It’s a gradient you can dial to fit the moment. You’re not escaping reality; you’re choosing how much of it to remix. That means AR/VR isn’t just for gamers. It’s for everyone.

You can also map how you use it across two axes:

	Low Immersion	High Immersion
Episodic	Phones, laptops	VR headsets, Quests
All-day / Ambient	AI glasses, Meta Rayban	(The holy grail?)

Most devices today live on the left. Phones and laptops are low-immersion and episodic. Headsets are high-immersion but uncomfortable for long sessions.

The goal isn’t just to make a holy grail device in the bottom-right corner. It’s adaptability along both axes. We want hardware that rides the gradient: shifting from light to deep immersion, from quick moments to all-day presence, based on what you’re doing and where you are.

2. The 3D Interface is Inevitable

We are 3D creatures. We move through space, we use eyes, hands, posture, proximity. Our brains evolved for spatial reasoning. So why is our software still trapped behind flat panes of glass?

2D software isn’t bad. Buttons, menus, text fields — these are elegant abstractions for symbolic work like writing, coding, or crunching numbers. But when the task itself is spatial, 2D becomes a translation layer. You compress an embodied intent like meet, place, move, into symbols on a flat screen.

Examples where the translation hurts:

• Windows & multitasking: juggling overlapping rectangles and alt-tabbing when what you really want is to spread tools around you.
• Video calls: faces flattened into identical boxes when what you want is to sit around a shared table with gaze and presence lined up.
• Online shopping: clicking through product photos when what you want is to drop the object into your room and see if it fits.

AR/VR removes that translation where it’s unnatural and unlocks interactions that were previously impossible:

• Home design: Instantly change wall colors, move virtual furniture in your real space, watch how afternoon light actually falls across your couch.
• Robot programming: Have a fully functional virtual robot you can program and mod in your room. Stand beside the robot and see its logic overlaid in space.
• Medical training: Instead of reading anatomy diagrams, walk through a beating heart. Practice procedures in mixed reality with real tools and virtual patients.

Every app you use today will sprout a 3D counterpart. Not because it’s flashy. Because it makes more sense. It’s the better mental model for how humans interact with the world.

One way to frame it: Viewports vs. Viewpoints.

• Old: Your device is the viewport—you peer into software.
• New: Your perspective is the viewpoint—software radiates outward from where you are.

Seen from your viewpoint, spatial content tends to show up in three formats: windows, volumes, and spaces.

• Windows are flat panels, they are familiar 2D apps that float where you pin them.
• Volumes are bounded 3D objects or tools that occupy a fixed chunk of your room—you can walk around them, and multiple can run at once.
• Spaces reshape the environment itself, dimming walls or transporting you to a brand new world.

Many experiences would fluidly shift among these: start as a window to inform, become a volume when manipulation helps, and escalate to a space when the moment truly benefits from immersion.

But as software becomes situated and a part of your environments, it introduces a lot more variables than just on a 2D screen. Where should objects be placed? When should the UI appear or disappear? Should the interface be ambient or immersive?

If users have to micromanage those choices, the experience will be very bad. The answer isn’t more buttons – it’s more context. That takes us from commands to context.

---

II. The Interaction Shift — From Commands to Context

1. The Great Gap: Input vs Output

Let’s look at software through a simple lens: input vs output.

User interfaces today are excellent at output. They’re designed to present information cleanly, buttons, layouts, dashboards. On Uber, for example, the information hierarchy is crystal clear: you can glance at your screen and immediately know when your ride is arriving.

But for input, understanding what you actually want, we give software dropdowns, search bars, and manual clicks. You have to translate fuzzy human intent into rigid sequences of taps and keystrokes.

LLMs flipped this equation. They’re great at input. You can ask them anything, no matter how vague or complex, and they’ll do their best to figure it out.

But their output? A giant wall of text. “I’ve confirmed your Uber ride. Your driver is Jack, driving a 2021 green Toyota Camry. They’ll arrive in 10 minutes at 234 Something Ave.” It’s not wrong. It’s just… too much.

This is why smart-speaker shopping never took off. People want to confirm delivery time, payment method, and shipping address visually before ordering. A pure voice-based or text-only output interface simply isn’t enough.

	Traditional 2D Apps	LLM-Based Apps
Input	Clicks & taps Dropdowns Search bars → ❌ Rigid & manual	Natural language → ✅ Flexible & expressive
Output	Predefined 2D UI → ✅ Clear & structured	Natural language text → ❌ Overwhelming & unstructured
Core Limitation	Software doesn’t know what you want	Software doesn’t know how to show it

2. Closing the Gap: UI + AI is the Next Interface

So the natural next question is: Why not combine them? Marry the flexibility of LLMs with the structure of UI.

We’re already seeing this happen in developer tools — ChatGPT Codex, Lovable, and other examples let you say things like “Add a like feature,” and the LLM figures out what code to change. It can highlight the diffs or even show a live preview. Suddenly, you’re steering complex software through natural language, and the system shows you what it did.

But these are still mostly text-based experiences. It’s an app split into two: one side with the UI and the other side is still a chat window. We haven’t yet seen good LLM + UI fusion in everyday, UI-heavy consumer apps.

Why? Because it’s hard. Really hard.

On the input side, when you’re interacting with a 2D app, the model need to take into account additional context — what’s the text on-screen, your prior actions, where you are at in a UI flow, etc. When you say “Send this to Jack”, what does “this” refer to? Which “jack” should we send it to through which app?

On the output side, it has to figure out how to act for you. It needs to pick the right UI template, modify the right elements, and present the right information — all on the fly.

The ultimate form of this is pure generative UI: no more templates, the AI system will build a brand new interface you need, right when and where you need it.

We’re going to see these kinds of apps emerge in the next few years — as startups explore this new paradigm and as on-device model capabilities catch up.

But why stop there?

3. The Next Leap: Contextual Understanding

What happens if your environment becomes part of the input?

With AR/VR hardware — camera arrays, directional mics, depth sensors — and with the all-day wear nature of these devices, software can begin to understand the world as you do: what you’re looking at, what you’re doing, where you are, and what’s around you.

If done wrong, it will be a privacy nightmare. But if done right, it unlocks continuous, ambient sensing — a shift from explicit input to implicit input.

Instead of software being confined to the context of a 2D viewport, it now operates from your 3D viewpoint. It sees what you see. It understands space, posture, gaze, movement.

And that unlocks a new UX paradigm. You don’t have to click, tap, or even speak — the system simply knows.

• You’re leaving the house and it’s raining → your glasses remind you to bring an umbrella.
• You slump at your desk for hours → a gentle nudge encourages you to stretch.
• You stare at a foreign menu → translations appear overlaid on top, aligned with each dish.

We’re moving from commands as input to context as input.

And this shift changes the fundamental design questions. Instead of: “What does the user click?” We now ask: “When should the system intervene? Based on what context? And in what form?”

4. The Output Shift: Generative Spatial UI

As input becomes ambient, the output will become adaptive. Because future software understands your environment, it doesn’t just generate what to show you — it also knows where to put it.

It can surface UI in the most effective — or least distracting — way. Maybe your calendar reminder appears gently in your periphery during a meeting. Maybe your to-do list reattaches itself to your fridge when you’re near the kitchen. Context isn’t just what you’re doing; it’s where and when and what matters right now.

And crucially, output is no longer limited to 2D elements.

Generative models are already producing 3D content from text prompts. But what if these 3D objects weren’t just art? What if they carried meaning — and were generated dynamically to help you?

• Choosing a hiking route → a 3D terrain model appears with the trail highlighted and a progress slider.
• Assembling IKEA furniture → animated arrows float over the exact parts for your next step.
• Learning guitar → a translucent hand hovers over your fretboard to show the chord.
• Understanding an LLM architecture → a labeled 3D diagram forms in mid-air, synced to the paper you’re reading.

Generative UI doesn’t just mean “the UI writes itself.” It means the world itself becomes the canvas for software to express itself.

	Traditional 2D Apps	LLM-Based Apps	Next-Generation Apps
Input	Clicks & taps Dropdowns Search bars → ❌ Rigid & manual	Natural language → ✅ Flexible & expressive	Natural language + Spatial context + On-screen context → ✅ Ambient & multimodal
Output	Predefined 2D UI → ✅ Clear & structured	Natural language text → ❌ Overwhelming & unstructured	Generative spatial UI → ✅ Adaptive, situated, multimodal
Core Limitation	Software doesn’t know what you want	Software doesn’t know how to show it	Software understands both intent and expression
Design Paradigm	“What does the user click?”	“What should I say back?”	“When should I help — and how should it appear?”

5. A Story: Food, Fitness, and Feedback

Let’s tie it all together.

I often take pictures of my meals and ask ChatGPT: “How many calories? How much protein? Does this align with my goal of muscle-building?”The answer usually comes back as a wall of text. Sometimes, if I’m lucky, it’s formatted as a table.

But imagine this instead: You’re wearing AR glasses. You sit down with a plate of steak and potatoes. You don’t need to ask anything.

The glasses already know your fitness goals, your recent meals, your dietary preferences. It quietly identifies the food, calculates the macros, and then presents the information right where it’s most useful:

• Protein count hovers over the steak.
• Calories appear over the potatoes.
• A subtle red glow warns you if you’re about to exceed your daily target.

That’s not an app you open. That’s software embedded in real world, responding in real time.

---

III. The Agency Shift — From Tools to Companions

1. The Capability–Hardware Gap

Over the last two years, there have been three major advances in AI:

• Multi-turn: where users can have back-and-forth conversations (GPT-3).
• Multimodal: where models can understand images, video, and voice (GPT-4o).
• Multi-hour autonomy: models can launch into long research sessions; they can think, plan, invoke tools, and rethink their plans (Claude 4, O3, R1, etc.).

But for most consumers, their experience of using AI is still stuck at the first level: multi-turn. Most still use default chat interfaces, or occasionally generate an image for fun, and never truly utilize the multimodal and agentic power.

Why? Because our hardware hasn’t changed. We’re trying to run superhuman cognition through phone-era plumbing.

• Multi-modal friction: On a phone, capturing “what you see” means: unlock → open app → switch modes → frame a photo → type a prompt → wait. You won’t hold up a camera for live video help for more than a few minutes.
• Multi-hour friction: Background agents exist, but they’re niche and developer-centric. For everyday life, laptops and phones are episodic, app-centric, and blind. They don’t live with you, so they can’t earn the right to act for hours on your behalf.

It’s not the models. It’s the hardware. Phones and laptops were built for intent-based, app-first, short-burst interactions. Agency needs perception-first, hands-free, always-on.

2. A Framework: Intent × Latency

We can map interactions on two axes: intent (explicit vs. implicit) and latency (low vs. high).

	Explicit Intent (you ask)	Implicit Intent (it notices)
Low Latency	“Set a timer for 12 minutes.” (Siri, Google assistant )	“You’re leaving without your keys.” (Just-in-time intervention )
High Latency	“Plan my 10-day Japan trip.” (ChatGPT, Deep research)	“You’ve been stress eating a lot in the past few days; are you doing ok?”(Long-horizon tracking)

• Explicit vs. Implicit Intent: whether you initiate the command or the system acts implicitly (see the previous chapter).
• Low vs. High Latency: your expectation for when a response will happen.

Phones are decent at Explicit + Low (voice assistants) and Explicit + High (chat-style research). But what can handle the implicit cases?

Most wearable has missing pieces. Headphones can’t show you anything. Smartwatches’s screen is too tiny and can’t fully speak or listen without you lifting your arm. Cameras for perception are absent in both. AI pins add cameras but strip away display and audio. Taken together, they’re stopgaps, unlikely to become the main agentic platform.

That’s why AR glasses matter. They close the gap: continuous sensors (cameras, mics, IMU), heads-up display, hands-free voice and gesture. The agent finally sees what you see, hears what you hear, and can place guidance where you are, not inside a rectangle.

3. From Apps to Agents

One joke I heard is that rich people don’t use apps — they just have a guy for each thing. Rather than Uber Eats, they have chefs who handle all the food. Rather than Expedia, they have a travel consultant who plans their trips. Rather than booking tickets, they have a concierge. These assistants know their preferences and get the job done.

In the future, this will be the case for everyone. We will stop “installing apps” and start hiring agents—clear roles with clear boundaries that can collaborate on your behalf. They’d share your lived experience.

And not just one, but multiple agents at once. With glasses, you could run several role-based companions concurrently: some focusing on low-latency actions, others on high-latency longitudinal tracking—each with permissioned access to the same live perception stream, but focusing on different signals:

• A Focus Steward detects your context-switching patterns at work, then proposes one small weekly experiment to reduce them.
• A Life-Goal Agent tracks your habits over months and checks whether you’re on track (did you practice Japanese today?).
• A Reminder Agent notifies you if you forget to bring the mail.

They coordinate or work independently through intent × latency: what deserves action now, what deserves reflection later, and what should stay invisible by default.

Since you can run multiple agents at once, just like you can have multiple apps open, the agents will need to negotiate. The Ad Blocker wants to mask an ad, but your Tour Guide needs that same wall to anchor a direction arrow. Which wins? Maybe an orchestrator like a “Meta-Agent” you trust will arbitrates based on your priorities.

4. Trust become the new currency

Underlying all of this is the matter of privacy and trust. Everything we’ve described would be a gold mine for advertisers and data brokers. How to securely make this future possible is beyond the scope of this post.

However, there are principles future agents need to adopt to foster trust:

• Ask-to-act. Anything with consequences gets explicit permission.
• Explain-why. Every agent’s action is inspectable: what it saw, why it acted, what it ignored.
• On-device by default. Perception runs locally whenever possible; cloud is opt-in and transparent.
• Easy-off. One gesture to pause, one tap to kill, one place to review.

You won’t browse an App Store of icons; you’ll enter an Agent Store of roles: “Home Coach,” “Trip Advisor,” “Weight Watcher.” You won’t ask, “How much time did I spend in this app?” You’ll ask, “How much trust did this agent earn this week?” That becomes the metric that matters.

Conclusion

Form, Interaction, and Agency build on each other. As interfaces move from screens to surroundings, input shifts from explicit commands to implicit conditions, and output becomes spatial and adaptive. With the right hardware plumbing, agents can finally act with you, not just for you — visible when needed, invisible when not, and accountable always.

As always, the views expressed here are entirely my own. They do not represent the positions of my employer, nor do they draw on any confidential information. My intention is simply to spark imagination and conversation about the future of AR/VR and AI.

I recognize that even though I’ve thought carefully about these ideas, time and progress may prove many of them wrong. Looking back, some views might seem naive. But that’s part of the process: exploration, trial, and error are what open the door to new possibilities and better solutions.

Most of all, I hope these reflections inspire you. If you’re building this future, we should talk.

Let’s have an adventure!