XR Prototyping with AI for 2025
Disclaimer: This document will be much more helpful if you have some coding understanding.
If you don’t, check this page instead 👉 Set Up XR with Zero Coding
The Empty Scene Problem
What is the XR empty scene problem?
XR developers face a unique challenge: unlike 2D apps that work with wireframes, spatial experiences require believable 3D environments from day one. Default Unity scenes contain only basic lighting and geometric shapes, creating lifeless prototypes that fail to engage users or stakeholders.
Key statistics:
More than 80% of initial XR prototypes built with primitive shapes require major redesigns (based on our interviews)
Empty scenes reduce user engagement by 300% compared to contextual environments
AI-enhanced prototypes achieve user feedback 10x faster than traditional methods
Why 3D Complexity Kills Prototypes
The Dimensionality Tax
Common XR Prototyping Failures
Barren World Syndrome: Using default Unity scenes with no environmental context (yes, I’ve been there and done that. It is challenging.)
Geometric Placeholder Trap: Testing interactions with cubes and cylinders instead of realistic objects
Technical Demo Mindset: Focusing on functionality over spatial experience
AI Tools That Can Solve XR Prototyping
Essential AI Tools for XR Development (2025)
Scene Generation
Unity Plugins: We have one too (Check it out)
Rodin AI/Meshy: Custom 3D asset creation from descriptions
DreamGaussian: Realistic object generation
Development Assistance
Cursor: AI-powered Unity development with natural language
Claude: XR design consultation and spatial planning
GitHub Copilot: XR interaction pattern generation
Asset Libraries:
Sketchfab: 200K+ free 3D models (Thank you Sketchfab for being there 🫶😭)
Poly Haven: AI-optimized materials and environments
Unity Asset Store: AI-curated XR components (We have one for you to create interaction without coding 👉 Check it out)
Cursor + Claude Workflow for XR
Again, you need to understand at least some basic coding.
First, AI Scene Planning
(Skip this part if you are focused on the interactions)
Step 1: Context Definition with Claude
Prompt: "I'm prototyping a **laparoscopic surgery simulation** for a **surgical resident**. The core interaction is **'dissecting tissue with an electrocautery hook'**. What environmental context, objects, and spatial layout would make this feel realistic and educational?"
Claude provides:
Essential interactive objects list: Electrocautery hook, grasper, trocar ports (entry points), target tissue model (e.g., gallbladder), vital signs monitor.
Environmental storytelling elements: An operating room setting, surgical drapes, an instrument tray, ambient beeping from the vitals monitor.
Spatial flow recommendations: Correct placement of the monitor in the surgeon's line of sight, ergonomic layout of the instrument tray.
User journey touchpoints: Introduce the patient case, insert tools through trocars, identify anatomy, perform dissection, manage a complication (e.g., a bleed).
Second, Cursor Implementation
Step 2: Iterative Natural Language Development
(Prerequisite: Create a new 3D project in a game engine like Unity and open the project folder in Cursor.)
Break the task into a logical sequence of prompts.
Detailed Prompt 1 (Scene Scaffolding):
C#
// In a new C# script called SceneGenerator.cs
"Create a public method called 'CreateOperatingRoom'. Inside this method, use Unity's GameObject.CreatePrimitive to generate a floor plane and four cubes for walls to form a simple 6x6x4 meter room. Add a basic directional light to the scene."
Detailed Prompt 2 (Asset Placement & Interaction Stubs):
C#
// In SceneGenerator.cs
"Create three empty GameObjects named 'OperatingTable', 'InstrumentTray', and 'VitalSignsMonitor'. Place them in logical positions within the room. On the tray, create two cylinder primitives named 'Scalpel' and 'Grasper'. Add a 'BoxCollider' to each. Then, create a new C# script named 'SurgicalTool'. The script should only contain a public enum 'ToolType' with options for Scalpel and Grasper."
Detailed Prompt 3 (Core Interaction Script):
C#
// In a new C# script called 'ToolController.cs'
"Using Unity's new Input System, write a script that casts a ray from the main camera's forward direction. If the ray hits an object with the 'SurgicalTool' script and the user presses the left mouse button, print the tool's type to the console and make the object a child of the camera so it moves with the view."
Expected Realistic Output: A collection of C# scripts and basic scene objects within your Unity project. You will still need to run the code and manually assign scripts in the Unity Editor. The scene will be functional but rudimentary, providing the framework for the surgical simulation.
Third, AI-Enhanced Iteration
Step 3: Text-Based Feedback Analysis
The correct workflow is to manually analyze footage from a surgical resident testing the prototype and feed your written observations to the AI.
Input (What you prepare):
Watch the user testing footage.
Write down key observations, user quotes, and points of friction in a text document.
Detailed Prompt Example
"I have conducted a user test for my surgical simulation with a resident. Here is a summary of my key observations and user feedback:
Observational Notes:
The user consistently tried to adjust the 'operating table' height, which is currently static.
They expressed difficulty differentiating between the target tissue and surrounding membranes due to uniform color.
The activation of the cautery tool was unclear; there was no visual or audio feedback, leading to accidental 'burns' on non-target tissue.
User Quotes
"I wish I could hear the sizzle of the cautery tool. It's a key feedback mechanism."
"How do I know if I'm cutting deep enough? The tissue doesn't change."
Task for you
Based *only* on the feedback above, suggest 5 specific, actionable improvements I can make to the prototype. For each suggestion, categorize it as an 'Interaction Change', 'Audio Feedback', 'Visual Feedback', or 'Environmental Change'."
Expected Realistic Output: A prioritized, text-based list of concrete suggestions (e.g., "Add an audio clip of a cauterizing sound," "Implement a shader that slightly darkens tissue on contact with the cautery tool," "Make the operating table vertically adjustable with UI buttons"). You would then take these suggestions back to Cursor to implement the changes.
Real Examples: Before vs After AI
Medical Training Simulation
Before AI (Traditional Method)
Environment: White room with floating anatomy models.
Development time: 2 weeks or more.
User feedback: "Feels like a test, not real training."
Issues: No spatial context, unrealistic interactions.
After AI Enhancement:
Environment: Hospital setting with proper medical lighting.
Development time: 4 hours with AI tools.
User feedback: "Felt like actual surgery."
Improvements: Contextual tools, ambient audio, realistic workspace.
The Hyper-focused Prototype: Cutting Skin with a Scalpel
Forget building the whole operating room. Let’s prototype the single most important interaction from our simulation: the incision. This directly connects to the feedback "How do I know if I'm cutting deep enough?".
A dynamic, deformable object representing "skin" and a tool representing a "scalpel." An interaction that allows a user to create a visible "cut" on the skin's surface.
The Tools
Game Engine: Unity
AI Code Assistant: Cursor or Unity's built-in AI Assistant
Step 1: Minimal Scene Setup (10-15 Minutes)
Open Unity and create a new 3D project.
Create a "Skin" Surface: Create a
Plane(3D Object > Plane). This plane needs more vertices to deform smoothly. You can use an AI prompt to generate a simple subdivision script to prepare it.Create the "Scalpel": Create a
Cube(3D Object > Cube), scale it to be long and thin, and name it "Scalpel".
Step 2: AI-Assisted Scripting (15-20 Minutes)
This is where AI handles the complex mesh code. Create a C# script called DeformableTissue.
Detailed Prompt for your AI Assistant:
C#
// In this Unity C# script named DeformableTissue:
// 1. Create public floats: 'cutDepth' (default 0.05f) and 'cutRadius' (default 0.5f).
// 2. I need a script that deforms a mesh where a raycast hits it.
// 3. In the Update() method, if the left mouse button is held down, cast a ray from the camera to the mouse position.
// 4. If the raycast hits this object, get the mesh from the MeshFilter component and its vertices.
// 5. Loop through all vertices. If a vertex's distance to the hit point is less than 'cutRadius', move the vertex down along the hit normal by 'cutDepth'.
// 6. After the loop, update the mesh with the new vertices and recalculate its normals.
Generate and Refine: The AI will generate a script that performs mesh deformation.
C#
using UnityEngine;
[RequireComponent(typeof(MeshCollider))]
public class DeformableTissue : MonoBehaviour
{
public float cutDepth = 0.05f;
public float cutRadius = 0.5f;
private Mesh mesh;
private Vector3[] vertices;
void Start()
{
mesh = GetComponent<MeshFilter>().mesh;
vertices = mesh.vertices;
}
void Update()
{
if (Input.GetMouseButton(0)) // Left mouse button held down
{
Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
if (Physics.Raycast(ray, out RaycastHit hit) && hit.collider.gameObject == gameObject)
{
DeformMesh(hit.point, hit.normal);
}
}
}
void DeformMesh(Vector3 point, Vector3 normal)
{
for (int i = 0; i < vertices.Length; i++)
{
if (Vector3.Distance(transform.TransformPoint(vertices[i]), point) < cutRadius)
{
vertices[i] -= transform.InverseTransformDirection(normal) * cutDepth;
}
}
mesh.vertices = vertices;
mesh.RecalculateNormals();
}
}
Step 3: Integration and Testing (5 Minutes)
Attach the
DeformableTissuescript to your "Skin" Plane object.Ensure the Plane has a
Mesh Collider.Press "Play".
Hold the left mouse button and "drag" the cursor across the plane.
The plane's surface will depress, creating a visible "cut" or incision. This hyper-focused prototype validates the core mechanic that can now be integrated back into the full surgical simulation.
Key Constraints of the AI-First Workflow
While resourceful and cost/time-effective, this approach is an accelerator, not an "easy button." Here are the realistic limitations.
You Still Need Core Skills. AI writes the first draft, but you are the final editor. You need foundational coding and 3D engine knowledge to debug, integrate, and optimize the AI's output. The AI won't fix its own mistakes.
Prompting is a Skill. Your results are only as good as your prompts. AI executes instructions literally; it doesn't have taste or design sense. You are still the creative director responsible for asking the right questions.
AI Tackles the First 90%. This approach is brilliant for getting a functional prototype running in record time. The final 10%—polish, nuanced "game feel," and fixing unique bugs—still requires intensive human effort and expertise.
Generated Assets Aren't Production-Ready. AI models are perfect for establishing a vibe but often have messy geometry (topology) and textures (UVs). They require manual cleanup by an artist before being used in a final product.
In short, AI doesn't replace expertise; it amplifies it. Think of it as a powerful tool that you, the skilled creator, must wield effectively.
Tools and Resources
Free AI Tools for XR Prototyping
Claude 3.5: Spatial design consultation
Cursor: AI-powered Unity development
Rodin AI: AI-powered 3D model generation
Spline AI: Basic 3D scene generation
Sketchfab: Free 3D asset library
Recommended Learning Resources
Unity XR Toolkit documentation
AI-assisted development best practices
Spatial design principles for VR/AR
Conclusion
AI tools have eliminated the traditional barriers to rich XR prototyping. The complexity tax of 3D development is disappearing, making contextual spatial experiences faster to build than empty scenes.
Key takeaways
Empty 3D scenes kill user engagement and stakeholder buy-in
AI tools like Cursor and Claude reduce XR prototyping time by potentially 80%
Contextual environments reveal user insights impossible to discover in abstract spaces
The future of XR prototyping is AI-enhanced from day one
Start building spatial experiences that feel alive, not software demos in digital voids.