Mind Palace XR
OCAD U Open Research Repository: https://openresearch.ocadu.ca/id/eprint/4639
The Idea
Inspired by the ancient "Method of Loci" mnemonic technique and motivated by the challenge of human forgetting, Mind Palace XR explores how Artificial Intelligence (AI) and Augmented Reality (AR) can be integrated to enhance memory. This innovative Unity-based application aims to transform the user's real-world surroundings into an interactive, personalized memory palace. By capturing the environment via a device camera, the system employs a multi-stage AI pipeline—using Google Gemini for scene understanding, FalAI for 2D artistic interpretation, and StabilityAI for 3D model generation—to create tangible 3D memory cues. These 3D models are then anchored within the user's augmented space, allowing them to leverage spatial memory principles for improved information recall and creating a dynamic, world-scale repository for memories
Development
Mind Palace XR was developed as the final prototype culminating from an iterative research-through-design process involving 15 distinct prototypes exploring related concepts. The core development involved:
- Platform: Built using the Unity engine for AR deployment.
- Input: Utilizes real-time video feed from the device's camera (tested with Meta Quest 3 HMD) to capture the user's environment (README Overview).
- AI Pipeline: Implemented a sequential workflow connecting multiple AI APIs:
- The captured image is sent to Google's Gemini Vision API for detailed scene description generation.
- The text description is passed to the FalAI API to generate a stylized 2D image.
- The 2D image is then converted into a 3D model using StabilityAI's 3D generation API.
- AR Integration: Generated 3D models (in glTF format) are loaded and displayed in the user's AR space using the GLTFast package in Unity.
- Features: Includes UI controls for initiating capture, provides feedback during processing, and offers local storage for the generated 3D memory objects with timestamps (README Features, Workflow).
- Assistance: AI coding assistants (OpenAI, Anthropic models) were used during code implementation and refinement.
Reflection
As a capstone thesis prototype, Mind Palace XR successfully demonstrated the technical feasibility of integrating complex, multi-modal AI services (vision-to-text, text-to-image, image-to-3D) within a real-time AR application to explore futuristic memory augmentation concepts. The project tangibly realized the idea of creating automated, spatially-anchored 3D memory cues inspired by the Method of Loci, showcasing a novel application of AI and AR. Public exhibition at DFX 2025 generated positive feedback and suggested potential real-world applications. However, as a speculative piece developed through research-through-design, it faced limitations, notably the absence of formal user testing to validate memory enhancement claims and a primary focus on technical integration over usability refinement or deeper exploration of aesthetic and emotional dimensions
What Worked
- Successful real-time integration of a multi-step pipeline involving camera capture, diverse AI APIs (Gemini, FalAI, StabilityAI), and AR rendering in Unity.
- Demonstrated the core concept of automatically generating and placing 3D memory representations in the user's augmented environment based on real-world scenes.
- Served as an effective proof-of-concept for the speculative research exploring AI/AR augmentation of mnemonic techniques.
- Received engaging feedback during public exhibition, validating the novelty of the concept.
What Did Not Work / Limitations
- Lack of empirical user studies to validate actual memory improvement or usability in real-world scenarios.
- Evaluation was primarily descriptive and based on the developer's reflection, lacking quantitative data.
- Performance, latency, cost, and consistency are dependent on external third-party AI APIs.
- Technical focus limited exploration of user experience nuances, artistic control, and potential ethical considerations.
- Requires further development to implement planned future improvements like conversational AI, VR support, and enhanced spatial understanding.