UX Case StudyInteraction DesignMulti-Sensory AccessibilityCo-Creative AI
Canvas Beyond Sight
An accessible, multimodal creative suite empowering visually impaired creators to illustrate, compose, and explore digital artwork through touch, spatial audio, and conversational AI.
Designer & ResearcherAnanya Bhagat
DisciplineInteraction Design & Multi-Sensory UX
Target HardwareiPad Pro, Apple Pencil & Spatial Audio
Bimanual Interaction in Practice: Inspired by two-handed Braille reading, creators hold the Apple Pencil for sketching while using the non-dominant hand for spatial orientation and boundary verification.
01 / The Shift
What This Changes
Canvas Beyond Sight rethinks digital art as a multi-sensory creative experience. By combining conversational interaction, semantic understanding, spatial audio, and touch, it enables visually impaired creators to independently create, explore, and refine digital artwork.
Traditional digital creative suites are overwhelmingly visual-first, isolating non-sighted artists from direct visual composition. This project transforms accessibility from passive consumption into active, self-directed artistic agency.
"What is art when vision is no longer the primary sense?"
This foundational question framed Accessible Canvas as a multimodal paradigm for non-visual creative expression.
02 / Context
The Creative Barrier
Visual art tools assume immediate visual perception at every stage. For creators who are blind or low vision, digital tools offer almost zero independent agency.
👁️
The Visual-First Bottleneck
Leading creative tools like Procreate, Photoshop, and Illustrator require visual feedback for every stroke, color blend, and layer position. Non-sighted creators are excluded from real-time spatial awareness.
🔊
Consumption vs. Creation
Standard assistive tech (like screen readers) is designed to read linear document trees or describe existing images. They cannot support the active, exploratory, non-linear act of freehand composition.
✋
Physical Art Trade-Offs
Physical tools (embossing styluses, Sewell drawing boards, wax sticks) provide tactile lines but lack digital non-destructive editing, unlimited undo, zooming, and spatial audio exploration.
03 / Architecture
Six Foundational Principles
To design an interface that does not depend on vision, six interconnected multi-sensory modalities were developed.
🎙️
Voice Interaction
Natural speech commands allow users to describe intent, request semantic region feedback, and refine elements hands-free while drawing.
👐
Bimanual Interaction
Inspired by Braille reading, one hand draws with the Apple Pencil while the non-dominant hand orients on screen edges and verifies landmarks.
💡
Semantic Drawing
Abstract geometric marks are assigned semantic meaning defined by the user rather than an automated guessing algorithm.
🔊
Spatial Audio
Vertical canvas placement is sonified via pitch (higher pitch at the top), while horizontal placement is mapped to stereo panning.
🧠
Scene Memory
The system maintains a relational scene memory model tracking object proximity, semantic depth, and contextual spatial arrangements.
💬
Conversational AI
The AI acts as an intentional collaborator that clarifies ambiguities and asks questions without stripping away artistic ownership.
Principles, Reflection & Future Scope Framework
04 / User Journey
The Creative Journey Flow
The creation process follows six continuous stages, guiding the artist from physical orientation through mark-making to an imaginative storybook finale.
1
Onboarding
System orients the user through audio cues and bimanual tactile interaction before drawing starts.
2
Drawing
User creates artwork through abstract mark-making paired with immediate voice commands.
3
Confirmation
AI confirms and semantically interprets each mark in dialogue with the user.
4
Exploration
Spatial audio and semantic regions guide users through their evolving composition.
5
Artwork Description
The full scene is described holistically, conveying composition and proximity rather than isolated tags.
6
Storybook
A narrative inspired by the artwork concludes the session, shifting from accessibility to imagination.
Documented User Journey Sequence
05 / System Design
Semantic Shape Understanding
From a simple sketch to an immersive artwork, the system operates as a co-creative conversational partner.
A Crucial Distinction: Traditional AI vs. Co-Creative Agency
Most AI drawing tools replace the user's intent with machine prediction. Canvas Beyond Sight ensures the user always defines the artistic meaning.
Traditional AI Recognition
System Decides the Meaning
A drawn triangle is rigidly classified as a geometric "Triangle". The artist's symbolic intent is ignored.
VS
Our Co-Creative Approach
"What does this mean to you?"
The system detects geometry, prompts the user for clarification, and semantically registers their answer (e.g. "Mountain") into scene memory.
Two Core Experience Modes: Create (Sketching) & Scene Building (Conversational Refinement)
06 / Interactions
Three Signature Interactions
The tactile, sonic, and conversational mechanics that make non-visual creation fluid and reliable.
Interaction 1
Semantic Region Exploration
Touch, speech, and spatial guidance work together to help users explore, understand, and refine artwork without relying on vision. Sliding a finger across the canvas announces objects mapped to a 3×3 spatial matrix.
Interactive 3×3 Semantic Grid Simulator
Touch or hover over regions to test spatial pitch & pan
💬
Hover or tap any region above to experience real-time spatial sonification and spoken object announcements.
As the user moves across the canvas, spatialised audio communicates position in real time. Higher pitch indicates the top of the canvas; stereo panning signals left-to-right placement. Boundaries are marked with distinct sonic cues.
STEP 01
Voice-Guided Drawing
User describes shapes and strokes aloud while the stylus is in motion.
STEP 02
Immediate Audio Feedback
System confirms actions instantly with distinct acoustic sound signatures.
STEP 03
Spatial Audio Canvas
Continuous pitch and stereo panning communicate exact coordinate placement.
STEP 04
Boundary Sonification
Distinct haptic clicks and edge frequencies warn when nearing canvas borders.
Spatial Audio Loop: Pitch, Panning, and Boundary CuesInteraction 3
Exploration, Description & Storybook
Addresses the two fundamental questions: "Where am I?" and "What am I touching?" Moving beyond utility to imaginative expression.
Exploration Mode
Provides rapid answers to position and content queries via continuous spatial audio guides.
Describe Artwork
Communicates the complete scene holistically (composition, proximity, and relationships) to build a mental map.
Storybook Closure
"Would you like to hear a story inspired by your artwork?" Shifting from accessibility to pure imagination.
Key considerations behind the interface design, sensory orientation, and hardware choices.
Final Prototype Specification Card
Bimanual Orientation
Bimanual interaction inspired by Braille reading: Apple Pencil in one hand, non-dominant exploring hand, microphone, and audio onboarding orient the creator before drawing starts.
Semantic Vocabulary
Abstract marks are mapped to concrete meanings chosen by the user: triangle → mountain, wavy line → river, horizontal line → ground or horizon.
Dark Mode UI for Residual Vision
High-contrast dark canvas reduces eye strain and supports users with partial vision. Scene memory stores object relationships without AI taking creative control.
08 / Reflection
Learnings & Reflections
Key reflections from designing a co-creative digital art experience with multi-sensory interactions.
1
Creativity Needs Freedom
Users did not want artwork generated for them. They wanted support, exploration, and the ability to shape their own creative outcomes.
2
Understanding Comes Before Creation
Clarifying intent through conversation proved more valuable than predicting what a sketch should become.
3
Accessibility Requires Orientation
Region awareness, voice guidance, and scene memory helped users build confidence and navigate the creative process independently.
