Semantic Ink: Intent-Based Graphics Authoring through Annotation-Driven Generative User Interfaces and Reusable Tools

Abstract

Designers face increasing challenges when translating high-level creative intent into precise graphics operations. Traditional authoring tools expose low-level primitives that require complex sequences of tool invocations, imposing steep learning curves on users. While AI-assisted systems can interpret high-level intent, existing tools primarily support single-shot generation or style transfer, lacking scaffolding for iterative refinement and semantic-level reuse. We present Semantic Ink, an intent-based interactive graphics authoring system that combines freehand annotation with generative UI. Users express intent through annotations directly on the canvas, and the system responds by generating structured parameter spaces as interactive controls, for progressive refinement without prompt iteration. Successful edits can be extracted as custom tools for semantic-level reuse. Informed by a formative study with five designers, we contribute a two-dimensional taxonomy crossing five intent types (add, transform, style, remove, scene) with four controllable parameter types (variation, magnitude, spatialization, surface) to enable multimodal LLMs to parse ambiguous annotations into editable parameter spaces. A user study with six experts demonstrates that Semantic Ink improves editing efficiency, reduces cognitive load from mode-switching, and enhances personalization through tool reuse.