Regardless of being educated on huge quantities of information, most LLMs are unable to reliably generate well-designed UIs. Designer suggestions is important to enhancing efficiency on UI technology; nevertheless, we discover that current RLHF strategies primarily based on rankings or rankings should not well-aligned with designers’ workflows and ignore the wealthy rationale used to critique and enhance UI designs. On this paper, we examine a number of approaches for designers to present suggestions to UI technology fashions, utilizing acquainted interactions comparable to commenting, sketching and direct manipulation. We first carry out a examine with 21 designers the place they gave suggestions utilizing these interactions, which resulted in ~1500 design annotations. We then use this knowledge to finetune a collection of LLMs to generate greater high quality UIs. Lastly, we consider these fashions with human judges, and we discover that our designer-aligned approaches outperform fashions educated with conventional rating suggestions and all examined baselines, together with GPT-5.
- ** Work performed whereas at Apple
