Session: DTM-02: Design Methods and Practice

Paper Number: 142788

142788 - Evaluating Design Rationale 

Design rationale captures the justification behind a design decision. How design rationale is represented has the potential to shape design outcomes. Often, rationale varies in the content and depth of information, making the study and comparison of rationales challenging. This project aims to compare the quality of rationale generated in two different representations and to develop a computational approach to evaluate the quality of design rationales at scale. In total, 2250 rationales were machine-generated using GPT, and a portion of the rationales (n = 512) were evaluated by two raters across five dimensions of quality. Natural language processing approaches were used to extract linguistic features. Eight models were assessed for each of the five dimensions. Moreover, comparisons of structured and unstructured rationales were conducted using statistical analyses. The main results show that structured rationales were rated higher than unstructured rationales across the five dimensions. Thus, the tested feature, specification, and evidence (FSE) framework was shown to be a worthwhile approach to represent the justification behind a design decision. Future work will explore how the quality of design rationale impacts design behavior and performance, particularly in a human-AI teaming context where generative design recommendations could benefit from the inclusion of generative design rationales.

Presenting Author: Yakira Mirabito University of California, Berkeley

Presenting Author Biography: Yakira Mirabito is a PhD candidate in the Department of Mechanical Engineering at UC Berkeley. Her research focuses on better understanding the what and why behind decision-making in engineering design in order to improve design processes. She uses a mixed methods approach to study (1) design behavior, (2) cognitive biases, and (3) communication across a range of engineering domains. Her work is impacting the way we teach design in education and practice. She is advised by Kosa Goucher-Lambert in the Cognition and Computation in Design (Co-Design) Lab. She holds a BSc in Materials Science and Engineering and Design Certificate from Northwestern University and an MSc in Mechanical Engineering from UC Berkeley.

Authors:

Yakira Mirabito University of California, Berkeley
Xiaowen Liu University of California, Berkeley
Kosa Goucher-Lambert University of California, Berkeley