Session: DAC-04-2: Data-Driven Design

Paper Number: 139024

139024 - DesignQA: Benchmarking Multimodal Large Language Models on Questions Grounded in Engineering Documentation 

This research introduces DesignQA, a novel benchmark aimed at evaluating the proficiency of multimodal large language models (MLLMs) in comprehending and applying engineering requirements in technical documentation. Developed with a focus on real-world engineering challenges, DesignQA uniquely combines multimodal data—including textual design requirements, CAD images, and engineering drawings—derived from the Formula SAE student competition. We focus on the development of a high-quality benchmark, as the question-answers are designed and reviewed by members of the MIT Motorsports team, industry professionals, and engineering researchers. Different from many existing MLLM benchmarks, DesignQA contains document-grounded visual questions where the input image and input document come from different sources. The benchmark features automatic evaluation metrics and is divided into segments—Rule Comprehension, Rule Compliance, and Rule Extraction—based on tasks that engineers perform when designing according to requirements. We evaluate state-of-the-art models like GPT4 and LLaVA against the benchmark, and our study uncovers the existing gaps in MLLMs' abilities to interpret complex engineering documentation. Key findings suggest that while MLLMs demonstrate potential in navigating technical documents, substantial limitations exist, particularly in accurately extracting and applying detailed requirements to engineering designs. This benchmark sets a foundation for future advancements in AI-supported engineering design processes. DesignQA is publicly available at: https://github.com/anniedoris/design\_qa.

Presenting Author: Anna Doris MIT

Presenting Author Biography: Anna is a graduate student in the DeCoDe Lab at MIT researching multimodal large language models for engineering design.

Authors:

Anna Doris MIT
Daniele Grandi Autodesk Research
Ryan Tomich Massachusetts Institute of Technology
Md Ferdous Alam Massachusetts Institute of Technology
Hyunmin Cheong Autodesk Research
Faez Ahmed Massachusetts Institute of Technology