The upcoming meeting, taking place on November 10 at Mila, will explore how we can collectively develop, govern, and deploy high-performing, reliable, and secure agentic systems by connecting academic researchers, industry experts, and practitioners.
Join us on the Venture Scientist Bootcamp, a full time, 4-month incubator at Mila, built specifically for deep tech founders with elite STEM backgrounds. You have until November 21, 2026 to apply!
We use cookies to analyze the browsing and usage of our website and to personalize your experience. You can disable these technologies at any time, but this may limit certain functionalities of the site. Read our Privacy Policy for more information.
Setting cookies
You can enable and disable the types of cookies you wish to accept. However certain choices you make could affect the services offered on our sites (e.g. suggestions, personalised ads, etc.).
Essential cookies
These cookies are necessary for the operation of the site and cannot be deactivated. (Still active)
Analytics cookies
Do you accept the use of cookies to measure the audience of our sites?
Multimedia Player
Do you accept the use of cookies to display and allow you to watch the video content hosted by our partners (YouTube, etc.)?
Background: Large language models (LLMs) could assist in streamlining health care instrument validation. This is particularly relevant as tr… (see more)aditional instrument development is time-intensive and resource-intensive. Currently, there is a significant gap in specifically developed and validated trust instruments for pediatric emergency and surgical contexts. Such tools are key because trust is foundational to the patient family-physician relationship and is associated with improved care-seeking and treatment adherence. Objective: This study aimed (1) to develop new trust instruments and conduct their preliminary content validation confirming item relevance through a Synthetic Instrument Validation approach integrating human and LLM capabilities and (2) to evaluate appropriate use cases for LLMs in psychometric assessment. Methods: Two trust instruments were developed (patient family and physician versions). In phase 1, instruments underwent a 2-stage content validation process using parallel synthetic and human expert panels (5 patient family members, 3 physician synthetic experts, and 7 unique human experts). Synthetic panels consisted of 3 persona-prompted LLMs (Claude Sonnet 4, GPT-5, and Grok 4), with human panels serving as comparators. The scale-content validity index (S-CVI) threshold was set at ≥0.80, percent agreement on item relevancy was used to compare panels. In phase 2, LLM performance in quantitative research tasks was evaluated. The patient family instrument underwent Flesch-Kincaid assessment; both instruments underwent cosine similarity analyses using algorithmic and LLM-based methods. Results: In phase 1, percent agreement between panels for the patient family and physician instruments was 90.9% and 88.9% for dimensional validation and 90.9% and 100% for contextual validation. Fleiss κ for within-panel agreement ranged from poor to almost perfect; 0.20 (95% CI -0.23 to 0.63) to 1.00 (95% CI 1.00-1.00). All instrument subsections met or exceeded S-CVI thresholds across both rounds and panels. Phase 2 Flesch-Kincaid metrics converged across all 3 methods to target benchmark (mean Grade Level 8.1, SD 1.1; mean Reading Ease 60.1, SD 5.6), while reference readability was near target, showing convergence across implementations of the same formula. In contrast, cosine similarity analyses revealed significant LLM performance limitations; cosine similarity analyses were, therefore, abandoned in favor of the algorithmic method alone, which achieved a maximum cosine similarity of 0.83, indicating acceptable item distinctiveness overall. Conclusions: Persona-prompted LLMs effectively assisted preliminary content validation and may help reduce timelines from months to weeks, but showed limitations in quantitative computations. This suggests that, in this study, persona-prompted LLMs performed better in subjective rating tasks than in deterministic cosine similarity computations. These findings help illustrate task-dependent boundaries for LLM integration in psychometric research, necessitating selective human-LLM collaboration. While these instruments achieved preliminary content validity, they require full psychometric testing before they can be considered validated for clinical use. Synthetic Instrument Validation offers a feasible approach to preliminary health care instrument development while preserving key elements of content validation.