Takeaways from Festival of Biologics (San Diego)

April 2024 – Festival of Biologics (Terrapin, San Diego) featured exciting talks on genAI for clinical trials. Here are a few intriguing takeaways:

Trial challenges are wide ranging; ethics, data, interpretability, dealing with deterministic systems in which answers are inexplicable, reproducibility, and the many resource-intensive tasks. Repetitive, error prone human activities can lead to data quality and interpretation issues. Many trials still require paper forms for monitoring and patient data collection. Manual inputs may compound errors across the 6-7 years of a trial.

GenAI is being utilized in beta or proof-of-concept scenarios with impressive gains, such as reducing study design development, site selection and operation by 25-40%, and improving data accuracy by up to 90%. LLMs are also great at sentiment analysis for gleaning insights from patient reports. GenAI systems are being used to clean and unify data formats from diverse inputs, and can flag potential errors for human validation before they’ve been incorporated into downstream analyses.

Anthropic’s Claude 3.0 was found to have the fewest hallucinations of modern LLMs. Prompt optimization can improve reproducibility and reduce hallucinations across the gamut of LLMs. Telling the model that, “I don’t know” is an acceptable answer can be useful, since models typically return some reply (even for low confidence conclusions). Improved responses were seen when the initial prompt was simpler (i.e. Give me the top five challenges…), with successive deeper questions asked of each response.

Similarly, this strategy helped complete tasks the LLM initially rejected. Asking, “Can you convert this R script to Python”, might be sidestepped with the retort, “No, however here are the rules and syntax for that task”. Shifting to a two-stage prompt (i.e. “Do you understand what this R script is doing” and if yes, “Can you convert it to Python”) lead to an accurate Python translation!

To make LLM responses more usable for downstream analysis and reports, the prompt can ask for each section to be wrapped in descriptive markup tags.

Thanks to presenters from Genentech, Readout.ai, Amgen and others for these insights!

Festival of Biologics San Diego – next date April 23-24, 2025