Senior Scientist, Clinical AI Research & Evaluation (Regulatory & Responsible AI) — Boston

We're building a connected, end-to-end  Enterprise AI  engine - uniting data foundations, AI technology, process reinvention, and business-facing AI to accelerate results across the whole value chain.  Success depends on being exceptional connectors : you'll actively leverage existing capabilities, celebrate and promote reuse, export breakthrough ideas across geographies and functions, and obsess over scaling impact rather than building in isolation. If you thrive in high-collaboration environments where your role is to turn complex, cross-functional problems into reusable, enterprise-wide capabilities - and where the measure of success is adoption and scale, not just innovation - you'll have the platform (and sponsorship) to make it real.  About AISI AI Science & Innovation (AISI) sits at the centre of AstraZeneca's R&D AI transformation within Enterprise AI Unit (EAI). Our remit is to build, buy, and deliver the AI models and agents that change pipeline outcomes across discovery, translational science, biomarkers, and clinical development, ultimately improving patients' lives. The opportunity We are building an end-to-end Enterprise AI engine that unites data foundations, AI models, platforms, and business-facing applications to accelerate results across the value chain. Success comes from reusing what already works, sharing ideas across teams, and scaling impact rather than building in isolation. Role overview AstraZeneca is building an AI capability for Clinical Development that will change how trials are designed, conducted, monitored, and analyzed. We are hiring a Senior Scientist, Clinical AI Research & Evaluation to do two things: advance the science of evaluating clinical AI systems as a hands-on researcher, and act as a technical AI advisor to Regulatory Affairs. The role is based in Cambridge, MA. This is primarily a researcher-first role with a secondary responsibility towards regulatory AI guidance. In your primary identity as an AI researcher, you design and run evaluations of LLM-based and agentic clinical systems such as benchmarks, evaluation harnesses, LLM-as-judge methods, failure analysis, and you do it in code, alongside the Clinical AI Applied Science & Translation team. Your second hat is technical advisor to Regulatory: you translate evaluation evidence, reliability, responsible-AI practice, and residual risk into language and evidence standards that Regulatory colleagues can act on, and you help bring validation thinking in early, before the design is locked. You will work day to day with the team building agentic workflows for clinical-trial planning, conduct, monitoring, and review, and you will spend a meaningful share of your time with Regulatory, Quality, and compliance colleagues. GxP expertise is not a must have for this role. We're looking for a strong researcher who wants to learn the evidentiary standards of regulated medicine and apply their evaluation skills there. What you'll do AI research & evaluation (primary, the majority of the role) Design and run evaluations of LLM-based and agentic clinical systems: benchmark construction, evaluation harnesses, LLM-as-judge methods with judge calibration, failure-mode taxonomies, red-teaming, and error analysis. Build reusable evaluation assets (test sets, scoring rubrics, annotation protocols, regression suites, analysis pipelines) that the portfolio can reuse across studies and therapeutic areas. Study how AI output shapes clinical judgment: expert-agreement studies, user studies, reliability under noisy, incomplete, or context-dependent ground truth. Investigate failures using traces, logs, qualitative review, and quantitative analysis; translate findings into model, prompt, tool, data, or workflow improvements. Publish and communicate findings internally and externally where appropriate. Technical advice to Regulatory (second hat) Serve as the technical bridge between the AI teams and Regulatory Affairs: translate evaluation evidence, residual risk, model limitations, and uncertainty into language Regulatory can act on. Advise on what evidence standards should look like for AI systems approaching regulated use: validation documentation, traceability, audit trails, intended-use clarity, and human-oversight definition. Track evolving health-authority expectations for AI in clinical decision support and regulated contexts, and brief both the team and Regulatory on what they mean for our systems. Support validation-planning and risk-assessment discussions with Quality and Regulatory from early design, so evidence is built in rather than retrofitted. Delivery with the team Write production-quality Python and SQL for evaluation and analysis; keep experiments reproducible, versioned, and tested. Partner with clinical subject-matter experts, product owners, and engineering to ground evaluations in real workflows and real stakes. Contribute to technical design reviews, code reviews, documentation, risk assessments, and release evidence. Essential skills/experience • PhD in Machine Learning, Computer Science, NLP, Information Science, Biomedical Informatics, Statistics, or a related quantitative discipline; recent PhD through ~3 years post-PhD (or a master's with an equivalent research record). • +3 years experience of hands-on Python and SQL: version control, tests, reproducible experiments, and the ability to turn research questions into clean, runnable code. • Demonstrated LLM or NLP evaluation experience (benchmarks, LLM-as-judge, error analysis, metric selection), evidenced by publications, released systems, or open-source work. • A research track record: peer-reviewed publications or equivalent evidence of rigorous, communicated science. • The ability to explain uncertainty and residual risk to non-specialists, including clinical, quality, and regulatory audiences, without overstating or obscuring. • Curiosity about clinical development and regulated medicine, and the motivation to learn GxP, validation, and health-authority context on the job. Desirable skills/experience • Clinical NLP experience: radiology reports, clinical notes, EHR, safety data, or biomedical language models. • Experience designing user studies or expert-agreement studies; causal inference or statistical rigor beyond predictive accuracy. • Industry research-lab experience or equivalent shipped AI systems used by real users. • Exposure to regulated environments: model risk management, audit readiness, validation plans, health-authority frameworks. • Familiarity with evaluating retrieval-augmented or agentic systems, red-teaming, or safety evaluation. What success looks like • The team's evaluation practice is measurably stronger: calibrated judges, reusable assets, failure analysis that drives engineering decisions rather than decorating them. • Regulatory colleagues can act on the technical advice: AI evidence questions get technically grounded answers before they become blocking issues. • Evidence standards are defined early in each system's lifecycle, and validation thinking is present from design. • Publications or standards contributions build external credibility for the team's evaluation approach. In-Office Working Requirements    When we put unexpected teams in the same room, we unleash bold thinking with the power to inspire life-changing medicines. In-person working gives us the platform we need to connect, work at pace and challenge perceptions. That's why we work, on average, a minimum of three days per week from the office. But that doesn't mean we're not flexible. We balance the expectation of being in the office while respecting individual flexibility. Join us in our unique and ambitious world.    The annual base pay for this position ranges from $ 124 713,60 - 187 070,40 USD Annual. Our positions offer eligibility for various incentives—an opportunity to receive short-term incentive bonuses, equity-based awards for salaried roles and commissions for sales roles. Benefits offered include qualified retirement programs, paid time off (i.e., vacation, holiday, and leaves), as well as health, dental, and vision coverage in accordance with the terms of the applicable plans.    Ready to make a difference? Apply now!   #EAI Date Posted 05-nov.-2026 Closing Date 13-nov.-2026 Our mission is to build an inclusive environment where equal employment opportunities are available to all applicants and employees. In furtherance of that mission, we welcome and consider applications from all qualified candidates, regardless of their protected characteristics. If you have a disability or special need that requires accommodation, please complete the corresponding section in the application form.