Conducting a Network Meta-analysis (NMA)
Project #1: Establishing a new standard for comparing data across clinical trials
For this project, we developed new methodological approaches to adjust networks of valid but disconnected data to create larger, more connected, and more accurate data models. Evidence bases often differ among clinical trials, producing information networks that are difficult or impossible to compare directly. In order to provide reliable estimates of relative efficacy among all relevant biological psoriasis treatments in a study population, researchers from Analysis Group developed an innovative and influential way to establish cross-trial similarity.
Conventional network meta-analyses (NMAs) that synthesize all available evidence from randomized clinical trials are able to deliver only partial comparisons using incomplete datasets. However, the reliability of this evidence depends on data similarity across trials. We noted that there was substantial variation in the response rates of reference arms (control groups of patients) across key psoriasis trials, which can be a source of significant bias in a traditional NMA. To overcome this potential bias, our team introduced a new methodology to adjust for variations in reference-arm response within a network meta-analysis.
The reference-arm-adjusted models fit the clinical trial data significantly better than unadjusted models, and led to a reliable estimation of relative efficacy among all relevant biological treatments in the study population. As a result, the reference-arm-adjusted model is now the standard model expected in psoriasis submissions to the National Institute of Health and Care Excellence (NICE) in the UK, which is the economic decision maker for the UK’s National Health Service. In the years since we developed this methodology, top-tier clinical journals have published a number of reference-arm-adjusted NMAs across multiple disease areas.
As the figure below shows, our models helped clinicians compare Humira to other therapies, even when clinical trials had not compared those drugs head to head. The standard we developed allows clinicians and patients to draw comparisons among drugs and make better-informed decisions when selecting treatments.
Project Team Member Profiles
Keith BettsAssociate (now Vice President), Los Angeles
I worked on this reference-arm adjustment project when I was an associate. I had completed my Ph.D. degree in biostatistics, and had a special interest in comparative-effectiveness research. I had previously discussed the idea of extending traditional meta-regression in the network meta-analysis setting with my colleagues on the Humira team. It wasn’t until we were trying to wrap our heads around surprising results in a traditional network meta-analysis of psoriasis treatments that we conceptualized the oversized impact of differences in reference-arm response. We shared our observations with the client, and partnered in this collaborative research project. The collaboration resulted in this successful innovation, which has since become the standard practice within psoriasis treatment and across network meta-analyses.
This model has been updated many times over the years. For each new psoriasis treatment that comes to market, Analysis Group again runs the model and assesses the comparative effectiveness across the current psoriasis landscape. Over time, a number of talented team members who began working on this project as junior staff members have transitioned into more senior project management roles.
This project required a vital combination of strong technical skills and collaboration. It provided an outlet for creativity and outside-the-box thinking. The project was a huge success for both the drug developer and Analysis Group, and it provided team members with excellent training and experience.
Junlong LiAssociate (now Manager), Boston
As an associate, my responsibility on this project was to develop statistical methods, conduct analyses (including literature searches, data extraction and processing, and coding), and manage workflow and the internal team. Since this was a long-running project, I worked with colleagues (including analysts, associates, a manager, and a vice president) across multiple offices (Boston, New York, Los Angeles, and Montreal). I enjoyed this project’s technical challenges – the more obstacles I faced, the more I learned (and the more fun I had). Some of the challenges we encountered included a sparse network, extreme and missing values, and huge heterogeneity across studies. To address these challenges, we investigated different statistical approaches and models.
On this project I worked with medical, statistical, epidemiological, and economics experts from academia and government. It was always inspiring and enjoyable, and I learned a lot from them. I was able to apply many technical and statistical skills, such as modeling, meta-analysis, Bayesian statistics, and missing data and longitudinal data analysis. I was also able to use my coding skills in R, SAS, WinBUGS, and Excel.
To succeed on an Analysis Group project team, technical skills are vital, but just as important are communication, teamwork, and multitasking skills. Also, everyone here is very collaborative. I consider this the firm’s key to guaranteeing high-quality and efficient work; it also makes it enjoyable to work here.
Jing ZhaoAnalyst/Senior Analyst (now Associate), Denver
This was one of the first projects I worked on at Analysis Group. As an analyst, my main responsibility was executing analytical tasks, such as collecting data, updating programming codes, preparing deliverables, helping to draft the manuscript, and disseminating the abstract, poster, and oral presentation. Later, as a senior analyst, I provided training for junior analysts, oversaw the work stream, provided quality control, participated in client calls, and brainstormed with the project manager and senior staff. We developed solutions to address the client’s needs.
I enjoyed working and communicating with consultants across levels and locations, and experiencing the teamwork and cooperation in our environment. We often discussed issues together, brainstorming ideas and solutions at the client’s request. From my perspective, the most challenging part of the job was keeping track of all the iterations of analyses along the way. This project lasted several years, so there were multiple rounds of updates and variations on the analyses, such as treatments, methods, and settings.
R programming skill was essential. Statistical understanding of network analysis was also important, as was knowledge of variations in study methods. The in-house R training course and the biostatisticians’ willingness to share their expertise really helped newer members of the team to understand the network analysis.
On any Analysis Group project team, it’s important to be an independent thinker, have good multitasking and organizational skills, and know how to locate help promptly. The best part of the firm’s culture is the emphasis on working together. Analysts worked closely to tackle tight timelines or analytical difficulties. Senior staff and our associates were very considerate about analysts’ workload. They consistently advocated for more reasonable timelines for the project team, and jumped in to help with hands-on tasks.