In my first post on Clinical Research, I mentioned how important it was for Investigators to adhere to the protocol, but I didn’t really go into detail on the whys and wherefores. Protocol Deviations (PDs) are often misunderstood, even by those who are tasked with identifying and categorizing them, so I wanted to write this breakdown of the concept and why it matters to everybody involved in clinical research.
By definition, a PD is anything that causes one or more subjects to deviate from the strictly defined procedures and events laid out in the protocol. You might imagine then that it wouldn’t be difficult for this to happen – for example, if a subject gets sick and cannot attend a study visit when they’re supposed to, that would be a PD. That wouldn’t be anybody’s fault, it’s simply something that couldn’t be avoided, but it is still a PD. More seriously if a pharmacy fridge fails without anyone noticing, and a subject is given a medication that might be ineffective, that would also be a PD.
There used to be a term of Protocol Violations (which were deemed more serious) and PDs, but that guidance has since been clarified/rescinded and now all should simply be reported as PDs, and further categorized into Important or Not Important. Important PDs are those that would impact the rights, welfare, or safety of subjects, or would potentially negatively impact the integrity of the data. Non-Important PDs are divergences from the protocol that do not significantly impact subject safety or data integrity. In part, I think this was to reduce the perception that PDs were a punitive act against the sites and Investigators. Investigators often feel that they are being blamed for events that are outside of their control (for example, sick subjects not showing up), or equipment that doesn’t work. There is a fear, which is in part founded on the reality that they are wholly responsible for the research conducted, that too many PDs will lead to their ability and integrity being called into question. It is true that in the very worst cases an IRB can revoke an Investigator’s ability to conduct scientific research, but in more than 99.99% of instances I would say that the PDs are meant as an administrative and a statistical tool, and certainly are never intended as punitive.
The true purposes of detecting PDs fall into two broad categories – regulatory, and operational. From a regulatory perspective, it is absolutely crucial to detect and identify a “per protocol population” – the group of subjects who, as best as reasonably possible, followed the protocol as agreed to by the regulatory body (e.g. the FDA in the USA). Without this, the sponsor for the study cannot possibly go back to the FDA and say “Look, we proved our drug is safe and effective – please approve it for us!”. If too few subjects followed the protocol, as agreed, then the FDA will simply refuse and tell the sponsor to go back and get more subjects. This is why it is so crucial that Investigators do not willingly deviate from the protocol, even if (and I cannot stress this enough) they would do something different under the normal standards of medical care. Losing one subject here and there is not too great a problem, but losing hundreds (or even thousands) of subjects due to PDs can be devastating to a clinical trial. For this reason some studies retain the “violation” distinction for the most serious PDs that would drop a subject from the per-protocol population, whereas others might opt for a designation of Major or Minor PDs. The distinctions are crucial, and must be clearly defined by the entire project team, including the Clinical Operations, Medical Monitoring, Data Management and Statistical team members. Many of the PDs will be detected by Clin Ops during their site visits or through remote data monitoring; a large chunk are detected during Medical data review (for example the use of prohibited medications, or potential exclusion criteria in medical history); and without the input of a statistician and data manager it isn’t possible to properly assess the impact of these PDs on the final analysis of the clinical trial. In some cases, sites may be asked to track additional information, or the data management team may have to code custom programming to identify and track certain PDs. In one study I was on, the out-of-window visits were subdivided into Major or not depending on how far out of window they were, and only for certain key study endpoints. This is the kind of nuance that can’t be simply tracked as an “out of window” visit – it needs to be “out of window for Visit X by Y days”, and would absolutely need custom programming (and very clearly written protocol deviation guidance) to detect, if the sponsor wanted to go down that route.
I was involved in a company-wide initiative to update and improve our Protocol Deviation process, to optimize the process and understanding of each team member on their role. Medical Monitors in particular have to weigh in on Important PDs, which are defined as any that might impact subject safety or data integrity – ultimately we have to decide if that subject should be dropped from the study or if they may continue. Sponsors often don’t appreciate the distinctions between the different levels of PD, and the work they entail, so it’s important to communicate that clearly during study start-up. PDs are also very much study-specific – for example an “expedited PD” may require the Medical Monitor to contact the site and intervene on an urgent basis, but it makes no sense to mark certain PDs as expedited in a single-dose study (such as a vaccine study) when the PD would be detected long after the subject was dosed. It also makes no sense to mark PDs that would be detected by the Medical Monitor as “expedited”, because they already know about them! If a missed study visit occurs then it’s reasonable to track that as a single PD, and not necessary to track separate PDs for every missing procedure during that visit (yes, I’ve seen this done inadvertently!) The goal of this process improvement was to streamline and align the PD process between all the relevant team members, and ultimate save time and money throughout the entire study timeline.
The second reason for detecting PDs is operationally – the Clin Ops team monitors and tracks PDs, and especially important is the categorizing of PDs. For example, if they find that there are many PDs for inclusion/exclusion criteria, it might be that the protocol is unclear and needs revision. If they find that one site has many PDs compared to their peers it might signal an issue that requires site staff retraining – and the flip side of course is that if a site has very few or no PDs that they aren’t being truthful in their data reporting… Ideally these are reporting back to the Sponsors regularly throughout the study, although I have come across some smaller biotech companies who have asked merely for updates perhaps one time during the study, and once at the end. In my opinion this is a recipe for disaster – it means that the operational benefits of the PDs are diluted, if not lost, and it also means that potential revisions to the protocol that could improve the quality of the data (or subject safety) are not made. The very best companies use PDs as “constructive criticism” to optimize their study sites and protocol. PDs can also be an absolute bear to properly collect and categorize, and the worst thing that could happen to a study team is to discover right at the very end of a study that there are hundreds (or thousands) of PDs to address when there are looming deadlines of database locks.
There are two further things I want to highlight about PDs. There is a hierarchy that actually stretches very high up – PDs can be at the study level, the country level, the site level, or the subject level. Study and country-level PDs are extremely rare. Site level PDs do occur, but they are often misidentified, and that can cause a lot of work. By definition, a site-level PD affects all subjects at that site. This is a statistical definition, not a common-sense definition. For example, a broken pharmacy fridge might be incorrectly identified and recorded as a single site level PD (failure to adequately maintain and monitor study equipment) – but the problem then arises in the final analysis, where EVERY subject from that site would be tagged with that PD! In truth, the clinical monitor who visits the site has to correctly identify the subject(s) impacted by the event, which usually means tracking specific dates, subject timelines, and drug deliveries to figure it all out. They would have to identify and report PDs on the affected subjects, and ONLY the affected subjects. This might mean reporting 23 separate PDs (one for each of 23 affected subjects) instead of one site level PD, unless the site has only enrolled 23 subjects! But trust me I say this is the correct way to do it – this is exactly how the FDA will ask the events to be captured if they were to visit the site for an audit. I perhaps misspoke above when I said that the worst thing that can happen is for a last minute deluge of PDs just prior to database lock – imagine a deluge of PDs after database lock because of an FDA audit findings…
And finally, the reason for the meme I made above (which I have actually presented in Investigator training slides…). Sites often ask for “exemptions” or “permission” for a PD ahead of time, for example if they know that a particular subject is going to be out of town and miss a study visit, or if a subject has a prohibited medical condition but would otherwise qualify for inclusion in the study. The thought is, if they ask ahead of time they won’t be “dinged” with a PD.
It is a very, very, very bad idea to grant exemptions to PDs. As I said above, PDs are there to ensure data integrity and that the per-protocol population is sufficient to be submitted for regulatory approval. If a significant PD is granted ahead of time, the entire clinical trial is put at risk. I was trained to never, ever, grant a PD exemption. We can acknowledge that a deviation has (or will) occur, but it will be reported, and it will be categorized, and if need be the local IRB will also have to hear about it. Now, there are some sponsors that will grant PD exemptions on a case-by-case basis but that is their risk to bear. Not every event reported as a PD is actually a deviation when fully reviewed, and it isn’t unreasonable to void a PD that shouldn’t be there, but they would have to defend that decision should a regulatory authority question it, and if you don’t think that a regulatory authority will detect discrepancies in individual protocol deviations and subject data, then you haven’t witnessed a regulatory audit… What the site/Investigator really needs to ask is “Can this subject continue in the study, should this PD occur?” That’s an entirely different question, and is probably what the site is really wanting to know, but even if the answer is “yes” then every effort needs to be made to follow the protocol – the Investigator literally signed a legal document saying they would do this (I talked about Investigator responsibilities last week…)
To summarize – PDs are inevitable, even if they should be avoided as much as possible, and they are very important tools used during the conduct and in the final analysis of a clinical trial. Clear PD guidance should be established very early on in the study start-up to identify areas of risk and mitigation steps, and PDs should be routinely reviewed with the sponsor so that course corrections or staff retraining can be done as needed. Proper categorizing and coding of PDs (Important, Non-Important, Major, Minor, Expedited…) will dramatically improve the quality of the data and project workflow for everybody involved. Sites should not fear PDs, and they absolutely should not attempt to cover them up, but please don’t ask for exemptions.
Drop me a line, or follow this blog, if you want to learn more about PDs and how best to avoid or manage them.
Nick Bennett MD 