What is an Investigational Device in the Context of Genomics Research? – Stephen Kingsmore
Articles Blog

What is an Investigational Device in the Context of Genomics Research? – Stephen Kingsmore


Stephen Kingsmore:
Thanks very much. Thank you for the invitation to come here. I’m operating on about four
hours sleep courtesy of American Airlines. [laughter] Okay, so I’m going to talk today based on
practical experience as an NIH funded principal investigator, who has gone through this process.
So, first of all, what does the world look like from my standpoint? We get happy news
from NIH; your grant has been funded. We formulate a proposal for our IRB to give us authority
to do human subjects research. Because it involves return of results into a medical
record we need to have a laboratory-developed test that conforms to the CLIA standards.
In California, we have the joyous duplication of this effort whereby the state also regulates
us independently and authorizes our LDT results return. And then most recently we have the
FDA pre-submission and IDE mechanism. As you can see that’s a lot of stuff. And
if you want to get research done in a hurry this is not the way to do it. I’m not saying
that there’s a problem here it’s just that’s a lot of stuff if we’re going to
make progress towards the realization of genomic medicine before it goes direct to consumer,
which is our great fear. So our hypotheses that we were funded to explore were relatively
simple. They were that genome sequencing if performed rapidly in acutely ill infants in
two locations, neonatal intensive care, pediatric intensive care units, who were thought likely
to have a single gene disease would increase the rate of diagnosis, decrease the time to
diagnosis, and improve the precision of acute medical care. The backdrop is that there are literally thousands
of these diseases. And so a new strategy was needed that would be comprehensive as opposed
to traditional testing which, as you can see below, is sort of on a case-by-case basis
decided in consultation with medical genetics. And really not something that as a nation,
as a healthcare industry, people can practice in most healthcare settings. So most babies
do not receive this type of testing and likely will not get a diagnosis during their hospital
stay, and may not get a diagnosis before they die. These diseases are the leading cause
of death in the NICU and the PICU and in infants in general. So September the 4th, I believe, it was we
started our U19 grant. U19’s are different from other NIH grants inasmuch as they give
a huge amount of oversight of the program by the program manager and by our peers so
there were four funded centers. And we were on phone calls at least once every two weeks
where we discuss pertinent aspects of our research. So this is a highly regulated mechanism
for doing clinical research. About a month after the grant started we had a call from
the FDA to ask could we enlighten them in terms of the work that we were proposing to
do. And subsequently, and in our other three centers had similar phone calls. And subsequently
our program managers were notified that we would need to go through a pre-submission
inquiry and maybe an IDE. None of us really knew what that meant and so we collectively
put on our thinking hats. We had numerous phone calls. We shared materials. And we submitted
our inquiry at month six. So let’s take a look at what that involved.
This is the process as David has shown you. For us that was a 13-page document with the
following contents. Specifically a device description; let’s show you that. This is
our device. We had many, many, many conversations about what the device meant. Because as academics
we don’t think device we think experimental design, we think technologies. We’re familiar
with the term laboratory developed tests but device is not something that we build. We
typically have many components which we hook together, typically informatically, to give
an end result. I’ll walk you through this incredibly fast;
and so we have an enrollment process followed — and before that an ascertainment process.
Information is collected in a REDcap database. We collect a blood sample if the patients
are enrolled; extract DNA. These are standard processes. Turn that into a sequencing library.
This is PCR free. And then sequence them. And in our case this was on a high SEQ, 2500
predominantly, in rapid run mode. The resultant information then is fed into a high performance
disc and then into a compute cluster, where it goes through these transitions and file
formats giving us a list of variants that may be causative of the infant’s symptoms.
Those are annotated with a separate software tool. And parallel with this process we enter
the clinical features of the baby and correlate those with the 8,200 known genetic diseases
to prioritize a differential diagnosis. And these are then integrated using a software
system that allows rapid variant interpretation. The information similarly goes to a database,
a master database, ultimately as part of the mechanism of the ward. We have a requirement
to deposit this in public databases. In terms of interpretation reporting by and large we
perform a confirmatory Sanger test to confirm what we see based on our genome sequence.
But in exceptional circumstances we provide a verbal report to the ordering physician
in advance of that confirmatory test. And I’ll explain a little bit more about it.
So that’s our take on the device. The proposed or intended use; this is a Randomized
Controlled Perspective Study. Inclusion and exclusion criteria are listed. The group sizes,
parents, and clinicians were given pre-test questionnaires to understand the Melia, if
you will, in terms of their hopes and fears regarding this novel type of testing. And
then rapid genome sequencing was performed. Return of diagnostic results was either verbal
if the child was about to die and the result could potentially change the outcome. But
in the vast majority of cases waited Sanger confirmation and was standard — a standard
report placed in a chart. In some cases the results were not clear in terms of whether
this was a diagnosis or not. In some cases we did not make a diagnosis. And there was
the opportunity for physicians to cross a patient over into the WGS arm if they felt
that the patient had to get that, which increasingly they did during the study. And then after test results and at some point
later there were additional follow up questions to get information related to diagnostic and
clinical utility; and also, again, parents’ social, legal, ethical type information. Another
part of the device was to understand had it been validated. And this was a device that
was in development. The grant had three components and component one was to continue to develop
the device during the five years of award. As I say we didn’t refer to it as a device
[laughs] but we started to. And so with time analytic performance improved; this is about
its current analytic sensitivity and specificity. And as you can see from the area under the
rock [spelled phonetically] or next gen sequencing is maybe the most sensitive and specific test
ever to be developed in terms of its analytic performance looking at nucleotide variation.
That’s both single nucleotide substitutions and small insertion and deletion events. We also had information on the diagnostic
performance and it’s comparator to all gold standard methods combined. And so we had reason
to believe that the diagnostic performance, albeit this is a retrospective study, was
superior. And then lastly did it make a difference to have that information? Yes it did. We had
one life saved. We had four cases in which there was a medication change, sort of the
definition of precision medicine, three cases in which major morbidity was avoided. This was maybe the most important slide which
was that there was not much room for time delays in a NICU or PICU environment. That
by 100 days of life over 50 percent of babies with a genetic diagnosis would have died.
And, therefore, delaying return of results was not in the best interest of the child.
And spoke to the relative risk and harms of returning an unconfirmed result, albeit one
with quite superb analytic specs versus Sanger confirmation taking about a week to perform. So after less than two months we had an initial
FDA response. The FDA responded to us with a series of questions. And we had a teleconference
shortly thereafter. I’ll just go through this very quickly. I don’t want to take
too much time. But this shows you the actual questions and our responses and how we formatted
this; and so largely these were clarifications of specific items in our 13-page document
that required additional explanation. And for many of these we had to actually go back.
This was a learning curve. It was a different language being spoken. And we needed to sort
of get our heads around the FDA thinks about research, which is fairly different from say
a CLIA CAP lab director’s thought or an IRB member’s thought. So there were questions about when and where
we would do confirmatory testing. This was back in 2014 remember when that was normative
in all cases. The FDA sought clarification about what was a high likelihood disease-causing
variant that we were able to show relatively simply based on published ACMG criteria. Process
for determining whether a verbal disclosure would occur and who did it and to whom and
we walked through that. And, you know, this was a rare event. I think we had two of these
events in a three-year period. A bit more information about this; a little
bit more information about the actual information we were placing in the medical record. Additional
clarification about VUS; whether there was any other mechanism to return an unconfirmed
result. This was something else. Lastly, there was questions about blood draws and whether
these would be higher than what our institution allowed in infants. And some of the infants
we were enrolling were premature. Were there other — any other samples being retained?
And then lastly our question for the FDA was do you require to proceed to the investigational
device exemption proposal? We were notified in writing a couple of days later that no,
that was not required; that that was a — not significant risk proposal. Our understanding
was that there was risk. This was a — an experimental device. However, in this somewhat
unique setting of high acuity illness in infants with high morbidity and mortality, that the
benefits exceeded the harms. And so we were then able to start enrollment. Now, I’m just going to finish up because
that’s not the end of the story. This is a five-year project. I have now moved to a
new institution. The technologies are now rather different than they were back in 2014.
For example, we can now decode a genome and return a result in 26 hours. Previously the
speed limit was 50 hours. This is our device these days. As you can see the level of complexity
has gone up at least in order of magnitude; that these days we use genome or exome sequencing
interchangeably. We have a variety of sequencing instruments. And furthermore, as is normative
now we often outsource this to CLIA and CAP approved external laboratories who will undertake
these processes for us. And we rely on their CLIA CAP accreditation and can see controls.
We do the analysis in-house. Some of the components are similar but, as you can see, there are
a lot more of them now. And, as you can see, they now involve both cloud-based solutions
as well as local solutions. And by and large there is a lot of duplication and redundancy
that we are rarely using a single tool to make a diagnosis. We still have the bottom line here that confirmatory
assays are normative. But increasingly we are giving our lab director oversight of that
as is generally the case around the country for diagnostic results return. In part, this
is based on the improved analytic performance which, as I showed you, has significantly
improved as we understand the components better. And partly by this fairly seminal paper where
Leslie Biesecker and colleagues showed that Sanger sequencing actually was not a good
test to validate next gen sequencing results. Inasmuch as it was more likely to incorrectly
refute a true positive than to correctly identify a false positive. And so information such
as this is making us take this on a case-by-case basis in individual laboratories and relying
heavily on our own experience. And I would say that we need to remember that this manuscript
spoke only to substitution events where we know that next generation sequencing really
does have 99.999 something sensitivity and specificity. For indel events the paper did
not speak to that. And I would argue that it’s still premature to not confirm those
by Sanger confirmation. But I think this is an area where different labs will have different
approaches. So thank you very much. I hope I didn’t
run over my time. Christina Kapustij:
Thank you, Stephen. And I think I really liked your analysis that everyone is speaking a
different language. And I hope today’s forum can act as sort of a translator and help to
develop a dictionary for the different dialogues between research and regulatory science. Next we have Dr. Paula Caposino who is a scientific
reviewer in the Division of Chemistry and Toxicology Devices in the Office of Invitro
Diagnostics at FDA. [end of transcript]

Leave a Reply

Your email address will not be published. Required fields are marked *

Back To Top