Maria Shepherd, President and Founder, Medi-Vantage03.13.20
When is the work of human factors over, and how does your team know they are prepared for summative testing? Fact is, human factors is never complete, it is only part of the iterative cycle FDA wants us to use for new product development of medical devices. There is always so much more to learn about your product. Even if the team has completed their analyses on the end-users who will be using the device, assessed the environment(s) where the medical device will be used, and characterized key device user interfaces, your work is still not finished.
Why This Is Important
A recent study from Johns Hopkins estimates greater than 250,000 Americans die each year due to medical error, making it the third leading cause of death after heart disease and cancer.1 Other studies put this figure as high as 440,000 (Table 1).2 The authors state the discrepancy exists because human errors/system failures are rarely noted on death certificates, which are the source of information for the CDC when estimating statistics for deaths nationwide.
Test Early, Test Often
A step many medical device companies skip is formative testing. This is a mistake made for cost control reasons and can have serious implications when it comes time to perform the summative study. Risk errors that may not have been anticipated can and will emerge in the summative (Table 2). Formative usability testing serves to validate the chosen user interface design controls are sufficient to eliminate as many risks to use safety and effectiveness as possible, bringing device performance to an acceptable level. Usability testing enables a medical device design team to improve the usability of their device to meet acceptable standards of risk.
Successful formative testing typically requires five to eight end-users, making the number of iterations flexible for the product development team. Use-related errors can be seen early in the formative usability process. In the early stages of product development, many use-errors can be identified utilizing a methodology that recruits end-users with no prior experience with the device to test how intuitive the device interface is. In the later stages, perform all usability testing with experienced end-users to ensure the product performs well against competitive devices on the market.
A series of formative tests can ensure the end-user will be able to manage use of the device successfully for its intended purpose. When designing formative studies, try to expose as many risks as possible on the road to a successful summative usability test.
Medical Errors in Orthopedics
A 2009 study of medical errors in orthopedics showed equipment errors (29 percent) and communication errors (24.7 percent) as those with the highest frequency (Table 3). In the study, two deaths were reported (a result of narcotic delivery errors). For location of errors, 78 percent happened in the hospital with 54 percent in the OR and 10 percent in the patient’s room. The reporting orthopedic surgeon was involved in 60 percent of the errors; a nurse in 37 percent; another orthopedic surgeon in 19 percent; another type of physician(s) in 16 percent; and general staff in 13 percent.5
What to Avoid
One of the biggest mistakes made in formative usability testing is the development team over-trains participants before testing, reducing the potential for use-errors to occur. Also, be sure to capture what end-users find easy and intuitive in order to note design features that do not need to be changed.
The Magic Number
Use-errors are a type of systematic error, so it is difficult to calculate their frequency of occurrence. ISO 62366-2 states many usability problems can be uncovered with sample sizes from five to eight end-users, driving the industry standard in medical device formative studies.
For summative testing, the ISO standards state 15 end-users per distinct user group (the minimum number recommended by the FDA) is likely to provide the data needed to measure risk. This number is based upon a study conducted in 2003 by Faulkner6 where empirical data was collected from a sample of 60 individuals. The study results indicated a sample of 15 people was enough to find a minimum of 90 percent (average 97 percent) of all problems with the product being tested.
IRB or Not?
When should you perform usability testing under institutional review board (IRB) oversight? An IRB is an organizational body with the goal to protect the rights and welfare of human research subjects participating in research studies. The IRB will want to see the following in your submission:
The IRB will evaluate the ethicality of your research based on the information you provide to ensure the methodology minimizes end-user risk and all recruited end-users are fully informed of the full scope of the research to be conducted before the end-user decides to participate. Of equal importance is the principal investigator chosen is qualified to manage the usability study.
Purpose of Submitting to IRB
Pre-approval by the IRB is important when potential risks are introduced by the usability study. An obvious example is usability testing of an injection device. This introduces the risk of a needlestick, and an IRB should be utilized to ensure the risk of a needlestick is minimized. Another example is a transdermal patch, where the end-user could have an allergic reaction to patch material. The IRB will ensure the protocol implements protection measures for the end-user participants.7
The Medi-Vantage Perspective
We are frequently asked if there are usability tests that do not require IRB submission. This decision is made on a case-by-case basis and there are reasons to waive an IRB submission, such as if the device is very low risk and the organization managing the usability test has taken all applicable protection measures. With an error rate of 29 percent, however, the equipment the surgeon is using should be examined to make the devices easier and more intuitive to use.
References
Maria Shepherd has more than 20 years of experience in medical device marketing in small startups and top-tier companies. After her industry career, including her role as VP of marketing for Oridion Medical, where she boosted the company valuation prior to its acquisition, director of marketing for Philips Medical, and senior management roles at Boston Scientific Corp., she founded Medi-Vantage. Medi-Vantage provides marketing, business strategy, and innovation research for the medical device, diagnostic, and digital health industries. The firm quantitatively and qualitatively sizes and segments opportunities, evaluates new technologies, provides marketing services, and assesses prospective acquisitions. Shepherd has taught marketing and product development courses and is a member of the Aligo Medtech Investment Committee (www.aligo.com). She can be reached at 855-343-3100. Visit her website at www.medi-vantage.com.
Why This Is Important
A recent study from Johns Hopkins estimates greater than 250,000 Americans die each year due to medical error, making it the third leading cause of death after heart disease and cancer.1 Other studies put this figure as high as 440,000 (Table 1).2 The authors state the discrepancy exists because human errors/system failures are rarely noted on death certificates, which are the source of information for the CDC when estimating statistics for deaths nationwide.
Test Early, Test Often
A step many medical device companies skip is formative testing. This is a mistake made for cost control reasons and can have serious implications when it comes time to perform the summative study. Risk errors that may not have been anticipated can and will emerge in the summative (Table 2). Formative usability testing serves to validate the chosen user interface design controls are sufficient to eliminate as many risks to use safety and effectiveness as possible, bringing device performance to an acceptable level. Usability testing enables a medical device design team to improve the usability of their device to meet acceptable standards of risk.
Successful formative testing typically requires five to eight end-users, making the number of iterations flexible for the product development team. Use-related errors can be seen early in the formative usability process. In the early stages of product development, many use-errors can be identified utilizing a methodology that recruits end-users with no prior experience with the device to test how intuitive the device interface is. In the later stages, perform all usability testing with experienced end-users to ensure the product performs well against competitive devices on the market.
A series of formative tests can ensure the end-user will be able to manage use of the device successfully for its intended purpose. When designing formative studies, try to expose as many risks as possible on the road to a successful summative usability test.
Medical Errors in Orthopedics
A 2009 study of medical errors in orthopedics showed equipment errors (29 percent) and communication errors (24.7 percent) as those with the highest frequency (Table 3). In the study, two deaths were reported (a result of narcotic delivery errors). For location of errors, 78 percent happened in the hospital with 54 percent in the OR and 10 percent in the patient’s room. The reporting orthopedic surgeon was involved in 60 percent of the errors; a nurse in 37 percent; another orthopedic surgeon in 19 percent; another type of physician(s) in 16 percent; and general staff in 13 percent.5
What to Avoid
One of the biggest mistakes made in formative usability testing is the development team over-trains participants before testing, reducing the potential for use-errors to occur. Also, be sure to capture what end-users find easy and intuitive in order to note design features that do not need to be changed.
The Magic Number
Use-errors are a type of systematic error, so it is difficult to calculate their frequency of occurrence. ISO 62366-2 states many usability problems can be uncovered with sample sizes from five to eight end-users, driving the industry standard in medical device formative studies.
For summative testing, the ISO standards state 15 end-users per distinct user group (the minimum number recommended by the FDA) is likely to provide the data needed to measure risk. This number is based upon a study conducted in 2003 by Faulkner6 where empirical data was collected from a sample of 60 individuals. The study results indicated a sample of 15 people was enough to find a minimum of 90 percent (average 97 percent) of all problems with the product being tested.
IRB or Not?
When should you perform usability testing under institutional review board (IRB) oversight? An IRB is an organizational body with the goal to protect the rights and welfare of human research subjects participating in research studies. The IRB will want to see the following in your submission:
- The usability testing protocol
- Informed consent and assent form
- Recruiting screeners with inclusion and exclusion criteria
- The CV of your designated principal investigator
- Research locations
- Details of your device
The IRB will evaluate the ethicality of your research based on the information you provide to ensure the methodology minimizes end-user risk and all recruited end-users are fully informed of the full scope of the research to be conducted before the end-user decides to participate. Of equal importance is the principal investigator chosen is qualified to manage the usability study.
Purpose of Submitting to IRB
Pre-approval by the IRB is important when potential risks are introduced by the usability study. An obvious example is usability testing of an injection device. This introduces the risk of a needlestick, and an IRB should be utilized to ensure the risk of a needlestick is minimized. Another example is a transdermal patch, where the end-user could have an allergic reaction to patch material. The IRB will ensure the protocol implements protection measures for the end-user participants.7
The Medi-Vantage Perspective
We are frequently asked if there are usability tests that do not require IRB submission. This decision is made on a case-by-case basis and there are reasons to waive an IRB submission, such as if the device is very low risk and the organization managing the usability test has taken all applicable protection measures. With an error rate of 29 percent, however, the equipment the surgeon is using should be examined to make the devices easier and more intuitive to use.
References
- bit.ly/odt200301
- bit.ly/odt200302
- bit.ly/odt200303
- bit.ly/odt200304
- bit.ly/odt200305
- Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, and Computers, 35(3), 379-383.
- bit.ly/odt200307
Maria Shepherd has more than 20 years of experience in medical device marketing in small startups and top-tier companies. After her industry career, including her role as VP of marketing for Oridion Medical, where she boosted the company valuation prior to its acquisition, director of marketing for Philips Medical, and senior management roles at Boston Scientific Corp., she founded Medi-Vantage. Medi-Vantage provides marketing, business strategy, and innovation research for the medical device, diagnostic, and digital health industries. The firm quantitatively and qualitatively sizes and segments opportunities, evaluates new technologies, provides marketing services, and assesses prospective acquisitions. Shepherd has taught marketing and product development courses and is a member of the Aligo Medtech Investment Committee (www.aligo.com). She can be reached at 855-343-3100. Visit her website at www.medi-vantage.com.