Maria Shepherd, President and Founder, Medi-Vantage08.17.21
Online, remote usability testing at home for orthopedic patients (e.g., braces, prosthetics, assistive devices, eHealth, etc.) has turned out to be a valuable option that goes beyond the need first demonstrated by the pandemic. Usability testing for patients in their home use environment is far more insightful than having them come to a research facility, because the home-use environment (under full HIPAA regulations, of course) presents many more distractions than those found in usability testing facilities. Dogs bark, deliveries require signatures, children are crying, and family members are watching television.
When the pandemic surged and the lockdown was mandated, an abrupt recession began in the healthcare industry that caused a cascade of economic issues from hospitals to medtech suppliers. Breaks in the links of the supply chain, which had grown over-dependent on cost-effective off-shore labor and device production costs, caused near panic when it came to essential products for healthcare workers such as masks, gloves, and protective gear. Many clinical studies were shut down or delayed, as were medical device human factors usability testing studies.
Medical Device Usability Testing Is Important
What we learned from the pandemic was medical device product development teams needed full-strength usability testing processes to preserve their device development schedules. The human factors process in the medical device domain requires a risk-centric approach, also called a “prevention through design” strategy throughout the product development lifecycle. Stopping usability testing meant halting the device development process, which experienced inevitable delays and created an impact on productivity of R&D teams.
The primary objective of human factors usability testing is to design out or improve the performance of features of a system or device that could lead to human error. The human factors engineering process uses three phases: (1) preliminary analysis (also called contextual inquiry), (2) a series of formative tests followed by design modification, and (3) validation testing (also called summative testing).
For example, in a 2019 study [An Electronic Health Tool to Prepare for the First Orthopedic Consultation: Use and Usability Study (n=144)], it was reported the use of an educational electronic tool to coach patients with knee and hip OA for the initial orthopedic consultation is feasible and improvements should be made to the content of the electronic tool to improve usability for all types of end users. There were three components in the eHealth tool (information, consultation, and medication), and the medication component was the least frequently used (34 percent). The mean SUS (system usability scale) score was 64.8. There was no difference comparing the SUS scores of active and superficial users (mean difference 0.04, 95 percent CI 7.69 to 7.77). Participants with a higher baseline knowledge of OA [odds ratio (OR) 1.2, 95 percent CI 1.0 to 1.4] and who used the internet less frequently in their daily life (OR 0.6, 95 percent CI 0.5 to 0.9) had a higher probability of using the eHealth tool. There were no differences between the clinical characteristics and demographics of active and superficial users.1
For comparative purposes, based on research, a SUS score greater than 68 is considered above average, while less than 68 is considered below average.2,3
Patient Usability Testing in the Home Environment
During the pandemic, the Medi-Vantage team employed remote, online usability testing as healthcare providers would not risk additional exposure by engaging in face-to-face testing at usability testing facilities. Patients also reported they were reluctant to go to testing facilities in person and run the risk of being infected by COVID-19. Our biggest barrier was overcoming a fundamental condition of medical device usability testing—the ability to get a video recording of the end-user using the device without showing that end-user’s face or other distinguishing features, which would violate HIPAA regulations. The Medi-Vantage team worked diligently to get the technology to record remote, online usability testing, and once we developed a successful methodology, we pivoted to remote, online usability testing.
What first seemed like a crushing obstacle turned out to have a silver lining. Patients in their home environments encounter distractions and issues that can interfere with their use of a medical device, which can cause unforeseen use errors that never would have been captured in the environment of a usability testing facility.
During 2020, we performed multiple online searches to see how other usability testing companies were managing the usability testing restrictions of the pandemic. We learned many had stopped usability testing since the A/V recording requirement could not be met. A/V recording is a must-have requirement for medical devices so R&D teams can see and reliably review the errors and successes of patients to gain the most successful outcomes of the entire human factors process. Our pivot strategy was developed to satisfy clients and help them meet their deadlines, recruit end-users into our remote and online usability testing studies so they didn’t have to visit an in-person facility (risking infection), and ensure FDA would get the results it needed. We were able to gain a glimpse into the home use of medical devices (under strict HIPAA guidelines) to learn what really happens when patients interact with our products.
How Many Usability Studies Are Needed?
As it turns out, the number of usability studies required depends on the complexity of the device. If the device has a graphic user interface; it can take multiple formative studies to achieve the end goal. In an insulin pump study, five formative studies (Table 1) were required to add almost 10 points to the SUS.1
Usability Testing Reduces Recall Potential
A recent report estimated device recalls, warranty costs, and lawsuits cost the medical device industry $2.5 billion to $5 billion annually.4 Device manufacturers must produce high quality devices to reduce the costs related to warranties and recalls. Even worse, the penalty of a single quality event, such as a large recall, has been as high as $600 million for medical device companies. Further, that figure is only the starting point as expensive lawsuits will likely follow. This represents a huge potential loss for medical device manufacturers.
Class I recalls—those FDA considers the most serious—increased substantially between 2016 and 2018. The average of 310,158 affected units per recall jumped to 511,017 affected units per recall between 2016 to 2017. Then, between 2017 and 2018, the total number of affected units surged to greater than 50 million per quarter. Responsible for 78 recalls, software issues were the leading cause in 1Q18, followed by mislabeling issues, which were attributed as the primary cause in 73 recalls.5
Risk Mitigation
The goal for human factors usability testing is risk mitigation in terms of device performance. While mislabeling may be difficult to detect in human factors usability testing, software issues are common root causes of use error and are frequently identified in medical device usability testing.
Many classes of medical devices transfer easily to a remote, online usability testing methodology. Online methods can detect at-home insights that provide the type of rich data that accurately links serious use errors to their original root cause. Online, remote usability testing allows patients and caregivers to assess products in their home environment, while demonstrating new insights to researchers to see exactly what end-users are doing with full video capture of their interactions with the device.
This process is an opportunity for medical device product development teams to gain a greater range of testing methods outside of those only seen in usability lab testing. Many of the challenges encountered in contextual inquiry, formative, and summative human factors studies can be mitigated earlier in the process of medical device development (and at lower costs) by researching early-stage product concepts and working with human factors professionals to adopt from a toolkit of new testing methodologies. This reduces medtech company product development cycle times and will reduce expenditures that can often occur with late-stage user testing.
The New Silver Lining
Sometimes, a silver lining appears when we think the worst has happened. Who hasn’t had to change strategies and lifestyles in the face of the pandemic? A great example is the rise in use and popularity of telemedicine, generated by COVID-19. New, unperceived benefits emerged during the rapid implementation of telemedicine, such as broader patient access. In human factors, we can now see into the patient’s home and use environment, while meeting HIPAA regulations, driving knowledge that will increase medical device safety, avoid product development delays, and ultimately improve patient outcomes.
References
Maria Shepherd has more than 20 years of leadership experience in medical device/life-science marketing in small startups and top-tier companies. After her industry career, including her role as vice president of marketing for Oridion Medical, where she boosted the company valuation prior to its acquisition by Medtronic, director of marketing for Philips Medical, and senior management roles at Boston Scientific Corp., she founded Medi-Vantage. Medi-Vantage provides marketing and business strategy and innovation research for the medical device industry. The firm quantitatively and qualitatively sizes and segments opportunities, evaluates new technologies, provides marketing services, and assesses prospective acquisitions. Shepherd has taught marketing and product development courses, speaks regularly at medtech conferences, and can be reached at mshepherd@medi-vantage.com. Visit her website at www.medi-vantage.com.
When the pandemic surged and the lockdown was mandated, an abrupt recession began in the healthcare industry that caused a cascade of economic issues from hospitals to medtech suppliers. Breaks in the links of the supply chain, which had grown over-dependent on cost-effective off-shore labor and device production costs, caused near panic when it came to essential products for healthcare workers such as masks, gloves, and protective gear. Many clinical studies were shut down or delayed, as were medical device human factors usability testing studies.
Medical Device Usability Testing Is Important
What we learned from the pandemic was medical device product development teams needed full-strength usability testing processes to preserve their device development schedules. The human factors process in the medical device domain requires a risk-centric approach, also called a “prevention through design” strategy throughout the product development lifecycle. Stopping usability testing meant halting the device development process, which experienced inevitable delays and created an impact on productivity of R&D teams.
The primary objective of human factors usability testing is to design out or improve the performance of features of a system or device that could lead to human error. The human factors engineering process uses three phases: (1) preliminary analysis (also called contextual inquiry), (2) a series of formative tests followed by design modification, and (3) validation testing (also called summative testing).
For example, in a 2019 study [An Electronic Health Tool to Prepare for the First Orthopedic Consultation: Use and Usability Study (n=144)], it was reported the use of an educational electronic tool to coach patients with knee and hip OA for the initial orthopedic consultation is feasible and improvements should be made to the content of the electronic tool to improve usability for all types of end users. There were three components in the eHealth tool (information, consultation, and medication), and the medication component was the least frequently used (34 percent). The mean SUS (system usability scale) score was 64.8. There was no difference comparing the SUS scores of active and superficial users (mean difference 0.04, 95 percent CI 7.69 to 7.77). Participants with a higher baseline knowledge of OA [odds ratio (OR) 1.2, 95 percent CI 1.0 to 1.4] and who used the internet less frequently in their daily life (OR 0.6, 95 percent CI 0.5 to 0.9) had a higher probability of using the eHealth tool. There were no differences between the clinical characteristics and demographics of active and superficial users.1
For comparative purposes, based on research, a SUS score greater than 68 is considered above average, while less than 68 is considered below average.2,3
Patient Usability Testing in the Home Environment
During the pandemic, the Medi-Vantage team employed remote, online usability testing as healthcare providers would not risk additional exposure by engaging in face-to-face testing at usability testing facilities. Patients also reported they were reluctant to go to testing facilities in person and run the risk of being infected by COVID-19. Our biggest barrier was overcoming a fundamental condition of medical device usability testing—the ability to get a video recording of the end-user using the device without showing that end-user’s face or other distinguishing features, which would violate HIPAA regulations. The Medi-Vantage team worked diligently to get the technology to record remote, online usability testing, and once we developed a successful methodology, we pivoted to remote, online usability testing.
What first seemed like a crushing obstacle turned out to have a silver lining. Patients in their home environments encounter distractions and issues that can interfere with their use of a medical device, which can cause unforeseen use errors that never would have been captured in the environment of a usability testing facility.
During 2020, we performed multiple online searches to see how other usability testing companies were managing the usability testing restrictions of the pandemic. We learned many had stopped usability testing since the A/V recording requirement could not be met. A/V recording is a must-have requirement for medical devices so R&D teams can see and reliably review the errors and successes of patients to gain the most successful outcomes of the entire human factors process. Our pivot strategy was developed to satisfy clients and help them meet their deadlines, recruit end-users into our remote and online usability testing studies so they didn’t have to visit an in-person facility (risking infection), and ensure FDA would get the results it needed. We were able to gain a glimpse into the home use of medical devices (under strict HIPAA guidelines) to learn what really happens when patients interact with our products.
How Many Usability Studies Are Needed?
As it turns out, the number of usability studies required depends on the complexity of the device. If the device has a graphic user interface; it can take multiple formative studies to achieve the end goal. In an insulin pump study, five formative studies (Table 1) were required to add almost 10 points to the SUS.1
Usability Testing Reduces Recall Potential
A recent report estimated device recalls, warranty costs, and lawsuits cost the medical device industry $2.5 billion to $5 billion annually.4 Device manufacturers must produce high quality devices to reduce the costs related to warranties and recalls. Even worse, the penalty of a single quality event, such as a large recall, has been as high as $600 million for medical device companies. Further, that figure is only the starting point as expensive lawsuits will likely follow. This represents a huge potential loss for medical device manufacturers.
Class I recalls—those FDA considers the most serious—increased substantially between 2016 and 2018. The average of 310,158 affected units per recall jumped to 511,017 affected units per recall between 2016 to 2017. Then, between 2017 and 2018, the total number of affected units surged to greater than 50 million per quarter. Responsible for 78 recalls, software issues were the leading cause in 1Q18, followed by mislabeling issues, which were attributed as the primary cause in 73 recalls.5
Risk Mitigation
The goal for human factors usability testing is risk mitigation in terms of device performance. While mislabeling may be difficult to detect in human factors usability testing, software issues are common root causes of use error and are frequently identified in medical device usability testing.
Many classes of medical devices transfer easily to a remote, online usability testing methodology. Online methods can detect at-home insights that provide the type of rich data that accurately links serious use errors to their original root cause. Online, remote usability testing allows patients and caregivers to assess products in their home environment, while demonstrating new insights to researchers to see exactly what end-users are doing with full video capture of their interactions with the device.
This process is an opportunity for medical device product development teams to gain a greater range of testing methods outside of those only seen in usability lab testing. Many of the challenges encountered in contextual inquiry, formative, and summative human factors studies can be mitigated earlier in the process of medical device development (and at lower costs) by researching early-stage product concepts and working with human factors professionals to adopt from a toolkit of new testing methodologies. This reduces medtech company product development cycle times and will reduce expenditures that can often occur with late-stage user testing.
The New Silver Lining
Sometimes, a silver lining appears when we think the worst has happened. Who hasn’t had to change strategies and lifestyles in the face of the pandemic? A great example is the rise in use and popularity of telemedicine, generated by COVID-19. New, unperceived benefits emerged during the rapid implementation of telemedicine, such as broader patient access. In human factors, we can now see into the patient’s home and use environment, while meeting HIPAA regulations, driving knowledge that will increase medical device safety, avoid product development delays, and ultimately improve patient outcomes.
References
Maria Shepherd has more than 20 years of leadership experience in medical device/life-science marketing in small startups and top-tier companies. After her industry career, including her role as vice president of marketing for Oridion Medical, where she boosted the company valuation prior to its acquisition by Medtronic, director of marketing for Philips Medical, and senior management roles at Boston Scientific Corp., she founded Medi-Vantage. Medi-Vantage provides marketing and business strategy and innovation research for the medical device industry. The firm quantitatively and qualitatively sizes and segments opportunities, evaluates new technologies, provides marketing services, and assesses prospective acquisitions. Shepherd has taught marketing and product development courses, speaks regularly at medtech conferences, and can be reached at mshepherd@medi-vantage.com. Visit her website at www.medi-vantage.com.