In brief

The UK’s National Health Service (NHS) partnered with the AI technology company and Google subsidiary, DeepMind, to develop a smartphone application that enables clinicians to detect early signs of kidney failure by using an instant alert system. The app, called Streams, directs clinicians to patients who are at risk of, or who have already developed, a serious condition called acute kidney injury (AKI).

Although it has been received with “great enthusiasm” as an AKI management tool, the collaboration suffered from a lack of clarity and openness regarding DeepMind’s business model and their interests, especially with regard to data protection.[1]

The challenge

Clinicians in the NHS do not have the tools to analyse each medical test result instantly, determine the right treatment for a patient, and make sure that the case of every patient who needs complex or urgent care is escalated to the right specialist unit straight away. Research published in the journal, Surgery, showed that half of NHS hospital patients do not get the care they need quickly enough, usually because of poor communication, particularly when one team of doctors or nurses hands over to another during shift changes.[2]

In addition to this, NHS staff are unable to use the latest tools that could aid them in dealing with these challenges. “Everybody has a smartphone and we all use apps all the time,” says Mustafa Suleyman, cofounder and head of applied AI at DeepMind. “But the people doing incredible work saving lives every day are hampered by using desktop computers and software designed a long time ago.”[2]

This can prove fatal when a condition such as AKI is not recognised in time. AKI is suddenly occurring kidney damage, usually as a result of another serious illness or injury. When the kidney stops working properly, the body's toxic waste products can build up in the blood and harm other organs. In the UK, AKI affects one in six in-patients and may result in fluid overload or respiratory failure and is thus strongly associated with adverse outcomes, including death, prolonged hospitalisation, requirement for renal replacement theory, and a need for high dependency or intensive care.[3]

It is also associated with an increased risk of chronic kidney disease. AKI is linked to 40,000 deaths a year in the UK, a quarter of which are preventable, according to NHS estimates – if clinicians were able to intervene earlier and therefore more effectively. AKI can be difficult to detect and treat quickly and is also expensive: it costs the NHS over GBP1 billion – which is greater than the annual cost of treating breast cancer.[4]

The initiative

To address this “failure to rescue”, the Royal Free London NHS Foundation Trust paired with DeepMind, a UK-based subsidiary of Google, and entered into a five-year partnership in November 2016.[5] This landmark agreement meant that specialists in healthcare and technology collaborated and began to develop the Streams application. It has been trialled by clinicians at the Royal Free Hospital since January 2017.[6] Streams is DeepMind’s first major healthcare project and primarily seeks to identify people who could be at risk of AKI and to assist in its management. However, “the Trust provided personal data of around 1.6 million patients [but they] were not adequately informed that their data would be used as part of the test”.[7]

The app’s design principle is simple: as with a breaking news alert on a mobile phone, the technology notifies relevant clinicians as soon as test results show that a patient is at risk of becoming seriously ill. It provides in one place all the necessary information to take appropriate action.[8] Finding an AKI early on can mean that it can be treated sooner, and so the damage to the kidney and other organ systems is minimised.[6]

Streams acts by analysing patient information, such as details about blood and liver function and then sends an alert to a clinician's smartphone to let them know that a patient needs their attention.[6] A clinical response team comprising nephrologists and critical care nurses responds to these AKI alerts by reviewing individual patients and administering interventions, according to existing clinical practice guidelines.[3]

The public impact

Streams’ ability to store all patient information in one place – accessible from a smartphone – has been shown to reduce the administrative burden on staff, meaning that they can dedicate more time to delivering direct patient care. “Prior to implementation, such results were normally batch reviewed by nonspecialists at the end of the day and may have been seen several hours after the results first become available. Clinicians may have opted to review results earlier, but this process relied upon repeated accessing of the results systems, as clinicians did not know when results were ready... However, this process was cumbersome and may have been unreliable.”[3] With the Streams app, the routine work of reading scans, for example, is done by the “NHS AKI algorithm to routinely collected serum creatinine data in hospital inpatients”, rather than by a healthcare professional.[3]

On a typical day at the Royal Free, Streams analysed 2,211 blood tests, detected 66 changes in kidney function, and ultimately issued 23 AKI alerts, of which 11 cases required immediate action. By freeing up doctors from paperwork, the Trust believes that it can deliver more than half a million hours of extra direct care.[9] “The instant alerts about some of our most vulnerable patients mean we can get the right care to the right patients much more quickly,” according to Sarah Stanley, a nurse at the Royal Free and one of the clinicians who uses Streams on a daily basis. Rather than have to locate a working desktop computer and sort through the patient's electronic record to find all the information she needs in order to decide if they have suffered an AKI, the information is delivered to her directly. "It's a massive time-saver. You save one to two hours a day just through filtering information.”[6]

The app also helps clinicians communicate with patients more effectively. Because most of the patient data involved in healthcare is unstructured, i.e. comprised of different test results, scans, consultation notes, and details of appointment follow-ups, this means that healthcare providers have a lot of data that needs collating and turning into usable information.[9] Clinicians use the app to explain to patients the changes in their kidney health. Because Streams shows changes through graphs and trend lines, patients can clearly see if certain markers of kidney health are spiking or falling, and by how much, making the progress of their condition easier to grasp.[6]

Despite these gains, the NHS has received huge criticism over its handling of patients’ data: the details of 1.6 million people were made available to DeepMind – including large amounts of data over and above that which relates to AKI. The Information Commissioner's Office (ICO) began an investigation into the collaboration between the NHS and DeepMind. The ICO found that the records of 1.6 million patients were being used in the Streams pilot without their being adequately notified.[9] According to the ICO, the Trust was not as transparent as it should have been, and it was instructed to take a number of measures to comply with the Data Protection Act, such as providing an opt-out form for patients on the Trust’s website.

However, the research project was allowed to continue, and Streams is now planned to be extended beyond AKI to help care for patients with other serious conditions, including sepsis and organ failure.[10] In a statement, the Trust said that the app will be used as a diagnostic support tool for a far wider range of illness, alerting doctors early on to patients who are at risk of becoming seriously ill. “Streams will be extended beyond AKI to help care for patients with other serious conditions including sepsis and organ failure.”[10]

Written by Julia Schnatz

Have an idea for a case study? Print

What did and didn't work

All cases in our Public Impact Observatory have been evaluated for performance against the elements of our Public Impact Fundamentals.

Legitimacy

Stakeholder Engagement Fair

DeepMind worked closely with clinicians at the Royal Free, “so it could understand how the technology could fit in their day-to-day lives”.[11] Consultant nephrologist and associate medical director for patient safety at the Royal Free, Dr Chris Laing, who helped design the app and oversaw the two initial pilots at the hospital, said: "using Streams meant I was able to review blood test results for patients at risk of AKI within seconds of it becoming available. This system of direct alerts and the ability to prioritise patients was just not possible previously. By using Streams, I intervened earlier and was able to improve the care of over half the patients Streams identified in our pilot studies."[12]

Even though Streams was welcomed by doctors and nurses, its development was marked by a failure to address patient privacy, and a lack of ex ante discussions and authorisations with relevant regulators such as the ICO.[1] “There’s no doubt the huge potential that creative use of data could have on patient care and clinical improvements, but the price of innovation does not need to be the erosion of fundamental privacy rights,” said Elizabeth Denham, the information commissioner.[7] Patients were not informed about the fact that their medical records would be transferred to Google’s servers and hence did not explicitly consent. "We've asked the Trust to commit to making changes that will address those shortcomings, and their cooperation is welcome. The Data Protection Act is not a barrier to innovation, but it does need to be considered wherever people’s data is being used," Denham continued. To address this, DeepMind confirmed that it was building an audit infrastructure that “will give our partners real-time visibility on how we’re processing data”.[4]

Political Commitment Strong

On the basis that the prompt and reliable identification of AKI cases to clinicians may trigger improved care, NHS England issued a national patient safety alert on “standardising the early identification of AKI”. This mandated the installation of a new detection algorithm in each NHS hospital, so that potential AKI incidents could be flagged to clinicians. NHS England’s national “Think Kidneys” programme has provided best practice examples of how AKI alerts may be clinically deployed.

However, simply alerting a clinician to the presence of a possible AKI incident is not necessarily enough to improve outcomes. A much richer clinical dataset is required to help clinicians prioritise, diagnose and manage patients. The UK’s National Institute for Health and Care Excellence (NICE) guidelines on AKI management suggest that patients with more severe AKI might benefit from care delivery by suitably expert clinicians, for example as part of a “rapid referral” nephrology service.[3]

The UK government has “also been keen to position itself as a supporter of AI, providing public funds for the sector and, in its Industrial Strategy white paper, identifying AI and data as one of four so-called ‘Grand Challenges’ where it believes the UK can ‘lead the world for years to come’ – including specifically namechecking DeepMind as one of a handful of leading-edge homegrown AI businesses for the country to be proud of”.[13] With this backing, in July 2015 clinicians from the Trust’s hospitals approached DeepMind to discuss developing AI software using the Trust’s patient data.[1]

The technology, which is run through a smartphone app, also has the support of Lord Darzi, a surgeon and former health minister in the Blair government, who is director of the Institute of Global Health Innovation at Imperial College London. “Innovation is the only solution we have for a sustainable NHS, both economically and in meeting the challenges and demands upon it,” Darzi told The Guardian in a preview of the technology.[2]

Public Confidence Fair

The project started to attract significant public interest in May 2016 when investigative journalists revealed the scope of the data-sharing agreement between DeepMind and the Trust, through which 1.6 million health records of identifiable patients were transferred to Google’s servers. Patients whose data was transferred were neither asked for their consent nor informed about the data transfer.[13]

This led to an investigation by Dame Fiona Caldicott, the national data guardian, who concluded that “deal had an inappropriate legal basis”,[14] and by the ICO, which raised a number of concerns. They concluded that the agreement between the Trust and DeepMind was insufficiently robust, that the Trust had not conducted a privacy impact assessment at the start of the project, and that it had not done enough to tell patients about these arrangements.[15] Added to this adverse publicity, privacy group medConfidential said that the information had been obtained without patient consent, and that it was excessive because the records covered information other than kidney function. Following these negative headlines, the Trust defended itself against these claims by stating that the data was used to enhance direct patient care, for which no patient consent is required. This defence was not accepted by the ICO.[12]

For its part, DeepMind promised to be more open with patients about its health plans and to hold regular patient engagement forums.[16] In a public statement, it stated that “the AI division had concentrated on building tools for clinicians rather than thinking about how the project should be shaped by the needs of patients and the public”.[4] In an independent review, DeepMind confessed to a “lack of work on public engagement particularly in relation to links between DeepMind and Google and public perception that data processed by DeepMind could be shared with Google”.[17] Several researchers also identify inadequacies in the architecture of the deal between the NHS and DeepMind, in its public communication, and in the processes of public sector oversight.[1]

Despite this controversy, patients’ experience has been largely positive. More than 26 doctors and nurses at the Royal Free Hospital are now using Streams, and it is alerting them to a daily average of 11 patients at risk of AKI.[18] One patient who had the benefit of the new app was Afia Ahmed, who had suffered complications following the birth of her daughter by emergency caesarean and developed sepsis during her labour, which then led to AKI. Using data from her blood test, the Streams app detected a problem with her kidney function and an alert was sent to a specialist kidney doctor.[19] The kidney specialist was able to provide guidance to the obstetric team on Afia’s condition and advised them to adjust her antibiotics dosage and her intravenous fluid treatment and stop using painkillers that might put a strain on her kidneys. Afia continued to be monitored by a kidney specialist until her kidney function recovered and she was discharged.[18]

Policy

Clear Objectives Weak

When DeepMind and the Royal Free London NHS Foundation Trust announced their partnership, Mustafa Suleyman stated that they intended “to figure out how we can jointly innovate in some of the systems most complex and tricky problems in healthcare”. David Sloman, chief executive of the Trust, said it would enable the organisation “to do what fundamentally every single one of our staff comes to work to do which is about saving lives and improving care”.[1]

However, there is no official Trust document that clearly outlines the partnership’s objectives nor why it chose DeepMind as a partner. A memorandum of understanding from January 2016 referenced plans for DeepMind to develop new systems for the Trust as part of a “broad ranging, mutually beneficial partnership... to work on genuinely innovative and transformational projects”. Presentations by DeepMind remained equally vague, elaborating a vision for a “truly digital NHS”, comprising “massively improved patient care, actionable analytics, advanced research both at the hospital-wide level and the population-wide level, and an open innovation ecosystem”.[1]

Neither DeepMind nor the Trust made reference to the use of patients’ data before they launched the app. The scope of the transfer of patients’ data from the Royal Free to DeepMind was defined only after the ICO’s investigation and report.[7]

Evidence Strong

A range of options to detect AKI were initially considered by leading clinicians at the NHS, including the use of AI. However, the Streams project eventually resulted in a much more straightforward software solution, using an existing NHS algorithm for early detection of AKI rather than AI technology. “The ‘Think Kidneys’ NHS England National programme has provided best practice examples of how AKI alerts may be clinically deployed.”[3]

A 12-week pilot phase began at the Royal Free in January 2017, and consisted of the observation of the response team, interviews with the response team, and improvements to the pathway of implementation.[3] Up to 40 clinicians started to use Streams in the first phase of the rollout, and the implementation was then phased across all Trust sites.[9] It will also be trialled at St Mary's Hospital in Paddington.

 

Feasibility Fair

The implementation of Streams “on the ground” in the hospital was made financially feasible through NHS funding. DeepMind focused on ensuring its technical feasibility, but Streams’ overall feasibility has been severely hampered by legal considerations, specifically the requirements of the Data Protection Act.

The app itself is, according to Nurse Sarah Stanley, "very user-friendly". This was a result of DeepMind’s efforts in the user interface design, for which they carefully scrutinised and refined every element of clinicians’ workflows.[6] “We wanted to understand, in as much detail as possible, what information they needed to have at their fingertips, and how we could present it so that it was most useful to them in an urgent care situation,” said Dr Dominic King, clinical lead at DeepMind Health.[11]

As the project unfolded, however, feasibility issues arose in terms of legal concerns that had not been properly evaluated before implementation.[1] The ICO ruled that processing the data 1.6 million patients without their explicit consent did not amount to processing for direct care purposes and so did not comply with data protection law. “Concerns were also raised that not enough was done to ensure that patients and the public were aware of the project prior to data processing.”[3] The ICO’s report concluded that "the collaboration has suffered from a lack of clarity and openness", adding that "if DeepMind and Royal Free had endeavoured to inform past and present patients of plans for their data, initially and as they evolved, either through email or by letter, much of the subsequent fallout would have been mitigated".[1]

Action

Management Good

Simply developing a smartphone app does not ultimately lead to an improvement in practice; rather, efficient management structures need to be in place to ensure that Streams provides improved outcomes for patients and clinicians alike. The Royal Free London NHS Foundation Trust in particular are responsible for the management of the applications implementation. Together with other NHS partners, they have developed clinical practice guidelines for the management of AKI and the use of the Streams app.[3]

In terms of the management mechanisms regarding data, however, the ICO clearly indicated that there were some issues in the development of the Streams app when it came to patient confidentiality. However, the Trust did accept the ICO’s findings concerning this lack of clarity and openness, and took a number of remedial steps to address them. These include putting a much stronger agreement in place with DeepMind and providing more information on the Royal Free’s website about the arrangements with DeepMind.[15]

Sir David Sloman confirmed that a respect for patient confidentiality was fundamental to the operation of the Royal Free Hospital. The senior leadership team as a whole confirmed that they had completed their information governance training in accordance with the annual training cycle. They described a number of steps the Trust had taken to improve information governance, including making a major investment in the hospital’s cybersecurity defences and inviting two lay members to take part in the Information Governance Group.[15] Together with DeepMind, they also arranged a meeting for an independent review panel of external reviewers.

After the ICO’s investigation and concluding report was published, the Trust put structures and processes in place to manage its use of confidential patient information, e.g. the Information Governance Group, which meets quarterly to discuss data protection and confidentiality issues. The Royal Free now has a data protection officer and an information governance manager to manage data protection and confidentiality issues.[15] Recently, the NHS announced that it will be strengthening the anonymisation of patient files in order to protect patients’ privacy further.[20]

Measurement Fair

So far, neither DeepMind nor the Trust has published data on Streams’ effect on patient health and how much time and money it can save. Following the initial 12-week pilot phase, an 18-week evaluation phase began, during which outcome data was collected, analysed and interpreted.[3] A service evaluation states that an “interim analysis is planned halfway through the evaluation phase”, of which outcomes will be used to inform future decisions and strategy regarding Streams’ further development.[3]

Alignment Fair

The evidence of alignment between the different stakeholders is mixed. While the Trust, the clinicians, and DeepMind shared common goals and interests in the collaboration, their interdependency has caused some issues and the lack of alignment with regulators has caused major problems.

The alignment of interests between the Trust and DeepMind initially appeared to be strong as regards to working towards more efficient healthcare operations. Both the general strategy and specific aims of the collaboration were made clear from the beginning of Streams’ development in a memorandum of understanding between the two parties.[14] That said, alignment issues have developed as the partnership has gone on. DeepMind and the Trust are dependent on each other and development cannot happen without progress from both partners. This interdependency has caused some issues. Royal Free have raised frustrations that they cannot develop anything further without DeepMind, when perhaps an open source solution would allow for much faster and reactive development. DeepMind has also raised frustrations about the slow pace and bureaucratic structures of the NHS and are thus thinking of scaling back development.

Regarding alignment of clinicians, the app itself was co-designed by specialists from both DeepMind and the Trust in order to meet specific clinicians’ needs. For example, the smartphone model chosen as the test device for Streams needed to be precisely the right size for the pockets of hospital scrubs, so that users could make sure it would not fall out while they were going about their work on the ward.[6] Mustafa Suleyman said that “within three weeks or so of meeting our first nurses… we had a working prototype (not connected to any data) that nurses and doctors could actually point to and say this button is in the wrong place, this colour is difficult to read, this menu hierarchy is sort of in the wrong order”.[21]

The lack of alignment with regulators, who represented the interests of the Trust’s patients, caused significant problems. The ICO criticised the arrangement whereby DeepMind had been given access to five years’ data of Royal Free Hospital patients, none of whom had given their consent. The Trust claimed that consent was not required, as the data was used for direct patient care and still controlled by the Trust. The ICO concluded that “the Royal Free could do more to bring information about Streams (and information about its processing more generally) to the attention of patients when those patients attend the Royal Free in person. For example, the Royal Free could include a signpost to this information on the electronic registration terminals.”[15]

Bibliography

[1] Google DeepMind and healthcare in an age of algorithms, Julia Powles and Hal Hodson, March 2017, Health Technology

(https://link.springer.com/content/pdf/10.1007%2Fs12553-017-0179-1.pdf)

[2] Smart care: how Google DeepMind is working with NHS hospitals, Sarah Boseley and Paul Lewis, 24 February 2016, The Guardian

(https://www.theguardian.com/technology/2016/feb/24/smartphone-apps-google-deepmind-nhs-hospitals)

[3] Service evaluation of the implementation of a digitally-enabled care pathway for the recognition and management of acute kidney injury, A. Connell et al, June 2017, F1000 Research

(https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5510018/)

[4] Streams in NHS hospitals, DeepMind

(https://deepmind.com/applied/deepmind-health/working-partners/how-were-helping-today/)

[5] A systematic review to identify the factors that affect failure to rescue and escalation of care in surgery, M. Johnston et al, 2015, Surgery, Volume 157 Issue 4

(https://www.sciencedirect.com/science/article/pii/S0039606014007132)

[6] DeepMind and the NHS: What it's really like to use Google's kidney health app, Jo Best, 10 January 2018, ZDNet

(https://www.zdnet.com/article/deepmind-and-the-nhs-what-its-really-like-to-use-googles-kidney-health-app/)

[7] Royal Free – Google DeepMind trial failed to comply with data protection law, 3 July 2017, Information Commissioner's Office

(https://ico.org.uk/about-the-ico/news-and-events/news-and-blogs/2017/07/royal-free-google-deepmind-trial-failed-to-comply-with-data-protection-law/)

[8] Our work with DeepMind, Royal Free London NHS Foundation Trust

(https://www.royalfree.nhs.uk/patients-visitors/how-we-use-patient-information/our-work-with-deepmind/)

[9] Google's DeepMind and the NHS: A glimpse of what AI means for the future of healthcare, Jo Best, 4 April 2017, ZDNet

(https://www.zdnet.com/article/googles-deepmind-and-the-nhs-a-glimpse-of-what-ai-means-for-the-future-of-healthcare/)

[10] Google DeepMind and Royal Free in five-year deal, Ben Heather, 22 November 2016, digitalhealth

(https://www.digitalhealth.net/2016/11/google-deepmind-and-royal-free-in-five-year-deal/)

[11] Google DeepMind’s Streams technology branded ‘phenomenal’, Hannah Crouch, 4 December 2017, digitalhealth

(https://www.digitalhealth.net/2017/12/google-deepmind-streams-royal-free/)

[12] Google's London AI powerhouse has set up a new healthcare division and acquired a medical app called Hark, Sam Shead, 25 February 2016, Business Insider

(https://www.businessinsider.de/google-deepmind-sets-up-healthcare-team-acquires-hark-and-builds-new-app-2016-2?r=UK&IR=T)

[13] UK Report warns DeepMind Health could gain 'excessive monopoly power', Natasha Lomas, 15 June 2018, TechCrunch

(https://techcrunch.com/2018/06/15/uk-report-warns-deepmind-health-could-gain-excessive-monopoly-power/)

[14] Google DeepMind 1.6m patient record deal 'inappropriate', Alex Hern, 16 May 2017, The Guardian

(https://www.theguardian.com/technology/2017/may/16/google-deepmind-16m-patient-record-deal-inappropriate-data-guardian-royal-free)

[15] Audit of the acute kidney injury detection system known as Streams, Linklaters LLP, 17 May 2018, for Royal Free London NHS Foundation Trust

(https://s3-eu-west-1.amazonaws.com/files.royalfree.nhs.uk/Reporting/Streams_Report.pdf)

[16] Google's DeepMind extends controversial Streams app, Jane Wakefield, 21 June 2017, BBC News

(https://www.bbc.com/news/technology-40352885)

[17] DeepMind Health Independent Review Panel Annual Report, June 2018, DeepMind Health

(https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=2ahUKEwiY56Xt57_cAhUIyIUKHR2WA3MQFjAAegQIABAC&url=https%3A%2F%2Fdeepmind.com%2Fdocuments%2F214%2FDeepMind%2520Health%2520%25E2%2580%2593%25C2%25A0%2520Independent%2520Reviewers%2520Report%25202018.pdf&usg=AOvVaw0YtWOlX0yv9OkhnXmTTi7v)

[18] New app helping to improve patient care, 27 February 2017, Royal Free London NHS Foundation Trust

(https://www.royalfree.nhs.uk/news-media/news/new-app-helping-to-improve-patient-care/)

[19] New mother receives pioneering kidney treatment after app detects life-threatening illness, Ross Lydall, 27 February 2017, Evening Standard

(https://www.standard.co.uk/news/health/new-mother-receives-pioneering-kidney-treatment-after-app-detects-lifethreatening-illness-a3476936.html)

[20] UK NHS will anonymise private medical data for use in AI systems, Ellen Tannam, 2 July 2018, siliconrepublic

(https://www.siliconrepublic.com/enterprise/google-deepmind-nhs-report)

[21] Google DeepMind: How, why, and where it's working with the NHS, Sam Shead, 10 July, 2016, Business Insider

(http://uk.businessinsider.com/google-deepmind-how-why-and-where-its-working-with-the-nhs-2016-7)

Sign up to stay updated on news about our meetings, our insights and our other activities.
Back to top