This article is from the source 'guardian' and was first published or seen on . It last changed over 40 days ago and won't be checked again for changes.
You can find the current article at its original source at https://www.theguardian.com/technology/2020/feb/05/welfare-surveillance-system-violates-human-rights-dutch-court-rules
The article has changed 4 times. There is an RSS feed of changes available.
| Version 1 | Version 2 |
|---|---|
| Welfare surveillance system violates human rights, Dutch court rules | Welfare surveillance system violates human rights, Dutch court rules |
| (32 minutes later) | |
| Government told to halt use of AI to detect fraud in decision hailed by privacy campaigners | Government told to halt use of AI to detect fraud in decision hailed by privacy campaigners |
| A Dutch court has ordered the immediate halt of an automated surveillance system for detecting welfare fraud because it violates human rights, in a judgment likely to resonate well beyond the Netherlands. | A Dutch court has ordered the immediate halt of an automated surveillance system for detecting welfare fraud because it violates human rights, in a judgment likely to resonate well beyond the Netherlands. |
| The case was seen as an important legal challenge to the controversial but growing use by governments around the world of artificial intelligence (AI) and risk modelling in administering welfare benefits and other core services. | The case was seen as an important legal challenge to the controversial but growing use by governments around the world of artificial intelligence (AI) and risk modelling in administering welfare benefits and other core services. |
| Campaigners say such “digital welfare states” – developed often without consultation, and operated secretively and without adequate oversight – amount to spying on the poor, breaching privacy and human rights norms and unfairly penalising the most vulnerable. | Campaigners say such “digital welfare states” – developed often without consultation, and operated secretively and without adequate oversight – amount to spying on the poor, breaching privacy and human rights norms and unfairly penalising the most vulnerable. |
| The UN special rapporteur on extreme poverty and human rights, Philip Alston, applauded the verdict and said it was “a clear victory for all those who are justifiably concerned about the serious threats digital welfare systems pose for human rights”. | The UN special rapporteur on extreme poverty and human rights, Philip Alston, applauded the verdict and said it was “a clear victory for all those who are justifiably concerned about the serious threats digital welfare systems pose for human rights”. |
| The decision “sets a strong legal precedent for other courts to follow”, he added. “This is one of the first times a court anywhere has stopped the use of digital technologies and abundant digital information by welfare authorities on human rights grounds.” | The decision “sets a strong legal precedent for other courts to follow”, he added. “This is one of the first times a court anywhere has stopped the use of digital technologies and abundant digital information by welfare authorities on human rights grounds.” |
| The verdict will be watched closely by welfare rights campaigners in the UK, where the government is accelerating the development of robots in the benefits system in a digitisation drive that vulnerable claimants fear could plunge them further into hunger and debt. | The verdict will be watched closely by welfare rights campaigners in the UK, where the government is accelerating the development of robots in the benefits system in a digitisation drive that vulnerable claimants fear could plunge them further into hunger and debt. |
| A Guardian investigation in October found the Department for Work and Pensions (DWP) had increased spending to about £8m a year on a specialist “intelligent automation garage” where computer scientists were developing more than 100 welfare robots, deep learning and intelligent automation for use in the welfare system. | |
| The Dutch government’s risk indication system (SyRI) is a risk calculation model developed over the past decade by the social affairs and employment ministry to predict the likelihood of an individual committing benefit or tax fraud or violating labour laws. | The Dutch government’s risk indication system (SyRI) is a risk calculation model developed over the past decade by the social affairs and employment ministry to predict the likelihood of an individual committing benefit or tax fraud or violating labour laws. |
| Deployed primarily in low-income neighbourhoods, it gathers government data previously held in separate silos, such as employment, personal debt and benefit records, and education and housing histories, then analyses it using a secret algorithm to identify which individuals might be at higher risk of committing benefit fraud. | |
| A broad coalition of privacy and welfare rights groups, backed by the largest Dutch trade union, argued that poor neighbourhoods and their inhabitants were being spied on digitally without any concrete suspicion of individual wrongdoing. SyRI was disproportionately targeting poorer citizens, they said, violating human rights norms. | A broad coalition of privacy and welfare rights groups, backed by the largest Dutch trade union, argued that poor neighbourhoods and their inhabitants were being spied on digitally without any concrete suspicion of individual wrongdoing. SyRI was disproportionately targeting poorer citizens, they said, violating human rights norms. |
| The court ruled that the SyRI legislation contained insufficient safeguards against privacy intrusions and criticised a “serious lack of transparency” about how it worked. It concluded in its ruling that, in the absence of more information, the system may, in targeting poor neighbourhoods, amount to discrimination on the basis of socioeconomic or migrant status. | The court ruled that the SyRI legislation contained insufficient safeguards against privacy intrusions and criticised a “serious lack of transparency” about how it worked. It concluded in its ruling that, in the absence of more information, the system may, in targeting poor neighbourhoods, amount to discrimination on the basis of socioeconomic or migrant status. |
| The system did not pass the test required by the European convention on human rights of a “fair balance” between its objectives, namely to prevent and combat fraud in the interest of economic wellbeing, and the violation of privacy that its use entailed, the court added, declaring the legislation was therefore unlawful. The Dutch government can appeal against the decision. | The system did not pass the test required by the European convention on human rights of a “fair balance” between its objectives, namely to prevent and combat fraud in the interest of economic wellbeing, and the violation of privacy that its use entailed, the court added, declaring the legislation was therefore unlawful. The Dutch government can appeal against the decision. |
| Christiaan van Veen, director of the digital welfare state and human rights project at New York University School of Law, said it was “important to underline that SyRI is not a unique system; many other governments are experimenting with automated decision-making in the welfare state”. | Christiaan van Veen, director of the digital welfare state and human rights project at New York University School of Law, said it was “important to underline that SyRI is not a unique system; many other governments are experimenting with automated decision-making in the welfare state”. |
| Van Veen cited Australia and the UK as countries where such concerns were particularly acute. “This strong ruling will set a strong precedent globally that will encourage activists in other countries to challenge their governments,” he said. | Van Veen cited Australia and the UK as countries where such concerns were particularly acute. “This strong ruling will set a strong precedent globally that will encourage activists in other countries to challenge their governments,” he said. |
| Alston predicted the judgment would be “a wake-up call for politicians and others, not just in the Netherlands”. The special rapporteur presented a report to the UN general assembly in October on the emergence of the “digital welfare state” in countries around the globe, warning of the need “to alter course significantly and rapidly to avoid stumbling, zombie-like, into a digital welfare dystopia”. | |
| In the UK, as well as contracts with the outsourcing multinationals IBM, Tata Consultancy and Capgemini, the DWP is also working with UiPath, a New York-based company co-founded by Daniel Dines, the world’s first “bot billionaire”, who last month said: “I want a robot for every person.” | |
| His software is being deployed in an effort to introduce machine learning to check benefit claims, which suggests welfare computers will autonomously learn and alter the way they make decisions with minimum human intervention. |