Collecting user data is a competitive disadvantage

a post by Cary Doctorow for the Boing Boing blog


Warren Buffet is famous for identifying the need for businesses to have "moats" and "walls" around their profit-centers to keep competitors out, and data-centric companies often cite their massive collections of user-data as "moats" that benefit from "network effects" to make their businesses good investments.

In a smart, eye-opening essay, Martin Casado and Peter Lauten from the VC firm Andreesen Horowitz dismantle the idea that data benefits from "network effects" and that it presents any kind of "moat" to protect businesses: instead, the VCs demonstrate how collecting data gets more expensive, and less useful, over time.

To understand why, think of Netflix's data-collection, performed in service to its famous recommendation engine, which suggests programs you might enjoy based on the preferences of people who are similar to you. When Netflix is starting out, it needs to develop a "minimum viable corpus" in order to produce recommendations, but once that data is in place, new data produces diminishing returns in recommendations. Going from 100 to 1,000,000 users allows Netflix to dramatically improve its recommendations, but going from 1,000,000 to 1,000,100 (or even 2,000,000) produces very little new benefit.

Continue reading

---

100,000 false positives for every real terrorist: Why anti-terror algorithms don't work

an article by Timme Bisgaard Munk (University of Copenhagen, Denmark) published in First Monday Volume 22 Number 9 (September 2017)

Abstract

Can terrorist attacks be predicted and prevented using classification algorithms? Can predictive analytics see the hidden patterns and data tracks in the planning of terrorist acts?

According to a number of IT firms that now offer programs to predict terrorism using predictive analytics, the answer is yes. According to scientific and application-oriented literature, however, these programs raise a number of practical, statistical and recursive problems. In a literature review and discussion, this paper examines specific problems involved in predicting terrorism.

The problems include the opportunity cost of false positives/false negatives, the statistical quality of the prediction and the self-reinforcing, corrupting recursive effects of predictive analytics, since the method lacks an inner meta-model for its own learning- and pattern-dependent adaptation.

The conclusion is algorithms don’t work for detecting terrorism and is ineffective, risky and inappropriate, with potentially 100,000 false positives for every real terrorist that the algorithm finds.

Full Text (HTML)

---