Weapons of Math Destruction by Cathy O’Neil

“..Models are, by their very nature, simplifications. No model can include all of the real world’s complexity or the nuance of human communication..”

– Cathy O’Neil, Math Nerd. Mum of 3, who still plans and cooks all the family’s meals and writes about modelling her kids’ meals (I’m guessing this blogger of MathBabe sounds more human (:D fine, is more able to relate) for being a “Mom” – little people after all have very little respect for PhDs. Harvard Math what? 🙂 )

Tangent alert: Remember this Nerd With Funky Hair –

Have Math Degree Will Crazy Hair? A thing about books and their covers comes to mind, and how some of the overly brainy seem to really enjoy toying with irony:

My absolute favourite quants are the ones who look like the aliens they are the ones who can be ironic. Irony requires being human enough enough of an understanding of human nature to be able to discern what makes the extraordinary dissent, the exquisite, just-over-the-top statement delivered while walking that fine line that qualifies wit from crass…

Then there’s the fact you simply have to understand the nature of the subject you are modelling because otherwise it’s all rubbish anyway – a person building a model has to make assumptions. Simplifications. Find proxies. Crucial decisions about what to include. What this means of course is that models and their blind spots are terrifying   evil  reflect the priorities and biases of their creators arguably as much as literature would.

People think numbers don’t lie, but in reality they lie all the time, because of how they were arrived at to begin with… Isn’t that just so exciting? 😀

If you don’t want to (potentially) be lied to then you cannot be afraid of numbers. Or of seeking knowledge. (And btw I don’t like people who jerk others around because of the little extra stuff they know… because no one can know everything ok, so either agree to share the way various team sportspeople must, in order to get the ball into the basket or goal so their team wins, or spend the eternity your work life feels like in a hell of insecurity of your own making. <shrugs>)

 

A la Nassim Taleb’s self-fulfilling prophecies, that is the first thing that tells you that deep down they themselves believe they don’t know enough… Taleb said “When a corporate feels the need to ‘instil (investor) confidence’ we know they are weak and therefore doomed.” When an investments salesperson who doesn’t know enough to make themselves understood to you starts regurgitating a bunch of hairy bull complicated stuff you may not be able to understand it simply because the person talking to you doesn’t either 😀  Knowing Realising it’s not just you, that the other person may not know what they’re talking about, is a most liberating phenom. I love Emperor’s New Clothes Phenomenon. Nor do I believe it a coincidence it was a little kid who has the gumption to call the emperor out, grownups have all had the “Muchness” educated and parented out of ’em.. Kids are just infuriatingly awesome and awesomely infuriating.) 

Anyway. You have to not be afraid to look inside the “black box” (that’s not just me, O’Neil really does say that). Y’know, like what they asked Jobs about the Mac decades ago: That’s Very Cool. Exactly What Does It Do?  😀

Perfectly good numbers don’t have to lie, sometimes they present perfectly legit opportunities that for a short time no one noticed because the trader with an axe was too busy well, trading, and the salesperson was too busy pushing the sexy new stuff to go take a second look in the older inventory that, by virtue of the latest happening in the market is now attractive. And sometimes predators are the way they are, for lack of imagination 🙂 (Think about it, if you could make the same money (well, get the same final payoff) legitimately, why would you ever cheat? Cheating simply “looks good” and appears to have a viable payoff because you haven’t seen beyond the immediate gratification – if you saw also how much you lost in valuable real skills-building time and energy (Get Good, Not Just Get Grades), if you saw the devastating cost of getting caught, factoring also the hit to your own self-esteem from believing more and more that you don’t have what it takes to win legitimately, the risk-reward would not make sense.) )

Sadly, “Solid values and self-regulation rein in only the scrupulous.” 

O’ Neil adds, “…I saw all kinds of parallels between finance and Big Data. Both industries gobble up the same pool of talent, much of it from elite universities like MIT, Princeton and Stanford. These new hires are ravenous for success…” 

“Ravenous”-ness for success by the usual benchmarks is exactly what caused the last financial crisis. We blame models, identify greed, revile self-serving natures in a way that implies we think this is relatively exclusive to “other” professions. (Wait for it….)

It starts with the kids (what ya think I was gonna say, Imma Mummy Blog). The character kids form before they become working professionals. The selfishness and ultra-competitiveness that might be particularly effective in squeezing out that little extra ounce of performance (no I don’t personally think so but lotsa people think pitting kids against each other or for that matter adults – the politically correct term I believe is “healthy competition” – produces satisfactory and better performance; I do think that comes at a cost though, because in all likelihood we’re gonna overuse that tool because it produces quick results… and it then creates a vulnerability of character that we will all end up paying for eventually – in school, college, then at work, in research and innovations….) The same personality traits some adults proverbially like to pluck at and play to in order to produce relatively quick results are the same traits that potentially cause financial crisis and colour goodness knows how much policy setting and research and funding…

We are the parents of the next generation of Type A professionals: How do we raise them?

(No I don’t mean the Fictional Mystic Arts Special Effects Trip) 😛 Dr Stephen Strange is a gifted but selfish surgeon who preserves his track record by picking and choosing his cases. There are of course many real-life professions, not just in the medical field, where you can manipulate your track record by picking cases likely to succeed in the first place. And so we use the safe fictional eg of Dr Strange 🙂 One day, Dr Strange is involved in a car accident that severely damages the nerve and muscle tissue in his hands. He can no longer perform his amazing surgeries as a result of this. As he expends his considerable wealth in an increasingly desperate attempt to regain his former livelihood, he encounters similarly gifted and selfish surgeons – who refuse to take his case because of the low likelihood of success, just as he once did with his own potential patients. 

See, we might think rewarding performance and competitiveness leads to well, greater performance…… but it also leads to cheating…. of the kind that can really hurt someone.

By that unwritten rule that says we perpetually strive to give our children better than we ourselves received – from hindsight, from the child always thinking they know better than their parents – we probably think we parent better today than our parents did 😉 – HOW will we measure that achievement? With… grades? Literacy tests?

Levitt and Dubner’s Freakonomics and now also O’Neil’s WMD, using as an example the American public school system and the (arguably too strong) stick-carrot approach (ref also Harvard Business Review’s Motivating Employees Is Not About Sticks And Carrots), show how instead of inducing dedication and performance teacher evaluation algorithms in Washington district public schools encouraged teachers to cheat.  (Cheating in itself is problematic, but when they’re changing test answers what could they have been better spending time and energy on?)

In an effort to overhaul D.C. public schooling, they threatened to either fire teachers or give them performance bonuses of up to USD 8,000 based on a teacher evaluation algorithm that relied heavily on student test scores. In a bad job market, desperate public school teachers started changing their students’ test answers to help them pass.

(I actually think this is a little funny – if too-strong stick-carrot approaches have been shown to induce perfectly decent adults to cheat (for one thing, performance by that yardstick might be too difficult to keep up legitimately in the given time frame each teacher has with a student), then WHY are there people who still expect it to work well with kids – are not kids simply juvenile versions of adults of the same species. (What, no? Kids are aliens? 😀 )) 

You guys seen this old Will Smith/ Tommy Lee Jones trilogy? Weird and whacky aliens… which, nonetheless, if you look closely enough are still modelled after predominantly Earth critters – cockroaches that love sugar, worms that can be cut in half – because y’know, Hollywood is limited by our human imagination just as much as mathematical models are coloured by our quants’ human biases  –

“The question is… whether we’ve eliminated human bias or simply camouflaged it with technology…”

Here’s another eg, this one being how bias made it alive and well into the prison models: In a 2013 New York Civil Liberties study, it was found that while black and brown (African American and Hispanic) males aged 14-24 years consisted just 4.7% of the total NYC population, they made up a whopping 40.6% of all stop-and-frisk searches by the NYC police.

But wait. Maybe there was a basis for those searches. Maybe the reason such a small percentage of the NYC population was searched so often was because said searches actually turned up something. Right?

The number you’re looking for is how often searches of that demographic, the 14-24 year old Black or Latino males, were actually innocent ie the searches turned up nothing.

Ready for it? Over 90%.

Over 90% of the time police searched this demographic, they didn’t turn up anything. You’d think said police would’ve then realised that they were wasting their time and they should turn their attention elsewhere…. but no, they kept right on searching that same demographic. This is because some human biases are very, very hard to fix.. Not… that it makes sense to let sleeping biases lie (sorry) when you actually need the police to catch the real dealers on the streets and they’re looking in the wrong place… Your friendly neighbourhood dealer meanwhile has probably figured out that they should dress not like a gangsta rapper, but like say, an Asian Science Nerd  😀

That’s not all – you’d think that algorithms run this bias out of data evaluations that affect policy setting, right? The problem however is in the collection of the data itself (Garbage In, Garbage Out). An actual prison inmate survey question to determine recidivism (the risk of a repeat offender and therefore affecting eligibility for early release) used to be, “Describe your first encounter with the police.”

Recall the >90% coloured male youth who get baselessly stopped and searched. What happens when they answer that question?

What about the heroin dealer who is never searched because she looks like an Asian Science Nerd (I got to type Asian Science Nerd three times now!) right up until they have to pull her Porsche (birthday gift from affluent parents) over because it’s weaving about unsteadily on the highway… and that’s when they find the glove compartment full of Blow.

More Up Is Down… In 2012, the British Telegraph reported the discovery of 99 identical copies of a single test in Zhongxiang province, prompting local authorities to come down hard on cheating – installing metal detectors, organising sting operations that uncovered transmitters in pencil erasers, individuals camped out in hotel rooms across the street, ready to communicate test answers……….

What d’you think the result of all that was – honesty? Yup.

Some 2000 stone-throwing protestors took to the streets chanting “We want fairness. There is no fairness if you don’t let us cheat.” 

They were honest about the (albeit) perceived need to cheat in order to level the playing field – because they truly believed everyone else was doing it.

Epilogue: Meantime on a lighter note, HN has been playing this on Youtube for days after picking it up in school assembly (bunch of different ways to be kind) and now the song’s stuck in our heads… From Life Vest Inside.
Lemme repeat that: Life Vest. INSIDE 🙂