Locklin on science

AI is not eliminating jobs

Posted in econo-blasphemy, machine learning by Scott Locklin on February 9, 2019

Midwits keep asserting that “AI” is going to eliminate jobs. They say things like “those jobs aren’t coming back because of AI” (or screws or whatever other dumb excuse: but they’re real sure about those jobs not coming back).  These are not statements of scientific or technological fact, or even a reasonable prediction based on present trends. These are ideological political statements. “AI” is a soundbite/fnord excuse for not doing anything about the policy problems of the present.

The ruling caste of American tech and FIRE lizard people continue to make these statements, not because they are inevitable, but because this is their desired future. Their preferred future is a population consisting of powerless, preferably drugged up serfs on the “universal basic income” dole, ruled over by our present ruling class of grifters, rentiers, pyramid scheme salesmen, watched over by a surveillance hellscape.  The lizard people would like to continue our present policy of de-industrializing the country, breaking what little labor negotiating power US citizens have, and atomizing people to their raw protoplasm. It’s almost like a Freudian slip. Don’t bother agitating for any rights, slave, we soon will have electric Golems and won’t need you!

The most murderous drug dealers who ever existed … Google’s AI dude thinks they’re great! https://twitter.com/JeffDean/status/1093953731756867584

Oh I am sure the Google doofs would like to develop and control some strong AI, and perhaps a robot maid to replace Juanita. I, too, would like to have a magic technology which gives me infinite power, and a robot maid to iron my shirts. If I were a Silicon Valley oligarch rather than a humble nerd, I might develop delusions the pile of C++, Javascript and tech drones which made me rich could become a Golem of infinite power. Personally,  I would build rockets. At least I could get away from lizard people who want to turn the world into a soy dystopia.

If these clowns really believed that “AI” were something actually like an “AI” which could replace humans in general tasks, they’d use it to replace computer programmers. At one point in history, people believed CASE tools would eliminate most programmer jobs. How’s that working out for the AI geniuses at Google? They can’t even automate devops jobs; devops being one of the most automatable roles in tech companies. Devops tasks don’t seem much different from a computer strategy game.

Google’s “AI” team can’t do this useful thing, which, even by my lights, actually seems  achievable. Yet somehow, google boobs think they’re going to violate Moravec’s paradox and replace drivers. Think about that for a minute. It’s becoming clear that autonomous vehicle “technology” as sold to people for the last 10 years is basically fraud, and is still stuck in the 1980s when Ernst Dickmanns was driving around the autobahn with Sun Workstations in his back seat. Demonstrations of this tech always have a human in the loop (remote or in vehicle), because moving automobiles without human control are death machines under most circumstances.

Inside of the UniBwM autonomous experimental vehicle VaMP, at the rear bench where the computing system was installed for easy access and monitoring. This was at the PROMETHEUS demonstration in Paris in October 1994 | Photo by Reinhold Behringer

Even assuming I’m wrong and the media hyperbole is right and full level 5 autonomous vehicles are “right around the corner” Google also has zero business interest in “disrupting” driving. Google is a tech driven advertising company with a  collection of loss leaders. Yet they go after this preposterously difficult, possibly impossible task. Why not disrupt a business they presumably know how to disrupt, like that of the lowly ops engineer? At least this would be good for their bottom line, and it would be a real step forward in “AI” rather than a parlour trick perpetuated by marketing nerds and started by obvious mountebanks.

From a semiotics point of view, this shows astounding hostility to the types of people who drive cars and trucks for a living. Drivers are … ordinary, usually uneducated, salt of the earth people who have a fairly independent lifestyle and make a decent living. Google overlords must really hate such people, since they’re dumping all this skrilla into ruining their lives for no sane business reason. They will almost certainly fail, but man, why would you try to blow up those people’s lives? If this country really wanted to get rid of driving, or considered it a serious problem that there are too many cars on the road, or thought that people now employed as drivers should do something else, we had a solution to this problem invented in the late 1800s.


The other professions  people “think” will be replaced always seem to be low caste irritations or lawyers (lol). You regularly hear “experts” talking about how presently common jobs won’t exist in 20 years because of “AI.”  I’ve said multiple times now that all estimates for delivery of something in 20 years are bullshit. A prediction that a technology will do X in 20 years means “we don’t know how to do this, but we want your money to fool around with anyway.” Controlled nuclear fusion researchers being the most amusing case of the perpetual 20 year rice bowl. 20 years is a magic number, as it’s plenty of time for a technological mountebank to retire; and it’s at least 2-3 generations of tenured academics, which is enough to turn a scam subject like “quantum computing” or “nanotech” into an actual field.

“AI” doesn’t exist. Machine learning is a force multiplier and productivity enhancer for statisticians. If you believe the “automation”=”no more jobs” ding dongs, machine learning should have at least automated away the job of statistician. Yet somehow, the  statistician (aka “data scientist”) jobs are among the best paid and most in-demand jobs out there at present.


The last job category I can think of which was automated away is Flight Engineer on airliners. It mostly went away because of automation of airliners, but it wasn’t even computer related; just normal improvements of systems monitoring and reliability; good old mechanical and systems engineering. Despite 1/3 fewer seats in airliner cockpits, there are now more people with airline flight officer jobs now than ever before. Planes got cheaper and there are more of them servicing vastly more people.

The example of Flight Engineer is how the world works. Technological advances increase human power over nature and makes more things possible. Actual “AI” advances, should any eventually materialize, will work exactly like this.

AI has eliminated exactly zero professions, and essentially no jobs. Since the best prediction tool for a market is generally a random walk, my forecast is, barring giant breakthroughs, this trend of “nothing important actually happened” regarding AI job destruction will continue. If you disagree with me and have an alternate prediction on a normal human (aka 5 or 10 year) timescale, I am happy to entertain any long bets on whatever platform you care to use.


Quantum computing as a field is obvious bullshit

Posted in non-standard computer architectures, physics by Scott Locklin on January 15, 2019

I remember spotting the quantum computing trend when I was  a larval physics nerdling. I figured maybe I could get in on the chuckwagon if my dissertation project didn’t work out in a big way (it didn’t). I managed to get myself invited to a Gordon conference, and have giant leather bound notebooks filled with theoretical scribblings containing material for 2-3 papers in them. I wasn’t real confident in my results, and I couldn’t figure out a way to turn them into something practical involving matter, so I happily matriculated to better things in the world of business.

When I say Quantum Computing is a bullshit field, I don’t mean everything in the field is bullshit, though to first order, this appears to be approximately true. I don’t have a mathematical proof that Quantum Computing isn’t at least theoretically possible.  I also do not have a mathematical proof that we can make the artificial bacteria of K. Eric Drexler’s nanotech fantasies. Yet, I know both fields are bullshit. Both fields involve forming new kinds of matter that we haven’t the slightest idea how to construct. Neither field has a sane ‘first step’ to make their large claims true.

Drexler and the “nanotechnologists” who followed him, they assume because we  know about the Schroedinger equation we can make artificial forms of life out of arbitrary forms of matter. This is nonsense; nobody understands enough about matter in detail or life in particular to do this. There are also reasonable thermodynamic, chemical and physical arguments against this sort of thing. I have opined on this at length, and at this point, I am so obviously correct on the nanotech front, there is nobody left to argue with me. A generation of people who probably would have made first rate chemists or materials scientists wasted their early, creative careers following this over hyped and completely worthless woo. Billions of dollars squandered down a rat hole of rubbish and wishful thinking. Legal wankers wrote legal reviews of regulatory regimes to protect us from this nonexistent technology. We even had congressional hearings on this nonsense topic back in 2003 and again in 2005 (and probably some other times I forgot about). Russians built a nanotech park to cash in on the nanopocalyptic trillion dollar nanotech economy which was supposed to happen by now.

Similarly, “quantum computing” enthusiasts expect you to overlook the fact that they haven’t a clue as to how to build and manipulate quantum coherent forms of matter necessary to achieve quantum computation.  A quantum computer capable of truly factoring the number 21 is missing in action. In fact, the factoring of the number 15 into 3 and 5 is a bit of a parlour trick, as they design the experiment while knowing the answer, thus leaving out the gates required if we didn’t know how to factor 15. The actual number of gates needed to factor a n-bit number is 72 * n^3; so for 15, it’s 4 bits, 4608 gates; not happening any time soon.

It’s been almost 25 years since Peter Shor had his big idea, and we are no closer to factoring large numbers than we were … 15 years ago when we were also able to kinda sorta vaguely factor the number 15 using NMR ‘quantum computers.’

I had this conversation talking with a pal at … a nice restaurant near one of America’s great centers of learning. Our waiter was amazed and shared with us the fact that he had done a Ph.D. thesis on the subject of quantum computing. My pal was convinced by this that my skepticism is justified; in fact he accused me of arranging this. I didn’t, but am motivated to write to prevent future Ivy League Ph.D. level talent having to make a living by bringing a couple of finance nerds their steaks.

In 2010, I laid out an argument against quantum computing as a field based on the fact that no observable progress has taken place. That argument still stands. No observable progress has taken place. However, 8 years is a very long time. Ph.D. dissertations have been achieved, and many of these people have gone on to careers … some of which involve bringing people like me delicious steaks. Hundreds of quantum computing charlatans achieved tenure in that period of time. According to google scholar a half million papers have been written on the subject since then.


There are now three major .com firms funding quantum computing efforts; IBM, Google and Microsoft. There is at least one YC/Andreesen backed startup I know of. Of course there is also dwave, who has somehow managed to exist since 1999; almost 20 years, without actually delivering something usefully quantum or computing. How many millions have been flushed down the toilet by these turds? How many millions which could have been used building, say, ordinary analog or stochastic computers which do useful things? None of these have delivered a useful quantum computer which has even  one usefully error corrected qubit. I suppose I shed not too many tears for the money spent on these efforts; in my ideal world, several companies on that list would be broken up or forced to fund Bell Labs moonshot efforts anyway, and most venture capitalists are frauds who deserve to be parted with their money. I do feel sad for the number of young people taken in by this quackery. You’re better off reading ancient Greek than studying a ‘technical’ subject that eventually involves bringing a public school kid like me a steak. Hell, you are better off training to become an exorcist or a feng shui practitioner than getting a Ph.D. in ‘quantum computing.’

I am an empiricist and a phenomenologist. I consider the lack of one error corrected qubit in the history of the human race to be adequate evidence that this is not a serious enough field to justify using the word ‘field.’ Most of it is frankly, a scam. Plenty of time to collect tenure and accolades before people realize this isn’t normative science or much of anything reasonable.

As I said last year

All you need do is look at history: people had working (digital) computers before Von Neumann and other theorists ever noticed them. We literally have thousands of “engineers” and “scientists” writing software and doing “research” on a machine that nobody knows how to build. People dedicate their careers to a subject which doesn’t exist in the corporeal world. There isn’t a word for this type of intellectual flatulence other than the overloaded term “fraud,” but there should be.

Computer scientists” have gotten involved in this chuckwagon. They have added approximately nothing to our knowledge of the subject, and as far as I can tell, their educational backgrounds preclude them ever doing so. “Computer scientists” haven’t had proper didactics in learning quantum mechanics, and virtually none of them have ever done anything as practical as fiddled with an op-amp, built an AM radio or noticed how noise works in the corporeal world.

Such towering sperg-lords actually think that the only problems with quantum computing are engineering problems. When I read things like this, I can hear them muttering mere engineering problems.  Let’s say, for the sake of argument this were true. The SR-71 was technically a mere engineering problem after the Bernoulli effect was explicated in 1738. Would it be reasonable to have a hundred or a thousand people writing flight plans for the SR-71  as a profession in 1760? No.

A reasonable thing for a 1760s scientist to do is invent materials making a heavier than air craft possible. Maybe fool around with kites and steam engines. And even then … there needed to be several important breakthroughs in metallurgy (titanium wasn’t discovered until 1791), mining, a functioning petrochemical industry, formalized and practical thermodynamics, a unified field theory of electromagnetism, chemistry, optics, manufacturing and arguably quantum mechanics, information theory, operations research and a whole bunch of other stuff which was unimaginable in the 1760s. In fact, of course the SR-71 itself was completely unimaginable back then. That’s the point.


its just engineering!

its just engineering!

Physicists used to be serious and bloody minded people who understood reality by doing experiments. Somehow this sort of bloody minded seriousness has faded out into a tower of wanking theorists who only occasionally have anything to do with actual matter. I trace the disease to the rise of the “meritocracy” out of cow colleges in the 1960s. The post WW-2 neoliberal idea was that geniuses like Einstein could be mass produced out of peasants using agricultural schools. The reality is, the peasants are still peasants, and the total number of Einsteins in the world, or even merely serious thinkers about physics is probably something like a fixed number. It’s really easy, though, to create a bunch of crackpot narcissists who have the egos of Einstein without the exceptional work output. All you need to do there is teach them how to do some impressive looking mathematical Cargo Cult science, and keep their “results” away from any practical men doing experiments.

The manufacture of a large caste of such boobs has made any real progress in physics impossible without killing off a few generations of them. The vast, looming, important questions of physics; the kinds that a once in a lifetime physicist might answer -those haven’t budged since the early 60s. John Horgan wrote a book observing that science (physics in particular) has pretty much ended any observable forward progress since the time of cow collitches. He also noticed that instead of making progress down fruitful lanes or improving detailed knowledge of important areas, most develop enthusiasms for the latest non-experimental wank fest; complexity theory, network theory, noodle theory. He thinks it’s because it’s too difficult to make further progress. I think it’s because the craft is now overrun with corrupt welfare queens who are play-acting cargo cultists.

Physicists worthy of the name are freebooters; Vikings of the Mind, intellectual adventurers who torture nature into giving up its secrets and risk their reputation in the real world. Modern physicists are … careerist ding dongs who grub out a meagre living sucking on the government teat, working their social networks, giving their friends reach arounds and doing PR to make themselves look like they’re working on something important. It is terrible and sad what happened to the king of sciences. While there are honest and productive physicists, the mainstream of it is lost, possibly forever to a caste of grifters and apple polishing dingbats.

But when a subject which claims to be a technology, which lacks even the rudiments of experiment which may one day make it into a technology, you can know with absolute certainty that this ‘technology’ is total nonsense. Quantum computing is less physical than the engineering of interstellar spacecraft; we at least have plausible physical mechanisms to achieve interstellar space flight.

We’re reaching peak quantum computing hyperbole. According to a dimwit at the Atlantic, quantum computing will end free will. According to another one at Forbes, “the quantum computing apocalypse is immanent.” Rachel Gutman and Schlomo Dolev know about as much about quantum computing as I do about 12th century Talmudic studies, which is to say, absolutely nothing. They, however, think they know smart people who tell them that this is important: they’ve achieved the perfect human informational centipede. This is unquestionably the right time to go short.

Even the national academy of sciences has taken note that there might be a problem here. They put together 13 actual quantum computing experts who poured cold water on all the hype. They wrote a 200 page review article on the topic, pointing out that even with the most optimistic projections, RSA is safe for another couple of decades, and that there are huge gaps on our knowledge of how to build anything usefully quantum computing. And of course, they also pointed out if QC doesn’t start solving some problems which are interesting to … somebody, the funding is very likely to dry up. Ha, ha; yes, I’ll have some pepper on that steak.


There are several reasonable arguments against any quantum computing of the interesting kind (aka can demonstrate supremacy on a useful problem) ever having a physical embodiment.

One of the better arguments is akin to that against P=NP. No, not the argument that “if there was such a proof someone would have come up with it by now” -but that one is also in full effect. In principle, classical analog computers can solve NP-hard problems in P time. You can google around on the “downhill principle” or look at the work on Analog super-Turing architectures by people like Hava Siegelmann. It’s old stuff, and most sane people realize this isn’t really physical, because matter isn’t infinitely continuous. If you can encode a real/continuous number into the physical world somehow, P=NP using a protractor or soap-bubble. For whatever reasons, most complexity theorists understand this, and know that protractor P=NP isn’t physical.  Somehow quantum computing gets a pass, I guess because they’ve never attempted to measure anything in the physical world beyond the complexity of using a protractor.

In order to build a quantum computer, you need to control each qubit, which is a continuous value, not a binary value, in its initial state and subsequent states precisely enough to run the calculation backwards. When people do their calculations ‘proving’ the efficiency of quantum computers, this is treated as an engineering detail. There are strong assertions by numerous people that quantum error correction (which, I will remind everyone, hasn’t been usefully implemented in actual matter by anyone -that’s the only kind of proof that matters here) basically pushes the analog requirement for perfection to the initialization step, or subsumes it in some other place where it can’t exist. Let’s assume for the moment that this isn’t the case.

Putting this a different way, for an N-qubit computer, you need to control, transform, and read out 2^N complex (as in complex numbers) amplitudes of N-qubit quantum computers to a very high degree of precision. Even considering an analog computer with N oscillators which must be precisely initialized, precisely controlled, transformed and individually read out, to the point where you could reverse the computation by running the oscillators through the computation backwards; this is an extremely challenging task. The quantum version is exponentially more difficult.

Making it even more concrete; if we encode the polarization state of a photon as a qubit, how do we perfectly align the polarizers between two qubits? How do we align them for N qubits? How do we align the polarization direction with the gates? This isn’t some theoretical gobbledeygook; when it comes time to build something in physical reality, physical alignments matter, a lot. Ask me how I know. You can go amuse yourself and try to build a simple quantum computer with a couple of hard coded gates using beamsplitters and polarization states of photos. It’s known to be perfectly possible and even has a rather sad wikipedia page. I can make quantum polarization-state entangled photons all day; any fool with a laser and a KDP crystal can do this, yet somehow nobody bothers sticking some beamsplitters on a breadboard and making a quantum computer. How come? Well, one guy recently did it: got two whole qubits. You can go read about this *cough* promising new idea here, or if you are someone who doesn’t understand matter here.

FWIIW in early days of this idea, it was noticed that the growth in the number of components needed was exponential in the number of qubits. Well, this shouldn’t be a surprise: the growth in the number of states in a quantum computer is also exponential in the number of qubits. That’s both the ‘interesting thing’ and ‘the problem.’ The ‘interesting thing’ because an exponential number of states, if possible to trivially manipulate, allows for a large speedup in calculations. ‘The problem’ because manipulating an exponential number of states is not something anyone really knows how to do.

The problem doesn’t go away if you use spins of electrons or nuclei; which direction is spin up? Will all the physical spins be perfectly aligned in the “up” direction? Will the measurement devices agree on spin-up? Do all the gates agree on spin-up? In the world of matter, of course they won’t; you will have a projection. That projection is in effect, correlated noise, and correlated noise destroys quantum computation in an irrecoverable way. Even the quantum error correction people understand this, though for some reason people don’t worry about it too much. If they are honest in their lack of worry, this is because they’ve never fooled around with things like beamsplitters. Hey, making it have uncorrelated noise; that’s just an engineering problem right? Sort of like making artificial life out of silicon, controlled nuclear fusion power or Bussard ramjets is “just an engineering problem.”

engineering problem; easier than quantum computers


Of course at some point someone will mention quantum error correction which allows us to not have to precisely measure and transform everything. The most optimistic estimate of the required precision is something like 10^-5 for quantum error corrected computers per qubit/gate operation. This is a fairly high degree of precision. Going back to my polarization angle example; this implies all the polarizers, optical elements and gates in a complex system are aligned to 0.036 degrees. I mean, I know how to align a couple of beamsplitters and polarizers to 628 microradians, but I’m not sure I can align a few hundred thousand of them AND pockels cells and mirrors to 628 microradians of each other. Now imagine something with a realistic number of qubits for factoring large numbers; maybe 10,000 qubits, and a CPU worth of gates, say 10^10 or so of gates (an underestimate of the number needed for cracking RSA, which, mind you, is the only reason we’re having this conversation). I suppose it is possible, but I encourage any budding quantum wank^H^H^H  algorithmist out there to have a go at aligning 3-4 optical elements to within this precision. There is no time limit, unless you die first, in which case “time’s up!”

This is just the most obvious engineering limitation for making sure we don’t have obviously correlated noise propagating through our quantum computer. We must also be able to prepare the initial states to within this sort of precision. Then we need to be able to measure the final states to within this sort of precision. And we have to be able to do arbitrary unitary transformations on all the qubits.

Just to interrupt you with some basic facts: the number of states we’re talking about here for a 4000 qubit computer is ~ 2^4000 states! That’s 10^1200 or so continuous variables we have to manipulate to at least one part in ten thousand. The number of protons in the universe is about 10^80. This is why a quantum computer is so powerful; you’re theoretically encoding an exponential number of states into the thing. Can anyone actually do this using a physical object? Citations needed; as far as I can tell, nothing like this has ever been done in the history of the human race. Again, interstellar space flight seems like a more achievable goal. Even Drexler’s nanotech fantasies have some precedent in the form of actually existing life forms. Yet none of these are coming any time soon either.

There are reasons to believe that quantum error correction, too isn’t even theoretically possible (examples here and here and here -this one is particularly damning). In addition to the argument above that the theorists are subsuming some actual continuous number into what is inherently a noisy and non-continuous machine made out of matter, the existence of a quantum error corrected system would mean you can make arbitrarily precise quantum measurements; effectively giving you back your exponentially precise continuous number. If you can do exponentially precise continuous numbers in a non exponential number of calculations or measurements, you can probably solve very interesting problems on a relatively simple analog computer. Let’s say, a classical one like a Toffoli gate billiard ball computer. Get to work; we know how to make a billiard ball computer work with crabs. This isn’t an example chosen at random. This is the kind of argument allegedly serious people submit for quantum computation involving matter. Hey man, not using crabs is just an engineering problem muh Church Turing warble murble.

Smurfs will come back to me with the press releases of Google and IBM touting their latest 20 bit stacks of whatever. I am not impressed, and I don’t even consider most of these to be quantum computing in the sense that people worry about quantum supremacy and new quantum-proof public key or Zero Knowledge Proof algorithms (which more or less already exist). These cod quantum computing machines are not expanding our knowledge of anything, nor are they building towards anything for a bold new quantum supreme future; they’re not scalable, and many of them are not obviously doing anything quantum or computing.

This entire subject does nothing but  eat up lives and waste careers. If I were in charge of science funding, the entire world budget for this nonsense would be below that we allocate for the development of Bussard ramjets, which are also not known to be impossible, and are a lot more cool looking.



As Dyakonov put it in his 2012 paper;

“A somewhat similar story can be traced back to the 13th century when Nasreddin Hodja made a proposal to teach his donkey to read and obtained a 10-year grant from the local Sultan. For his first report he put breadcrumbs between the pages of a big book, and demonstrated the donkey turning the pages with his hoofs. This was a promising first step in the right direction. Nasreddin was a wise but simple man, so when asked by friends how he hopes to accomplish his goal, he answered: “My dear fellows, before ten years are up, either I will die or the Sultan will die. Or else, the donkey will die.”

Had he the modern degree of sophistication, he could say, first, that there is no theorem forbidding donkeys to read. And, since this does not contradict any known fundamental principles, the failure to achieve this goal would reveal new laws of Nature. So, it is a win-win strategy: either the donkey learns to read, or new laws will be discovered.”

Further reading on the topic:

Dyakonov’s recent IEEE popsci article on the subject (his papers are the best review articles of why all this is silly):


IEEE precis on the NAS report:

https://spectrum.ieee.org/tech-talk/computing/hardware/the-us-national-academies-reports-on-the-prospects-for-quantum-computing (summary: not good)

Amusing blog from 11 years ago noting the utter lack of progress in this subject:


“To factor a 4096-bit number, you need 72*40963 or 4,947,802,324,992 quantum gates. Lets just round that up to an even 5 trillion. Five trillion is a big number. ”

Aaronson’s articles of faith (I personally found them literal laffin’ out loud funny, though I am sure he is in perfect earnest):



Machine learning & data science: what to worry about in the near future

Posted in machine learning by Scott Locklin on July 9, 2018

Henry Kissinger  recently opined about machine learning. OK, he used the ridiculously overblown phrase “AI” rather than “machine learning” but the latter is what he seemed to be talking about. I’m not a fan of the old reptile, but it is a reasonably thoughtful piece of gaseous bloviation from a politician. Hopefully whoever wrote it for him was well compensated.


There are obvious misapprehensions here; for example, noticing that chess programs are pretty good. You’d expect them to be good by now; we’ve been doing computer chess since 1950. To put this in perspective; steel belted radial tires and transistor radios were invented 3 years after computer chess -we’re pretty good at those as well. It is very much worth noting the first important computer chess paper (Shannon of course) had this sentence in it:

“Although of no practical importance, the question is of theoretical interest, and it is hoped that…this problem will act as a wedge in attacking other problems—of greater significance.”

The reality is, computer chess largely hasn’t been a useful wedge in attacking problems of greater significance.  Kissinger also mentioned Alpha Go; a recent achievement, but it is something which isn’t conceptually much different from TD-Gammon;  done in the 1990s.

Despite all the marketing hype coming out of Mountain View, there really hasn’t been much in the way of conceptual breakthroughs in machine learning since the 1990s.  Improvements in neural networks have caused excitement, and the ability of deep learning to work more efficiently on images is an improvement in capabilities. Stuff like gradient boost machines have also been a considerable technical improvement in usable machine learning. They don’t really count as big conceptual breakthroughs; just normal improvements for a field of engineering that has poor theoretical substructure. As for actual “AI” -almost nobody is really working on this.

None the less, there have been progress in machine learning and data science. I’m betting on some of the improvements having a significant impact on society, particularly now that the information on these techniques is out there and commodified in reasonably decent software packages. Most of these things have not been spoken about by government policy maker types like Kissinger, and are virtually never mentioned in dopey “news” articles on the subject, mostly because nobody bothers asking people who do this for a living.

I’d say most of these things haven’t quite reached the danger point for ordinary people who do not live in totalitarian societies, though national security agency type organizations and megacorps are already using these techniques or could be if they weren’t staffed with dimwits. There are also areas which we are still very bad at, which are to a certain extent keeping us safe.

The real dangers out there are pretty pedestrian looking, but people don’t think through the implications. I keep using the example, but numskull politicians were harping on the dangers of Nanotech about 15 years ago, and nothing came of that either. There were obvious dangerous trends happening in the corporeal world 15 years ago which had nothing to do with nanotech. The obesity rate was an obvious problem back then, whether from chemicals in the environment, the food supply, or the various cocktails of mind altering pharmies that fat people need to get through the day. The US was undergoing a completely uncommented upon and vast demographic, industrial and economic shift. Also, there was an enormous real estate bubble brewing. I almost think numskull politicians talk about bullshit like nanotech to avoid talking about real problems. Similarly politicians and marketers prefer talking about “AI” to issues in data science which may cause real problems in society.

The biggest issue we face has a real world example most people have seen by now. There exists various systems for road toll collection. To replace toll takers, people are encouraged to get radio tags for their car like “ezpass.” Not everyone will have one of these, so government choices are to continue to employ toll takers, removing most of the benefit of having such tools, or use an image recognition system to read license plates, and send people a bill. The technology which underlies this system is pretty much what we’re up against as a society. As should be obvious: not many workers were replaced. Arguably none were; though uneducated toll takers were somewhat replaced by software engineers. The real danger we face from this system isn’t job replacement; it is Orwellian dystopia.

Here is a list of  obvious dangers in “data science” I’m flagging over the next 10-20 years as worth worrying about as a society.

1) Face recognition software  (and to a lesser extent Voice Recognition) is getting quite good. Viola Jones  (a form of boosted machine) is great at picking out faces, and sticking them in classifiers which label them has become routine. Shitbirds like Facebook also have one of the greatest self-owned labeled data sets in the world, and are capable of much evil with it. Governments potentially have very good data sets also. It isn’t quite at the level where we can all be instantly recognized, like, say with those spooky automobile license plate readers, but it’s probably not far away either. Plate readers are a much simpler problem; one theoretically mostly solved in the 90s when Yann LeCun and Leon Bottou developed convolutional nets for ATM machines.

Related image

2) Machine learning  and statistics on large data is getting quite respectable. For quite a while I didn’t care that Facebook, google and the advertisers had all my data, because it was too expensive to process it down into something useful enough to say anything about me. That’s no longer true. Once you manage to beat the data cleaning problems, you can make sense of lots of disparate data. Even unsophisticated old school stuff like éclat is pretty helpful and various implementations of this sort of thing are efficient enough to be dangerous.

3) Community detection. This is an interesting bag of ideas that has grown  powerful over the years. Interestingly I’m not sure there is a good book on the subject, and it seems virtually unknown among practitioners who do not specialize in it. A lot of it is “just” graph theory or un/semi-supervised learning of various kinds.

Image result for community detection algorithm

4) Human/computer interfaces are getting better. Very often a machine learning algorithm is more like a filter that sends vastly smaller lists of problems for human analysts to solve. Palantir originated to do stuff like this, and while very little stuff on human computer interfaces is open source, the software is pretty good at this point.

5) Labels are becoming ubiquitous. Most people do supervised learning, which … requires labels for supervision. Unfortunately with various kinds of cookies out there, people using nerd dildos for everything, networked GPS, IOT, radio tags and so on; there are labels for all kinds of things which didn’t exist before. I’m guessing as of now or very soon, you won’t need to be a government agency to track individuals in truly Orwellian ways based on the trash data in your various devices; you’ll just need a few tens of millions of dollars worth of online ad company. Pretty soon this will be offered as a service.


Ignorance of these topics is keeping us safe

1) Database software is crap. Databases are … OK for some purposes; they’re nowhere near their theoretical capabilities in solving these kinds of problems. Database researchers are, oddly enough, generally not interested in solving real data problems. So you get mediocre crap like Postgres; bleeding edge designs from the 1980s. You have total horse shit like Spark, laughably insane things like Hive, and … sort of OK designs like bigtables… These will keep database engineers and administrators employed for decades to come, and prevent the solution of all kinds of important problems. There are people and companies out there that know what they’re doing. One to watch is 1010 data; people who understand basic computing facts, like “latency.” Hopefully they will be badly managed by their new owners. The engineering team is probably the best to beat this challenge. The problem with databases is multifold: getting at the data you need is important. Keeping it close to learning algorithms is also important. None of these things are done well by any existing publicly available database engines. Most of what exists in terms of database technology is suitable for billing systems, not data science. Usually people build custom tools to solve specific problems; like the high frequency trader guys who built custom data tee-offs and backtesting frameworks instead of buying a more general tool like Kx. This is fine by me; perpetual employment. Lots of companies do have big data storages, but most of them still can’t get at their data in any useful way. If you’ve ever seen these things, and actually did know what you were doing, even at the level of 1970s DBA, you would laugh hysterically. Still, enough spergs have built pieces of Kx type things that eventually someone will get it right.


2) Database metadata is hard to deal with. One of the most difficult problems for any data scientist is the data preparation phase. There’s much to be said about preparation of data, but one of the most important tasks in preparing data for analysis is joining data gathered in different databases. The very simple example is the data from the ad server and the data from the sales database not talking to each other. So, when I click around Amazon and buy something, the imbecile ad-server will continue to serve me ads on the thing that Amazon knows it has already sold me. This is a trivial example: one that Amazon could solve in principle, but in practice it is difficult and hairy enough that it isn’t worth the money for Amazon to fix this (I have a hack which fixes the ad serving problem, but it doesn’t solve the general problem). This is a pervasive problem, and it’s a huge, huge thing preventing more data being used against the average individual. If “AI” were really a thing, this is where it would be applied. This is actually a place where machine learning potentially could be used, but I think there are several reasons it won’t be, and this will remain a big impediment to tracking and privacy invasions in 20 years. FWIIW back to my ezpass license plate photographer thing; sticking a billing system in with at least two government databases per state that something like ezpass works in -unless they all used the same system (possible), it was a clever thing which hits this bullet point.

3) Most commonly used forms of machine learning requires many examples. People have been concentrating on Deep Learning, which almost inherently requires many, many examples. This is good for the private minded; most data science teams are too dumb to use techniques which don’t require a lot of examples. These techniques exist; some of them have for a long time. For the sake of this discussion, I’ll call these “sort of like Bayesian” -which isn’t strictly true, but which will shut people up. I think it’s great the average sperglord is spending all his time on Deep Learning which is 0.2% more shiny, assuming you have Google’s data sets. If a company like google had techniques which required few examples, they’d actually be even more dangerous.

4) Most people can only do supervised learning. (For that matter, non-batch learning terrifies most “data scientists” -just like Kalman filters terrify statisticians even though it is the same damn thing as linear regression). There is some work on stuff like reinforcement learning being mentioned in the funny papers. I guess reinforcement learning is interesting, but it is not really all that useful for anything practical. The real interesting stuff is semi-supervised, unsupervised, online and weak learning. Of course, all of these things are actually hard, in that they mostly do not exist as prepackaged tools in R you can use in a simple recipe. So, the fact that most domain “experts” are actually kind of shit at machine learning is keeping us safe.



A shockingly sane exposition of what to expect from machine learning, which I even more shockingly found on a VC’s website:


I don’t want to work on your shitty blockchain project: especially you, Facebook

Posted in fun, privacy by Scott Locklin on May 24, 2018

At the moment, I appear to be some kind of unicorn. I’m a no bullshit dozen year veteran of using math and machine learning to solve  business problems. I’ve also got some chops in blockchain which I am considerably more humble about. I am a real life machine learning blockchain guy. I don’t actually ride to work on a unicycle while wearing silver pants, but I probably could get away with it. As such, recruiters looking to cash in on the blockchain chuckwagon  seem  unable to leave me alone, despite my explicitly asking them to do so.

Image result for blockchain unicorn

It boggles my mind that there even exist recruiters for blockchain. After the blockchain annus mirabilis of 2017, anyone who knows a few useful things about the subject is almost certainly productively employed and probably fairly unconcerned with stuff like money. I’d posit that any blockchain type who can’t find productive employment on socially useful projects or isn’t in danger of financial independence either  doesn’t feel like working, doesn’t care about money or doesn’t actually know anything about blockchain. In the former cases you can’t recruit them, and in the latter case, you really shouldn’t.

Of course there are no shortage of faux “experts” who wouldn’t know a Merkle-tree from a KD-tree. Usually these same “experts” were or would have been touting themselves as “AI” or machine learning thought leaders a few months prior, and IoT, augmented reality, clean tech, “dat cloud” and … I don’t remember what the litany of  marketing diarrhea was being squirted out of Silly Con Valley’s corporate orifices before then. I have better things to use that brain cell for.

On the off chance that someone who is competent in this subject were looking for a job, there are obvious places to go. The crypto currency exchanges are decent places that will  incubate many new ventures; Gemini would be my pick. Their exchange is technologically far and away the best there is, and based on my experiences so far, it’s also the best run. There is good reason for this; the Winkelvii struck me as a couple of smart, honest and diligent guys. Better than the exchanges are the companies and foundations running the various blockchain projects themselves. Crypto investment funds will be an interesting place to make a buck. Right now it’s shooting fish in a barrel and there are a lot of morons doing it, but some of them are going to accumulate tremendous wealth, and there are direct, obvious and not so obvious ways a blockchain expert can help them do this. Or, start your own blockchain project. There is much work to do, and even though it is more difficult to fund new projects than last year, good projects will be funded, and now is the time. Whatever solutions win either already exist or they will shortly.  Other decent ideas: one of the big accounting firms, the banks, various corporate contributors to hyperledger fabric.

Of course, I don’t want any of this: I’m exactly where I want to be. I am helping good people fix the internet and save it from corporate weasels. Every day I get up and help do my bit to make things better. It’s a nice feeling. Problems are pretty interesting too.

But if I did want another job, the very last place on earth I would work is Facebook. Facebook is corporate syphilis. I keep telling them this. I even went through the process of quitting their service and wrote a whole blog on it. They don’t listen. It’s almost like they don’t give a shit when people tell them things. I was polite the first time, joking they could have my services if they buy my company. No more.

When I say Facebook is corporate syphilis, I am not engaging in hyperbole. I consider tobacco companies to be more ethical and serving a higher social purpose. Tobacco companies employ factory workers, farmers, shopkeepers and .. they keep doctors in business. Tobacco is more sociable than Facebook; smokers must meet face to face now that they are banished to the outdoors. Hell, smoking is probably physiologically healthier than spending hours a day noodling with your nerd dildo on ‘tardbook; at least you get up and walk around once an hour.  Supposedly nicotine is a prophylactic against Parkinsons disease, even if the most popular delivery method does kind of give you cancer. Facebook isn’t prophylactic against anything but having a life. Unlike Facebook,  some people want and enjoy nicotine. Nobody in the history of the human race has ever decided they want something like Facebook in their lives. “Gee I want a fraudulent advertising service that ruins and commodifies my relationships, wastes my time, makes me depressed, decays the moral fiber of entire civilizations, causes mass hysteria, spies on me and sells me out for pocket change, is as addictive as heroin,  is the bones of a hellscape surveillance state and is impossible to live without in the modern world; SIGN ME UP YO.”

Even gambling syndicates serve a higher social purpose than Facebook. The gambling rackets provide subsidies for entertainment, jobs for hundreds of thousands of decent working class people, and they somehow manage to employ more and more interesting applied math types than Facebook does. Facebook has all of the addictive and time wasting qualities of gambling, applied to more people, causing more social corrosion and employing fewer people. Facebook really is corporate syphilis.



Their excuse for existence is that Facebook “brings people together.”  CBS news used to bring people together; everyone would watch 60 minutes and talk about it at the water cooler. Facebook is a narcissism factory which causes moral panics, ridiculous rumor propagation, argument between friends, social fragmentation, alienation and even mass suicide. It’s also so obviously rotting the social fabric of the internet and society at large, even the debauched whores in the media are noticing. Facebook’s walled garden is wrecking the economics of the content providers and entertainers that make the internets interesting and worthwhile. It’s run by opportunistic mountebanks and sinister robots who … well, assuming they aren’t actual comic book villains, they sure do a reasonable impersonation. The PR these yoyos get is at best Stalinistic nonsense; at worst, people just sucking up to money and power. Speaking of Stalinism, Facebook employs literal former Stasi agents to censor and snitch on people for … saying things. Think about that. They expect me to work for a company that employs East German Secret Police; in precisely the same capacity as they were used in the former East German Workers paradise. I wonder what their dental plan is like? Maybe the one described in Marathon Man?

Kim Jong Il backed by officers visits the July 18 Cattle FarmImage result for zuck and cows


The recruiters (4 so far counting outside contractors) tell me there is some little Eichmann at Facebook who suffers under the delusion I would work in their cubicle jonestown. I will not. Not as long as I have a kidney I can sell to Ukrainian kidney merchants,  hands to shovel shit, or a sword to fall on. Facebook needs blockchain and machine learning people the same way they need a Manhattan project on biological warfare.

I am no boy scout, but I do still harbor a vague moral sense. Facebook is bad and anyone who works there who is not an active saboteur or malingerer should be deeply ashamed of themselves.   The only way I will ever return to their once pleasant campus (it was pleasant when Sun Microsystems was there) is at the head of a column of tanks.


Edit add: look at what they came out with today; a press release from their own internal ministry of truth. I’m going to assume it is either the Demons they keep in the basement, or the electroshock therapy they administer in the “art rehab center” which causes the total lack of self awareness which makes crap like this possible: https://newsroom.fb.com/news/2018/05/facing-facts-facebooks-fight-against-misinformation/