Standards: a ratchet of progress
In my study of the history of scientific and technological progress, there are a few obvious drivers which help push us forward, ones which nobody can argue against. Standardization is one of these. Standardization moves a field from one of creative craftsmanship to one of routine engineering. Engineering can be planned; craftsmanship may run into unexpected problems.
I’ve documented this a little bit; the standardized screws made the industrial revolution possible. Screws had existed since the time of the ancient Greeks, but their contributions to man’s power over nature was supercharged by having standards. If you take a tour of old armories, they’ll likely tell a story of pre-standardization firearms and how standardization made mass production and mass adoption of cheap firearms possible. This changed human history forever: those societies who had mass produced firearms had peasant armies armed with cheap rifles with interchangeable parts: they conquered the world.
Humans have known about this for millennia of course; the Roman Empire had standardized brick sizes which made all manner of architectural innovation possible. There may be more ancient examples, but the concept of having a set standard which enabled planning through interchangeability is a very old idea. It continues to be an important source of innovation and increase in productivity: the shipping container is an obvious modern example which transformed logistics.
Consider the early 19th century physicists; restricting it to E&M, Ampere, Ohm, Faraday. Ampere was fortunate enough to be born in France in time to take advantage of the metric system. He was also fortunate to not be killed by the revolution, though it was a near thing as his father was. Faraday had an even better system of weights and measures: the Imperial system -you know, the measurement system that landed on the moon. Poor Ohm didn’t live in a real country and his weights and measures literally varied by the village.
The Imperial system and the metric system were both standards which helped fuel the scientific and industrial progress of the UK and France. While the Germans made some discoveries and industrial progress before the introduction of the metric system it arguably didn’t really achieve escape velocity until it adopted the metric system in 1872 (admittedly a year after the Empire was founded which I am sure also helped).
There are other smaller scale more or less contemporary examples. The CAMAC crate made the computerization of experimental physics possible. I don’t know what people used before CAMAC (NIM was an earlier related standard for general electronics) but it was ad-hoc and probably each installation used one or more grad student careers worth of brainpower and labor to set up and use. The ConFlat flange similarly made ultra high vacuum chamber construction from a sort of form of creative engineering project to a routine engineering practice any bozo could cobble together. These standards have probably saved tens or hundreds of thousands of wasted careers which would have otherwise been spent reinventing the wheel for a customized computerized experimental interface or vacuum chamber. Similarly I’m pretty sure there were lots of microcontroller systems in cars which didn’t use the CAN bus, but the ubiquity of the latter has made more interesting things possible and spread beyond automotive use. Of course a mature field like steel metallurgy and engineering with steels: there are like 3500 various standardized varieties of steel to build with: unlike the pre-industrial past where you were in the ridiculous situation of needing to be some kind of wizard to make decent steels like wootz out of charcoal, iron and horse piss.
The internet and various programming languages have standards committees which are of towering importance. I think it’s pretty obvious that there is much more work to be done here in computer land. As I have said, the profusion of serialization protocols continues to be an embarrassment for the profession of computard. This happens for sound sociological and business reasons of course; but there is no reason we should put up with it. Similarly for the various forms of shared object spaghetti. Of course, most programming languages are really only distinguishable from one another by what is written in them at this point: otherwise it’s 101 flavors of repl or compiled Algol, along with the mostly irrelevant paradigms of Lisp, Forth and APL. Honestly virtually everything in “software engineering” is non-standard craftsmanship spaghetti bullshit. Where companies succeed they generally have at least internal standards: Amazon most famously turned their internal software standards (which are horrible, but at least are a standard) into their most profitable product line.
I think most of software engineering will remain ad-hoc piles of craftsmanship bullshit, mostly because old software is valuable. There is otherwise absolutely no reason for it. People talk about “AI” replacing programmers; while aliens and immigrants have replaced many, LLMs haven’t, and almost certainly won’t for the foreseeable future. What could replace entire segments of programmers is standardization of more of the pieces which represent “software engineering.”
For a long time, LAMP was a software stack representing about 90% of what net companies do. It was a sort of defacto standard because it represented 4 technologies which were sort of “first and easy to use” -Linux, Apache, MySQL, PHP/Perl/Python. Now of course this stuff is enormously out of date and replaced by a bewildering bucket of cobbled together brainworms that have various alleged special qualities which supposedly make them better or solve some subset of problems more efficiently. LAMP type problems, of course, could be made to go away with an efficient standardization, or even a product meant to standardize these sorts of workflows. Labview did something like this back in the 1980s; more or less an extinction event for a certain kind of data acquisition programmer. Of course, people who solve these kinds of LAMPy problems the way they presently do are highly paid and vastly over privileged craftsmen who enjoy licking each other’s butts, so it won’t happen until some founder goes even farther than Bezos did and creates the Labview for LAMP patterns.
Mind you, while I am a happy Linux user, even Linux is a preposterous warthog which is more archaeological dig and hoarder therapy session than operating system. At least it doesn’t spy on me like all the commercial ones do; not if it is working right anyway. I’d like to talk total revolution here, sending most of what exists to the disintegration tubes -even the existing chip architectures. But even a stripped down Linux or Microkernel with network-first (preferably graphical) language and two or three flavors of syntactically identical data store and a secure IPC standard would destroy most Silicon Valley engineering careers. Everyone who has used something like Heroku knows this is possible. I await the future Napoleon who wipe away the vile old world; cleanse it with fire, bring an orderly and beautiful Napoleonic code.
I think you can measure the vitality of a field by its creation of standards. Software standards are pretty weak; even for things which should obviously be standardized, people seem to have a hard time accepting this. So, rather than a form of engineering, software languishes as a sort of ad-hoc craftsmanship. Dorks standing around and ‘mirin your ‘leet code are like woodcarvers, not engineers. The ironic and absurd thing about it is most of them are woodcarvers of things like screws; the screws that ancient Greeks used for wine and olive presses were created by woodcarvers, not engineers.
I figure nuclear annihilation is more likely than fixing software development with more powerful standards, but it’s possible to do, and the future belongs to whoever does this. It’s a shame the Fifth Generation computing project didn’t have ambitions more in this vein, as that would have been a great time to do it, and apparently there was the political will to do so. I suppose most of software was still academic in those days, and so they didn’t really know what to build, but there ought to be enough people around by now with ideas of what needs to be done. It’s funny software dweebs talk about automation and software eating the world, but they never talk about replacing themselves, which is what actually needs doing.
Locklin on science 
Sometimes everyone standardizes on a retarded standard. Docker as a solution for the fact that python has centralized dependency management and therefore some bizarre recapitulation of dll hell is one of them. That we need to simulate an entirely separate computer with an entirely separate operating system to deconflict your dependencies is obviously malformed.
Javascript – it became a standard, but should it have?
Sometimes, as with Labview and the Windows operating system, the standard is someone’s proprietary fiefdom that they can then use to charge rent, gatekeep, or jerk people around. What would have happened to the industrial revolution if you had to pay a license fee to create a standard screw thread pattern? If you published a paper using the metric system, you owed Paris money? If the code of conduct for using ANSI gear splines forbade use by certain people that were being persecuted by society?
Apparently the standard that we keep our money in banks and transact via credit card transactions has morphed into a monopoly by a few payment processors, and can be used to exile people from society.
The advantages that everyone collapsing to a standard can provide can also be abused as a means of gatekeeping and control if the standard isn’t freely available and freely replicable by anyone that wants to use it.
There’s nothing stopping people from writing an open source Labview or other high level programming language.
There’s nothing stopping people from writing open source libraries that work like Labview modules for the kind of crap the EC2 does with any of the 100 solutions on offer there.
I mean, there is something stopping people: open source projects are generally run by people who aren’t solving a real problem, they’re just goofing off.
There’s also nothing wrong with paying money for software: people do it all the time.
> > Sometimes, as with Labview and the Windows operating system, the standard is someone’s proprietary fiefdom..
>
> There’s nothing stopping people from writing an open source Labview..
I feel like you didn’t really respond to the statement.
Standardization in atoms had to combat two forces:
* Technically, the ability to manufacture to fine tolerances.
* Socially, the incentives of manufacturers to lock in their customers.
Standardizing bits is easy from the technical side, but the social problem is just as difficult. And it will follow the same trajectory as pre-digital technologies: pioneers who try to extract rents from new stuff (veeeerrry gradually, tens to hundreds of years) get commoditized by their complements, then vendors of a commodity organize into a standard.
FWIW, I think there are ways to accelerate this process. And they start with more enlightened consumers: https://akkartik.name/freewheeling
I mean, why bother having an operating system? We could just sell people computers. We managed to standardize that much; doing something to replace the pestiferous plague “engineers” who afflict us with all manner of things nobody wants (I know, I’ll write a bloom filter in RUST so my app can be 50% more technological) seems at least in the best interests of investors?
It’s obvious applying even some weak form of standardization is a win, otherwise people wouldn’t ship all their apps in electron containers. Imagine not having to even think about that. It’s certainly possible.
I use Slack on the free plan, but Slack doesn’t exist to be a free product. Me getting free use out of it is an accidental side-effect that may disappear one day.
Operating systems have a standardizing effect, but operating systems don’t exist to be standards. Our getting that use out of them is an accidental effect that may disappear one day.
What operating systems exist for is to enable rents. Even the free ones. All software is geopolitics, even the open source ones. These days I see every new software product as an attempt to create a new Strait of Hormuz.
You don’t know what you’re talking about.
> What operating systems exist for is to enable rents. Even the free ones. All software is geopolitics, even the open source ones. These days I see every new software product as an attempt to create a new Strait of Hormuz.
Do you want to unpack this or do we have to guess at your meaning? What rents are the linux devs extracting, since you can run it without paying? Where are the geopolitics in such a wide and randomly distributed developer base as the linux kernel? Note that when the US banned Huawei from using US tech like Android or Qualcomm’s 5G chips, they simply forked AOSP for their HarmonyOS (they laughably claim they didn’t) and stuck to using 4G chips for their exported smartphones.
I mostly agree with Scott that some standard is going to come along and obsolete most software developers, as there is a ridiculous amount of waste now.
My own idea is some kind of OSS WeChat, with built-in open protocols for everything from ride-hailing to grocery shopping. You could obsolete 90% of Silicon Valley with such an app, but of course, you’d need to be smart and start with one protocol and some good tech choices (no Java or XML) and incrementally grow your way into an “everything app,” unlike the morons who try to build it all first.
I don’t think enlightened consumers are that important, though they help, what matters much more are enlightened software programmers, of which there are almost none.
Didn’t you hear? I don’t know what I’m talking about.
Y’all are too aggressive for my taste, so I’ll see myself out.
I did post a link in a different comment which might help triangulate on where I’m coming from.
(I mean, of course I don’t know what I’m talking about. But it seems to help to air opinions and debate them.)
(I was in Berkeley many years ago airing opinions when the guy in the next table interrupted me to tell me I didn’t know what I was talking about. He did it more kindly though.)
By the same token, it’s not clear to me what you mean and you’ll have to unpack it. As a programmer, it seems to me programmers are plenty enlightened. We’re enlightened to who pays our salary and how to keep it coming. We choose what we don’t understand based on where our salary comes from. If that seems sub-optimal, the way to get us to understand something new is by collecting enough power to force us to do so.
I was right then which you presumably eventually realized, and I’m right now.
Programmers are mostly self-serving bureaucrats who ruin almost everything they touch (like all bureaucrats). They should be replaced with software like other bureaucratic jobs eventually will be.
You’re no fun in dinner conversations anymore, are you? 😀
We agree on the problem. If you think the solution to bureaucracy is software — you don’t know what you’re talking about.
Most bureaucracy is self serving bullshit and you can indeed get rid of it using software and standards.
> Y’all are too aggressive for my taste, so I’ll see myself out.
That’s Scott’s style, see previous posts’ comment threads for much worse: you got off light by comparison.
All I did is question your statements: if that’s too “aggressive” for you, sounds like you can’t take real debate.
I had clicked on your link before responding, interesting presentation with which I agree and indeed apply a lot of that advice myself, but no mention there of geopolitics or linux’s rents. I wouldn’t bother questioning you if I wasn’t hoping to learn something.
> By the same token, it’s not clear to me what you mean and you’ll have to unpack it. As a programmer, it seems to me programmers are plenty enlightened. We’re enlightened to who pays our salary and how to keep it coming. We choose what we don’t understand based on where our salary comes from. If that seems sub-optimal, the way to get us to understand something new is by collecting enough power to force us to do so.
This is precisely what I mean: almost nobody would call “enlightened” a bunch of drones simply chasing a paycheck. That is why most software is a complete waste of time and money that is unceremoniously axed and dumped in the back. I’m all for experimentation, but other than Wave and one or two others, that is pure waste, as many knew and said those were dumb projects and why from the beginning.
To be enlightened would be to actually understand why all this is happening, which as you yourself seem to admit, no programmer does. Scott’s not a programmer by education, no CS PhD like you, but he gets closer to the root of the problem with this post.
As for forcing programmers to understand, you flatter yourself that they matter. I agree with Scott that standards would put 95+% of programmers out of work, at least out of the current form of make-work jobs.
> If you think the solution to bureaucracy is software — you don’t know what you’re talking about.
Again, you provide no argument to buttress your claim. I don’t always agree with Scott’s idiosyncratic views on software, but I completely agree with his substantive responses to you.
Most bureaucratic jobs consist of slavishly following rules most of the time, particularly the bureaucrats we call programmers. But most of those rote jobs are being automated away by software, whether taxi dispatchers or assembly-line workers. There is a little human discretion in all these jobs that cannot be automated away, but most of the work is.
You may believe you are right in that software companies have become giant bureaucracies while automating away other jobs, but that’s only because these methods, like standards, haven’t been applied to software production itself… yet.
You’re right, I was being unfair, sorry. I’ve seen this pattern a few times where an edgy, outspoken style works well for a blog but destroys the comments because we all naturally tend to talk like the host. Hence my knee-jerk desire to back away. But you were/are fine. With a renewed commitment to be myself, I’ll continue.
The fact that several of of you share this opinion I’d never seen before (I don’t often read comments here) certainly gives me pause, but I guess the question I have is: where do you think standards come from? My sense is that standards in the world of atoms historically all came from engineers. So standards in the world of bits will come from programmers. Even if standards put programmers out of work after they come into being, you still need programmers to create them. And why would programmers collaborate in their own destruction?
The pattern I observe is that programmers love creating standards — and so are loath to subject themselves to them too much. Why should a single person have all the fun? Hence https://xkcd.com/927. It’s not enough to create the standard, we have to commit to follow the standard. That’s really the most important part. And there are currently too many forces against that.
I think the same pattern happened in physical processes as well. I suspect (based on what little I know) that if you look back in history you always find many standards during a certain era of experimentation. Over time the world settles on one or three, but that happens when the thing being standardized is on its way to becoming a commodity. It takes time. It takes however long it takes. Societies have to first converge on some level of institutional architecture that is capable of constraining the new technology.
So I think the post gets the causality at least partly wrong. Germany didn’t “achieve escape velocity” because it discovered the power of standards. Converging on a single standard was just a (key) part of the process of achieving escape velocity. Germany had to be molded to become worthy of a standard just as every other country has had to.
Programmers generally won’t develop standards which replace them. In fact they rarely develop any standards or tools of any utility at all. Asking a programmer to develop something useful is like asking a yogi who sucks his own dick to reproduce.
Engineers, on the other hand.
The nature of bureaucracy and bureaucrats is a whole other topic, and one where I feel much more certain about my opinions than those about standards.
> Programmers are mostly self-serving bureaucrats [and] should be replaced with software like other bureaucratic jobs eventually will be.
> Most bureaucratic jobs consist of slavishly following rules most of the time, particularly the bureaucrats we call programmers. But most of those rote jobs are being automated away by software, whether taxi dispatchers or assembly-line workers.
Bureaucrats are not at all the same thing as workers. To persuade yourself of this, ask yourself how the population of assembly-line workers has changed over time, and how that compares with the population of university administrators. Taxi dispatchers are going extinct. University administrators are seeing explosive growth. When the dynamics are so divergent, surely it makes sense to consider the underlying entities as different species.
When you go to the DMV, what fraction of the bureaucracy do you think you see? It’s infinitesimal. Bureaucracy isn’t made of rule-followers, that’s all of humanity. What distinguishes bureaucracy is the rule creators. Bureaucrats create rules. It’s a machine of drones all busily creating new rules, often within the constraints old ones, with varying impacts depending on the status of the bureaucrat. Poor, ill-considered, myopic rules. Rules that would never survive the selection pressures of the environment the bureaucracy ostensibly serves. Rules that continue to exist long after the last sign of any reason for their original creation has disappeared from the environment.
When I read that paragraph, as a programmer my reaction is: that’s software! Rules, check. Exploding growth, check. That’s bloat! No selection pressure to weeding out expired rules, check.
Software has so far been a great boon to bureaucracy. It greatly expands the range of things bureaucracies can create rules about while having no business doing so. And all those rules will continue to persist long after the programmers that created them. Even if you got rid of every single programmer, you wouldn’t have less bureaucracy. You’d have more.
I highly recommend this paper: https://law.mit.edu/pub/interpretingtherulesofcode/release/4 Automation can’t eliminate bureaucracy. Bureaucracy is automation. They’re kindred species, each providing optimal conditions for the flourishing of the other.
A goal of automation without bureaucracy seems quite out of reach at the world’s current level of maturity. I hazily think the path to such a world must pass through much wider oversight. No creating rules without the (extremely conservative, grudgingly given) consent of the governed. My talk is basically about this if you squint. Perhaps the first 4 minutes of this follow-up video will help distill things: https://www.youtube.com/watch?v=aD6vmbmzdBo.
It’s absolutely not another topic. Most programmers are just bureaucrats; self serving ones at that. How many times do I have to say this?
Millions of bureaucrats have been vaporized by technology already. It’s not like you have human calculators or ladies plugging in your phone calls any more.
FWIIW I don’t go to the DMV; it’s been replaced by technology where I live.
We could just sell people computers. We managed to standardize that much;
I’m not sure what you mean by that.
Computers are far from standardized. Yeah, I can (did) buy an Intel Nuc, and I can install about 5 different operating systems on it (counting all linux distros as 1, FreeBSD, Windows (various), ESXi (on some), and Open Solaris). This is not because the hardware is standardized across the industry, it’s because there’s a brazillian drivers out there that asphalt over the non-standardized nature of it.
Hell, even *virtualized* hardware isn’t standard. You can’t reliably take a VMWare Workstation image and run it under a different virtualization tool. There is conversion software, because (at least within a given version) the *file format* is standard, but it would generally have to be.
One of the ironic things, as far as Linux is concerned, is that Microsoft does use their market dominate to actually force the PC to be a relatively open platform/standard platform through programs like “Windows Hardware Certification”. While drivers are sometimes necessary to unlock all features, get compatibility with oddball storage controllers, or get the most performance out of GPUs… most things function relatively well with what is shipped by Microsoft by default.
And Linux has benefited from this. Through a general policy of “Do what would windows do” Linux they have gotten to the point were most things function on most PCs out of the box if you install Linux on it. If it works in Windows and not in Linux then they consider it a bug in Linux. In the past when hardware folks tried to make accommodations through things like Linux-specific BIOS settings they generally only mucked it up. I understand that a huge amount of work has gone into drivers and I am not trying to discount this. I understand that the bulk of Linux kernel code is dedicated just for drivers. The only thing that I am trying to say is that things could be a LOT worse.
For example: take the Android phone situation. With the exception of iPhones Linux is utterly dominate. If it doesn’t run Linux kernel you can’t sell it. Even among non-Android Phones most are still going to be running a Linux kernel.
But because Linux is open source and hardware makers can modify Linux to be compatible with their phones and there is no licensing program forcing the hardware to be “open”… then installing or upgrading your own Linux/Android system on most phones is mostly impossible. Most android phones are very closed platform and are actively hostile to user modifications. (for both good and bad reasons). Some individuals can get things working through herculean efforts thanks in part to GPL licensing, but by and large for us casuals only a relatively small handful of phones can have third party Android OSes installed on them.
Now am I going to trust Microsoft because of this? No, not enough to run their OS for my personal systems. But I trust Microsoft more then I trust Apple… Probably more then I trust Google, too.
Resituating my response here to avoid the shrinking nesting:
> an edgy, outspoken style works well for a blog but destroys the comments because we all naturally tend to talk like the host.
Heh, that’s Scott alone, haven’t seen anybody else mouth off like him in the other comment threads.
> My sense is that standards in the world of atoms historically all came from engineers. So standards in the world of bits will come from programmers. Even if standards put programmers out of work after they come into being, you still need programmers to create them. And why would programmers collaborate in their own destruction?
So you admit programmers are not engineers then? 😉
> The pattern I observe is that programmers love creating standards
Wait, you say they are loath to participate in their own destruction by creating standards, then you say they love creating them: which is it? You are arguing from both sides of the fence here.
> and so are loath to subject themselves to them too much. Why should a single person have all the fun? Hence https://xkcd.com/927.
Are they engineers or attending some kind of perpetual LAN party, what does fun have to do with it? Your answers highlight the fundamental unseriousness of most programmers that Scott is pointing out.
> It’s not enough to create the standard, we have to commit to follow the standard. That’s really the most important part. And there are currently too many forces against that.
There are, the most important being what you pointed out above: it is too easy to create a software “standard” unmoored from reality, and even a reference implementation, which then naturally nobody uses.
The big problem is that software standards have to be formed differently from prior physical standards, because they do different things, but most software people simply ape the prior physical standard-making process.
Because software is infinitely extensible and our needs and even the underlying hardware are always changing, software standards have these different attributes:
1. Physical standards have to be agreed upon ahead of time because of the much greater cost, both of fabrication and interoperation. Whereas software standards are evolved and only de facto software standards matter: written software standards can only codify and clarify what is already a de facto standard, else it doesn’t matter (that’s why most of the 14 standards in your xkcd link are irrelevant). Obviously, I’m only talking about most software that is easily installed and updated in downloadable apps and operating systems: the small niche of software that goes in firmware for routers or other embedded boxes that are not easily updated, like a TCP/IP stack, must still be standardized like the old physical standards.
2. Software standards have to be versioned and continually updated, again because software is much more flexible.
3. Software standards really have to be a collection of independent modules with a small common core, with some of those modules grown or thrown out in subsequent versions, eg all the ciphers that have been thrown out or added to the SSL/TLS standards.
Most software standards don’t acknowledge this basic reality, hence they are ignored.
> I think the same pattern happened in physical processes as well. I suspect (based on what little I know) that if you look back in history you always find many standards during a certain era of experimentation. Over time the world settles on one or three, but that happens when the thing being standardized is on its way to becoming a commodity. It takes time. It takes however long it takes. Societies have to first converge on some level of institutional architecture that is capable of constraining the new technology.
Sure, DC versus AC and so on. However, software has become a widespread commercial product for four decades now, since the PC revolution. The web was invented in 1989 and HTML 2 was standardized in 1995, despite massive ongoing change in the ’90s, so we know it can be done quickly with very fast-changing tech too.
I don’t know what constraining institutional architecture you have in mind, but software will never become a commodity. Instead, we have to standardize what we can and update the standard as we keep experimenting, as the web guys once did.
> So I think the post gets the causality at least partly wrong. Germany didn’t “achieve escape velocity” because it discovered the power of standards. Converging on a single standard was just a (key) part of the process of achieving escape velocity. Germany had to be molded to become worthy of a standard just as every other country has had to.
This is a variant of the old “nature vs. nurture” argument, ie was it German culture or technical standards? And the truth is in between: both are important and in fact reinforce and build on each other.
Unless you’re arguing that software standards today are irrelevant because no current group of programmers is serious enough to abide by them, not sure how you bring that back to today. If that is what you mean, trust me, there are some out there, and they will obsolete the rest.
> The nature of bureaucracy and bureaucrats is a whole other topic, and one where I feel much more certain about my opinions than those about standards.
Why are you more certain about that?
>> Most bureaucratic jobs consist of slavishly following rules most of the time, particularly the bureaucrats we call programmers. But most of those rote jobs are being automated away by software, whether taxi dispatchers or assembly-line workers.
> Bureaucrats are not at all the same thing as workers.
I didn’t say they were: read my quote that you pasted above carefully. I situated bureaucrats within the larger class of “rote jobs” and noted that the “worker” rote jobs have already been automated away.
> To persuade yourself of this, ask yourself how the population of assembly-line workers has changed over time, and how that compares with the population of university administrators. Taxi dispatchers are going extinct. University administrators are seeing explosive growth. When the dynamics are so divergent, surely it makes sense to consider the underlying entities as different species.
Those dynamics are different, but not in the way you highlight. The bureaucrats in the taxi companies weren’t the dispatchers, but the managers who created the rules, according to you. Well, with all the old taxi companies going bankrupt, those taxi manager-bureaucrats are all out of work just like the dispatchers.
Yes, university bureaucrats have saved themselves from this process for now, but that’s only by keeping out all automation through propaganda and their accreditation cartel, and bribing Biden and other govt officials to get the govt to pay off the massive student debt they laid on the youth for worthless classes, which cannot be paid back by those now jobless students.
In other words, the distinction in your examples isn’t between bureaucrats and workers: it is between bureaucrats whose markets have been automated away by software already, and those that will soon be.
> Bureaucracy isn’t made of rule-followers, that’s all of humanity.
No, it isn’t. Most of humanity today and through most of time has never followed any outside rules. This is a delusion of the social class you live in now. Go visit the street outside the Nancy Pelosi federal building or some rural area a couple hours away to see what I mean.
> What distinguishes bureaucracy is the rule creators. Bureaucrats create rules.
You are right that bureaucracies constantly create new rules, but most bureaucrats don’t create rules. A few have that legislative power, but most don’t. A lawyer’s job is simply to interpret and apply those rules, yet the much more numerous lawyers are just as much bureaucrats as the few legislative bureaucrats.
> It’s a machine of drones all busily creating new rules, often within the constraints old ones, with varying impacts depending on the status of the bureaucrat. Poor, ill-considered, myopic rules. Rules that would never survive the selection pressures of the environment the bureaucracy ostensibly serves. Rules that continue to exist long after the last sign of any reason for their original creation has disappeared from the environment.
> When I read that paragraph, as a programmer my reaction is: that’s software! Rules, check. Exploding growth, check. That’s bloat! No selection pressure to weeding out expired rules, check.
OK, so you agree with Scott and me that software and programmers have become giant bureaucracies, so go the next step: what happened to the giant bureaucracies at IBM or the British Empire? They certainly still exist in some much reduced form, but the selection pressure did weed them out from power.
> Software has so far been a great boon to bureaucracy. It greatly expands the range of things bureaucracies can create rules about while having no business doing so.
True, software is merely a tool, and just like a pickaxe can be used by both a communist and a capitalist, software can be used to either demolish bureaucracies or enable them. Scott and I are arguing for the demolition.
> And all those rules will continue to persist long after the programmers that created them. Even if you got rid of every single programmer, you wouldn’t have less bureaucracy. You’d have more.
Let’s go back to our concrete examples from before: thousands of taxi companies all over the globe have been replaced by a handful of ride-hailing app companies, like Didi, Uber, Lyft, and so on. That was a massive reduction in bureaucracy, thousands of taxi company rule-creating manager-bureaucrats replaced by a hundred or so Didi and Uber executives. These new executives have significantly fewer rules, eg they don’t employ the drivers and pay them benefits, but the drivers also set their own hours, working when or as long as they want. They also flout many of the rules that local govt bureaucracies put in.
So in this particular taxi market, we see a massive reduction in bureaucracy, in part because it was highly regulated and protected by city taxi cartels before software invaded. What makes you think the same won’t happen to the bureaucratic holdouts you trumpet, like university administrators and software companies themselves?
I see the same happening to them.
> I highly recommend this paper: https://law.mit.edu/pub/interpretingtherulesofcode/release/4
Why? I’m not interested in the post-modernist meanderings of some postdoc on whether law can be replaced by code, which has little to nothing to do with this discussion.
> Automation can’t eliminate bureaucracy.
You are arguing against the lived experience of the taxi company bureaucrats here. 😀
> Bureaucracy is automation. They’re kindred species, each providing optimal conditions for the flourishing of the other.
There is some truth here, in that only highly automated societies can afford the massive waste of large bureaucracies, so that is where they flourish. But bureaucracy is certainly not automation, and automation has laid waste to many bureaucracies. Similarly, countries or markets that become bureaucratic after automation will eventually be laid waste to by others who aren’t as bureaucratic, just as Microsoft and Intel once did to IBM.
> A goal of automation without bureaucracy seems quite out of reach at the world’s current level of maturity.
I disagree, it has already happened to a large extent in several markets. If you’re arguing that Uber still has some, much smaller bureaucracy, just wait: that will be automated away too.
>I hazily think the path to such a world must pass through much wider oversight. No creating rules without the (extremely conservative, grudgingly given) consent of the governed. My talk is basically about this if you squint. Perhaps the first 4 minutes of this follow-up video will help distill things: https://www.youtube.com/watch?v=aD6vmbmzdBo.
Sure, but you now raise a different matter of precisely what rules will be chosen once the few legislative bureaucrats are gone. Automation merely ensures that once the rules are chosen, they are carried out as efficiently as possible, ie rather than a giant bureaucracy carrying out those rules manually, they are implemented in software by a few coders, eg Uber’s geolocation software. Suffice to say that the rules will be chosen much more democratically once the bureaucrats are gone.
To give an indication of the challenge: LabVIEW 1.0 dates from 1986 and ran on a 68K Mac, back when the best Mac you could buy was a Mac Plus with 4MB of memory and a hard drive the size of a large web page.
Hypercard dates from about a year later and while superceded, has never really been equalled in its domain.
Really good software design – and I’m at best a grudging LabVIEW proponent – is hard.
Incidentally NodeRED looks nice and while I have not used it in anger, it looks like it could be extended to do a lot of things. I would give it a spin if I were prototyping some IoT doohickey solo.
It’s funny back end shit should be easier using an eventy thing like labview. So should front end shit. Anything involving squirting data into a computard and squirting it back is basically a flow-charty FSM. Why don’t people just execute their flow charts? But there is electron, and we can build a back end out of OPEN SORES doodads that run on oligarch Bezos’ mainframes!
I know, writing graphical anything is hard. It also enrages code nerds who think they’re artists because it reduces them to what they’re really doing: plumbing. I wish I were a hard nosed Elon type I’d build it just to watch all those artist code monkeys queue up in the unemployment line.
PS: What happens if you ratchet into a grossly globally-suboptimal system?
Every mammal on Earth has 4 limbs, because that was first past some post of viability. Mutations that alter body plans/segmentation are pretty much always fatal to the organism at our stage of complexity. But starting from a sufficiently simple common ancestor, there were any number of other limb-plans/layouts. Insects and cephalopods have other arrangements. With 6 limbs, the development of manipulators and tool use would have been a far shorter and less fraught path than needing to develop bipedal locomotion. The development of intelligence might have been easier.
https://en.wikipedia.org/wiki/Carcinisation
I’m surprised that you have a positive view of standardisation, that kind of activity (making everything accord to standard) doesn’t sound like the Klein type 1 organisations (make something unique) that you like so much
However I agree that computards need more standardisation, industrial standards would help a much more than open source to escape monopolistic software firms that focuse on rent extraction
There’s innovation, such as inventing bittorrent, and there’s writing a back end for a nerd dildo app so teenagers can send each other nudes. Most of silly con valley and fellow travellers basically do the latter. Hundreds of thousands of people do this. It’s like hundreds of thousands of people making their own GBIB interface for an HP voltmeter in C++ or Forth or whatever instead of using the labview module, except times 10000 more effort.
This offends me from a wasted effort standpoint, wasted money and lives, and because it’s frankly retarded. Seeing the abominations people build to keep something like twitch running, or to make sure you get your nudes on time is just mind boggling. Like everyone building railroad engines using different home-made screw threads and boilers (railroad engines were all pretty standardized early on). Like everyone having their wives knit them hand made suits from raw wool.
Yes, but in what sort of world can the innovators that you like survive and do their work? Innovators have to grow into their role, and can they do it in a world where everything they would want to initially build and try is considered an “irrelevant waste of time”? If your entire life you’re just supposed to grab legos out of boxes from the container ship from China, can you ever really learn how to invent all the pieces that aren’t there?
On the one hand, “making your own (thing that has been done 1000 times already)” is wasted effort. On the other hand, refusing to learn an interface and insisting on learning the thing itself (by knocking together your own possibly uncompetitive example) is an attempt to keep the sort of frontier open where creative work and *learning* can exist.
IMO, once the frontier is closed, the learning stops. You don’t get the learning back. Things get frozen in whatever forms were dominant (good or horrible), and people can’t make their living there anymore. This may be something that happens, but not everything that is natural is good.
I suppose we’re talking about different things. Basic application logic isn’t algorithm development, but I think a push for “efficiency” in all things is probably the kiss of death for the sort of organization where someone can “waste” time on a deep problem.
I remember someone yelling at poor abused high school kids something to the effect of: “You had better be the next Edison or Einstein, because if it has been done before, it’s being done in China.” (And do it snappily, and for low pay, and without reinventing the wheel. Ignore the stuff that is “irrelevant”, and cough up “innovation” – not whatever bullshit you wanted to work on, the stuff *we* want. Just what sort of country do they think Edison, and the people Edison needed, could have survived in?)
I don’t really care what computards end up doing if/when there’s 1000x in programmer productivity; maybe we’ll need just as many because that would unlock much more useful stuff than helping teenagers send nudes to each other. I don’t worry about what ancient greek wood carvers of screws are doing now either, whether or not carving olive press screws was, like, totally a cool thing that made them happy. Innovation like bittorrent or blockchain stuff matters. Innovation like making your fucked up plumbing work at AmaGoogFaceSoft is not innovation in any meaningful sense: it’s just time wasting craftsmanship. I think it’s obvious by now those people would be better off taking up painting, carpentry or making jewelry in their spares if they enjoy craftsmanship.
It’s probably fair to say that I ran software shops. Not for the sake of software. I found it works to separate the innovators and the standardizers by a fuzzy chinese wall. They fight alot but eventually settle down. The farmer writes down the math, solves the problems, codes things up. Ugly, scrappy, spaghetti code. Then it goes over the wall and the latter adheres to coding standards and documentation standards. They make the code faster in dev. The commit to prod. Every time I’ve mixed these groups / workflows it’s been an utter failure.
Your article made me realize there are standards most persons do not even think about since they are so common and effective such as Arabic numerals, the Gregorian calendar and the creation of time zones when railroads allowed relatively rapid travel between cities.
I was thinking about making a post about mathematical notation (but I don’t want to spam the comment section.) It seems when people first attempt some new thing they are reasoning about, there is a period where they struggle through with some awkward notation, before inventing something more elegant to the purpose. Einstein summation notation comes to mind.
I wonder, though, what the choices we’ve made wrt notation are obscuring from us? (And what alternative notation might reveal.) Roman vs. arabic numerals, etc. Maybe there is something a little less awkward than the Christoffel symbol out there.
I tried learning differential geometry recently and while I in principle understand Einstein notation, looking at the older way and having it painstakingly re-explained (in Levi-Civita’s book) was helpful. Still don’t know it. Can see a way forward at least.
I highly recommend Principles of Relativity Physics by James Anderson. Not well known, extremely high quality, a tad dated, but well ahead of its time. Solving problems slowly like g_0j,k, g_1j,k etc. very useful for people that don’t love or aren’t good with the modern notation and like to write out the elements leading up to R_ij=0.
It’s all about the Benjamin’s. Look at Apple. Pentalobe screws. Lightning port.
Look at Dell, Non-standard mobo in pre-built PCs. I haven’t even touched software yet.
Everybody has a taken a page out of Microsoft’s playbook, “Embrace, Extend, Enslave”.
Desktop PC is so accessible and ubiquitous because of standards. The motherboard, the RAM/Storage, the I/O. It’s cheap, replaceable/repairable (relatively speaking) but nobody has much pricing power, at best, some have brand recognition and reputation.
Exact opposite in Apple/Google/Microsoft land. Showering computards like obscene amounts of money and marketing their way to profits. They consider it “Digital Innovation”.
However, there are standards for all the serious stuff. Consider C/C++, POSIX, Khronos (although applel and microsoft said F U with Metal and DirectX). Open Source Dingdongs have other priorities like “software activism” so you best bet for software standardisation is companies/orgs who’d actually save money with it (something like ANSI) and Congress/Parliament.
Another thing.
Consider luxury watchmakers or bespoke shoemakers. That’s what computards think of themselves. I guess they can charge more money that way. More proprietary spaghetti = more “features” to market = potentially more $$$ with vendor lock-in.
Another interesting thing.
https://www.wikiwand.com/en/Fordism
A paragraph from the article linked above:
Historian Thomas Hughes has detailed how the Soviet Union, in the 1920s and the 1930s, enthusiastically embraced Fordism and Taylorism by importing American experts in both fields as well as American engineering firms to build parts of its new industrial infrastructure. The concepts of the Five-Year Plan and the centrally-planned economy can be traced directly to the influence of Taylorism on Soviet thinking. Hughes quotes Joseph Stalin:
“American efficiency is that indomitable force which neither knows nor recognizes obstacles; which continues on a task once started until it is finished, even if it is a minor task; and without which serious constructive work is inconceivable…The combination of the Russian revolutionary sweep with American efficiency is the essence of Leninism.”
Fantastic example of the effects of standarisation. This is certainly the definition of leninism I’d use.
If we standardized marketing, and eliminated all other forms of it, everything would be way better and smoother. I don’t want 1000 ways of accessing the same sort of information, including duplicates, when trying to buy a second hand gas canister in my area.
It also seems like rent seekers gravitate to things that should be standardized but aren’t.
There is no doubt that you are right about the dismal state of modern software development. To its essential complexity we have now added a ton of self-inflicted complexity driven by the low quality of current software developers. But considering the essential complexity by itself and ignoring the cultural decline, yes, standardization would be beneficial if it were possible. But is it actually possible? You didn’t establish that it is. What for instance is the “screw’ of software? Or by analogy with electronics, the resistor, capacitor, etc. of software? The “LAMP” stack isn’t it because that isn’t the right level of granularity, not even close. What are the standard software “parts” that apply to every and all programming domain?
The truly great (and never to be seen again) programmers of the 1970s and 1980s already tried and failed. If they couldn’t do it with Objects/SmallTalk or Lisp, it is because it cannot be done. Software is complex far beyond mechanical devices. The Space Shuttle is said to have had about 2.5 million parts. Linux has almost 30 million lines of code making it–if we take the LOC as a “moving part”–12 times more complex than the Space Shuttle. I’m temperamentally on your side but this idea of “standardization” seems, well, ungrounded. “I can think of it therefore it is possible”, is not a persuasive argument, especially in this age of technology’s twilight.
Linux is better than the alternatives but it’s an abomination. Just the network stack is an abomination; great place to start (I use DPDK which is also shit, but at least it allows me to use the network like it’s …. I dunno, 1998 or so). Anything with a goddamned code of conduct that isn’t written by IRON SHEIK needs to go.
It’s abundantly obvious to me that all my silly con valley friends are essentially plumbing some horse shit from a database to the web (aka “LAMP”); almost everything there can be standardized and made to scale automatically without resorting to the insanity that they do. I don’t know what the state of FPGA network stacks is at present, but you could probably do it all in hardware, giving people some thin crap to customize. Replacing tens of thousands of engineers with a couple of $10,000 devices.
This isn’t anything fancy, and people have had the thought before. Twitch’s doofus C-team was impressed with the video chips on planes and so they use some unholy hardware/software abomination which started out as the idea I’m talking about. It has nothing to do with “software gods trying to create some perfect universe that pleases them” -as you say, we tried this and it failed back in the 80s. Recognizing that most of it is self inflicted complexity, and that writing more python or rust or whatever isn’t the right answer is all you need to get it right.
Just looking at how much programming languages suck ought to be enough to make you realize there’s plenty of room for innovation here. How come K is still so damn fast on numeric queries?
Comparing standardization in computing with standardization in physics is interesting. How long did it take for standards to appear in physics? For context, we’re around 80 years from the discovery of electronic computers (von Neumann machine is 1940s). UNIX is say, sixty years old. I would guess that standards in physics took less than 80 years, but it’d be nice to have a number to compare.
Also, what parts of computing would you standardize — x86? UNIX? POSIX? LISP/APL? IEEE-754?
CAMAC was developed AFAIK a few years before people starting using stuff like Altair computers, so probably only 2-3 years after they were using cobbled together stuff with PDP-8’s or whatever they did back then.
I’d standardize LAMP like stacks for one thing; possibly replacing it with cheap hardware and definitely with better network stacks. Posix is already a good standard with mostly lousy implementations.
I’m with you on standardized protocols and serialization. I don’t think these matter much. I doubt there is new ground to be broken here so most of what we’re doing at this point is fiddling for novelty’s sake.
In languages, let alone operating systems or user interface design, god help us if we try to standardize around the current state of the art. I’d rather go back to Lisp and C and call it good. And if anybody ever tries to make a drag and droppy mousy clicky graphical programming language some kind of defacto standard I swear on all that is holy I will make it my life’s mission to kill it and the committee that designed it with fire. This kind of trash is for people who can’t operate a keyboard, let alone their wetware. I would rather go pound rocks together in the woods than try to build programs with one-handed duplos. I like almost everything you write here but this thing that you said is a personal fetish that should stay in your bedroom.
You will be replaced by drag and drop monkeys, and it will be an improvement. Programmers used to just flip switches for machine code; Fortran and Algol were an improvement, and python and JS lets bona fide retards write spaghetti. Labview for devops will eventually happen if the russians don’t nuke us.
Flipping switches for machine code isn’t an art form, and neither is writing Lisp haikus.
That has been the dream of boomers in suits for more than 30 years now and all it has produced is more shitty languages and more shitty software built with them. Someday standards will hopefully coalesce around software development to the point that we can call it engineering and not craftsmanship, but the difficult part of the job has never been and will never be typing or understanding syntax.
When our salaries inevitably implode it will be because we finally figured out what software was good for, which didn’t include anything prefixed by “smart”, “e-“, or “i”, not because we reduced software engineering to something you could teach in a “boot camp” to teens kidnapped from Nigeria.
Everyone wants to feel special, but the reality is software nerds and their “ideas” are abominations who will be wiped out as thoroughly as hardware interface guys were wiped out by Labview, and microsoft certified administrators were wiped out by companies buying (retarded but at least not dangerous) Apple products.
Well, we’ll see. You might be right. If the whole software world burned down I certainly wouldn’t miss it. I could be making more money and be in a lot better shape if I had stuck with welding instead of getting into this whole gig. But as it stands right now, seeing this mess from the inside for 20 years, I’m deeply skeptical that it’s going to happen or that anyone has the foggiest idea how to even start making that happen.
I can’t help but feel that the main, heavy lifting applications of tech, were already there with text interfaces. Each “Innovation” seems to have slowed things down, and increased the noise/signal ratio.
Seems like a field long overdue some forest management.
Well I’m hoping pointing out the possibility will inspire someone to do it for us.