DjVu and its connection to Deep Learning

DjVu is a vastly superior file format for books, mathematical papers and just about anything else you can think of to original PDF (current year PDF adopted some of its innovations, but they’re only used to break into your ipotato afaik). PDF is mostly postscript with a bunch of weird metadata and layers. This is fine if the PDF is generated by LaTeX and is designed to be something that comes out a printer. But most of the world’s useful text is still on pieces of paper that have to be scanned to be on the interbutts. DjVu is good at sharing compressed book scans, and PDF is not. It shows its superiority when someone makes a big image scan in PDF, which is just a bunch of photographic images in jpeg (which is absolute shit at representing text in part because of how the FFT works) or tiff. DjVu assumes that the data is some kind of mix of text and images and as such most of the data can be safely thrown away.  This is a good assumption; usually I just want the text and plots, and DjVu captures those well. PDF generally clownishly captures everything in a scan more or less as a bitmap, or using jpeg’s silly rastered cosine transform.

Why jpeg sucks on text

Yann LeCun, Léon Bottou and Yoshua Bengio were creators of DjVu along with some other guys you’re less likely to have heard of (Patrick Haffner, Bill Riemers etc etc). All three are also  fathers of Deep Learning (along with Geoffrey Hinton before he developed his peculiar fear of ballpoint pens). Leon and Yann are also creators of my favorite little programming language, Lush -the Lisp Universal Shell, where they did much of their pioneering work back in the 90s. While I know enough about programming languages to understand why current year Torch migrated from Lush to Lua to Python, it will always remain one of my all time favorite designs; as interesting in its own way as the K family, and since it never had to do crap like maintain order books …. one of the comfiest languages I’ve ever used. When I retire I’ll probably revive it and use it to power superior robot vacuum cleaners or something. It’s really that good. There’s so much cool shit hanging around in it from their R&D days as well; just mind boggling stuff -like looking at Leonardo’s notebooks. I ain’t even talking about the neural stuff; all of it from the Ogre UI to the codebook editor is genius.

Since deep learning models is all the bugman talks about any more, the older work product of their creators should interest people, at least for historical perspective. It was an important problem: in 1998 the internet was still pretty new and stuff like PDFs didn’t quite work right. We mostly downloaded LZ77/Huffman coded postscript files when we wanted to use the internet for its original purpose of sharing scientific papers. Those were awful. It wasn’t awful because you had to unzip the file before you could look at it, but you did, but  because they were quite large (maybe 4x what PDFs delivered from compiled LaTeX), and the internet in those days was very slow. It would take minutes to download a couple of shitty jpeg files with boobs in them, let alone the 40 megs of javascript that websites now make you download now so they can track you.

At the time DjVu solved an important problem, allowing very good compression ratios and even allowing scanned stuff to be efficiently shared online, potentially making the internet into a super library including all printed books as well as generated net.content. The problem was most operating systems didn’t come with a DjVu reader, but Adobe made sure everyone had a PDF reader. Finding and installing a DjVu reader was a pain in the ass. Browsers in those days mostly couldn’t display either PDFs or DjVu, so that wasn’t even an option.

One of the cool things about DjVu is it internally uses an image format very similar to JPEG2000 for image backgrounds (called IW44). You have probably never seen a JPEG2000 image (unless in a DjVu file), but it’s a fantastic idea using wavelet compression, so if you only get the first quarter of the file you’ll get a pretty nice low-resolution image. It provides a natural way of doing lossy compression; just drop the higher order wavelets. It also compresses better than regular JPEG. The wavelets are further compressed with arithmetic coding which is also a mighty cool idea.

There is another format it used for foreground text (it looks for text) called JB2 which is related to the thing in PDF  which was buffer overflowed on the ipotato by Pegasus. You have to be careful with your document formats; I strongly suspect PDF has more holes like this in it, just because it has so much going on inside. JP2 is cool because it’s a sort of clustering algorithm where it looks for bitmaps which are around the size of characters, then looks for things which are geometrically similar to them; effectively doing a quick and dirty map of pixel clusters into symbols (not necessarily text symbols: the idea is textually agnostic). Then the document is arithmetically code compressed with the symbols.

The arithmetic coding system used is also innovative; it’s called the ZP-coder. It’s similar to other simple run length coding systems in its use of probability tables, but oriented towards decoding performance. It is a shame the ZP-coder isn’t a universal coder, as if it was it might make convincing fake documents based on the corpus in the document (aka do generative prediction the way openai does with neural nets, using a considerably cheaper algorithm). Pretty cool it works well on the wavelets and the text though.

It’s a shame it didn’t catch on better, and there is probably a HBS case study for the full story of why the objectively superior tool failed in the market. It even failed in Internet Archive use, which it was also well suited for. DjVu  still has utility in scanned documents and reading scanned documents. The main problem with it is the problem it had of old: lack of support. Black and white e-book readers like the kindle and the kobo don’t support it natively despite it being just about the perfect format for scanned documents on a limited processor greyscale e-book reader. I personally use a Kobo-Forma rooted with the excellent koreader to get access to the many useful DjVu files I have (basically all my textbooks available on the road). It’s ridiculous that I have to hack a device to get access to physically portable DjVu files, but I suppose scanned books don’t make anybody money.

I’ve long held that most of the knowledge developed since the advent of the internauts is basically anti-knowledge, meaning those scanned books in DjVu are potentially more valuable than all the PDFs in the universe. It would be nice to see it used by more mainstream publishers, but the lack of a DjVu target for things like LaTeX means it probably won’t be. I guess in the meanwhile DjVu is the most punk rock document format.

 

https://en.wikipedia.org/wiki/DjVu

 

Further books (May 2023 )

Assembleywomen –Aristophanes. Imagine if women had political power; what would happen? Aristophanes thought they’d vote for communism, stop shaving their bits, dress like men, enforce handsome men to sexually service old and ugly women and destroy society in hopeless search for equality. Of course it was just a silly 2400 year old play; what did those Greeks know.

 

King Lear -this is more like it; something which strikes pure like Julius Caesar did. Somehow I don’t find the characters as irritatingly inconsistent as in Othello, though they are probably just as much so -perhaps because the motivations are more clear and relatable.  In this one we get the full proverbial brass marching band with absurdly dramatic scenes, soliloquys and open air settings. Othello by contrast was sort of squalid and claustrophobic as if it all took place in bedchambers. I’ll try a Shakespeare comedy next, then another Marlowe play to see if his few plays are indeed better than the best of Shakespeare.

“Experimental Mathematics” by VI Arnold.  A series of Arnold lectures in which he uses computer examples to motivate proofs. This is something I always assumed happened. I mean, when you talk to a topologist they’re always taking examples on the unit circle or sphere or whatever, what’s the difference if you use computard to do something similar? I’m not going to pretend to be any kind of mathematician; I got up to applied Hilbert space methods and complex analysis and pretty much gave up. My coursework in advanced dynamics gave a lot of examples which could be applied to learning about point topology theorems (I didnt realize until reading Nash and Sen on the train to go visit Gunnar’s house of wonders), but stuff like algebraic geometry, number theory and all that…. I only ever ran into names of things -the ideas mostly didn’t make their way into didactics of classical physics at my level. FWIIW Arnold  came up with a lot of what we now know as “chaos theory.” While Arnold is an idol of mine for opposing the sinister Bourbakians and developing his mathematical trivium, this book is fairly over my head. I can thumb through it and appreciate some of the definitions (at last a satisfying definition of Betti numbers) and ideas, but the road which motivated all this is a distant hazy land to me. My mathematical ambitions are really ultimately only to understand probability theory a little better in its most applied forms. Just a word on the trivium, it gives me hope there is actually a mathematician inside of me; I would have done decently well at it in my early 20s, and managed to solve a couple of them in my head even in my present advanced state of cerebral decomposition. Of course this is why people like me love Arnold and hate Bourbakians: his work is always hammer and tongs stuff, or at least has its basis in hammer and tongs stuff. That’s probably why his results matter, and the Bourbakians are best known for sueing each other. I recently bought a copy of his “mathematical methods of classical mechanics” as well as his differential equations book and will likely work through them in coming months. The diff EQs book looks to be amusing in that it appears to be a restatement of ideas  in physics on Poincare surfaces of section, or something that looks like them, excepting in the more general case.

“Development of Mathematics of the 19th Century” Felix Klein -mentioned by everyone’s mathematical hero, Vladimir Arnold in the aformentioned book. Arnold suggested the book for one of his students (mentioned in the book above). Some nice enterprising Chinese people study hard differential equation and reprinted it for us. Klein was a math bro of the generation before Hilbert. Most known for the Klein bottle and contributions to group theory, he also wrote this cool book. I’m not going to give you the outline, but 19th century math is as superior to the 20th century as 19th century physics is superior to 20th century physics. People were concerned with important problems in those days; ones pertaining to ideas of practical interest to people. It was also a vastly smaller world; the names are mostly familiar, but the relationships were new to me.

“Sex & Deviance” Guillaume Faye. Faye is some kind of right wing frenchman, which means that he over analyzes everything, and …. notices things that shitlib types wish you wouldn’t notice. I have nothing against this; I am a noticer of things. Hell, I used to get paid to use machine learning to notice things (now a freebooter noticer; fuck helping megacorps notice things). Faye notices things which are real -current year is vile and disgusting and probably the least sexy time in human history, ethnicities and religions have different approaches to sexuality, men and women are different, feminists are imbeciles …. and so on.  On the other hand he has fucked up spergy views on, say, what Catholic sexual morality is. Does anyone besides Faye think people are undersexed in Catholic countries as opposed to protestants or whatever? This isn’t my observation. Lots of other weird little falsities like this, studded with excellent insights. I’m not even sure how or why I ended up reading the thing. Maybe /pol ran out of humor threads.

“Great Astronomers” Robert Ball. One of those mini-biography books listing some of the great men of astronomical science; from Claudius Ptolemy to Herschel. Written in the 1860s, and fairly jingoistic it is heavy on Englishmen you never heard of (afaik they’re mostly not hidden figures; they’re just filler that seemed appropriate when the book was written). It also contains a few I hardly think of as astronomers (physicists like Laplace, Hamilton and Newton), and leaving out people like Charles Messier, or, say, Delaunay. I can’t recommend it unless you’re really bored, which I was: the author doesn’t seem to understand the accomplishments of the people he’s talking about. Might be an artifact of its time, but I think it’s just bad. I had figured he was some schoolmaster type ratcheting up the jingo for young students, but in fact he was an Irish astronomer of some note. Interestingly he developed a theory of screws around the time of the great manufacturers of screws. Anyway his astronomy book sucks, but maybe I’ll read his masterpiece on screws. Screws are underrated.

Why Homer Matters Adam Nicolson -heard about this on the bapcast. I’m not sure I buy all of his speculations, comparisons or even his argument why Homer Matters, but it is a passionate and personal account featuring some explorations into archaeology, history and his, erm account of arab man touching him. Was definitely worth it for me, as it is rare for people to engage seriously with Homer except in very broad strokes. Reviews of this book elsewhere are unintentionally hilarious: women chimping out because …. Nicolson is a man rather than a eunuch basically.

“The Iliad” translated by Peter Green. This is a newish translation done by a guy who started at age 94. It has similar ambitions to the classic Lattimore translation (which I haven’t read; Butler and Chapman in the past) of hewing strictly to the Greek text. It takes a bit of mental effort to spin up on it as it does Shakespeare, but this is probably the most rewarding translation. He does annoying hipsterisms like calling Ajax Aias; while technically correct (Ajax is a latinism) it’s still annoying. Otherwise this is great. Difficult due to its closeness to the original, but great. I’ll read his Odyssey soon. If you read my blog and have never read the Iliad; fix that. This is an acclaimed translation, but if you’re not used to Homeric diction, read Butler or Lombardo instead. This is a book to inspire men to great deeds.

“The History and Culture of Japanese Food” -Naomichi Ishige. I read this quite a long time ago, but was reminded of it again as I have a friend visiting Japan. In general, national cuisines have a rich and varied history and can be related to events and discoveries of their peoples. The financial system of most of Japan’s history was based on rice, the religions preached variations on vegetarianism, and different specialities like beef sukuyaki and different kinds of sushis were innovations that came at very specific dates and times and for historical reasons which are certainly not clear without the explanations of the book. I wish I had this sort of history book for everything, or even everything Japanese. If anyone reading can suggest such a wonderful piece of autism on other aspects of Japanese culture, I sure would appreciate it.

“Mishima on Hagakure: The Samurai Ethic and Modern Japan” -I’ve the Hagakure before, but had a hard time understanding why anybody gave two shits about it in current year beyond as a historical curiosity. The author literally recommends badger underpants on long campaigns to avoid lice. Some of Mishima’s thoughts on the topic of Hagakure are enlightening enough, though sometimes obvious. The Way of the Samurai is indeed the Way of Death. I don’t really need Mishima to tell me this: it’s said directly in Hagakure. The one really valuable thing I got out of this book was the historical context of Hagakure; it was apparently mostly forgotten before the Pacific War. It makes sense it was used as wartime propaganda by the Tojo regime, and it makes sense Mishima was inspired by it as a result.  It would be as if some modern  Napoleon made de Charny’s Book of Chivalry an important part of every soldier’s mess kit (BTW I haven’t done more than thumb through de Charny, but it sits on the shelf next to Hagakure and Mishima). I thumbed through Hagakure again after this, and haven’t changed my opinion of the thing, but I guess I understand better why it was important to Mishima. Tsunetomo-san, author of Hagakure, had never seen battle and worked as a scribe his entire life; his entire social class of Samurai had not seen war for a century. The idea of a book like this is to revive the warrior spirit among people who have been soft for too long.

“Slide Rule: autobiography of an engineer” -Nevil Shute. Shute is turning into new favorite comfy reads; it starts out with his work on the R100 airship, made by the Vickers company in competition with the R101 made in a government lab. Shute notices some of the urgency and lack of bureaucratic restraints which made the R100 a much more successful project than the bloated government R101. Classic stuff for would be technologists: no fucking around with experimentation and innovation: just build the thing with maximum dispatch and minimum fuss. The Wikipedia entry doesn’t have Shute’s inside view, but gets the rough outline right: the R101 did too much innovation and R&D and was a comparative failure (killing its designers), where the R100’s tight budget and decisive management made it a comparative success. Also very charming management advice on hiring test pilots (prefer happily married men to hot dogs) and factory workers (avoid farm girls with filthy habits). The first hand account of flying around in a dirigible is priceless; I wrote my aesthetic appreciation for these devices as a sort of apotheosis of the Faustian spirit over a dozen years ago now. The later half of the book relates to his founding a successful aviation company. Very interesting stuff in that you could build an airplane company with a couple of mechanics and craftsmen for a period of time. Like most startups they were always one step from financial disaster; it was interesting that it was kept afloat in part by wealthy board members who were landed aristocracy.

Reinventing Your Life: The Breakthough Program to End Negative Behavior Young Klosko & Beck. -a book suggested by a friend for another pal’s woman problems. It wasn’t appropriate for my pal but I read it anyway. A lot of “life traps” people can fall into because their parents were assholes and didn’t raise them right, or, like, if they are just assholes. I think, in fact, most of these personality types are out of date if you’re not Gen-X. Younger people have more spectacular personality disorders, half of which aren’t even officially recognized as dysfunctional (they may even have political representation). I suppose if you have one of the problems listed, it looks like  helpful, actionable stuff to do about it. You probably don’t though. My biggest problem with it is one of the “life traps” listed is being a productive busy person. I suppose some such people are miserable, but …. who cares? Really “stop doing that” is pretty easy in almost all life trap problems. The bro whose predicament consists in dating crazy ladies,  I doubt he’ll need more than “hey, they’re crazy: stay away from those.” BTW this isn’t considered a “life trap” by this book, but it totally is.

Foxhound

I’ve been meaning to write about the MiG-25/31 complex since the last time I wrote about Soviet interceptors in 2013. The soviet interceptor philosophy was much different from the dominant US philosophy, but oddly the US philosophy came from a misapprehension of the MiG-25.

Originally the MiG-25 Foxbat was considered a super fast super maneuverable aircraft based on the giant inlets and giant wings. It was observed in Israel going Mach-3 in 1967 and was considered a super fighter well beyond western capabilities until a defector (Victor Belenko) gave us one. This was wonderful for the Macdonald Douglass company who came up with the F-15, which at the time was considered a sort of knock off. In reality the F-15 was a completely different kind of aircraft, which the Russians eventually knocked off in the Su-27, which is probably a better F-15 airframe than the F-15. The big wing and big engines made people think the MiG-25 was a highly maneuverable super dogfighter and air superiority fighter. It wasn’t: the big wing was to go high using lift, and the stainless steel construction was to protect the airframe from heat, so it could fly fast and high for long periods of time. The US used (Soviet, amusingly) titanium for its Mach-3+ interceptor designs.

The MiG-25 had its own goals and lessons; it was effectively the same philosophy as the other Soviet interceptors; just better at it. Go fast and high, meet up with enemy bombers using land and onboard radar, and shoot it down using the most advanced radar guided missiles. There’s scuttlebutt that the airframe was derived from that of the North American A-5 Vigilante, a forgotten nuclear attack jet which does vaguely resemble it in later forms. I don’t think there is any relation between them beyond the superficial high wing and vaguely similar intake manifolds. It was Gurevich’s (the G in MiG) last design and he didn’t need to steal anybody’s airframes. After all the F-15 looks a lot more like the MiG-25 which it was designed to combat, yet they have nothing to do with each other in mission, performance, airframe or much of anything.

Though the MiG-25 is an interesting jet, it was misused by export partners and mostly had an uneventful Soviet career shooting down spy balloons and trying to catch SR-71s like it was supposed to do. The MiG-31 evolution of it is more interesting as it is still used. It ultimately replaced all the other Soviet interceptors. Russians were still making them in 1994. Superficially it’s a MiG-25 with turbofan engines instead of turbojets -an important evolution in efficiency and reliability which quietly happened in the 1970s all over the world. They had the world’s first phased array radar system which allowed the MiG-31 to shoot down cruise missiles, and a wizzo operator in the back. This radar is ultimately probably why the US went stealth crazy in the 80s. It also had a stronger fuselage so it could operate at lower altitudes than the MiG-25 was useful in.

The MiG-31 also has a very interesting and innovative “airplane internet” where the weapons systems of a group of MiG-31s can coordinate attacks. I think the F35 is supposed to be the first US plane to have something like this, but I doubt it actually works, since nothing else about the flying dodo bird works properly. This is pretty impressive as it’s using Space Invaders (1970s) era electronics and software.

The R-33 is its main weapon as an interceptor: it’s for big bomber targets. B-1B, B-2, Avro Vulcan, SR-71, B-52, Mirage IV. It’s a big orc of a missile and it depends on the radar system. It has a hypersonic successor in the R-37; something the US can’t field because it has other priorities. There’s also a 23mm cannon and the capability for carry the normal contingent of Russian air to air IR and short range missiles as well as some weird ones nobody ever heard of.

Very current year is using the MiG-31 as a launcher for their Kinzhal hypersonic missile. Apparently one has been used in the war in Ukraine. It basically can’t be tracked or defended against, and every time there is a MiG-31 within a few thousand miles of the border (aka all the time), people in bunkers get nervous.

One of the most interesting developments from the war is the success of the MiG-31 against the Ukrainian Su-27. As stated above; the US originally thought the MiG-25 ancestor of the ’31 was something like an F-15; a very maneuverable dogfighter which does air superiority. The Su-27 is more or less a Russian F-15. Western observers always said the MiG-25 was a goofy paper tiger and the MiG-31 an outdated silly idea, but in actual air combat, the MiG-31 is smoking the Su-27s. They’re using the combat doctrine of the high speed high altitude interceptor with look-down radar, and it works. They seem to mostly be using the R-37 hypersonic missile, at least according to reports. Also important are the classic interceptor characteristics of powerful radar, extremely high speed and altitude. For what it is worth these are  the exact qualities the West thought would be important in the early supersonic era. The XF-108 rapier, Avro Arrow, YF-12 were designed with this doctrine in mind. Fly super high and fast and use long range missiles. Well, apparently this “outdated” doctrine works quite well when your missiles work properly.

The West bet big on Stealth technology a long time ago, mostly because Soviet radars were top of the pops. It worked against the Iraqi air force operating earlier Soviet machinery. It might not work so well against the Russians with modernized radar and missile technology. It may work particularly badly when the Russian plane is a MiG-31 flying 15,000 feet above, as it’s not super clear that “stealth” works well against look-down. It is interesting that the Rooskies have announced a replacement which is essentially the same thing, but higher and faster. This is an aspirational one to be sure, but early work on a hypersonic jet was undertaken by MiG in the 70s and 80s as a replacement for the MiG-31, and they’ve been fielding some wild stuff lately.

Management consultants as Soviet apparatchiks

The management consultant is one of those phenomena that people take for granted, but which are really  ridiculous as a sociological phenomenon.  Consider a standard example of a management consulting engagement: you have a giant corporation with tens or hundreds of thousands of talented people that are good at doing its corporate crap, whatever it might be. You are going to make an important decision in bigCo, so you hire a management consulting firm: a company which knows nothing about what you do as a company, has no inborn expertise about anything, and which  hires fresh out of Yalevard imbeciles to wear nice clothes and give powerpoint telling you what you want to hear. You then make your decision and proceed to feed millions of dollars to the  management consultancies.

The management consultants often get paid to do long term planning, as if the firm itself can’t do this. Most of the firms they advise are already old, so they must be at least historically adequate at long term planning. The management consultants won’t be there to take the consequences at the end of the long term planning, unless the planning fails and short-memory bureaucrats hire the consultants to sort out the long term disaster they created.

There are obvious benefits to working for a consulting firm as an individual: you get to belong to a powerful mafia and you get trained in the latest in corporate rhetoric and polish. Look at Sundar Pichai; he went from McKinsey consultant to Google CEO. And everyone knows Sundar Pichai is the best Google manager of all time. It’s certainly not clear they bring any value to their clients.

Now, let us examine what the apparatchik was. It’s a vaguely derogatory term, but most of us have no real knowledge of what these people did. We have an idea that they’re brown-noser loyalist commies who sit somewhere in the upper-middle tier of the Soviet bureaucracy. The Wakipedia entry has a few more choice words to put color on it:

An apparatchik (/ˌæpəˈrættʃɪk/; Russian: аппара́тчик) was a full-time, professional functionary of the Communist Party of the Soviet Union or the Soviet government apparat ….. Members of the apparat (apparatchiks or apparatchiki) were frequently transferred between different areas of responsibility, usually with little or no actual training for their new areas of responsibility. Thus, the term apparatchik, or “agent of the apparatus” was usually the best possible description of the person’s profession and occupation.

McKinsey  reached its present form via the leadership of someone named Martin Bower. George Booz was another inventor of the modern approach, though Booz Allen Hamilton is basically a publicly traded branch of the American intelligence services at this point, so they are more like nomenklatura than apparatchiks. Both men were from the finest Ivy League lizard backgrounds, and believed in the idea of firms taking outside advice from gentlemen of a certain education and social class. Probably this originated in some social club where these fellows were already doing this with their CEO pals, without the formal engagement.

Of course the types of firms these guys did business with were being led by …. people who get the job by being of a certain social class, rather than people who actually created the company or made it grow.  Since we’re good Kleinian economists here, pretty much all victims of management consulting are Klein type 3 or type 4 organizations. To remind everyone of the organization types of Burton Klein:

Type 1: “Happy warrior rationality” is associated with ideological outbreeding and is commonly employed in making fast history.

Type 2: “Middle-class rationality” is associated with ideological inbreeding and is commonly employed in making slow history.

Type 3: “Accounting rationality” is associated with a zero rate of ideological change and is commonly employed by profit maximizing firms in a temporary equilibrium with an unchanging outside environment.

Type 4: “Conservation-of-power rationality” is associated with organizations which have such a low ability to deal with unpredictability that they must manipulate the rules of the game if they are to survive.

So it was in the time of the fathers of Management Consulting and Burton Klein: and so it is now. There are no management consultants in Type 1 startups, and few to none in Type 2 organizations. If you have a sclerotic 25, 50 or 100 year old American firm though: you hire management consultants to sort out your problems. Essentially the management consultant tells higher management what they want to hear by doing an end run around middle management bureaucracy. At best they may add value by asking people in the trenches what they think: something that could be done by proactive management. At worst they just regurgitate some bullshit they read in HBR. There are  variations on this formula: Accenture does this with a specialization in technology. They were probably a normal tech consulting firm at one point, but now they’re McKinsey with Eclipse editors.  Bain seem to specialize in extracting value from functioning companies by bankrupting them or sending their factories overseas. They also dabble in government corruption and insider trading. BCG also government corruption (aka colonialism) with a tech arm. McKinsey offspring Kearney is known for insider trading, colonialism and being WEF lackeys. McKinsey of course; they also do government corruption (aka colonialism), drug dealing and insider trading. They want to get into big data as well.

Let’s forget about the numerous ethical monstrosities of these management consulting firms for the moment and look at these firms for what they are. All of these firms are made up of members of a certain social class. The social class they’re drawn from are top bureaucrats and “leaders” in the American establishment. They’re people who went to Harvard and Northwestern, not U-Mass or U-Michigan. They have an ideology: pretty much it is the ruling class ideology -whatever it might be at the time. All that Davos/WEF “eat the bugs” horse pookey? That’s all management consultant stuff; I think WEF even has their favorite firm. I’m not sure what ruling class organ the apparatchiks of the Soviet Union were reading: Management Consultants read the Economist and HBR for their ideological updates. While the Economist and HBR are bad enough by themselves, these clown cars also have their own ideological tools, as lurid and ridiculous as anything out of communist apparatchiks.

Consider the DICE framework of BCG. It is a heuristic I guess designed to assess successful outcomes for consulting engagements.

Duration (D)either the total duration of short projects, or the time between two milestones on longer projects

Team Performance Integrity (I)the project team’s ability to execute successfully, with specific emphasis on the ability of the project leader

Commitment (C) levels of support, composed of two factors: C₁ visible backing from the sponsor and senior executives for the change C₂ support from those who are impacted by the changeEffort

(E)how much effort will it require to implement (above and beyond business as usual)

D + (2 x I) + (2 x C₁) + C₂ + E

These are all subjectively scored on 1-4 basis. People actually take this gorp seriously. The scientastic veneer of having an equation apparently confers credibility. It is, of course, the sheerest nonsense (they optimize for revenue like everyone else). The entire field is filled with goofy rubbish like this; for example the BCG “growth share matrix” I can imagine people in 1970s three piece suits with large collars to match long sideburns coming up with this nonsense. They probably went to a sex orgy on an orange shag carpet afterwords to celebrate. I’m certain the apparatchiks had their own such tools based on some weird Hegelian gorp.

The role they’re supposed to fill is to fix dysfunction in sclerotic type 3 and type 4 organizations. This is always going to be a problem facing human beings. Large organizations are going to become disorganized and have ill defined self-serving groups within. Giving a pipe-fitter or machinist supreme power to hire, fire and reorganize the sclerotic organization is more likely to work than hiring McKinsey. Pipe-fitter is a regular no bullshit kind of Joe. Executives could explain their problems in minute detail to no-bullshit working man, working man will give advice, then get a million dollars if it works out. Unlike McKinsey he’d get paid on performance. Unlike McKinsey associates, Mr. hypothetical pipe-fitter isn’t afflicted with the degree or kinds of ideology and nonsense that afflict current year management consultants. His solution will likely have a lot of common sense baked into it, possibly even including the kinds of pool room Solomonic wisdom utterly foreign to management consultants. Plus, everyone’s incentives are aligned. I can go find legions of these people: they’re in Boston, drinking beer in taverns.

For what it is worth, this was the original Soviet idea, and it wasn’t a terrible one. Killing everyone to implement it was rude and listening to that gibbering dunderhead Marx and his squalid followers about economics was moronic, but the basic idea of having practical men give feedback on complex systems is quite sound. Note that the Soviet system never had a problem with its type-1 and type-2 organizations; they were arguably better at it than the West was; it was always a difficulty with type-3 and 4 organizations. Of course, all the practicality of having a construction worker or tractor mechanic look at your pantyhose factory was forgotten after the war, and they educated a class of peasant-souled bureaucrat ideological apparatchiks who proceeded to destroy the place. Much like the parasitic management consulting apparatchik bugmen are destroying the West.

While the technical oriented consultancies may do some good, it’s  obvious at this point that the management consultancies do not serve any socially useful purpose in modern society. Mostly, they are a self serving group who exist to support their in-group and their customers. By “customers” I don’t mean the corporations and shareholders, I mean incompetent bureaucrats who have been promoted beyond their capabilities. Mind you, people from these groups have a lot of polish: they dress up nice and give good powerpoint. If you wanted to hire, say, a lobbyist or something that requires diplomatic skills, someone formerly at one of these firms would probably be ideal.

However, they should mostly be laughed at, as people eventually laughed at communist apparatchiks. They’re the same phenomenon; ridiculous, self regarding goblins who get paid lots of money to, for example, get Americans addicted to pain pills to up your margins. At some point in the happy future, if people aren’t living in nuclear rubble, the management consultant will become a sort of punch and judy  comedy stereotype the way the communist apparatchik did. As it is, they should be shunned from all decent society for what they are.