(Like this article? Read more Wednesday Wisdom! No time to read? No worries! This article is also available as a podcast).

People who are getting on a bit in years (like me), like to discuss how nothing is like it was anymore. In that line of conversation, there is a fine distinction between outright complaining and a potentially useful analysis of changes in society and how these changes impact the things that are happening. Last week, during a dinner with my former manager at a big tech company (who is also past the target age for an AARP membership), we started discussing how we got into the field, how that was different from the situation people find themselves in today, and what the consequences of that are.

When I started getting interested in computers, everything was an uphill battle in all directions: Computers were scarce, they were underpowered, and they were expensive. There was little information available and what was available was difficult to get and, again, expensive. I regularly bought computer magazines and tried to get my hands on any and all books that discussed computers and programming. It was also a very lonely endeavor, as I knew almost nobody who knew anything about computers.

Then, when I went to college, access to computers, books, and people improved significantly, but it was still not a panacea. For instance, during my first two years at college, we had to reserve time slots to use the terminals of our school’s mini-computer. Bookings were for 30 minutes time slots you could have at most three slots outstanding. Yes, you read that right, we could have a max of 1 1⁄2 hours of terminal time reserved up front! When that time was spent, you had to go find the reservation terminal and then you could book another 1 1⁄2 hours, subject to availability. In these circumstances, when you get to sit down in front of the terminal, you better know what you are going to type in.

Additionally, the Pascal and COBOL compilers ran on a batch system overnight, so you had to carefully plan a ½ hour slot for the next day to see if your program had compiled, and if it hadn’t, make any corrections and resubmit, so that you could get another attempt at maybe running it the day thereafter. As you can imagine, this system did not do wonders for velocity and there was a bonus for getting your work in early in the trimester.

To make matters worse, whenever the college IT department was faced with a low free disk space situation, they ran a program called “DailyReport” which removed “temporary files” from the disk. Unfortunately, “DailyReport” considered student’s binaries temporary files, because they could be recreated from the sources. On at least one occasion, I logged on in the morning to find that my compilation had succeeded at 2am but that DailyReport had come round at 4am to remove my binary. Try submitting your code for a software engineering class on time under these circumstances!

All of that hassle was to get a professional qualification for a job that didn’t even pay that well. It wasn’t bad, but it was not “doctor or lawyer” good. I have written about this before, but when I graduated, I could maybe look forward to a comfortable middle class existence. As a matter of fact, in my first job, my yearly income was so low that I qualified for healthcare under the state’s social health fund plans. Nothing wrong with that, I guess about 75% of the country earned under the threshold for the funds, but it just goes to show that nobody, not even me, thought I was heading for a cushy tech job.

There are some consequences to this state of affairs. First of all, in that environment, only people who really (and I mean really really) like toying with computers get into the field. It’s not just that coding wasn’t cool yet; it was actually difficult and cumbersome. Only people with a deep interest in the subject matter and with lots of passion studied computer science and they had to struggle mightily to become any good at it. But, good they typically became, because: Pressure makes diamonds.

Another consequence of this story is that, tech skills being rare, it was very easy to let it go to your head.

In the small village I grew up in, I knew literally nobody who was into technology. That’s quite lonely as it means that you have nobody to talk to about your passion and it doesn’t make you very popular either. But it also makes you think you might be a wizard. Myths and legends are full of wizards and they are always solitary figures that are holed up somewhere and doing things that nobody understands while speaking strange languages. I was a solitary figure that was holed up in my room all day doing things that nobody understood in strange languages, so I thought I was a wizard. Stands to reason.

This feeling persisted into college. Sure, there were more people there, but in the meantime computer science had become somewhat hip and so I had a fair amount of fellow students (especially at the start of the first year) who did not have my deep pre-college experience hacking with computers. Fortunately, a handful of them did and we hung out together, because it is expected for wizards to form a conventicle, especially in wizard school (forming our own little pre-HP Ravenclaw).

Most of the rest did not make it very far in college. We started the first year with 220 students and four years later, about 60 would graduate. During the introductory program, the professors helpfully said: “Look left and right of you, only one of you will be here next year. Even then, of the people who graduated in the same year as me, I would trust approximately a quarter of them to touch a computer that I care about.

The problem with feeling you are a wizard is that it can lead to arrogance. Being in the possession of secret knowledge and arcane spells, I would look down on the muggles who were struggling to set their VCR’s clock to adapt to daylight savings time or who were trying to tame the dragon of WordPerfect without knowing the incantations and wand movements required to do so.

One day, a couple of guests came in to dine in our restaurant. They were obviously exhausted and my mother asked what was going on. They explained that they ran a small bookkeeping firm and they had just bought their first computer. For the whole day, they had been trying to set it up, install some software, and make it print. My mother graciously offered the services of the local wizard (me). Next day, I went over there and did the needful, showing off my mastery of the arcane “MODE 9600,N,8,1” and the contents of CONFIG.SYS.

Until I started working, I really did not meet anyone who knew more about computers than I did. Not even the college professors. However, when I did start working, I soon found myself surrounded by real wizards who knew much more than I did. This was a humbling experience. It’s not just that they knew and understood things I didn’t know and understand, they knew and understood so much more that I feared I might never catch up…

This is of course the young person’s underestimation of the value of time. Time heals all wounds and it affords compound interest (in both money and knowledge). Steadily grinding over a period of time does wonders.

After my first job, I kept finding people who were much smarter and more knowledgeable than me and I started to thoroughly enjoy their presence. At almost every company or contract, I found people, often older than me, who had knowledge and experience that I didn’t have. Fortunately, I had never become so full of myself that I resented meeting these people. It also didn’t break any fundamental feelings of superiority because, being Dutch, I really didn’t have these to begin with; all things considered, feeling like a wizard was a really thin layer of veneer.

Holland is the country where people say: “If you behave like a normal person, you are already crazy enough.” Dutch parents tell their kids to “Doe normaal!” (Act normal!) It is one of the two Dutch sentences the lovely Mrs Wednesday Wisdom can pronounce. The other one being: “Hand voor je mond!” (Hand in front of your mouth!), which is what we say if someone yawns while providing access to the light source at the other end of the tube that goes through your body all the way to the other end.

When I joined Google, I finally found my place. Here was a company where everyone was a wizard. When people asked me what it was like, I used to say: “Well, everyone’s a wizard and I am the dumbest person in the building”. Being in that exalted company robbed me from any last vestiges of feeling special that might still have been lingering. My colleagues had written books on a range of topics, had authored open source software that everyone in the world used, patented new stuff left and right, invented new (and useful) programming languages, and together we did things that had never been seen before. I cannot lay claim to any significant contributions to any of that, but I am really good at taking other people’s great ideas and running with them, which is a skill in itself. And also, I can grind, which in a world where every success is 1% inspiration and 99% perspiration is also quite useful.

Being the dumbest person in the building means that you experience a lot of pressure from everyone above you. But, you have to realize that the easiest way to get better at something is if you have the examples right in front of your eyes to copy and learn from. It might be a lot of pressure, but pressure makes diamonds!

I kept running into people that looked like the younger me, but “gone wrong”. I once interviewed a young gentleman from South America who had applied for an SRE role at Google. I started the interview with a lowball Linux system administration question, which he answered satisfactorily. After giving the answer he looked at me proudly and said: “Have you ever met anyone aged 26 who knows as much about Linux as I do?” “Yes,” I answered, “everyone in this building.”

He did not make the hiring bar, but he became legendary for asking our lovely receptionist out on a date on the way out of the building.

In the hiring committees we regularly rejected candidates who we felt were probably smart enough, but who had spent too much time being a big fish in small ponds. It is not hard to shine in a dim room and these people had grown complacent and not continued to develop as engineers. I’ll say it again: Pressure makes diamonds, without that pressure you just have some carbon…

The times have really changed a lot. Compared to 40 years ago, there is an abundance of everything you need to get going in the field: Powerful computers are cheap and widely available; there’s the Internet, and, until recently, getting into technology was a surefire way to get a high paying job. Consequently, it appears to me that many people choose a career in tech not because of a passion for the subject matter, but for purely monetary reasons. I do not begrudge anyone that choice though and I totally understand parents who give their children three career options: Doctor, lawyer, software engineer. But I do regularly see that lack of passion and without that passion you do not have the intrinsic motivation to grind and without that grinding it is hard to become good.

I regularly told people who asked how to land a job at Google that the only people who could do that were the ones who had thoroughly misspent their childhood.

Personally I miss the days of pouring over a book on Z80 assembler and trying to make heads or tails of it. I got into this field because it seemed to me that making the idiot box do something useful was one of the greatest puzzles around and it has never disappointed in that sense. Choosing my employers so that I was always the dumbest person in the building worked extremely well for me. As a strategy for selecting your next gig, I can highly recommend it.

Download audio: https://api.substack.com/feed/podcast/160757610/b7d0f0e19f2c8578404ca5afbb4f1891.mp3

The Department of Government Efficiency, Musk's vehicle. made news by "discovering" the General Services Administration uses tapes, and plans to save $1M by switching to something else (disks, or cloud-based storage). Long time readers of this blog may remember I used to talk a lot about storage and tape backup. Guess it's time to get my antique Storage Nerd hat out of the closet (this is my first storage post since 2013) to explain why tape is still relevant in an era of 400Gb backbone networks and 30TB SMR disks.

The SaaS revolution has utterly transformed the office automation space. The job I had in 2005, in the early years of this blog, only exists in small pockets anymore. So many office systems have been SaaSified that the old problems I used to blog about around backups and storage tech are much less pressing in the modern era. Where we have stuff like that are places that have decades of old file data, staring in the mid to late 1980s, that is still being hauled around. Even when I was still doing this in the late 2000s the needle was shifting to large arrays of cheap disks replacing tape arrays.

Where you still see tape being used here are offices with policies for "off-site" or "offline" storage of key office data. A lot of that stuff is also done on disk these days, but some offices still kept their tape libraries. I suspect a lot of what DoGE found was in this category of offices retaining tape infrastructure. Is disk cheaper here? Marginally, the true savings will be much less than the $1M headline rate.

But there is another area where tape continues to be the economical option, and it's another area DoGE is going to run into: large scientific datasets.

To explain why, I want to use a contrasting example: A vacation picture you took on an iPhone in 2011, put into Dropbox, shared twice, and haven't looked at in 14 years. That file has followed you to new laptops and phones, unseen, unloved, but available. A lot goes into making sure it's available.

All the big object-stores like S3, and file-sync-and-share services (like Dropbox, Box, MS live, Google Drive, Proton Drive, etc) use a common architecture because this architecture has been proven to be reliable at avoiding visible data-loss:

• Every uploaded file is split into 4KB blocks (the size is relevant to disk technology, which I'm not going into here)
• Each block is written between 3 and 7 times to disk in a given datacenter or region, the exact replication factor changes based on service and internal realities
• Each block is replicated to more than one geographic region as a disaster resilience move, generally at least 2, often 3 or more

The end result of the above is that the 1MB vacation picture is written to disk 6 to 14 different times. The nice thing about the above is you can lose an entire rack-row of a datacenter and not lose data; you might lose 2 of your 5 copies of a given block, but you have 3 left to rebuild, and your other region still has full copies.

But I mentioned this 1MB file has been kept online for 14 years. Assuming an average disk life-span of 5 years, each block has been migrated to new hardware 3 times in those years. Meaning each 4KB block of that file has been resident on between 24 and 42 hardrives; or more, if your provider replicates to more than 2 discrete geographic region. Those drives have been spinning and using power (and therefore requiring cooling) the entire time.

These systems need to go to all of this effort because they need to be sure that all files are available all the time, when you need it, where you need it, as fast as possible. If a person in that vacation photo retires, and you suddenly need that picture for the Retirement Montage at their going away party, you don't want to wait hours for it to come off tape. You want it now.

Contrast this to a scientific dataset. Once the data has stopped being used for Science! it can safely be archived until someone else needs to use it. This is the use-case behind AWS S3 Glacier: you pay a lot less for storing data, so long as you're willing to accept delays measurable in hours before you can access it. This is also the use-case where tape shines.

A lab gets done chewing on a dataset sized at 100TB, which is pretty chonky for 2011. They send it to cold storage. Their IT section dutifully copies the 100TB dataset onto LTO-5 drives at 1.5TB per tape, for a stack of 67 tapes, and removes the dataset from their disk-based storage arrays.

Time passes, as with the Dropbox-style data. LTO drives can read between 1 and 2 generations prior. Assuming the lab IT section keeps up on tape technology, it would be the advent of LTO-7 in 2015 that would prompt a great restore and rearchive effort of all LTO-5 and previous media. LTO-7 can do 6TB per tape, for a much smaller stack of 17 tapes.

LTO-8 changed this, with only a one version lookback. So when LTO-8 comes out in 2017 with a 9TB capacity, a read restore/rearchive effort runs again, changing our stack of tapes from 17 to 12. LTO-9 comes out in 2021 with 18TB per tape, and that stack reduces to 6 tapes to hold 100TB.

All in all, our cold dataset had to relocate to new media three times, same as the disk-based stuff. However, keeping stacks of tape in a climate controlled room is vastly cheaper than a room of powered, spinning disk. The actual reality is somewhat different, as the few data archive people I know mention they do great restore/archive runs about every 8 to 10 years, largely driven by changes in drive connectivity (SCSI, SATA, FibreChannel, Infiniband, SAS, etc), OS and software support, and corporate purchasing cycles. Keeping old drives around for as long as possible is fiscally smart, so the true recopy events for our example data is likely "1".

So another lab wants to use that dataset and puts in a request. A day later, the data is on a disk-array for usage. Done. Carrying costs for that data in the intervening 14 years are significantly lower than the always available model of S3 and Dropbox.

Tape: still quite useful in the right contexts.

Today’s episode of Untaxing is about Jaffa cakes and VAT. With the benefit of hindsight, that’s a very small VAT issue on a day when VAT has become a very large geopolitical issue. Donald Trump is considering applying tariffs to much of the world because of VAT. He believes it’s a tariff. You’ll be unsurprised […]

Source

(Like this article? Read more Wednesday Wisdom! No time to read? No worries! This article will also become available as a podcast on Thursday)

Everywhere I go, there are things that are suboptimal. That’s not unexpected; the universe is a cold, dark and lonely place that was clearly not created for either our comfort or our convenience. Accompanying these suboptimalities are people who can explain in detail what is wrong with the world; newspapers and social media are full of them. But does all that talk help?

The people who are explaining all the world’s problems seem to operate under the assumption that people either do not know that the problem exists or that they don’t care. They are usually wrong about the first thing, but usually right about the second. However, in my experience, just explaining the problem does not help one iota.

Every morning, I wake up and read a high quality Dutch newspaper. And every morning, without fail, it is full of stories that describe something that is wrong with the world. Here is a selection of today’s picks: American scientists are not allowed to join an IPCC panel on climate change, one of our dumber ministers (and that is saying something these days), called Zelensky “not democratically elected”, our lame duck prime minister is powerless in Europe, female athletes who complain about harassment from their coaches are not listened to, kids spend too much time on TikTok, and one of our public broadcasting societies (VPRO, of which I used to be a member) is an inward focused mess. Here is what is quite rare though: Any talk about solutions. It is usually just a whole lot of complaining and then sometimes a vague call to action: Someone should do something! But what exactly should be done? And by whom?

Let’s take an example.

In Europe there is considerable worry about dependence on American firms for cloud-based services, especially by governments and government agencies. There are newspaper articles about this problem, blogs, policy briefs, LinkedIn posts, and even questions in parliament. And that is usually where it ends. There are big unrealistic vague calls to action and then the next thing we read is that the Dutch Internet domain registrar is planning to move a big part of their operation to AWS.

This problem is of course massive. First of all, there is no European cloud alternative that can provide a level of service that is anywhere near what Microsoft, Amazon, and Google can offer. Having worked at or with all three of these cloud providers, I can say with some confidence that suggestions that any current European provider can offer a competitive service are simply laughable. But, it is not physically impossible. Europe has the money and it definitely has the talent. Sure, it doesn’t help that some of their greatest talents have moved to the US or work for the Americans, but that can probably be overcome. It is a gargantuan task though and it is not at all clear that there exists enough willpower to solve it. I mean, after more than a decade of signals that it is really time to get more self-sufficient in defense and energy, the Europeans were still caught like deer in the headlights when Russia invaded Ukraine and more recently when Donald Trump got elected and immediately started rolling out his maffia-like “NATO as a protection racket” policies. If we cannot fix things that are that important, will we really get our act together when it comes to, I don’t know, running our own virtual machines?

And thus, after many articles that explain how really big and really pressing and really important the problem is, nothing happens. As we say in Holland: Everyone takes a leak and things continue as they were.

```

This sounds better in Dutch because it rhymes: Iedereen doet een plas en alles is weer zoals het was.

You can see this same lack of tangible action at work on much smaller scales too. Many teams that I joined in the past ran complicated infrastructures where there are lots of things going wrong. Things are on fire, customers complain, and the pager rings off the hook. In situations like this there is no shortage of people who can explain what the problem is. But, like with the big problems, everyone takes a leak, and things continue as they were.

I get it though. All of these problems are relatively huge, often complicated, without obvious solutions, and mostly not urgent, where “urgent” is defined as: Our comfortable existence will be upended tomorrow if we don’t fix this today. But because of this, we are stuck a bit in a loop of problems that are complained about but not solved unless they become so urgent that we need to drop everything and declare a code red, because eventually our comfortable existence will actually be under threat of ending on short notice.

The root cause of the paralysis that often follows the explanation of a huge problem is that these explanations are not accompanied by reasonable calls to action. For example: People worried about the dominance of US cloud providers tell everyone that our government should stop using Microsoft ‘s cloud solutions. That is quite simply not possible on short notice and it is not even possible to determine where to begin. The people who say it is possible clearly have no experience providing IT-solutions to large bureaucracies. The annals of IT history are littered with projects on a much smaller scale gone horribly wrong. Did anyone say Horizon? Or KEI? In a similar vein, people who say that a team should stop everything they are doing and attack some problem are equally unrealistic. The team has lots to do this quarter and pretty much no annoying and urgent problem is so annoying or so urgent that the team can afford to drop everything and refocus.

So, what to do?

First all, realize that problems that take decades to build up cannot be solved in years. Similarly, problems that were years in the making, cannot be solved in weeks. If it took decades to get to where we are, it will take decades to solve. That doesn’t mean we shouldn’t start today, but it does mean that we should be realistic in our goals for the short term.

Next: Most people truly underestimate the effects of work compounding over time. Much like compound interest, doing something and then keeping at it really leads to amazing results even after a relatively short amount of time. That’s why I advise teams that are in fire fighting mode not to drop everything, but instead to commit to spending a manageable amount of their time to solving something, anything, but also to keep at that quarter after quarter. Even if you consistently spend only 10-15% of your time solving the things that you can solve, that work really adds up.

There are other advantages of this approach too: By setting your goals at an attainable level, you increase people’s belief that this is something that can and eventually will be solved. That’s the kind of positivity that begets more positivity. You are not only solving things, you are also creating confidence that this is a problem that can and will be solved. So when the time comes for bigger or more painful investments, you have some work that you can point to that will inspire support for your program. Nobody is going to spend oodles of money on an ambitious project, but people will be ready to fund the next phase of a project that has been going on for a while and that has been delivering results.

For the European dependency on US cloud providers I suggest a similar approach. Getting rid of all US clouds for all services and all government agencies is of course an impossible project. So here is my suggestion: Start with email. Make a principled decision that you are going to switch all government email to a newly built European service. That’s not a comprehensive solution by any means, but it is a start, and one that I think is doable. It will probably take ten years or so, but it is something that is attainable. For all its importance as the backbone of government information exchange, email is mostly a solved problem. The technologies are well understood, there is ample software available, and wherever we are going in the future, email will probably play a part. Remember: The whole Google cloud started with GMail! So why not do the same.?Don’t get me wrong, it will still be a massive project, but if we can’t even do that, there is nothing left but total and utter despair 🙂.

So, don’t just complain about the status quo. Talk is cheap, always offer solutions too. They don’t have to be all encompassing and solve every possible problem, but propose something and get going.

Download audio: https://api.substack.com/feed/podcast/157829048/254f4be9c6d776da4683cc363a343725.mp3

This is my response to Rob Pike’s words On Bloat.

I’m not surprised to see this from Pike. He’s a NIH extremist. And yes, in this aspect he’s my spirit animal when coding for fun. I’ll avoid using a framework or a dependency because it’s not the way that I would have done it, and it doesn’t do it quite right… for me.

And he correctly recognizes the technical debt that an added dependency involves.

But I would say that he has two big blind spots.

1. He doesn’t recognize that not using the available dependency is also adding huge technical debt. Every line of code you write is code that you have to maintain, forever.

2. The option for most software isn’t “use the dependency” vs “implement it yourself”. It’s “use the dependency” vs “don’t do it at all”. If the latter means adding 10 human years to the product, then most of the time the trade-off makes it not worth doing at all.

He shows a dependency graph of Kubernetes. Great. So are you going to write your own Kubernetes now?

Pike is a good enough coder that he can write his own editor (wikipedia: “Pike has written many text editors”). So am I. I don’t need dependencies to satisfy my own requirements.

But it’s quite different if you need to make a website that suddenly needs ADA support, and now the EU forces a certain cookie behavior, and designers (in collaboration with lawyers) mandate a certain layout of the cookie consent screen, and the third party ad network requires some integration.

What are you going to do? Demand funding for 100 SWE years to implement it yourself? And in the mean time, just not be able to advertise during BFCM? Not launch the product for 10 years? Just live with the fact that no customer can reach your site if they use Opera on mobile?

I feel like Pike is saying “yours is the slowest website that I ever regularly use”, to which the answer is “yeah, but you do use it regularly”. If the site hadn’t launched, then you wouldn’t be able to even choose to use it.

And comparing to the 70s. Please. Come on. If you ask a “modern coder” to solve a “1970s problem”, it’s not going to be slow, is it? They could write it in Python and it wouldn’t even be a remotely fair fight.

Software is slower today not because the problems are more complex in terms of compute (yet they very very very much are), but because the compute capacity of today simply affords wasting it, in order that we are now able to solve complex problems.

People do things because there’s a perceived demand for it. If the demand is “I just like coding”, then as long as you keep coding there’s no failure.

Pike’s technical legacy has very visible scars from these blind spots of his.

(Like this article? Read more Wednesday Wisdom! No time to read? No worries! This article will also become available as a podcast on Thursday)

My first job was at a bank . When it came time to discuss compensation, HR sent me a letter that explained that my future job was categorized as a level 8 job and that per the collective agreement negotiated between the unions of bank employees and the organization of bank employers, my salary would be X. It was my first job, I had no idea what X meant and I still lived at home, so I barely needed money anyway. The collectively bargained agreement also came with standard increases every year to account for growing experience and of course there was the yearly inflation correction. It was highly organized and nobody worried about compensation.

A few years later, I worked for another bank and by that time the shortage of IT talent was starting to hurt. Because of the constraints of the wage tables negotiated by the unions, it was getting hard for banks (and other sectors of the economy that were ruled by similar agreements) to pay enough money to stay competitive. There was a healthy (or perhaps unhealthy) stream of people who left these organizations to work for “software bureaus”. These were body shops that paid a higher salary than organizations in sectors run by collectively bargained agreements could offer and then contracted staff out to their original employers.

Example: One of my colleagues left on Friday and came back on Monday as a contractor, but with a higher salary and a nice company car. He was very good and very nice so we didn’t begrudge him that.

Unions were unwilling to make allowances for certain types of expertise, all in the name of fairness. In their eyes, a software engineering role that required a four year college degree should be paid about the same as any other role that required a four year college degree, market conditions be damned. In later years, the unions grudgingly agreed to a ”labor market bonus”. With this bonus certain types of jobs could be offered a higher salary than other jobs that required about the same amount of education and expertise.

A cursory web search indicated that this is still a thing in Holland. In one sector’s agreement I found a clause that said that the labor market bonus could be as much as 10%. This being Holland, the consent of the workers council is of course required because generally people hate it when other people earn more than they do.

Even with the labor market bonus, employers had a difficult time hiring into certain jobs because there were other benefits that the body shops could offer and they couldn’t, such as a company car, luxurious trips, and relatively generous expense accounts. For people who thought their skills were worth even more, there was the option to start their own one-man body shop. This came with the obvious advantage that you could bag all of the contracting hourly rate. On top of that, you did not have to pay certain types of social security taxes that applied to employees but not to individuals who ran their own small businesses. I started my own one-man operation in 1992 (and didn’t have a regular job again until I joined Google in 2006).

Examples of such social security taxes are the unemployment tax and the disability tax. Most of these “one-cylinder” operations figured they would never be unemployed. Some (like myself) insured themselves privately for disability. Being a one-cylinder operation also came with options to lower your total tax burden by stashing any money you made but did not need in an LLC, thereby hiding it from income tax.

Then the 90s hapened and the official motto of society became: “Laissez les bons temps rouler”. The whole economy moved in the direction of laissez-faire capitalism and everyone in the IT sector benefited, as tech companies enjoyed the benefits of the network effect and disrupted one market after another. The world went from “a few computers, somewhere” to “lots of computers, everywhere” and so the demand for people who knew how to build, program, and maintain these computers grew and grew. On top of that, social democracy was out and neoliberalism was in, and that ideology prescribes that everyone must get as much out of the market for themselves as possible, whatever the consequences.

This had two effects: First and foremost, salaries went up because the regulations went down and the job market is, well, a market. Secondly: Lots of new people entered the market, drawn in by the high prices even a junior engineer could demand; these were the times where we had to offer equity to interns! The result of this was that anyone working in information technology did phenomenally well as the price of employment went through the roof. Everyone got accustomed to high and growing compensation as well as to other forms of remuneration: Free food, fun offices with pinball machines, gym memberships, lots of flexibility to work from home, free shuttles to the office, unlimited PTO, generous 401(k) programs, employee stock purchase programs, free phones, paid-for Internet connection at home, and free snacks. No idea was too wild if it could help get scarce talent through the door.

One often-neglected insight from this is that it is really expensive to be poor. If you are struggling in a low wage job, you probably also have to buy all of your own food, pay for your own gym membership (if you can even afford it), pay for your own Internet, and pay for your own commute. Nobody will invite you to cushy trips or give you $1500 for Christmas just because. On the other hand at Google we had people complaining that they had to arrange for their own food during the weekend and people got terribly upset when the catering staff organized a meatless Monday, with the money not spent on meat donated to the World Food Program. One engineer was so upset that he loudly complained that “it is impossible to work under these circumstances!”

It is not hard to understand that these very pleasant conditions made software engineers a happy lot. Unfortunately, part of that happiness was caused by having a really short memory and most people had forgotten what working conditions were actually like before this holiday from history.

One of the more annoying aspects of our hyper anxious news cycle is that whenever things are going well, there are lots of pundits that will declare that things have fundamentally shifted and that this time, really, the trees will grow into the heavens forever and will never stop growing. We regularly hear economists confidently declare that there will never be a recession again. Other economists have been know to state equally confidently that the economy will continue to grow at this amazing pace and that everybody will be rich. The understanding that many aspects of the human experience are a sine wave seems to go right out of the window every time we are nearing the top of the curve.

Book tip: Y2K: How the 2000s Became Everything (Essays on the Future That Never Was) by Colette Shade is a must-read book for people who grew into adulthood in the 1990s and early 2000s. One of the essays is about the outrageous economic claims that were made.

These people are obviously clowns and have conveniently forgotten that in a perfect market all profits eventually go to epsilon. If there is money being made anywhere, that is either a temporary condition or a sign that there are unnatural barriers to entry, such as borders, limited availability of work visas, or state supported monopolies. That said, companies that were on the hunt for scarce talent until very recently continued to offer ever more outrageous compensation: Free massages in the office, a concierge service to help you make restaurant reservations at the French Laundry, or a 10% pay rise for everyone. The “bon temps” were really “rouler-ing”!

But, pilots know that everything that goes up, must eventually come down, and in the case of the demand/supply ratio for software engineers, that landing was rough! Many people had started to believe that the good times would go on forever and, even more weirdly, had started to believe that the cushy job conditions that they had become accustomed to was some sort of natural state of affairs. I can potentially forgive you for these mistaken beliefs if you were born after the year 1994 or so, but everyone older than that really should have known better.

With the supply/demand ratio suddenly inverted, companies were freed from having to offer high compensation and cushy work conditions, so things quickly reverted to a more normal state of affairs, with downward pressure on compensation, downleveling on hire, layoffs, and “voluntary” exits.

Here is a fun side story about voluntary exits. Years ago, a large Dutch bank I was working for wanted to let go of some people but the unions strenuously objected. They struck a deal where 25% of people were categorized as A-class employees and the rest as B-class employees. The A-class employees got a letter saying: “You are in the A-class, we are happy with you, please don’t leave”. The B-class employees got a letter that said: “You are in the B-class, we are not going to fire you, but if you leave of your own accord, you’ll get a package”. The result: 100% of employees were unhappy! The A-class people were unhappy because they could easily get a job somewhere else and they wouldn’t mind a package. The B-class people were unhappy because, well, they were now officially labeled as B-class.

Local VPs had significant leeway in assigning people to the A or B class. Some VPs had a rule that if you were with the bank for less than one year, you were automatically in the B class. A friend of mine who was very, very, good at his job had been with the bank for 11 months and so was declared to be in the B-class. During meetings, when asked difficult questions, he would answer: “That seems more of a question for someone in the A-class 🙂”. Of course some A-class employees left because they were upset and some of the best of the B-class also left for other jobs, but with a package. The real stragglers in the B-class stayed of course because they knew they would never get a job anywhere else. So overall: Lots of employees were upset, some very good people left, and the people they most desperately wanted to get rid off, stayed.

I see lots of stories on LinkedIn where people are complaining about companies engaging in practices that decrease the comfort and remuneration of employees. Return to the office is a big hot button topic, but so are complaints about wage decreases, lay-offs, and other practices that are inspired by companies’ desire to lower their cost base in a market where there is ample supply.

I fear that most of these people have forgotten that the job market is, in fact, a market. During the holiday from history, companies did not offer high salaries and cushy benefits because they wanted to be nice, they did so because otherwise they would not be able to hire any staff! Of course you should not be afraid to ask a high price for your skills when they are in high demand, but you should also be ready to flex when market conditions change. You might think it dumb for companies to get all of their employees back into the office, but all things considered, that is not your call to make. If it is dumb, then the market will eventually punish these companies because they will not be able to get the best people at the best price. But if a free market means anything, it definitely also means that companies are free to offer the working conditions that they want to (bar their responsibility vis-a-vis the law of course); freedom also means the freedom to be wrong.

It is of course possible to change the law and to make some of the cushier conditions that we like mandatory. For instance in Holland you have a legal right to part-time work and in Sweden the parents have 480 days of parental leave (with a minimum of 90 days for the father, giving rise to the phenomenon of the Latte Dad). Everything is possible, really the only laws are the laws of physics. But you are not free from the consequences of any choices that are made. These countries have higher taxes and lower wages for the upper echelons of the job market. They have near-free education, cheaper healthcare and better retirement for most people, but less access to experimental treatments and drugs. Really everything is a market when you look at it from the right angle. If you think we should change the laws to change the deal between employees, employers, and the state, my advice is to become politically active. I know I am…

Anyway, back to employment conditions. If for instance working from home is super important to you, that is your good right and if it was important to me I would definitely find an employer who would offer that. But, the market being what it is, I might want to compromise on that in favor of other things. I look at my entire employment package holistically; there are things in there I like, things I don’t care about, and things that I do not like but which I can stomach because, all things considered, I want a good-sized paycheck and this is the best way to get one. Telling companies they are doing it all wrong is your good right, but I fear it is not very productive in a market where you also want to be hired by these companies. Remember: You are selling, the companies are buying, and as my dear old mother used to say: “Nobody has ever won an argument with a customer.”

We are sadly (for us) in a time where software engineers have less market power and companies are over eager to exploit that situation to the fullest, just like we were ready to exploit the situation to the fullest when the situation was reversed. But, rejoice, as long as the engines are still working, everything that comes down pretty much always goes back up again. Again, pilots know that when you are in an unaccelerated flight and push on the throttle, the plane will go up; that’s just physics.

That said, I am very happy I am not a new-grad these days, because that surely must suck major balls. But then again, when I graduated from college in 1988 we thought that there would be people who would never ever have a job and clearly most everyone did. It might be a bit of a rough ride for a while, but the good times will come back. They always do. In the meantime, make sure you have a competitive offering for any prospective employers!