Programming note: I recently launched a weekly podcast, Complex Systems with Patrick McKenzie. About 50% of the conversations cover Bits about Money's beat. The remainder will be on other interesting intersections of technology, incentives, culture, and organizational design. The first three episodes covered teaching trading, Byrne Hobart on the epistemology of financial firms, and the tech industry vs. tech reporting divide. Subscribe to it anywhere you listen to podcasts. If you enjoy it, writing a review (in your podcast app or to me via email) helps quite a bit.

On July 19th, a firm most people have sensibly never heard of knocked out a large portion of the routine operations at many institutions worldwide. This hit the banking sector particularly hard. It has been publicly reported that several of the largest U.S. banks were affected by the outage. I understand one of them to have idled tellers and bankers nationwide for the duration. (You’ll forgive me for not naming them, as it would cost me some points.) The issue affected institutions across the size spectrum, including large regionals and community banks.

You might sensibly ask why that happened and, for that matter, how it was possible it would happen.

You might be curious about how to quickly reconstitute the financial system from less legible sources of credit when it is down. (Which: probably less important as a takeaway, but it is quite colorful.)

Brief necessary technical context

Something like 20% of the readership of this column has an engineering degree. To you folks, I apologize in advance for the following handwaviness. (You may be better served by the Preliminary Post Incident Review.)

Many operating systems have a distinction between the “kernel” supplied by the operating system manufacturer and all other software running on the computer system. For historical reasons, that area where almost everything executes is called “userspace.”

In modern software design, programs running in userspace (i.e. almost all programs) are relatively limited in what they can do. Programs running in kernelspace, on the other hand, get direct access to the hardware under the operating system. Certain bugs in kernel programming are very, very bad news for everything running on the computer.

CrowdStrike Falcon is endpoint monitoring software. In brief, “endpoint monitoring” is a service sold to enterprises which have tens or hundreds of thousands of devices (“endpoints”). Those devices are illegible to the organization that owns them due to sheer scale; no single person nor group of people understand what is happening on them. This means there are highly variable levels of how-totally-effed those devices might be at exactly this moment in time. The pitch for endpoint monitoring is that it gives your teams the ability to make those systems legible again while also benefitting from economies of scale, with you getting a continuously updated feed of threats to scan for from your provider.

One way an endpoint might be effed is if it was physically stolen from your working-from-home employee earlier this week. Another way is if it has recently joined a botnet orchestrated from a geopolitical adversary of the United States after one of your junior programmers decided to install warez because the six figure annual salary was too little to fund their video game habit. (No, I am not reading your incident reports, I clarify for every security team in the industry.)

In theory, you perform ongoing monitoring of all of your computers. Then, your crack security team responds to alerts generated by your endpoint monitoring solution. This will sometimes merit further investigation and sometimes call for immediate remedial work. The conversations range from “Did you really just install cracked Starcraft 2 on your work PC? … Please don’t do that.” to “The novel virus reported this morning compromised 32 computers in the wealth management office. Containment was achieved by 2:05 PM ET, by which point we had null routed every packet coming out of that subnet then physically disconnected power to the router just to be sure. We have engaged incident response to see what if any data was exfiltrated in the 47 minutes between detection and null routing. At this point we have no indications of compromise outside that subnet but we cannot rule out a threat actor using the virus as a beachhead or advanced persistent threats being deployed.”

(Yes, that does sound like a Tom Clancy novel. No, that is not a parody.)

Falcon punched

Falcon shipped a configuration bug. In brief, this means that rather than writing new software (which, in modern development practice, hopefully goes through fairly extensive testing and release procedures), CrowdStrike sent a bit of data to systems with Falcon installed. That data was intended to simply update the set of conditions that Falcon scanned for. However, due to an error at CrowdStrike, it actually caused existing already-reviewed Falcon software to fail catastrophically.

Since that failure happened in kernelspace at a particularly vulnerable time, this resulted in Windows systems experiencing total failure beginning at boot. The user-visible symptom is sometimes called the Blue Screen of Death.

Configuration bugs are a disturbingly large portion of engineering decisions which cause outages. (Citation: let’s go with “general knowledge as an informed industry observer.” As always, while I’ve previously worked at Stripe, neither Stripe nor its security team necessarily endorses things I say in my personal spaces.)

However, because this configuration bug hit very widely distributed software running in kernelspace almost universally across machines used by the workforce of lynchpin institutions throughout society (most relevantly to this column, banks, but also airlines, etc etc), it had a blast radius much, much larger than typical configuration bugs.

Have I mentioned that IT security really likes military metaphors? “Blast radius” means “given a fault or failure in system X, how far afield from X will we see negative user impact.” I struggle to recall a bug with a broader direct blast radius than the Falcon misconfiguration.

Once the misconfiguration was rolled out, fixing it was complicated by the tiny issue that a lot of the people needed to fix it couldn’t access their work systems because their machine Blue Screen of Death’ed.

Why? Well, we put the vulnerable software on essentially all machines in a particular institution. You want to protect all the devices. That is the point of endpoint monitoring. It is literally someone’s job to figure out where the devices that aren’t endpoint monitored exist and then to bring them into compliance.

Why do we care about optimizing for endpoint monitoring coverage? Partly it is for genuinely good security reasons. But a major part of it is that small-c compliance is necessary for large-C Compliance. Your regulator will effectively demand that you do it.

Why did Falcon run in kernelspace rather than userspace?

Falcon runs in kernelspace versus userspace in part because the most straightforward way to poke its nose in other programs’ business is to simply ignore the security guarantees that operating systems give to programs running in userspace. Poking your nose in another program’s memory is generally considered somewhere between rude and forbidden-by-very-substantial-engineering-work. However, endpoint monitoring software considers that other software running on the device may be there at the direction of the adversary. It therefore considers that software’s comfort level with its intrusion to be a distant secondary consideration.

Another reason Falcon ran in kernelspace was, as Microsoft told the WSJ, Microsoft was forbidden by an understanding with the European Commission from firmly demoting other security software developers down to userspace. This was because Microsoft both a) wrote security software and b) necessarily always had the option of writing it in kernelspace, because Microsoft controls Windows. The European Commission has pushed back against this characterization and pointed out that This Sentence Uses Cookies To Enable Essential Essay Functionality.

Regulations which strongly suggest particular software purchases

It would be an overstatement to say that the United States federal government commanded U.S. financial institutions to install CrowdStrike Falcon and thereby embed a landmine into the kernels of all their employees’ computers. Anyone saying that has no idea how banking regulation works.

Life is much more subtle than that.

The United States has many, many different banking regulators. Those regulators have some desires for their banks which rhyme heavily, and so they have banded into a club to share resources. This lets them spend their limited brainsweat budgets on things banking regulators have more individualized opinions on than simple, common banking regulatory infrastructure.

One such club is the Federal Financial Institutions Examination Council. They wrote the greatest crossover event of all time if your interests are a) mandatory supervisory evaluations of financial institutions and b) IT risk management: the FFIEC Information Technology Examination Handbook's Information Security Booklet.

The modal consumer of this document is probably not a Linux kernel programmer with a highly developed mental model of kernelspace versus userspace. That would be an unreasonable expectation for a banking supervisor. They work for a banking regulator, not a software company, doing important supervisory work, not merely implementation. Later this week they might be working on capital adequacy ratios, but for right now, they’re asking your IT team about endpoint monitoring.

The FFEITC ITEH ISB (the acronym just rolls off the tongue) is not super prescriptive about exactly what controls you, a financial institution, have to have. This is common in many regulatory environments. HIPAA, to use a contrasting example, is unusual in that it describes a control environment that you can reduce to a checklist with Required or Optional next to each of them. (HIPAA spells that second category “Addressable”, for reasons outside the scope of this essay, but which I’ll mention because I don’t want to offend other former HIPAA Compliance Officers.)

To facilitate your institution’s conversation with the examiner who drew the short straw, you will conduct a risk analysis. Well, more likely, you’ll pay a consulting firm to conduct a risk analysis. In the production function that is scaled consultancies, this means that a junior employee will open U.S. Financial Institution IT Security Risk Analysis v3-edited-final-final.docx and add important client-specific context like a) their name and b) their logo.

That document will heavily reference the ITEH, because it exists to quickly shut down the line of questioning from the examiner. If you desire a career in this field, you will phrase that as “guiding the conversation towards areas of maximum mutual interest in the cause of 'advanc[ing] the nation’s monetary, financial, and payment systems to build a stronger economy for all Americans.'” (The internal quotation is lifted from a job description at the Federal Reserve.)

Your consultants are going to, when they conduct the mandatory risk analysis, give you a shopping list. Endpoint monitoring is one item on that shopping list. Why? Ask your consultant and they’ll bill you for the answer, but you can get my opinion for free and it is worth twice what you paid for it: II.C.12 Malware Mitigation.

Does the FFEITC have a hugely prescriptive view of what you should be doing for malware monitoring? Well, no:

Management should implement defense-in-depth to protect, detect, and respond to malware. The institution can use many tools to block malware before it enters the environment and to detect it and respond if it is not blocked. Methods or systems that management should consider include the following: [12 bullet points which vary in specificity from whitelisting allowed programs to port monitoring to user education].

But your consultants will tell you that you want a very responsive answer to II.C.12 in this report and that, since you probably do not have Google’s ability to fill floors of people doing industry-leading security research, you should just buy something which says Yeah We Do That.

CrowdStrike’s sales reps will happily tell you Yeah We Do That. This web page exists as a result of a deterministic process co-owned by the Marketing and Sales departments at a B2B software company to create industry-specific “sales enablement” collateral. As a matter of fact, if you want to give CrowdStrike your email address and job title, they will even send you a document which is not titled Exact Wording To Put In Your Risk Assessment Including Which Five Objectives And Seventeen Controls Purchasing This Product Will Solve For.

CrowdStrike is not, strictly speaking, the only vendor that you could have installed on every computer you owned to make your regulators happy with you. But, due to vagaries of how enterprise software sales teams work, they sewed up an awful lot of government-adjacent industries. This was in part because they aggressively pursued writing the sort of documents you need if the people who read your project plans have national security briefs.

I’m not mocking the Federal Financial Institutions Examining Council for cosplaying as having a national security brief. (Goodness knows that that happens a lot in cybersecurity... and government generally. New York City likes to pretend it has an intelligence service, which is absolutely not a patronage program designed to have taxpayers fund indefinite foreign vacations with minimal actual job duties.)

But money is core societal infrastructure, like the power grid and transportation systems are. It would be really bad if hackers working for a foreign government could just turn off money. That would be more damaging than a conventional missile being fired at random into New York City, and we might be more constrained in responding.

And so, we ended up in a situation where we invited an advanced persistent threat into kernelspace.

It is perhaps important to point out that security professionals understand security tools to themselves introduce security vulnerabilities. Partly, the worry is that a monoculture could have a particular weakness that could be exploited in a particular way. Partly, it is that security tools (and security personnel!) frequently have more privileges than is typical, and therefore they can be directly compromised by the adversary. This observation is fractal in systems engineering: at every level of abstraction, if your control plane gets compromised, you lose. (Control plane has a specific meaning in networking but for this purpose just round it to “operating system (metaphorical) that controls your operating systems (literal).”)

CrowdStrike maintains that they do not understand it to be the case that a bad actor intentionally tried to bring down global financial infrastructure and airlines by using them as a weapon. No, CrowdStrike did that themselves, on accident, of their own volition. But this demonstrates the problem pretty clearly: if a junior employee tripping over a power cord at your company brings down computers worldwide, the bad guys have a variety of options for achieving directionally similar aims by attacking directionally similar power cords.

When money stops money-ing

I found out about the CrowdStrike vulnerability in the usual fashion: Twitter. But then my friendly local bank branch cited it (as quote the Microsoft systems issue endquote) when I was attempting to withdraw cash from the teller window.

My family purchased a duplex recently and is doing renovation prior to moving in. For complex social reasons, a thorough recitation of which would make me persona non grata across the political spectrum, engaging a sufficient number of contractors in Chicago will result in one being asked to make frequent, sizable payments in cash.

This created a minor emergency for me, because it was an other-than-minor emergency for some contractors I was working with.

Many contractors are small businesses. Many small businesses are very thinly capitalized. Many employees of small businesses are extremely dependent on receiving compensation exactly on payday and not after it. And so, while many people in Chicago were basically unaffected on that Friday because their money kept working (on mobile apps, via Venmo/Cash App, via credit cards, etc), cash-dependent people got an enormous wrench thrown into their plans.

I personally tried withdrawing cash at three financial institutions in different weight classes, as was told it was absolutely impossible (in size) at all of them, owing to the Falcon issue.

At one, I was told that I couldn’t use the tellers but could use the ATM. Unfortunately, like many customers, I was attempting to take out more cash from the ATM than I ever had before. Fortunately, their system that flags potentially fraudulent behavior will let a customer unflag themselves by responding to an instant communication from the bank. Unfortunately, the subdomain that communication directs them to runs on a server apparently protected by CrowdStrike Falcon.

It was not impossible at all financial institutions. I am aware of a few around Chicago which ran out of physical cash on hand at some branches, because all demand for cash on a Friday was serviced by them versus by “all of the financial institutions.” (As always happens during widespread disturbances in infrastructure, there quickly arises a shadow economy of information trading which redirects relatively sophisticated people to the places that are capable of servicing them. This happens through offline social networks since time immemorial and online social networks since we invented those. The first is probably more impactful but the second is more legible, so banking regulators pretend this class of issues sprang fully formed from the tech industry just in time to bring down banks last year.)

I have some knowledge of the history of comprehensive failures of financial infrastructure, and so I considered doing the traditional thing when convertibility of deposits is suspended by industry-wide issues: head to the bar.

A hopefully unnecessary disclaimer: the following is historical fact despite rhyming with stereotype.

Back in 1970, there was a widespread and sustained (six months!) strike in the Irish banking sector. Workers were unable to cash paychecks because tellers refused to work. So, as an accommodation for customers, operators of pubs would cash the checks from the till, trusting that eventually checks drawn on the accounts of local employers would be good funds again. 

Some publicans even cashed personal checks, backed by the swift and terrible justice of the credit reporting bureau We Control Whether You Can Ever Enjoy A Pint With Your Friends Again. This kept physical notes circulating in the economy.

As I told my contractors, to their confusion, I was unable to simply go down to the local bar to get them cash with the banks down. I don’t have sufficient credit with the operator of the local bar, as I don’t drink.

I told them, to their even greater confusion, that I had considered going down to the parish and buying all their cash on hand with a personal check. Churches, much like bars, have much of their weekly income come through electronic payments but still do a substantial amount of cash management through the workweek heading into the weekend. I’m much more a known quantity at church than I am at the friendly neighborhood watering hole. (Also, when attempting to workaround financial infrastructure bugs to get workers their wages, consider relying on counterparties with common knowledge of James 5:4.)

I eventually resolved the issue in a more boring fashion: I texted someone I reasonably assumed to have cash and asked them to bring it over.

Financial infrastructure normally functions to abstract away personal ties and replace favor-swapping with legibly-priced broadly-offered services.

Thankfully, while this outage was surprisingly deep and broad, banks were mostly back to normal on the following Monday.

...will never happen more than once at a company.

I say this knowing that chunks of Germany's civil infrastructure managed to standardize on SuSE desktops, and some may still be using SuSE. Some might view this as proof it can be done, I say that Linux desktops not spreading beyond this example is proof of why it didn't happen. The biggest reason we have the German example is because the decision was top down. Government decision making is different than corporate decision making, which is why we're not going to see the same thing happen, a Linux desktop (actually laptop) mandate from on high, more than few times; especially in the tech industry.

it all comes down to management and why Linux laptop users are using Linux in the first place.

You see, corporate laptops (hereafter referred to as "endpoints" to match management lingo) have certain constraints placed upon them when small companies become big companies:

• You need some form of anti-virus and anti-malware scanning, by policy
• You need something like either a VPN or other Zero Trust ability to do "device attestation", proving the device (endpoint) is authentic and not a hacker using stolen credentials from a person
• You need to comply with the vulnerability management process, which means some ability to scan software versions on and endpoint and report up to a dashboard.
• The previous three points strongly imply an ability to push software to endpoints

Windows has been able to do all four points since the 1990s. Apple came somewhat later, but this is what JAMF is for.

Then there is Linux. It is technically possible to do all of the above. Some tools, like osquery, were built for Linux first because the intended use was on servers. However, there is a big problem with Linux users. Get 10 Linux users in a room, and you're quite likely to get 10 different combination of display manager (xorg or wayland), window manager (gnome, kde, i3, others), and OS package manager. You need to either support heterogeneity or commit to building the Enterprise Linux that has one from each category and forbid others. Enterprise Linux is what the German example did.

Which is when the Linux users revolt, because banning their tiling window manager in favor of Xorg/Gnome ruins their flow -- and similar complaints. The Windows and Apple users forced onto Linux will grumble about their flow changing and why all their favorite apps can't be used, but at least it'll be uniform. If you support all three, you'll get the same 5% Linux users but the self-selected cranky ones who can't use the Linux they actually want. Most of that 5% will "settle" for another Linux before using Windows or Apple, but it's not the same.

And 5% Linux users puts supportability of the platform below the concentration needed to support that platform well. Companies like Alphabet are big enough the 5%  is big enough to make a supportable population. For smaller companies like Atlassian, perhaps not. Which puts Enterprise Linux in that twilight state between outright banned and just barely supported so long as you can tolerate all the jank.

The U.S. Department of Justice (DOJ) today said they arrested the alleged operator of 911 S5, a ten-year-old online anonymity service that was powered by what the director of the FBI called “likely the world’s largest botnet ever.” The arrest coincided with the seizure of the 911 S5 website and supporting infrastructure, which the government says turned computers running various “free VPN” products into Internet traffic relays that facilitated billions of dollars in online fraud and cybercrime.

The Cloud Router homepage, which was seized by the FBI this past weekend. Cloud Router was previously called 911 S5.

On May 24, authorities in Singapore arrested the alleged creator and operator of 911 S5, a 35-year-old Chinese national named YunHe Wang. In a statement on his arrest today, the DOJ said 911 S5 enabled cybercriminals to bypass financial fraud detection systems and steal billions of dollars from financial institutions, credit card issuers, and federal lending programs.

For example, the government estimates that 560,000 fraudulent unemployment insurance claims originated from compromised Internet addresses, resulting in a confirmed fraudulent loss exceeding $5.9 billion.

“Additionally, in evaluating suspected fraud loss to the Economic Injury Disaster Loan (EIDL) program, the United States estimates that more than 47,000 EIDL applications originated from IP addresses compromised by 911 S5,” the DOJ wrote. “Millions of dollars more were similarly identified by financial institutions in the United States as loss originating from IP addresses compromised by 911 S5.”

From 2015 to July 2022, 911 S5 sold access to hundreds of thousands of Microsoft Windows computers daily, as “proxies” that allowed customers to route their Internet traffic through PCs in virtually any country or city around the globe — but predominantly in the United States.

911 S5 built its proxy network mainly by offering “free” virtual private networking (VPN) services. 911’s VPN performed largely as advertised for the user — allowing them to surf the web anonymously — but it also quietly turned the user’s computer into a traffic relay for paying 911 S5 customers.

911 S5’s reliability and extremely low prices quickly made it one of the most popular services among denizens of the cybercrime underground, and the service became almost shorthand for connecting to that “last mile” of cybercrime. Namely, the ability to route one’s malicious traffic through a computer that is geographically close to the consumer whose stolen credit card is about to be used, or whose bank account is about to be emptied.

The prices page for 911 S5, circa July 2022. $28 would let users cycle through 150 proxies on this popular service.

KrebsOnSecurity first identified Mr. Wang as the proprietor of the popular service in a deep dive on 911 S5 published in July 2022. That story showed that 911 S5 had a history of paying people to install its software by secretly bundling it with other software — including fake security updates for common programs like Flash Player, and “cracked” or pirated commercial software distributed on file-sharing networks.

Ten days later, 911 S5 closed up shop, claiming it had been hacked. But experts soon tracked the reemergence of the proxy network by another name: Cloud Router.

The announcement of Wang’s arrest came less than 24 hours after the U.S. Department of the Treasury sanctioned Wang and two associates, as well as several companies the men allegedly used to launder the nearly $100 million in proceeds from 911 S5 and Cloud Router customers.

Cloud Router’s homepage now features a notice saying the domain has been seized by the U.S. government. In addition, the DOJ says it worked with authorities in Singapore, Thailand and Germany to search residences tied to the defendant, and seized approximately $30 million in assets.

The Cloud Router homepage now features a seizure notice from the FBI in multiple languages.

Those assets included a 2022 Ferrari F8 Spider S-A, a BMW i8, a BMW X7 M50d, a Rolls Royce, more than a dozen domestic and international bank accounts, over two dozen cryptocurrency wallets, several luxury wristwatches, and 21 residential or investment properties.

The government says Wang is charged with conspiracy to commit computer fraud, substantive computer fraud, conspiracy to commit wire fraud, and conspiracy to commit money laundering. If convicted on all counts, he faces a maximum penalty of 65 years in prison.

Brett Leatherman, deputy assistant director of the FBI’s Cyber Division, said the DOJ is working with the Singaporean government on extraditing Wang to face charges in the United States.

Leatherman encouraged Internet users to visit a new FBI webpage that can help people determine whether their computers may be part of the 911 S5 botnet, which the government says spanned more than 19 million individual computers in at least 190 countries.

Leatherman said 911 S5 and Cloud Router used several “free VPN” brands to lure consumers into installing the proxy service, including MaskVPN, DewVPN, PaladinVPN, Proxygate, Shield VPN, and ShineVPN.

“American citizens who didn’t know that their IP space was being utilized to attack US businesses or defraud the U.S. government, they were unaware,” Leatherman said. “But these kind of operations breed that awareness.”

I just got an email from a store I bought something at about 4 months ago. It happens, they have my mail, somewhere in their fine print they said that they were gonna send me product recommendations or I clicked a thing because it looked like the box you have to click to get the thing.

I’m not even mad about it, it does sometimes make sense. The store where we order cat food sends out coupons and I can save a bit on chewy treats for our libertarian overlords here, for example. The company that just send me the email about new products? The sell automatic standing desks. How many of those do I buy every quarter? About zero now that I have one – I have only so many rooms to fill and so many bodies to place at tables.

Now of course they could think that I have this huge company with staff who all need standing desks (get one, they’re great, I wished the office I work in had them) but how big are the chances really?

People know the scary stories of “surveillance capitalism” and how Facebook and other platforms knows us better than we do. How they can predict our needs with sophisticated algorithms, shape our world and behavior. Because we are just these simplistic animals that if they see something they need it and buy it.

The standing desk company isn’t the most sophisticated in the world when it comes to data analytics and algorithms I guess. They probably just use some ready-made ecommerce solution that sends out emails. But there are more sophisticated players in that space, with limitless resources and the smartest programmers and statisticians on staff. There is Amazon.

You might have heard for example that Amazon ships products already to fulfillment centers near hotspots before they have been ordered to ship faster and that they then use their algorithmically sorted page to push the things they already transported to your neightborhood to you. That they track and analyze every click you make to try to get you to buy things, especially things you did look at.

These things then follow you around and keep being presented to you. Buy this, you did check it out, right? You want this!

I do buy stuff on Amazon at times (I try to avoid it but sometimes it’s hard) so I have products following me around. I remember a few years ago I was looking to buy a TV, I browsed some and all my recommendations were TVs and then I bought the one I wanted.

But the recommendations didn’t go away. Amazon kept showing me more TVs. Now I needed exactly one TV. I don’t need more of those things in this home. But these things kept following me around in spite of me buying a TV an Amazon based on my comparison.

Digital systems have a tendency to flatten everything. Because flat is easy to implement. We see that more than anywhere with modern “AI” systems but other systems also have the same tendency. What do I mean by that? Flatten?

Amazon for example doesn’t care why you look at an object. Maybe you want to buy something, maybe you just wanted to look up the technical specs or someone sent you a link to a funny review. All the same to Amazon. The complexity of the world flatted to a little “but you looked at this” flag. Same for buying stuff. Amazon builds your profile to suggest things that might be relevant to you but then just adds everything you buy (you can modify it but it’s an annoying process) to your profile. Even the thing you bought for your niece as a birthday gift. Or your dad’s last Christmas present. All these things are being flattened. Stripped of context and nuance and messiness. Perfectly flat and neat. A model build not to understand you but to enable an algorithm to work efficiently. You are just a data provider.

I think this shows us another crack in tech’s narrative of being so uber-powerful and data being the perfect source of truth and future forecasting (another “AI” related narrative here, “AI” is really just the continuation of the tech development of the last 10-15 years, just a bit more wasteful). Because while the algorithms might be smart and efficient and might scale. Might build profiles of people in nanoseconds while comparing products to millions of people in milliseconds all that is built on a flattend world view.

I don’t have a grand point to make here. Just a random observation while deleting mails from my inbox: A lot of discourse might be lead by people thinking themselves to be the smartest people in the room with access to the only real magic there is, data. But when we look at what all that data analytics does it quickly just looks very mediocre. Like something that goes through some motion without understanding context and the world. Like something mimicking something real but without any understanding for it and therefore failing. Kinda like modern “AI” systems.

James Bond films have a certain formula to them. It is more interesting when seen from the perspective of the villain.

He has long been adjacent to money and power, but craves more. Several years ago, he successfully escaped his low-on-the-ladder job at an existing institution. He built a base of power that is independent of institutions. From it, he successfully puppets any organization he needs to. He and his organization are from elsewhere, everywhere, all at once. They have no passport and fly no flags; these concepts are thoroughly beneath them. They move around frequently and are always where the plot requires them to be, exactly when it requires them to be there. No law constrains them. Governments scarcely exist in their universe. To the limited extent they come to any government’s attention, no effective action is taken. The villain rises to the heights of influence and power.

This continues for years.

Then we suddenly hear E minor major 9. We begin the film, telling the end of the story, mostly from Bond’s perspective. The villain is just another weirdo who dies at the climax in act three.

Life imitates art

For years I have used the phrase “Bond villain compliance strategy” to describe a common practice in the cryptocurrency industry. 

In it, your operation is carefully based Far Away From Here. You are, critically, not like standard offshore finance, with a particular address in a particular country which just happens to be on the high-risk jurisdiction list. You are nowhere because you want to be everywhere. You tell any lie required to any party—government, bank, whatever—to get access to the banking rails and desirable counterparties located in rich countries with functioning governments. You abandon or evolve the lie a few years later after finally being caught in it.

Your users and counterparties understand it to be a lie the entire time, of course. You bragged about it on your site and explained it to adoring fans at conferences. You created guides to have your CS staff instruct users on how to use a VPN to evade your geofencing. The more clueful among your counterparties, who have competent lawyers and aspirations to continue making money in desirable jurisdictions, will come to describe your behavior as an “open secret” in the industry. They will claim you’ve turned over a new leaf given that the most current version of the lie only merely rhymes with the previous version of the lie.

And then we begin the third act.

So anyhow, Binance and its CEO Changpeng Zhao (known nearly universally as CZ) have recently pleaded guilty to operating the world’s largest criminal conspiracy to launder money, paying more than $4 billion in fines. This settles a long-running investigation involving the DOJ, CFTC, FinCEN, and assorted other parts of the U.S. regulatory state. Importantly, it does not resolve the SEC’s parallel action.

How’d we get here?

A brief history of Binance

Binance is, for the moment, the world’s largest crypto exchange. Its scale is gobsmacking and places it approximately the 100th largest financial institution in the world by revenue. The primary way it makes money is exacting a rake on cryptocurrency gambling, in particular, leveraged bets using cryptocurrency futures. To maintain its ability to do this, it runs a worldwide money laundering operation with the ongoing, knowing, active participation of many other players in the crypto industry, including Bitfinex/Tether, the Justin Sun empire, and (until recent changes in management) FTX/Alameda.

In his twenties CZ worked in Japan (waves) and New York for contractors to the Tokyo Stock Exchange then Bloomberg. In about 2013 he got interested in crypto and then joined a few projects, including becoming CTO of OKCoin, another Bond villain exchange. Being a henchman is an odd job, so he decided to promote himself to full-fledged villain. In 2017 he did an unregistered securities offering (then commonly spelled “ICO”) for Binance. 

Binance rose meteorically from then until recently, essentially gaining share at the expense of waning Bond villains. To oversimplify greatly, it carved up the less-regulated side of the crypto market with FTX, with Binance mostly taking customers in geopolitical adversaries of the U.S. (most notably greater China) and FTX mostly taking them in geopolitical allies (most notably, South Korea, Singapore, and the U.S.). But the cartels did not partition the globe in a way which fully insulated them from each other.

These operations were intertwined and coordinated. How intertwined? Binance was a part-owner of FTX until SBF decided successfully capturing U.S. regulators was a lot more likely if his cap table named more Californian trees and fewer Bond villains. How coordinated? The name of the Signal chat was Exchange Coordination.

This eventually led to grief as CZ (mostly accurately) perceived SBF was using the U.S. government as a weapon against Binance. He retaliated by strategic leaking, leading to a collapse in the value of FTX's exchange token, a run on the bank, and FTX's bankruptcy.

Where was Binance in all of this?

Binance did a heck of a lot of business in Japan in the early years. This officially ended in March 2018 when the Financial Services Agency, Japan’s major financial regulator, made it extremely clear that Binance was operating unlawfully in serving Japanese customers without registering in the then-relatively-new framework for virtual currency exchange businesses.

As an only-sometimes-following-crypto skeptic, this was the thing which brought Binance to my attention. Binance was piqued, saying that they had engaged the FSA in respectful conversations and then learned they were being kicked out of the country from a news report. Having spent roughly twenty years getting good at understanding how Japanese bureaucratic procedures typically work, I surmised  “...that is a very plausible outcome if you start your getting-to-know-you chat with ‘Basically I am a James Bond villain.’” I think that was the first time the metaphor came to me.

The order expelling them listed their place of business as Hong Kong, with a dryly worded asterisk stating that this was taken from their statements on the Internet and “...there exists the possibility that [this information] is not accurate as of the present moment in time.”

That, Internets, is how a salaryman phrases “I am absolutely aware that you maintain a team and infrastructure in Japan.”

Did Binance exit Japan? Well, that depends. Did CZ personally return to Japan? Probably not. Does Binance continue to serve Japanese customers? Yes, though (Bond villain!) it pretends not to. Where does Binance’s exchange run as a software artifact? As a statement of engineering fact: in an AWS data center in Tokyo. ap-northeast-1, if you want to get technical.

 (Someone needs to write an East Asian studies paper on how Tokyo became Switzerland for Asian crypto enthusiasts due to a combination of governance, network connectivity, latency, and geopolitical risk. I nominate anyone other than me.)

Binance also maintained an office in Shanghai, with many executives working there. It was raided by the Chinese police. Binance denied that the office existed. The spokesman’s quote was pure Bond villain: “The Binance team is a global movement consisting of people working in a decentralized manner wherever they are in the world. Binance has no fixed offices in Shanghai or China, so it makes no sense that police raided on any offices and shut them down.”

This was a lie wrapped around a tiny truth. Internet-distributed workforces containing many mobile professionals do not exactly resemble a single building with all your staff and your nameplate on the door.

Of course, on the actual substantive matter, it was a lie.

We know it was a lie, because (among many other reasons) we have the chat logs where the parts of their criminal conspiracy that operated in the U.S. complain that the parts of their criminal conspiracy that operated in Shanghai kept information from them that they needed to do their part to keep the crime operating smoothly. Coworkers, man.

Binance’s Chief Compliance Officer, one Samuel Lim, apparently is not a fan of The Wire and never encountered Stringer Bell’s dictum on the wisdom of keeping notes on a criminal conspiracy. He writes great copy, most memorably “[We are] operating as a fking unlicensed security exchange in the U.S. bro.” He and many other Binance employees have helpfully documented for posterity that their financial operations teams were, for most of corporate history, working from Shanghai.

Binance also operated in the state of Heisenbergian uncertainty, sometimes known as Malta. Malta has a substantial financial services industry, which welcomed Binance with open arms in 2018 and then pretended not to know him in 2020. This continues Malta’s proud tradition of strategic ambiguity as to whether it is an EU country or rentable skin suit for money launderers. ¿Por qué no los dos? Despite this, Binance would continue claiming to customers and other regulators for a while that it was fully authorized to do business by Malta.

Binance operates in Russia, to enable its twin businesses of cryptocurrency speculation and facilitating money laundering. In 2023 it pretended to sell its Russian operations.

Binance operated in many jurisdictions. The U.K.: kicked out. France: under investigation. Germany, the Netherlands, etc, etc, they required non-teams of non-employees at non-headquarters to keep track of all the places they weren’t registered doing their non-crimes.

A core cadre of the Binance executive team is currently in the United Arab Emirates, where CZ hopes to return. He professes that he will await sentencing there, and pinkie swears that he will totally get back on a plane to the U.S. to show up to it. For reasons which are understandable by anyone with more IQ than a plate of jello, the U.S. is skeptical he will make good on this promise, and is currently, as of Thanksgiving 2023, attempting to keep him in the U.S. He is physically present to sign what Binance advocates believe is the grand compromise to put all his legal worries behind them.

A defining characteristic of Bond villains is that they think they are very smart and everyone else is very stupid. To be fair, when you play back the movie of the last few years of their life, they keep winning and their adversaries look like nincompoops.

Then, they get extremely confident and begin to make poor life choices.

How did this work for so freaking long?

Much like the optimal amount of fraud is not zero, the global financial system institutionally tolerates (and actively enables) some shenanigans at the margin. You can think of Binance, Tether, FTX, and all the rest as talented amateurs capable of engaging the services of professionals. They followed advice and grew like a slime mold into the places where shenanigans are wink-and-a-nudge tolerated.

Why tolerate shenanigans? Some shenanigans are necessary to keep the world spinning.

China has grown into an economic superpower via capitalism while also at times officially having private property ownership be illegal. That circle cannot be squared. We, the global we, want Chinese people to not live in grinding poverty. That requires economic growth. Economic growth required making things the world wanted. Selling those things required integration into the global economic order. That required a willingness to ignore things the Communist dictatorship said were crimes, while simultaneously saying “Oh, bankers definitely, definitely shouldn’t facilitate billionaires committing crimes.”

As I’ve remarked previously, we similarly have complicated preferences with regards to Russian oligarchs. In some years, money laundering for them is, how might a gifted speaker phrase this, “[b]ringing our former adversaries, Russia and China, into the international system as open, prosperous and stable nations.” In other years, money laundering for them is described as funding Russia’s war machine.

Finance is messy because the world is messy.

Some of the shenanigans aren’t strictly necessary or planned, but society considers an expenditure of effort required to curtail them to be wasteful or to compromise our other goals. We had all the technology required to CC regulators on every banking transaction years before slow database enthusiasts decided all transactions would eventually be publicly readable and persisted forever. We simply chose not to implement it. It would have been quite expensive and infringed on the privacy of many ordinary people and firms.

But Binance, and others, forgot the critical step, to the annoyance of their engaged professionals: you have to eventually stop growing and keep to a low profile. You have to simply be content with being fantastically rich. If you do, you can continue showing up to the nicest parties in New York, owning expensive real estate in London, and commuting to a comfortable office in Hong Kong or the Bahamas or many other places.

But crypto kept growing until the control systems could not ignore them any longer. And the control systems cannot continue to avoid knowledge of the crimes.

So, so many crimes. Many of them are what crypto advocates consider as utterly inconsequential, like serially lying on paperwork. And also Binance gleefully and knowingly banked terrorists and child pornographers. That’s not an allegation; that has been confessed to. There is no line a Bond villain will not cross. They will cross them performatively.

And, surprising even me, some crypto characters consciously adopt the aesthetic of Bond villains. Le Chiffre, the villain in Casino Royale, owns a fictional house. That house exists in the physical world, where a location scout said “This certainly looks like the sort of place a Bond villain would live.” Jean Chalopin owned that literal, physical house. (c.f. Zeke Faux’s Number go Up, Kindle location 1175.) As previously discussed, Chalopin is a professional bagman, and his largest client was previously Tether.

What happens to Binance now?

Some believe that Binance admitting to being a criminal conspiracy is actually good news, not merely in the memetic “good news for Bitcoin” sense, but because this upper-bounds Binance’s exposure somewhere below “The United States forcibly dismantles the most important crypto exchange and much of the infrastructure it touches.”

The immediate consequences are about $4 billion in fines. Despite being one of the world’s largest hodlers, the U.S. will not accept payment in Bitcoin, and Binance has agreed to pay in installments over the next two years. CZ and Binance will be sentenced in February.

Some people think the grand bargain was to avoid him getting imprisoned. The actual text of the agreement says that Binance gets to walk away from some parts of it if he is sentenced to more than 18 months. (Senior officials told the NYT they are contemplating asking for more than that.)

Probably more consequentially, the settlements are going to force Binance to install so-called monitors internally. Those monitors are effectively external compliance consultants, working at the expense of Binance in a contractual relationship with them, but whose true customer is the United States. The monitors have pages upon pages of instructions as to exactly how they are to reform Binance’s culture by implementing recommendations to bring onboarding, KYC, and AML processes into compliance with the law everywhere Binance does business, and sure, that is part of the job.

But the other part of the job is that they’re an internal gateway to any information Binance has ever had, or will ever have. This can be queried essentially at will by law enforcement, with Binance waiving substantially all rights to not cooperate.

You might reasonably ask “Hey, doesn’t the U.S. typically require a warrant to go nosing about in the business of people who haven’t been accused of a crime?” And, to oversimplify half a century of jurisprudence, one loses one’s presumption of privacy if one brings a business into one’s private affairs. All of Binance’s customers and counterparties gave up their privacy to Binance by transacting with it. The U.S. has Binance’s permission to examine all of Binance’s historical, current, and future records, at will, for at least the next three years. It also secured a promise that Binance would assist in any investigation.

And so, if one were hypothetically not yet indicted by the U.S., but one had hypothetically done business with one’s now-confessed money launderer, one’s own Fourth Amendment protections do not protect the U.S. from hoovering up every conversation and transaction with Binance.

All of this is certainly good news and we can put this messy chapter behind us, say crypto advocates.

How are Bond villains actually regulated?

Was the Bond villain strategy ever going to work? Did Binance have a reasonable likelihood of prevailing on jurisdictional arguments, like telling the U.S. that the Binance mothership had no U.S. presence and so it should not be subject to U.S. law? No. Crikey, no. The system has to be robust to people lying or acting from less-salubrious jurisdictions, at least to the extent it cares about being effective, and at least some of the time it does actually care about being effective.

The U.S.’s point of view on the matter, elucidated at length in any indictment for financial crimes, is that if you have ever touched an electronic dollar, that dollar passed through New York, and therefore you’ve consented to the jurisdiction of the United States. Dollarization is very intentionally wielded like a club to accomplish the U.S.’s goals.

There exist some not-very-sympathetic people one could point to who ran afoul of this over the years who are still much more sympathetic than Binance. Binance intentionally used the U.S. market and infrastructure to make money. The U.S. was essential to their enterprise. Many peer nations can, and will, make a similar argument.

Binance had tens of percent of their book of business in the U.S. They were absolutely aware of this, knew that some of those users were their largest VIPs or otherwise important, and took steps to maximize for U.S. usage while denying they served Americans.

Their engineers didn’t accidentally copy the exchange onto AWS or deploy it to Tokyo by misclicking repeatedly. The crypto industry playbook for doing sales and marketing looks like everyone else’s playbook for doing sales and marketing. They get on planes, present at events, send mail, hire employees (or otherwise compensate agents), open offices, etc etc.

If having an email address meant you didn’t exist in physical reality anymore, the world would be almost empty.

CZ personally signed for bank accounts for some of his money laundering subsidiaries at U.S. banks, like Merit Peak and Sigma Chain. The SEC traced more than $500 million through one of those accounts.

One major rationale for KYC legislation, as discussed previously, is that it makes prosecuting Bond villains easier. Even if compliance departments at banks are utterly incompetent at detecting Bond villains at signup, having extracted the Bond villain’s signature on account opening documents is very useful to prosecutors a few years down the road. Why have to do hard work quantifying exactly how many engineers work on which days at Binance’s offices in San Francisco when you can do the easy thing and say “Hey, fax me the single piece of paper where the Bond villain signed up for responsibility for all the crimes, please.”

Why do Bond villains sign for bank accounts in highly regulated jurisdictions? Partly it is because of beneficial ownership KYC requirements to open bank accounts. Partly it is because finding loyal, trustworthy subordinates is very hard if you’re a Bond villain, and Bond villains (sensibly!) worry that if the only name on the paperwork is a henchman, eventually that henchman might say “You know, actually, I would like to withdraw the $500 million I have on deposit with you.”

(This is why Bond villains frequently have e.g. the mother of their children sign for bank accounts. Bond villains, again, think everyone else is stupid, and that no one will cotton onto this.)

A subgenre of challenges in people management for Bond villains: you have to hire experienced executives in the United States to run the U.S. fig leaf for your global criminal empire. The people you hire will, by nature, be experienced financial industry veterans who are extremely sophisticated and have access to good lawyers. This combination of attributes is the recipe for being the best in the world at filing whistleblower claims. I expect a few previously executives at Binance U.S. are eventually going to take home the most generous pay packages in the entire financial industry for a few years between 2018 and 2022.

To make this palatable to the American public, those whistleblower rewards are not courtesy of the taxpayer; they’re courtesy of money seized from previous Bond villains. A portion of Binance’s settlement(s) will go to pay the whistleblowers at the next Bond villain. It’s a circle of life.

News that will break shortly

Different regulators have differing ability to prosecute complex cases, but they basically all have the ability to read simple legal documents. That is one of the things they are best at doing.

Binance will suffer a wave of tag-along enforcement actions, in the U.S. and globally. Partly this will be for face saving; global peers of the U.S., which Binance has transacted billions of dollars in, will largely not want to signal “Oh we’re totes OK with money laundering for terrorists and child pornographers”, and so they’re going to essentially copy/paste the U.S. enforcement actions. They will then play pick-a-number with Binance’s new management team, who will immediately cave.

The earliest version of this is probably only weeks away, but Binance will deal with it for years.

More interestingly, and likely more expensively, the SEC is going to hit Binance like a ton of bricks. They were one of the few regulators which opted out of consolidating with the DOJ’s deal. They think they have Binance dead to rights (they do), and tactically speaking, the deal makes their life even easier. Binance has waived ability to contest some things the SEC will argue. The SEC can now proxy requests for evidence to Binance’s monitors through other federal agencies.

Binance has had the enthusiastic cooperation of many people who walk in light in addition to their co-conspirators who walk in shadow. Those people, lamentably inclusive of some in the tech industry who I feel a great deal of fellow-feeling for, are going to start cutting off access to Binance. Compliance departments at their corporate overlords, which were either entirely in the dark or willing to be persuaded that a new innovative industry required some amount of flexibility with regards to controls, are (today) having strongly worded conversations which direct people to lose Binance’s number.

Binance has pre-committed to helping with the efforts to cauterize them from the financial system. They also pre-committed to assisting in, specifically, the investigation of their sale of the Russia business. That investigation will conclude that the sale was a sham (a Bond villain lied?) designed to avoid sanctions enforcement. Ask your friends in national security Washington how well that is going to go over.

Binance is going to be slowly ground into a very fine paste.

Many crypto advocates believe the U.S. institutionally wants to see Binance reform into a compliant financial institution. They are delusional. The U.S. is already practicing their lines for the next press conference. This course of action allows them to deflate Binance gradually while minimizing collateral damage, which responsible regulators and law enforcement officials actually do care quite a bit about.

The U.S. is aware that many high-status institutions and individuals, which are within the U.S.’s circle of trust, actively collaborated with Binance. Most of them will escape serious censure.

A few examples will be made, especially in cases where it is easy to make an example, because the firm is no longer operating financial infrastructure. This will take ages to happen and be public but relatively quiet, insofar as senior U.S. regulators will not get on TV to make international headlines announcing it. It will be one of the stories quietly dribbled out on a Friday to the notice mostly of people who draft Powerpoint decks for Compliance presentations. If you want a flavor for these, join any financial firm and pay attention during the annual training; you’re stuck going to it, anyway.

You seem a little smug, Patrick

I’m not breaking out the Strategic Popcorn Reserves yet, but I will admit to a certain amount of schadenfreude here. The world was grossly disordered for many years. It has corrected a relatively small amount.

We are a nation of laws. I’d support reforming some of them; a lot of the AML/KYC regulatory apparatus harms individuals who have done no wrong. Much is not well-calibrated in terms of societal costs versus occasionally facilitating a Bond villain’s self-immolation.

However, in the interim, one cannot simply gleefully ignore the laws because the opportunity to do so allows you to become wealthy beyond the dreams of avarice. Even staunch crypto advocates looking at Binance’s conduct see some things they are not happy to be associated with. Not all of the crimes were victimless crimes.

There exists the possibility that there is some salvageable licit business in crypto. People enjoy gambling. But if you factor out the crime, the largest casino in the world is not that interesting a business relative to the one that Binance et al ran the last few years.

I do not know if we’ll ever have a world with this scale of crypto businesses without the crime. The crime was the product. An opportunity to transform global financial infrastructure was greatly overstated and has not come to pass. I do not expect this to change.

I'm seeing more and more folk post, "we got out of IE, we can get out of Chrome," on social media, referencing the nigh-monopoly Chrome has on the browsing experience. Unless you're using Firefox or Safari, you're using Chrome or a Chromium-derived browser. For those of you too young to remember what internet life under Internet Explorer was like, here is a short list of why it was not great:

• Once Microsoft got the browser-share lock in, it kind of stopped innovating the browser. It conquered the market, so they could pull back investment in it.
• IE didn't follow standards. But then, Microsoft was famous for "embrace and extend," where they adopt (mostly) a standard, then Microsoftify it enough no one considers using the non-MS version of the standard.
• If you were on a desktop platform that didn't have IE, such as Apple Macintosh, you were kinda screwed.

Google Chrome took over from IE for three big reasons:

• They actually were standards compliant, more so than the other alt-browsers (Mozilla's browsers, Opera, and Safari)
• They actually were trying to innovate in the browser
• Most important: they were a megacorp with a good reputation who wanted everyone to use their browser. Mozilla and Opera were too small for that, and Apple never has been all that comfortable supporting non-Apple platforms. In classic dot-com era thinking, Google saw a dominant market player grow complacent and smelled a business opportunity.

This made Chrome far easier to develop for, and Chrome grew a reputation for being a web developer's browser. This fit in nicely to Google's plan for the future, which they saw as full of web applications. Google understands what they have, and how they got there. They also understand "embrace and extend," but found a way to do that without making it proprietary the way Microsoft did: capture the standards committees.

If you capture the standards committees, meaning what you want is almost guaranteed a rubber stamp from the committee, then you get to define what industry standard is. Microsoft took a capitalist, closed-source approach to embrace and extend where the end state was a place where the only viable way to do a thing was the thing that was patent-locked into Microsoft. Google's approach is more overtly FOSSY in that they're attempting to get internet consensus for their changes, while also making it rather harder for anyone else to do what they do.

Google doesn't always win. Their "web environment integrity" proposal, which would have given web site operators far greater control over browser extensions like ad-blockers, quietly got canned recently after internet outrage. Another area that got a lot of push back from the internet was Chrome's move away from "v2 manifest" extensions, which include ad-blockers, in favor of "v3 manifest" plugins which made ad-blockers nearly impossible to write. The move from v2 to v3 was delayed a lot while Google grudgingly put in abilities for ad-blockers to continue working.

Getting off of Chrome

The circumstances that drove the world off of Internet Explorer aren't there for Chrome.

• Chrome innovates constantly and in generally user-improving ways (so long as that improvement doesn't majorly affect ad-revenue)
• Chrome listens, to a point, to outrage when decisions are made
• Chrome is functionally setting web standards, but doing so through official channels with RFCs, question periods, and all that ritual
• Chrome continues to consider web-developer experience to be a number one priority
• Alphabet, Google's parent company, fully understands what happens when the dominant player grows complacent, they get replaced the way Google replaced Microsoft in the browser wars.

One thing has changed since the great IE to Chrome migration began, Google lost its positive reputation. The old "don't be evil" thing was abandoned a long time ago, and everyone knows it. Changes proposed by Google or Google proxies are now viewed skeptically; though, overtly bad ideas still require internet outrage to delay or prevent a proposal from happening.

That said, you lose monopolies through either laziness of the monopolist (Microsoft) or regulatory action, and I'm not seeing any signs of laziness.