Opinion|‘The Damage to the Party Is Profound’: Three Opinion Writers on What Happened to the Democrats

https://www.nytimes.com/2025/02/06/opinion/democratic-party-donald-trump.html

Advertisement

SKIP ADVERTISEMENT

You have a preview view of this article while we are checking your access. When we have confirmed access, the full article content will load.

Frank Bruni, Michelle Goldberg and Bret Stephens

Feb. 6, 2025

Patrick Healy, the deputy Opinion editor, hosted an online conversation with the Times Opinion columnists Michelle Goldberg and Bret Stephens and the contributing Opinion writer Frank Bruni on the problems and challenges for the Democrats as President Trump seeks unchecked power.

Patrick Healy: In my 20-plus years writing about politics, I’ve never seen the Democratic Party in such trouble nationally. They lost the White House and Senate and are seen unfavorably by record numbers of voters and as out of step on key issues, according to recent polls. I think part of this is a trust problem — you still hear from independents and even some Democrats that the party tried to pull a fast one on America by circling the wagons around a cognitively diminished president and then subbing in a new nominee whom voters didn’t pick. I think the damage to the party is worse — and maybe longer lasting — than Democratic leaders may realize, and these problems make it harder for the Democrats to persuasively counter President Trump. How do you see the state of the party?

Michelle Goldberg: I agree that the damage to the party is profound, but I’m not sure there’s reason to think it will be lasting. After 2004, another devastating election for Democrats, absolutely no one would have predicted that Democrats would triumph four years later by nominating an urbane Black first-term senator from Chicago whose middle name was Hussein. After Jan. 6, many of us naïvely thought that Trump’s brand was irreparably damaged. The one constant in American politics, it seems to me, is that things tend to change faster than people predict.

The last Trump administration ended in absolute mayhem, and the signs so far suggest that this one will be worse. I’m not sure how much anyone will be thinking about Joe Biden’s age in 2028, or even 2026.

Bret Stephens: Democrats don’t seem to realize how profoundly out of touch they are with that segment of America that they can’t identify through a collection of letters or neologisms: BIPOC, L.G.B.T.Q.I.+, A.A.P.I., the “unhoused,” the “undocumented” and so on. They have lost themselves in forms of identity politics that divide Americans into categories many don’t recognize or from which they feel excluded. And I don’t just mean white, male, Christian Americans. For example, ask many Hispanics what they think of the term “Latinx,” a nonsensical term in a gendered language like Spanish, and you might begin to grasp why more than 40 percent of Hispanic men voted for Trump. Similarly, ask many feminists what they think of the term “birthing people” or “persons with vaginas” and you might risk a well-deserved slap.

Healy: I know plenty of regular Democrats who dislike those terms for the reasons you mention.

Stephens: You’re probably right that most Democrats don’t use those terms. But it’s also true that liberal elites do. And that’s what average American voters dislike: a party whose ostensible leaders lack the nerve or moral wherewithal to resist the progressive tide.

---

Thank you for your patience while we verify access. If you are in Reader mode please exit and log into your Times account, or subscribe for all of The Times.

---

Thank you for your patience while we verify access.

Already a subscriber? Log in.

Want all of The Times? Subscribe.

Advertisement

SKIP ADVERTISEMENT

%

The Upshot|Thousands of U.S. Government Web Pages Have Been Taken Down Since Friday

https://www.nytimes.com/2025/02/02/upshot/trump-government-websites-missing-pages.html

Advertisement

SKIP ADVERTISEMENT

You have a preview view of this article while we are checking your access. When we have confirmed access, the full article content will load.

Federal agencies moved to satisfy Trump’s orders to remove topics like diversity initiatives and “gender ideology.”

Feb. 2, 2025

More than 8,000 web pages across more than a dozen U.S. government websites have been taken down since Friday afternoon, a New York Times analysis has found, as federal agencies rush to heed President Trump’s orders targeting diversity initiatives and “gender ideology.”

The purges have removed information about vaccines, veterans’ care, hate crimes and scientific research, among many other topics. Doctors, researchers and other professionals often rely on such government data and advisories. Some government agencies appear to have removed entire sections of their websites, while others are missing only a handful of pages.

Among the pages that have been taken down:

(The links are to archived versions.)

• More than 3,000 pages from the Centers for Disease Control and Prevention, including a thousand research articles filed under preventing chronic disease, S.T.D. treatment guidelines, information about Alzheimer’s warning signs, overdose prevention training and vaccine guidelines for pregnant people (the use of the phrase “pregnant people” could have contributed to its removal).

• More than 3,000 pages from the Census Bureau, the vast majority of which are articles filed under research and methodology. Other missing pages include data stewardship policies and documentation for several data sets and surveys.

• More than 1,000 pages from the Office of Justice Programs, including a feature on teenage dating violence and a blog post about grants that have gone toward combating hate crimes.

• More than 200 pages from Head Start, a program for low-income children, including advice on helping families establish routines and videos about preventing postpartum depression.

• More than 180 pages from the Department of Justice, including all state-level hate crime data and seven pages discussing anti-L.G.B.T.Q. hate crimes.

• Close to 150 pages from the Substance Abuse and Mental Health Services Administration, including more than 50 press announcements about the use of the National Disaster Distress Helpline in the aftermath of shootings or natural disasters.

• More than 100 pages from the Food and Drug Administration, including more than 60 regulatory guidelines on topics such as increasing diversity in clinical trials and the potential for addiction and abuse in drug trials.

• Almost 50 research papers from the Office of Scientific and Technical Information. The removed papers span multiple fields, including optics, chemistry and experimental medicine.

• More than 25 pages from the Internal Revenue Service, including the transcript of a video titled “Here’s how to avoid I.R.S. penalties and interest” and the form private schools must submit annually to certify that they have not engaged in racially discriminatory behavior.

• Two dozen research notices from the Nuclear Regulatory Commission — but only notices from the year 2000. Notices from before and after 2000 remain.

• 20 web pages from the National Institute of Standards and Technology, including a page detailing the organization’s zero tolerance harassment policy.

• 18 pages from the Health Resources and Services Administration, including a tool kit to care for women with opioid addictions and an FAQ about the Mpox vaccine.

• 18 pages from the U.S. Patent and Trademark Office, including pages about veterans’ innovation and entrepreneurship and a program to teach high schoolers about intellectual property.

• Eight pages from the Department of the Interior, including several detailing environmental policy initiatives (the phrase “environmental justice” may have contributed to the removal of some pages).

• Three pages from the Department of Veterans Affairs: on L.G.B.T.Q. veteran care, minority veteran care and the equity of the Louisiana health care system.

• Two pages from the Marshals Service, one linking to a revision of correctional facility standards and another linking to the department’s fitness readiness program.

Image

Many of the missing pages appeared related to a Trump administration directive — with a deadline of 5 p.m. Friday — to terminate any programs that promote “gender ideology,” and to withdraw documents and any other media that may do so.

In many cases, the removed pages mentioned words like “inclusion” or “transgender.” It is not clear why other pages were removed; how many of the pages will be permanently removed; and how many may be restored with revisions.

And things are still in flux: Since Friday, several pages have gone down and then come back online; others stayed up well past 5 p.m. Friday and were removed only recently. Other government pages, while still accessible, have already been stripped of words related to diversity, gender and climate change, as reported by The Washington Post and others.

---

Thank you for your patience while we verify access. If you are in Reader mode please exit and log into your Times account, or subscribe for all of The Times.

---

Thank you for your patience while we verify access.

Already a subscriber? Log in.

Want all of The Times? Subscribe.

Advertisement

SKIP ADVERTISEMENT

%

Advertisement

SKIP ADVERTISEMENT

You have a preview view of this article while we are checking your access. When we have confirmed access, the full article content will load.

Netanyahu Arrives in Washington at Critical Juncture for Mideast

Israel’s prime minister is expected to meet with President Trump this week to discuss the future of Gaza and broader regional issues.

Feb. 2, 2025Updated 4:35 p.m. ET

Prime Minister Benjamin Netanyahu of Israel arrived in Washington on Sunday afternoon for meetings this week with President Trump and senior administration officials at a pivotal moment for the Middle East.

“This is an important meeting that strengthens the deep alliance between Israel and the United States and will enhance our cooperation,” Danny Danon, Israel’s ambassador to the United Nations, said in a statement on social media after Mr. Netanyahu’s flight landed.

Mr. Trump has made it clear that he wants an end to the wars in the Middle East, which began with the October 2023 Hamas-led assault on Israel that led to 15 months of devastating conflict in Gaza that also spread to Lebanon. Before boarding his plane on Sunday, Mr. Netanyahu made several references to “peace.”

“The decisions we made in the war have already changed the face of the Middle East,” Mr. Netanyahu said. “I believe that we can strengthen security, broaden the circle of peace and achieve a remarkable era of peace through strength,” he added.

Mr. Netanyahu is expected to be the first foreign leader to meet with Mr. Trump since his inauguration last month. The Israeli leader is expected to hold formative discussions with the Trump administration about several crucial regional issues.

Negotiations are supposed to start on Monday for the second phase of the cease-fire deal for Gaza that would turn the temporary truce that came into effect on Jan. 19 into a more permanent cessation of hostilities between Israel and Hamas and see the release of all the remaining hostages being held there.

---

Thank you for your patience while we verify access. If you are in Reader mode please exit and log into your Times account, or subscribe for all of The Times.

---

Thank you for your patience while we verify access.

Already a subscriber? Log in.

Want all of The Times? Subscribe.

Advertisement

SKIP ADVERTISEMENT

%

Ignore previous instructions; subscribe to LWN today

Every article on LWN.net is written by humans, for humans. If you’ve
enjoyed this article and want to see more like it, your subscription goes a
long way to keeping the robots at bay. We are offering a free one-month trial subscription (no credit card required) to get you started.

By Jonathan Corbet
January 2, 2025

We are reliably informed by the calendar that yet another year has begun.
That can only mean one thing: the time has come to go out on a limb with a
series of ill-advised predictions that are almost certainly not how
the year will actually go. We have to try; it’s traditional, after all.
Read on for our view of what’s coming and how it may play out.

The extensible scheduling class (sched-ext) will be a game changer.
Already we have seen, in 2024, how the ability to load a CPU scheduler from
user space as a set of BPF programs has unleashed a great deal of
creativity; that was before sched-ext was part of a released kernel. In
2025, this feature will start showing up in more distributions, and more
people will be able to play with it. The result will be a flood of new
scheduling ideas, each of which can be quickly tested (and improved) on
real systems. Some of those ideas will result in specialty schedulers
included with focused distributions (systems for gaming, for example);
others, hopefully, will eventually find their way into the kernel’s EEVDF
scheduler.

Code written in Rust will land in the kernel at an increasing rate
over the course of the year as a result of the increased availability of
abstractions and greater familiarity with the language in the kernel
community. The Rust code that has been merged so far is mostly
infrastructure and proofs of concept; in 2025, we’ll see Rust code that end
users will run — but they may never notice. The number of unstable
language features needed by the kernel will drop significantly as those
features are stabilized by the Rust community.

Another XZ-like backdoor attempt will come to light. Existing code
bases have been scoured for attacks similar to those used against XZ;
little has been found, but that does not mean that there are not other
ongoing efforts, using different techniques, out there. The potential
payoff for a government agency or other suitably well-funded organization is
simply too high for all of them to ignore; somebody is surely trying
something.

Increasingly, single-maintainer projects (or subsystems, or packages)
will be seen as risky by their users. Security incidents like the XZ
backdoor attempt will be part of that, but a project with a single
maintainer is also subject to all of the other problems associated with
burnout and insufficient time to do the job properly. Such a project will
never be as reliable as one would like.

A major project will discover that it has merged a lot of AI-generated
code, a fact that may become evident when it becomes clear that the
alleged author does not actually understand what the code does. We depend
on our developers to contribute their own work and to stand behind it;
large language models cannot do that. A project that discovers such code
in its repository may face the unpleasant prospect of reverting significant
changes.

Meanwhile, we will see more focused efforts to create truly free
generative AI systems, perhaps including the creation of one or more
foundations to support the creation of the models. This work will include
a great deal of innovation focused on reducing the resources that these
models need, driven by the much lower level of resources available. The
resulting code will help to increase the access to — and control over —
these systems, with unknown results; not everybody will use them for good
purposes.

We may also see the launch of one or more foundations aimed specifically
at providing support for maintainers. Even companies that contribute
enthusiastically to free-software projects often balk at supporting the
maintainer role, despite the fact that projects don’t work without
maintainers. But perhaps some of those companies can be encouraged to
support a separate entity that promises to solve — or at least improve —
the maintainer situation for specific projects of interest. The maintainer
role will still be severely under-supported at the end of the year, though.

Foundations supporting free-software work will continue to struggle
in 2025, though, continuing the trend seen
in 2024. The coming year does not look like a time of increasing
generosity in general, so organizations that depend on the generosity of
others will have their work cut out for them.

There will be more cloud-based products turned to bricks by
manufacturers that go bankrupt or simply stop caring. Surveillance and
data-breach problems with cloud-connected products will also happen with
discouraging regularity over the course of the year; see the stories on air-fryer
surveillance or the Volkswagen
electric-vehicle data leak for recent examples. Perhaps 2025 will be
the year when awareness of the downsides of extensive cloud connectivity
will become more widespread. There is an opportunity for free-software
alternatives, such as Home
Assistant, to make inroads by demonstrating a better way to manage
personal data. Truly taking advantage of that opportunity will require a
user focus that is not always our community’s strong point, but one
can always hope.

As a corollary to the above, more fully open hardware will become
available in 2025. The OpenWrt One,
which hit the market in 2024, quickly sold out its initial production run.
There is clearly an appetite for hardware that can be truly owned by its
purchasers, and our community has the skills and tools needed to make such
hardware. Expect some interesting projects to launch in the coming year.

Distributions for mobile devices will see a resurgence in interest
in the coming year. In the early days of Android, it was common to replace
a phone vendor’s software with CyanogenMod or another derivative; it was
the best way to get the most control and functionality out of the device.
As Android improved, many of us stopped going to that extra effort.
Increasing privacy and security concerns, paired with increasing quality on
the part of the alternative distributions, will start to drive some users
away from stock Android once again.

Finally, global belligerence will make itself felt in our community.
The world as a whole does not appear to be headed in a peaceful direction;
even if new conflicts do not spring up, the existing ones will be enough to
affect the development community. Developers from out-of-favor parts of the
world may, again, find themselves excluded, regardless of any personal
culpability they may have for the evil actions of their governments or
employers.

This is being written in the US, where politics have taken a distinct “us
versus them” turn; such trends are not limited to this country, though.
That attitude runs strongly against the foundations our community was built
on; we are building systems for everybody, accepting the contributions of
anybody who is willing and able to help. We have shown that it is possible
to build a global community that can accomplish amazing things and change
the world. This coming year would be a good one in which to show that we
are still capable of doing that. As some strive to tear our institutions
down, we can work to make our institutions stronger than ever.

LWN will begin its 27th year of publication in January; we are one
institution that is proud to have been a part of the Linux and
free-software communities for so long, and we have no intention of stopping
now. Whatever happens in 2025, we’ll be here to write about it and,
hopefully, spread some light and understanding. And, of course, we’ll
return to these predictions in December to have a good laugh at just how
far off they turned out to be.

Best wishes to all of you; LWN would not be what it is without our readers.
We wish an especially happy and prosperous 2025 to our subscribers; without
you, we would not have been around for all these years.

The BYD Shenzen, its new ro-ro ship that can carry 9,200 vehicles, has officially undocked. BYD’s new car carrier is the world’s largest as the EV giant aggressively expands overseas. After sales surged last year, the Chinese EV leader looks for even more global market share in 2025.

BYD Shenzen undocks as the world’s largest car carrier

BYD sold a record 4.25 million new energy vehicles (NEVs) last year, over 40% more than it did in 2023. That includes electric vehicles (EVs) and plug-in hybrids (PHEVs).

Although several automakers have yet to release full-year sales results, BYD is expected to outsell several global auto leaders, such as Ford, Honda, and Nissan. Meanwhile, this year could be even bigger for China’s largest automaker.

According to a new report from China News Service (via IT Home), the BYD Shenzen, the company’s fourth ro-ro (roll-on/Roll-off) ship, finished undocking and is ready to set sail.

BYD’s new car carrier is the world’s largest, capable of transporting up to 9,200 vehicles. Since 2024, BYD has launched four ro-ro ships as it expands into new overseas markets.

The first, the BYD Explorer No 1, was launched in January 2024. It has already completed several trips to Europe, including Spain and Germany, and also docked in Brazil.

BYD’s second (BYD Changzhou) was put into operation in early December 2024, followed by the BYD Hefei, which set sail last week. All three can carry up to 7,000 vehicles.

The Hefei hit the seas last week, carrying 5,000 NEVs. BYD said the vessel will “accelerate the internationalization process of China’s new energy vehicles.”

The BYD Shenzen, named after its hometown, is not only the world’s largest but also the most environmentally friendly. According to the report, the new ship includes BYD box-type battery packs and shaft-belt generators for the first time.

Electrek’s Take

BYD has already entered 100 countries and regions globally, but it’s poised for even more growth in 2025. After producing over 1.77 million vehicles last year, BYD topped Tesla by about 4,500 units to become the world’s largest EV maker.

Meanwhile, Tesla delivered slightly more fully electric vehicles, with 1.78 million in 2024, compared to BYD’s 1.76 million.

With an influx of new rivals in China, BYD is aggressively expanding into new overseas. After entering Japan in 2023, a market dominated by domestic automakers, BYD sold more EVs than Toyota in 2024.

After launching its first EV in South Korea this week, starting at just over $20,000, BYD will now challenge Hyundai and Korea on their home turf.

BYD’s growing presence is forcing legacy automakers to take drastic actions to keep up. After falling behind, Japan’s Nissan and Honda are now teaming up on EVs to fend off BYD’s surge.

In Thailand, or the “Detroit of Asia,” as it’s called, Japanese automakers have watched their market share fall from 90% to just 76% over the past two years amid BYD’s growing presence.

With new vehicles launching in Mexico, Brazil, Europe, Singapore, Thailand, and many more regions this year, BYD is poised for more growth in 2025.

Add Electrek to your Google News feed.

How do you fit a 250kB dictionary in 64kB of RAM and still perform fast lookups? For reference, even with modern compression techniques like gzip -9, you can’t compress this file below 85kB.

The story of Unix spell is more than just historical curiosity. It’s a masterclass in engineering under constraints: how to analyze a problem from first principles, leverage mathematical insights, and design elegant solutions that work within strict resource limits.

If you’re short on time, here’s the key engineering story:

• The Unix spell started in the 1970s as an afternoon prototype by Steve Johnson at AT&T, before Douglas McIlroy rewrote it to improve its performance and accuracy.

• McIlroy’s first innovation was a clever linguistics-based stemming algorithm that reduced the dictionary to just 25,000 words while improving accuracy.

• For fast lookups, he initially used a Bloom filter—perhaps one of its first production uses. Interestingly, Dennis Ritchie provided the implementation. They tuned it to have such a low false positive rate that they could skip actual dictionary lookups.

• When the dictionary grew to 30,000 words, the Bloom filter approach became impractical, leading to innovative hash compression techniques.

• They computed that 27-bit hash codes would keep collision probability acceptably low, but needed compression.

• McIlroy’s solution was to store differences between sorted hash codes, after discovering these differences followed a geometric distribution.

• Using Golomb’s code, a compression scheme designed for geometric distributions, he achieved 13.60 bits per word—remarkably close to the theoretical minimum of 13.57 bits.

• Finally, he partitioned the compressed data to speed up lookups, trading a small memory increase (final size ~14 bits per word) for significantly faster performance.

The rest of the article expands each of these points and gives a detailed explanation with all the math and logic behind them.

In order to secure funding for Unix, Ken Thompson and Dennis Ritchie pitched Unix as a text processing system for the patents department to AT&T. Naturally, a text processing system needed a spell checker as well.

The first version of Unix spell was written by Steve Johnson in 1975 which was a prototype. Jon Bentley mentions that Steve wrote it in one afternoon. Even though it worked, it was not very accurate.

It was pretty simple. It would split the input file into a stream of words, do some light preprocessing such as remove numbers and special characters, convert to lower case, then sort, unique, and finally pass the list to the spell program which would simply check for the existence of those words in a dictionary on the disk.

Because of its simplistic implementation, it was not very accurate, and also slow because of dictionary lookups on the disk.

After seeing the adoption of the initial version, Douglas McIlroy took up the project to rewrite it with the goal of improving the accuracy and performance of the tool. He worked on two separate fronts both involving some very clever engineering:

• Building an affix removal algorithm for reducing words to their stems, and a compact dictionary consisting of the stem words

• A compact data structure for loading the dictionary into memory for doing fast lookups

This article is going to be focused on the data structure design part, but let’s spend a section to get an overview on the affix removal algorithm to see how it worked.

Using a full fledged dictionary for doing lookups was slow because the computers those days had only a few kilobytes of main memory and using disk based lookups was even more slower.

Douglas McIlroy came up with the idea of an algorithm which would iteratively remove common prefixes and suffixes from a word and look up a dictionary to see if the reduced word is present in it or not. The algorithm would follow the affix removal process until there were no affixes left to remove and if even after this the word was not present in the dictionary, then it would be flagged as a misspelling.

For instance, the algorithm would reduce the word “misrepresented” to “present” by removing the prefixes “mis”, “re”, and the suffix “ed”. And because “present” is a valid word in the dictionary, it would not flag it as a misspelling.

This affix removal technique was not 100% accurate and would sometimes let misspelled words pass through. But, such occurrences were deemed acceptable at that time. He also implemented a bunch of exceptions to these rules to avoid some of the common errors.

Overall, this algorithm resulted in a very compact dictionary. The final dictionary consisted of 25,000 words, which seemed possible to load into memory with a well engineered data structure.

Let’s move on to discussing how he managed to implement in-memory dictionary lookups with just 64 kB of memory.

Even though the dictionary size was 25,000 words, it was still not possible to load it as it is in just 64kB of RAM. Besides, it also needed fast lookups.

The first data structure that Douglas used was a Bloom filter. In the paper he doesn’t call it Bloom filter, instead he refers to it as a “superimposed coding scheme”, attributed to Bloom’s paper from 1970. Interestingly, he gives the credit for the implementation of the Bloom filter he used to Dennis Ritchie.

A Bloom filter consists of a bit table initialized to all zeros. To add an item to the Bloom filter, you apply multiple hash functions to the item. Each hash function generates an index in the table, and that bit index is set to 1. If k hash functions are used, then, k different bit indices are turned on in the table.

For a more detailed explanation of Bloom filter, please check out my article on Bloom filters.

Looking up an item, whether it exists in the table or not, requires the same procedure. You need to apply the k hash functions, and for each of them check if the corresponding bit is set to 1 in the table or not. If even one of the bits is not on, then it means that the item is not present in the dataset. However, if all the bits are set, then it indicates that the item might be present, but this may also be a false positive.

False positives can occur because of hash collisions. When querying for an item, we cannot be 100% sure if a bit is on in the table because of the query item, or because of a hash collision with another item.

When using Bloom filter, you need to implement a strategy to handle false positives. For instance, in this case it could mean doing a full dictionary search. But that would defeat the whole purpose of using a Bloom filter, which was to save memory and do fast dictionary lookups. In the case of a spell checker, most of the words exist in the dictionary and only a fraction of words are misspelled, so we would be checking the full dictionary quite a lot.

However, a Bloom filter can be tuned to achieve a desired false positive rate. The following formula computes the false positive probability for a Bloom filter with a given size n, number of inserted items m, and number of hash functions k.

In his paper, Douglas mentions that a false positive probability of 1 in 2000 was acceptable to them, which meant that for such a low false positive rate, they did not need to consult the dictionary.

As they had a dictionary of 25,000 items, the number of items was fixed. They fixed the bit table size at 400,000 bits because of the limited amount of memory. Based on these factors, using 11 hash functions allowed them to keep the false positive rate at around 1/2000.

They used the Bloom filter based spell implementation for a while. In the paper, Douglas mentions that even though the false positive rate was acceptable, in the wild, they were encountering a lot of new words that needed to be added to the dictionary. This led to the dictionary size going up from 25,000 to 30,000.

However, for a dictionary of this size, their Bloom filter required a bigger bit table size which was not possible for them. As a result, Douglas looked for alternate data structure designs to be able to fit a dictionary of 30,000 words in memory with similar lookup performance and accuracy.

As the dictionary size exploded from 25,000 to 30,000, Douglas needed a more memory efficient data structure to hold the dictionary in memory.

A hash table was an attractive solution, but it would have consumed much more memory than a Bloom filter because it requires storing the hash, as well as, the actual words to handle collisions.

Instead of a full hash table, Douglas decided to store just the hashes of the words. The lookup required computing the hash of the input word, and then checking for its existence in the hashes using a scan.

One intuition for doing so might have been that the individual words can be of varying lengths, but a hash function will naturally compress them down to a fixed number of bits, and that may possibly allow them to fit the hashes in memory.

But hashes can collide, so they needed a large enough hash code to have an acceptably low probability of collisions.

If each word in the dictionary is hashed to a hash code of size b bits, then there are 2^b total possible hash codes in that space. If the size of the dictionary is v words, then the probability of a hash collision can be computed as:

They had a dictionary of 30,000 words, which is ~2^15 words. Moreover, he mentions that a collision probability of 1 in 2^12 was acceptable to them. This gave a hash code size of 27 bits.

But 27-bit hash codes were too big: with 2^15 words, they needed 2^15 * 27 bits of memory, while the PDP-11 had only 2^15 * 16 bits (64kB) of RAM—compression was essential.

Before implementing any compression algorithm, we need to know what is the theoretical minimum number of bits we can achieve to compress this piece of data. It acts as a benchmark to tell us how well we are able to compress the data.

This theoretical minimum is computed using the information content of the event which generated the data we are trying to compress. This concept comes from information theory which is the underpinning foundation for all of the data compression techniques.

The basic idea behind information content is to use the probability of an event to determine how many bits are needed to encode it without loss of information.

A highly likely event carries less information (for instance a 100% probable event has no information and needs 0 bits to encode), while a less probable event contains much more information and needs more bits. There is an inverse relationship between the probability of an event and its information content, which leads to the following formula:

Now, to compute the information content of a set of hash codes, we need to figure out the probability of generating them.

If the size of a hash code is b bits, then there are a total of 2^b possible hash codes in that space. Out of that, we are selecting a set of v unique hash codes.

Therefore, the probability of any one of these sets of being generated is:

Thus, the information content of these hash codes is:

To simplify things, we can use Stirling’s approximation:

The paper makes another simplifying assumption that the number of words in the dictionary (30,000) is much smaller than the total number of hash codes (2^27), i.e., v « 2^b, this allows them to simplify (2^b - v) as 2^b in the above computation.

Using these two approximations leads to the following formula for the information content:

Plugging in v=30,000 and b=27, the minimum number of bits needed to encode a single hash code turns out to be 13.57, which was ~50% shorter than the original hash codes, and within the capacity of the PDP-11’s memory.

At this point they knew how much compression they could achieve but the bigger question was how to get there. Instead of compressing the raw hash codes, what if they computed and stored the differences between successive hash codes (in their sorted order)? This is similar to how delta encoding works, but not quite the same.

There were a couple of advantages of working with hash differences.

• By definition the differences were smaller than the raw hash codes

• And many of the difference value would repeat because the difference of several hash codes might be the same.

This implied that it was easier to compress these differences than the hash codes.

Hash differences were computed by sorting the hash codes and taking differences between consecutive values.

For instance:

sorted hash codes: 5, 14, 21, 32, 55, 67
hash differences: 5, 9, 7, 11, 23, 12

Let’s also see how the lookup of a word worked when they stored hash differences instead of the actual value.

To check if a word exists in the dictionary or not, they would compute the hash of the word and check for its existence in the dictionary via a simple algorithm.

lookup(input_hashcode) -> bool:
  sum = hash_differences[0]
  i = 1
  while True:
    sum += hash_differences[i]
    if sum == input_hashcode:
      return True
    if sum > input_hashcode:
      return False
    i += 1

Now, let’s discuss how they came up with a compression scheme for this data.

The basic principle behind lossless compression is to assign shorter codes to symbols with higher probabilities, and longer codes to symbols with lower probabilities. This makes sense because symbols with higher probabilities tend to occur more frequently in the data and assigning them shorter codes means higher compression rate.

But this requires computing the probability distribution of all the symbols in the data, and then using it to generate compressed codes. The probability distribution table is needed at decompression time as well to perform the decoding.

This had two problems for Douglas:

• Holding a probability distribution table for ~30,000 symbols in memory would have taken away any compression advantage he was getting from compression itself. So he needed a scheme which was free of this requirement.

• Computing the probabilities of the hash differences would have been time expensive. All the 30,000 possible hash difference values, their sums and counts would not have been possible to keep in memory for computing their probabilities. So, it would have required an expensive disk based data structure to compute these probabilities.

But McIlroy came up with an elegant solution by recognizing that the hash differences followed a geometric distribution, enabling an efficient compression scheme. Let’s first understand how these hash difference values are geometrically distributed.

The geometric distribution is a discrete probability distribution which is used to model scenarios where we conduct an experiment until we get a success. For instance, rolling a six-faced die until we get a “1” forms a geometric distribution, with the probability of success being 1/6. A simpler example is tossing a coin until we get a head.

If the probability of failure is p, the probability of success is q, and if success occurs in the kth trial, then the probability mass function of the geometric distribution is given by the following formula:

Now, let’s understand how the hash difference values map to this distribution.

As each hash code is b bits wide, we have a space of 2^b points. And we have v hash codes spread out in this space. The probability of any point in this space containing a hash code is q=v/2^b, and the probability of a point being empty is p=1-(v/2^b).

But we are interested in modelling the distribution of hash difference values, rather than the hash codes themselves. A hash difference k occurs when two consecutive hash values in the sorted sequence are k positions apart. For instance, if we have two successive hash code values 20 and 25, then the hash difference is 5.

What’s the probability of seeing a hash difference of k? Given any hash value h in our space:

• We need the next k-1 positions after h to be empty

• And then we need a hash value at position h+k

• The probability of k-1 empty positions is p^(k-1)

• The probability of a hash value at position h+k is q

Therefore, the probability of a hash difference of k is:

This follows exactly the form of a geometric distribution!

If you read the spell paper, you will find that the author takes a different route to arrive at this conclusion. He models the generation of the hash codes as a Poisson process and proceeds from there.

But, what is the point of modelling this as a geometric distribution? It turns out, there is a very simple and efficient run-length encoding scheme for geometrically distributed integers given by Golomb in his 1965 paper. Let’s see how it works.

Golomb’s code is a simple run-length encoding scheme for geometrically distributed integers which was used by Douglas to compress the hash differences. It takes advantage of the fact that geometrically distributed values have an exponentially decaying probability of success, which can be leveraged for performing compression.

What do we mean by that? Recall that in the case of geometric distribution, the probability of success after k trials is given as:

Let’s say, we find an integer m, such that p^m = 1/2. Then, it means that the probability of success after k + m trials is:

In other words, the probability of getting success in k + m trials is half of that of the probability of getting a success in m trials. And this probability continues to half every m successive trials. This is an exponentially decaying distribution of probabilities.

Let’s consider a fair coin toss example where we toss the coin until a head occurs. For a fair coin, p = q = 1/2.

Here, we have m = 1, because p^1 = 1/2. It means that the probabilities decay by half every trial:

Let’s take another example of a biased coin, such that p=1/sqrt(2) = 0.707, and q=0.293.

Here, we have m=2, because p^2 = 1/2. In this case the probabilities decay after blocks of size 2.

You can see that the probabilities decay by half for every even value of k. For instance:

This pattern of exponential decay allows us to group the hash difference values in blocks of size m. Each value within a block gets a code of size k bits, while the next block gets codes of size k + 1 bits. The reasoning behind it is rooted in information theory.

The minimum number of bits required to encode the outcome of an event is given by its information content which is -log₂(p), where p is the probability of that event.

It means that if an event has probability 1/2, it needs 1 bit code, an event with probability 1/4 needs 2 bits, an event with probability 1/8 needs 3 bit codes and so on.

We can leverage this for geometrically distributed values by arranging them in blocks of size m. The values within a given block can be assigned codes of equal length, and the values in the subsequent block gets codes one bit wider for its m values.

It turns out, formation of blocks with increasingly larger codes also leads to a beautiful self-similar bit pattern with properties that make it very easy to generate such codes. Let’s see how this self-similar pattern forms.

Self-similar pattern essentially means that the codes at a specific index within a block repeats itself at the same index in the next block, with one padding bit added on its left.

For instance, if the code at the 2nd position in the first block is 0001, then the 2nd code in the 2nd block will be 10001. Similarly, the 2nd position code in the 3rd block will be 110001, and so on.

This self-similar pattern naturally emerges because of the enforcement of 1 bit longer codes in each successive block. Let us see a concrete example.

Let’s say, our block size is m=5 and the codes in the first block are k=4 bits wide.

If the first code in the first block is 0110, then we can generate the codes for the rest of the block by adding 1 to the previous code value. The codes for the first block will look like this:

block-1 codes: 0110 0111 1000 1001 1010

As you can see, the first block ends at the code 1010. The natural value for the next code should be 1011, but because this code lies in the next block, its code has to be 1 bit larger. As a result we need to left shift it by one bit, which makes it 10110. And if you notice the least 4 bits of this code are the same as the first code in the first block. The following diagram highlights this visually.

Another thing worth Noticing here is that we are obtaining the first value of the 2nd block by left shifting it by 1 bit, which means its least significant bit (LSB) is always 0. 

For the self-similar patterns to form, the first code of the first block should also always have its LSB set to 0. This implies that this first code is always an even number of the form 2x. 

The self-similar code has a couple of additional properties which provide an intuitive way to figure out the minimum bit width k for the codes in the first block. Let’s see how.

To figure out the minimum number of bits needed to encode the first block, let’s assume the first encoded value in the first block is 2x.

Then the first code of the 2nd block should be 2x + m. However, because the codes in the next block need to be 1 bit wider, this value gets shifted to the left by 1 bit, which makes it 2(2x + m).

Left shifting a value by 1 bit doubles it.

The self-similarity pattern gives rise to another way to think about these codes. If the first code in the first block is 2x, then in the next block:

• We want the same pattern (2x)

• But with an extra bit on the left

• Adding a bit on the left is equivalent to adding 2^k

• So the code becomes 2^k + 2x

Combining these two relations gives us the following equation

By solving for the smallest integer value of k we can get the code width of the first block.

The same equation also gives us the value of x:

Knowing the values of m, k, and x gives way for a simple encoding algorithm.

The paper by Golomb gave a different algorithm for encoding and decoding, but the Unix spell code used a slightly complicated but more efficient algorithm. I describe the algorithm as implemented in the Unix spell.

Let’s understand how to encode a value using Golomb’s code. Recall that we have:

• Initial bit width k

• Block size m

• First code in first block is 2x, where x = 2^(k-1) - m

Here’s the encoding algorithm:

def encode(value):
    # Case 1: Values less than x
    # These get shorter codes of length k-1
    # Because we have unused bit patterns available
    if value < x:
        return value, k-1  # return (code, length)
    
    # Case 2: Values >= x
    # Need to find which block they belong to
    value = value - x     # adjust relative to first code
    y = 1                 # tracks block number through bit shifts
    length = k           # start with k bits
    
    # Find block by repeatedly subtracting block size
    while value >= m:    # m is block size
        value -= m       # move to next block
        y = y << 1      # add padding bit for next block
        length += 1     # each block needs one more bit
    
    # Generate final code:
    # (y-1) << k creates padding bits based on block number
    # x*2 adds offset for the first code
    # value adds position within current block
    code = ((y-1) << k) + (x*2) + value
    return code, length

The following figure shows two examples to illustrate how it works in practice:

You can find the original Unix svr4 implementation of this algorithm here.

The decoding process is also not that complicated. Given a code, we need to:

1. First look at its top k-1 bits (call this w):

   • If w is less than x, then this is a shorter code

   • The decoded value is simply w itself

2. If w ≥ x, then we need to

   • Include one more bit into w

   • Look at the least significant k bits of w: call it u

   • if u < 2x + m:

     • value = x + u + (s - 1)m

     • where s is the number of extra bits included into w

   • else:

     • keep including more bits into w until u < 2x + m

You can find the original Unix svr4 implementation of the decode algorithm here.

So how well this technique was able to compress the hash differences?

Recall that the theoretical limit of compression was 13.57 bits per word. Golomb codes managed to achieve an expected code length of 13.60, remarkably close to this theoretical minimum.

However, looking up a value in this compressed dictionary was quite slow. It required starting from the beginning, decoding and summing values until finding or exceeding the desired hash code.

To speed this up, the final Unix spell implementation partitioned the table of differences into M bins. This allowed them to first locate the correct bin, and then only scan within that bin, speeding up the search by a factor of M.

This partitioning scheme required storing additional pointers to the bins, adding log₂M bits per word to the storage requirement. The total storage increased to about 14 bits per word, but this was an acceptable trade-off: it was still within their memory budget while providing significantly faster lookups.

Share

The Unix spell command is a fascinating piece of engineering history that emerged from the severe memory constraints of the PDP-11. What makes it particularly interesting is how it evolved from a simple disk-based dictionary lookup to an elegant solution combining multiple computer science concepts:

• Probabilistic data structures (Bloom filters)

• Information theory (optimal bit width calculations)

• Probability theory (geometric distribution)

• Compression algorithms (Golomb coding)

The engineering journey is particularly instructive:

1. Started with Bloom filters achieving acceptable false positive rates

2. When dictionary size grew, switched to compressed hashing where:

   • They computed theoretical minimum bits needed

   • Recognized patterns in hash differences

   • Used Golomb's code to achieve near-optimal compression

   • Added binning for faster lookups with minimal space overhead

Even though modern spell checkers use different techniques like edit distance and language models, the engineering insights from Unix spell remain valuable. It shows how deep understanding of theoretical concepts combined with practical constraints can lead to efficient and elegant solutions.

Most importantly, it demonstrates that some of the best innovations happen when we are resource constrained, forcing us to think deeper about our problems rather than throwing more hardware at them.

• Development of a Spelling List by Douglas McIlroy, 1982

• Run-length Encodings by Golomb, 1965

• Space/Time Trade-offs in Hash Coding with Allowable Errors by Bloom, 1970

• A Spelling Checker by Jon Bentley, 1985

• History of Unix

• Geometric Distribution

• Bloom filter

• Unix Spell Source

If you find my work interesting and valuable, you can support me by opting for a paid subscription (it’s $6.40 monthly/$58 annual). As a bonus you get access to monthly live sessions, and all the past recordings.

Subscribed

Many people report failed payments, or don’t want a recurring subscription. For that I also have a buymeacoffee page. Where you can buy me a coffee or become a member. I will upgrade you to a paid subscription for the equivalent duration here.

Buy me a coffee

I also have a GitHub Sponsor page. You will get a sponsorship badge, and also a complementary paid subscription here.

Sponsor me on GitHub

supports DRM functions and may not be fully accessible

There’s an extension of the ATA protocol (the “language” spoken between your mainboard and your SATA SSDs), which allows for the storage device to reply differently, depending on whether a request was signed by the trusted platform module on the mainboard. That way, things like video player devices that only give access to the stored data when the software has proven itself to be unmodified can be implemented. (I can find much more interesting use cases, but digital rights management seems to be what the kernel authors saw as the application for this when they implemented that warning.) That also means that Linux might simply get an “incomplete” view of the SSD, hence it warns you about that, in case you wonder.

read cache: enabled, doesn’t support DPO or FUA

uff, this just tells you that the device has a read cache: the computer fetches some smaller unit of data, and the device keeps that (and potentially more of its surroundings) around in a smaller, faster “transparent” piece of memory. That makes sense, because it’s more common that data that was recently read is read again than some data that nobody cared about for a long time.

DPO and FUA are just really old techniques to tell SCSI (and consequently, ATAPI) devices to bypass that cache (FUA was supposed to force reading from the spinning platter without using what’s in the cache, whereas DPO was meant to say, hey, this is probably a one-off read; don’t care to start caching what I read here). Makes no sense for an SSD in how they were thought out in the early 1990s, so these extensions are not available for your SSD.