Ctrl+Z Can't Save You: The Accelerating Erasure of the Web's Past
Photo by Photo by Egor Komarov on Unsplash on Unsplash
There's a cruel irony baked into the modern web. We built a global network capable of copying and distributing information at zero cost, in milliseconds, to anyone on the planet — and somehow we're losing the internet's history faster than a library fire ever could.
It's not dramatic. There's no smoke. One day a link works. The next day it returns a 404, or a redirect to a homepage, or — worst of all — the exact same URL now points to something completely unrelated. The content is just gone. And increasingly, it's taking entire ecosystems with it.
The Platforms Eating Their Own History
Let's name names, because this problem has addresses.
Reddit's 2023 API pricing overhaul didn't just kill third-party apps. It wiped out years of archiving infrastructure that volunteer developers had quietly maintained. Tools like Pushshift, which let researchers and archivists index Reddit's full comment history, were effectively shut down when Reddit pulled API access. Millions of threads — some of them the only surviving record of niche communities, technical discussions, and cultural moments — became unsearchable or inaccessible overnight.
Twitter, now rebranded as X, has been on a similar trajectory. The platform gutted its free API tier, broke dozens of archiving bots, and has made mass account deletions and suspensions routine. When a high-profile account disappears, so does every link ever shared through it. Every embedded tweet in a news article becomes a gray void. Every citation in an academic paper turns into a ghost.
Medium presents a different flavor of the same problem. Writers who built audiences there have watched their work vanish behind paywalls, get quietly unpublished due to policy changes, or simply disappear when the platform pruned inactive accounts. The URLs remain indexed in Google — but click them and you hit a wall.
This isn't accidental. It's structural.
Why Preservation Is Losing the Economic Argument
Here's the uncomfortable truth that most think pieces on link rot skip over: preservation is expensive, and nobody wants to pay for it.
Storage costs money. Bandwidth costs money. Legal review of archived content — especially when platforms claim copyright over user-generated posts — costs a lot of money. The Internet Archive, the closest thing the web has to a national library, operates as a nonprofit and has faced multiple high-profile lawsuits from publishers challenging its right to archive and lend digital content. In 2024, a federal appeals court ruled against the Archive in a case brought by major book publishers. The legal and financial pressure on the organization is immense.
Meanwhile, the platforms generating the content have every economic incentive to not make it easy to archive. Archived content doesn't serve ads. It doesn't drive engagement metrics. It doesn't keep users logged in. A link that routes through a live platform page is worth something to an advertiser. A link to an archived snapshot is worth nothing.
The math is brutal and simple: the entities best positioned to preserve the web's history profit from letting it decay.
The Technical Side of the Rot
Beyond economics, there's a technical reality that makes link rot nearly inevitable at scale. Most of the web's URL structure was never designed for permanence. Dynamic content management systems generate URLs tied to database IDs, session states, or content slugs that change when sites are redesigned. Short links — including the kind we work with every day here at GoWWW — add another layer of dependency: if the shortening service shuts down, every link it ever generated dies simultaneously.
That last point deserves emphasis. When a URL shortener goes dark, it doesn't just break one link. It can break millions of them, all at once, across every platform where those links were ever shared. It's a single point of failure with catastrophic blast radius.
Researchers at Harvard's Berkman Klein Center have documented link rot rates in legal citations, academic journals, and news articles consistently running between 20% and 50% over five-to-ten-year periods. A 2021 study found that roughly 25% of links in New York Times articles published between 1996 and 2019 were already dead. These aren't obscure corners of the web — these are flagship publications.
What Archivists Are Actually Doing About It
The people fighting this battle are largely volunteers running on caffeine and stubbornness.
The Wayback Machine's Save Page Now feature allows anyone to manually trigger an archive crawl of a URL — it's imperfect and crawl depth is limited, but it's the most accessible tool most users have. Browser extensions like Zotero and SingleFile let researchers and developers save local snapshots of pages they depend on. Academic institutions are increasingly building internal link-checking pipelines that automatically flag and re-archive citations before they break.
On the developer side, a growing movement around "perma.cc" style permanent citation services is gaining traction in legal and academic publishing. The concept is straightforward: instead of linking directly to a live URL, you link to a verified snapshot stored on a neutral third-party server. The original source can disappear; the citation doesn't.
Some archivists are pushing for a more radical solution — treating certain categories of web content with the same legal preservation mandates we apply to broadcast media. In the US, the FCC requires broadcasters to maintain records of their transmissions. There's a reasonable argument that major social platforms, as the dominant public record of our era, should face similar obligations.
What You Can Do Right Now
If you're a developer, a content creator, or just someone who cares about the information they publish surviving longer than a typical app's lifespan, here's a practical playbook:
Use canonical, self-hosted URLs for anything important. If you're building a product, a portfolio, or a knowledge base, own your domain. Don't build your permanent presence on a subdomain of someone else's platform.
Archive before you link. Before you drop a critical external link into documentation, a blog post, or a legal brief, run it through the Wayback Machine's Save Page Now. It takes ten seconds and creates a fallback you can reference if the original disappears.
Audit your links on a schedule. Tools like Screaming Frog, Broken Link Checker, and even simple shell scripts using curl can scan your site or documentation for dead outbound links. Make it a quarterly habit.
Think twice about URL shorteners for permanent content. Short links are fantastic for campaigns, social sharing, and tracking click data — that's literally what we do here. But if you're embedding a link in something meant to last — a whitepaper, a citation, a product README — link to the canonical long URL and archive it separately.
Back up your own content. If you publish on Medium, Substack, Reddit, or any platform you don't control, export your content regularly. Most platforms offer data export tools. Use them.
The Bigger Picture
The web's link rot crisis isn't just a technical inconvenience. It's a cultural and historical problem. The debates, discoveries, arguments, and communities that defined the first two decades of the internet are becoming unverifiable — referenced in surviving articles but unreachable through any live link. Future researchers studying this era will face gaps that no algorithm can fill.
The internet promised permanence. What it delivered was the most fragile archive in human history, maintained by volunteers, threatened by lawsuits, and deprioritized by every platform that profits from keeping users in the present tense.
We can't fix the structural economics overnight. But we can stop being passive about our own digital footprints. Archive aggressively. Own your URLs. And when you shorten a link, know what you're trading away in exchange for the convenience.
