CS Fundamentals Trainer · Guides

Persistent identifiers and link rot

Why a URL fails as a name, how a DOI is built and resolved, normalising and checking identifiers (measured: ISBN-13 and ORCID catch typos, DOIs cannot), why a 301 breaks updates in a real browser, content negotiation, and rules for designing your own identifiers.

19 min read

A web address does two jobs at once: it names a thing ("this paper") and it says where that thing is today ("this server, this path"). That works until the location changes. A university moves its website, a publisher changes platform, a company is acquired, and every link that used the old location now leads to an error page. Persistent identifiers are the fix: a name that stays the same while the location behind it is allowed to change. The DOI (Digital Object Identifier) on a scientific paper is the best-known example, and the ideas apply to any system that hands out names people will keep for years. This guide explains the problem, how a DOI is built and resolved, why check digits matter (and why DOIs have none), and what to copy when you design identifiers of your own. The companion guide, Design a DOI resolver, builds the service that answers the lookups.

1. The problem: a name that is also an address

Three words describe how links fail:

• Link rot: the address no longer answers (404, domain gone, server shut down).
• Content drift: the address still answers, but the page now says something different from what was cited.
• Reference rot: either of the two, seen from the reader's side: you cannot get back to what the author meant.

How common is it? A large study of scholarly articles (Klein and colleagues, 2014, over 1 million references to web resources taken from over 3.5 million articles) found that one out of five articles in science, technology and medicine suffered from reference rot, and about seven out of ten among the articles that cite web resources at all [1]. Those numbers are for web references, not for DOIs, and they describe 2014; treat them as the order of magnitude of the problem, not a current rate.

Tim Berners-Lee's advice, "Cool URIs don't change" [2], is correct, and it is also what fails in practice: organisations reorganise, and a path that was a good name last year becomes a broken link. The robust answer is not to promise that addresses will never change, but to stop using the address as the name.

2. The idea: separate the name from the location

Add one level of indirection. Publish and cite a stable name; keep a table that maps the name to the current location; when the location changes, update one row of the table, and every citation keeps working.

Diagram: A stable name, resolved to a location that can move

Indirection moves the problem; it does not remove it. Someone has to keep the table current. A DOI that points to a dead page is still a broken link, and only the organisation that registered it can fix that, so persistence is a social and contractual commitment backed by software, not a property of the string. What the software buys you is that the fix is one update at one place, instead of asking every citing author to edit every document.

3. How a DOI is built

A DOI has two parts separated by the first slash:

10.1145 / 362686.362692
└──┬──┘   └─────┬──────┘
 prefix      suffix

• The prefix starts with 10. and is followed by a number that identifies the registrant, the organisation that registered the name (here 1145 is the Association for Computing Machinery). It is assigned by a registration agency.
• The suffix is chosen by the registrant. It can be a counter, an opaque string, or an older style that embeds other identifiers. It may contain characters such as parentheses, angle brackets and semicolons.
• DOIs are case-insensitive: 10.1145/ABC and 10.1145/abc are the same name.

Underneath, the DOI system uses the Handle System (RFC 3650 to 3652 describe its architecture, namespace and protocol [3]): a global service that resolves a handle by routing its prefix to the service that holds that prefix's records. DOI is a standard in its own right (ISO 26324). Different registration agencies run the registrations for different kinds of content; Crossref registers most scholarly articles and books, and DataCite is the agency commonly used for datasets and software. The agency also keeps the metadata (title, authors, date, license) next to the location.

For display, the convention is to show a DOI as a full link, https://doi.org/10.1145/362686.362692, so that it is clickable and unambiguous. The bare 10.… form is for databases.

4. Normalising: one key for each DOI

Anything that stores or compares DOIs (a bibliography tool, a duplicate check, a resolver) needs one canonical key. People type and paste DOIs in many shapes: with https://doi.org/, with the old dx.doi.org host, with doi:, in upper case, percent-encoded, with a full stop from the end of a sentence stuck on. The rule is to strip the wrapper, decode, lower-case, and check the shape:

import re
from urllib.parse import unquote

DOI = re.compile(r"^10\.\d{4,9}/\S+$")          # a loose shape check, not a validity check

def normalise_doi(s):
    s = s.strip()
    s = re.sub(r"^(?:https?://(?:dx\.)?doi\.org/|doi:\s*)", "", s, flags=re.I)
    s = unquote(s)
    return s.lower() if DOI.match(s) else None

I ran it on these inputs:

Two cautions. First, the trailing punctuation problem is real: when you extract DOIs from running text, a final . or ) may or may not belong to the suffix, because suffixes can legitimately end in both. Strip a trailing ., , or ; when the text is prose, and when in doubt look the candidate up instead of guessing. Second, lower-casing is right for comparison; for display keep what the registrant published.

5. Checking identifiers: check digits, and why DOIs have none

A check digit is a character computed from the others, so that a typo makes the check fail. ISBN-13 uses a weighted sum modulo 10; the ORCID iD (the identifier researchers use) ends in a character computed with ISO 7064 MOD 11-2, where the value 10 is written X [4]. The ORCID algorithm in code:

def orcid_check_digit(first15):
    total = 0
    for ch in first15:
        total = (total + int(ch)) * 2
    r = (12 - total % 11) % 11
    return "X" if r == 10 else str(r)

def orcid_ok(s):
    s = s.replace("-", "")
    return len(s) == 16 and s[:15].isdigit() and orcid_check_digit(s[:15]) == s[15]

def isbn13_ok(s):
    d = [int(c) for c in s if c.isdigit()]
    return len(d) == 13 and sum(x * (1 if i % 2 == 0 else 3) for i, x in enumerate(d)) % 10 == 0

The known examples pass (978-0-306-40615-7, and ORCID's documented test identity 0000-0002-1825-0097) and the same strings with the last digit changed fail. Then I asked how well each scheme notices mistakes, with 100,000 random identifiers each:

Both catch every single-digit error. Swapping two neighbours is where they differ: in ISBN-13 a swap of two digits that differ by 5 leaves the sum unchanged (the weights are 1 and 3, so the change is a multiple of 10), so roughly one swap in ten slips through (9% in my run); MOD 11-2 catches every swap.

A DOI has no check digit. Its suffix is whatever the registrant chose, so there is nothing to compute. The shape check above accepted 20,000 of 20,000 single-digit changes to a real DOI: a typo produces another syntactically valid name, and syntax cannot tell it from a different paper. The only reliable check is to resolve it (ask the resolver whether the name exists) or look up its metadata and compare the title. That is also why a copy-paste of a DOI into a manuscript is safer than retyping it.

Design lesson for your own identifiers: if people will type or read them aloud (an order number, a tracking code, a membership id), add a check character; the cost is one character and it removes most typos. If machines move them around (links, API ids), a check digit adds little and a lookup is the real validation. A check digit is not a security feature: anyone can compute it.

6. The resolver must not be cached for ever

A resolver answers with a redirect. Which redirect matters, because browsers and intermediaries cache some of them. I started a small server that redirected /r/x to /a, visited it in a real Chromium, moved the target to /b, and visited the same address again in the same browser:

A permanent redirect (301, 308) is cacheable by default and, with no freshness information, browsers keep it for as long as they like [5]. For a mapping that is supposed to change (that is the whole point of a persistent identifier), a cached permanent redirect turns the resolver's promise upside down: readers who visited once keep going to the old place after the publisher moved. So a resolver answers with a temporary redirect (302 or 307) and chooses its caching on purpose with Cache-Control: a short lifetime trades a little staleness for far fewer requests.

7. One name, several representations: content negotiation

The same DOI can be asked for in different formats. A browser sends Accept: text/html and is redirected to the publisher's page; a reference manager sends another Accept value and gets metadata instead. The Crossref documentation lists these values for requests sent to the DOI resolver [6]:

curl -L -H "Accept: application/vnd.citationstyles.csl+json" https://doi.org/10.1145/362686.362692
curl -L -H "Accept: application/x-bibtex" https://doi.org/10.1145/362686.362692

The -L matters: the resolver works out which registration agency holds the record and redirects the request there, so a client that does not follow redirects sees only the redirect.

One HTTP detail to remember: when the same URL gives different answers depending on a request header, caches must be told, with Vary: Accept, or a shared cache will serve BibTeX to a browser. The caching strategies guide covers Vary.

8. Other ways to name things

Two different strategies hide in that table. Indirection (DOI, ORCID) keeps the name stable and updates where it points. Preservation (an archive snapshot) keeps a copy of the content itself, so the citation survives even if the source disappears. They solve different failures: indirection cannot help if the publisher goes out of business and nobody updates the record; a snapshot cannot help if you want the live, current version. Careful scholarly writing uses both.

9. Designing identifiers for your own system

If you are going to hand out names that people store (in documents, bookmarks, spreadsheets, other systems), these rules save you from regret:

1. Make the name opaque. Do not put the title, the owner, the folder, the date or the status in it. Anything meaningful will one day be wrong (the title changes, the owner leaves), and the identifier cannot change.
2. Never reuse an identifier, even after deleting what it named. Someone's old link must not open a stranger's document.
3. Separate the name from the location with a mapping you control, and make the mapping updatable by the owner with an audit trail.
4. Decide on case and character rules up front (case-insensitive, an alphabet without look-alikes such as 0 and O) and normalise at the edge.
5. Add a check character if humans will type it.
6. Keep "gone" a real answer. When the thing is deleted, keep the name and answer with a tombstone page ("this was removed on …, here is why") and a 410, not a 404. DOI-style systems typically keep the record and the metadata even when the content is gone.
7. Decide what a new version means. Either the name always points to the latest version, or every version gets its own name plus one name that tracks the latest. Some repositories, such as Zenodo, give each version its own DOI and one more DOI that always points to the newest, so both citing a precise version and citing "the software" work.
8. Write down the policy: who may change a mapping, how fast, and what happens to the identifiers if your organisation stops existing.

The shortener in Design a URL shortener is the lightweight cousin of this: a short code mapped to a long URL. The difference is the promise: a shortener is allowed to rot; a persistent identifier service exists to keep that promise.

10. Common mistakes

• Using the page address as the identifier and promising that it will not change.
• Putting meaning in the name (/2024/physics/smith-thesis), then reorganising.
• Answering a mutable mapping with a 301 or 308. Clients cache it and never see the update (section 6).
• Treating a syntactically valid DOI as a real one. Resolve it or look up its metadata.
• Comparing DOIs as raw strings. Case, wrappers and percent-encoding produce false "different" results.
• Re-using identifiers after deletion.
• Deleting the record when the content disappears, so the name now means nothing instead of "removed".
• Assuming the registrant will keep the mapping current. Monitor targets and chase dead ones.

11. Questions interviewers like to add

"Why not just use URLs?"

A URL fuses name and location, so any move breaks every citation. A persistent identifier puts a table between them so a move is one update. The cost is that someone must keep the table current, which is an organisational commitment, not code.

"Why does the resolver answer with a 302 and not a 301?"

Because the mapping is meant to change. Browsers and proxies may cache a 301 indefinitely (in my test, a visit after the change still went to the old place and never reached the resolver), so readers would stay stuck on the old location. A temporary redirect, with a deliberate Cache-Control, lets updates reach people.

"How would you detect a typo in an identifier?"

Add a check character if people type it (ISBN, ORCID). For an identifier without one, like a DOI, you can only check the shape and then resolve it. I measured shape checks accepting every single-digit change.

"What happens to a DOI when the publisher disappears?"

Nothing happens by itself: the name still resolves to the last known location, which may be dead. In practice, agencies require registrants to keep records current, and may point the name to a tombstone page; the metadata can live on. This is why archives and publishers' succession arrangements matter.

"How would you store DOIs in a database?"

Store the normalised, lower-case key in a unique column (and the original spelling beside it for display), index it, and put the registration agency and the last verified target next to it. Compare keys, never raw strings.

"Where would you use content negotiation?"

Wherever one name has several representations and a machine, not a person, is asking: citation exports, API vs HTML for the same resource, language variants. Remember Vary for caches.

12. Things to mention in an interview

• Name and location are two things; link rot, content drift and reference rot are the failures.
• Indirection (a mapping table) fixes the technical half; persistence is also a commitment.
• DOI structure: prefix, registrant, opaque suffix, case-insensitive, resolved through the Handle System by an agency.
• Normalise before you compare or store.
• Check digits (ISBN, ORCID) versus none (DOI); resolve to validate.
• 302/307 for mutable mappings; 301/308 get cached.
• Content negotiation and Vary: Accept.
• Opaque, never reused, tombstones, versioning policy.

Quick recap

• A URL is both name and address; moving it breaks every link. About one in five scholarly articles had reference rot in one large 2014 study.
• A persistent identifier separates the stable name from the current location; one table update fixes all citations, provided someone maintains it.
• A DOI is 10.<registrant>/<suffix>, case-insensitive, with no check digit: validate by resolving it.
• Normalise (strip wrappers, decode, lower-case) before storing or comparing.
• ISBN-13 and ORCID have check characters: both caught 100% of single-digit errors, and ORCID caught 100% of adjacent swaps against 91% for ISBN-13 in my test.
• Resolvers redirect with 302 or 307: browsers cached a 301 and 308 and kept sending readers to the old target.
• One DOI can return a web page or machine-readable metadata, chosen by the Accept header.

Appendix: the experiment code

Python 3 standard library plus Playwright for the redirect test (pip install playwright, playwright install chromium). Timings and counts come from my run; the random tests use fixed seeds.

ids.py: check digits, error detection and DOI normalisation:

import random, re
from urllib.parse import unquote

# --- check digits ---
def isbn13_ok(s):
    d=[int(c) for c in s if c.isdigit()]
    return len(d)==13 and sum(x*(1 if i%2==0 else 3) for i,x in enumerate(d))%10==0
def isbn10_ok(s):
    ch=[c for c in s.upper() if c.isdigit() or c=="X"]
    if len(ch)!=10: return False
    v=[10 if c=="X" else int(c) for c in ch]
    return sum((10-i)*x for i,x in enumerate(v))%11==0
def orcid_check_digit(first15):
    total=0
    for ch in first15: total=(total+int(ch))*2
    r=(12-total%11)%11
    return "X" if r==10 else str(r)
def orcid_ok(s):
    s=s.replace("-","")
    return len(s)==16 and s[:15].isdigit() and orcid_check_digit(s[:15])==s[15]

print("known good:", isbn13_ok("978-0-306-40615-7"), isbn10_ok("0-306-40615-2"), orcid_ok("0000-0002-1825-0097"), "| known bad:", isbn13_ok("978-0-306-40615-8"), isbn10_ok("0-306-40615-3"), orcid_ok("0000-0002-1825-0098"))

# --- how often does each scheme notice a typo? ---
rnd=random.Random(1)
def rand_isbn13():
    d=[9,7,8]+[rnd.randrange(10) for _ in range(9)]
    c=(10-sum(x*(1 if i%2==0 else 3) for i,x in enumerate(d))%10)%10
    return "".join(map(str,d+[c]))
def rand_orcid():
    f="".join(str(rnd.randrange(10)) for _ in range(15)); return f+orcid_check_digit(f)
def mutate_sub(s,digits_only=True):
    i=rnd.randrange(len(s)); alt=[c for c in "0123456789" if c!=s[i]]; return s[:i]+rnd.choice(alt)+s[i+1:]
def mutate_swap(s):
    while True:
        i=rnd.randrange(len(s)-1)
        if s[i]!=s[i+1]: return s[:i]+s[i+1]+s[i]+s[i+2:]
N=100_000
for name,gen,ok in (("ISBN-13",rand_isbn13,isbn13_ok),("ORCID iD",rand_orcid,orcid_ok)):
    ids=[gen() for _ in range(N)]
    sub=sum(not ok(mutate_sub(x)) for x in ids)/N; sw=sum(not ok(mutate_swap(x)) for x in ids)/N
    print(f"{name}: one wrong digit noticed {100*sub:.2f}% | two adjacent digits swapped noticed {100*sw:.2f}%  ({N} random ids each)")

# --- DOIs have no check digit: only the syntax can be checked ---
DOI=re.compile(r"^10\.\d{4,9}/\S+$")
def normalise_doi(s):
    s=s.strip()
    s=re.sub(r"^(?:https?://(?:dx\.)?doi\.org/|doi:\s*)","",s,flags=re.I)
    s=unquote(s)
    return s.lower() if DOI.match(s) else None
dois=["10.1145/362686.362692","10.1145/509907.509965","10.1016/S0169-7552(98)00110-X","10.1371/journal.pone.0115253","10.1002/(SICI)1097-4571(199806)49:8<693::AID-ASI4>3.0.CO;2-O"]
d0="10.1145/362686.362692"
pos=[i for i,c in enumerate(d0) if i>=3 and c.isdigit()]      # digits after "10." : registrant and suffix
acc=0
for _ in range(20_000):
    i=rnd.choice(pos); alt=rnd.choice([c for c in "0123456789" if c!=d0[i]]); acc+=bool(DOI.match(d0[:i]+alt+d0[i+1:]))
print(f"syntax accepts {acc} of 20000 single-digit changes to {d0} (it cannot tell a typo from another DOI)")
for t in ["https://doi.org/10.1145/362686.362692","HTTP://DX.DOI.ORG/10.1145/362686.362692","doi: 10.1145/362686.362692","10.1016/S0169-7552(98)00110-X","https://doi.org/10.1016/s0169-7552%2898%2900110-x","not a doi","10.12/short"]:
    print(f"  {t!r:60} -> {normalise_doi(t)!r}")

redir.py: which redirects a real browser caches after the target moves:

import http.server, socketserver, threading
from playwright.sync_api import sync_playwright
TARGET={"url":"/a"}; HITS=[]; CFG={"code":301,"cc":None}
class H(http.server.BaseHTTPRequestHandler):
    def log_message(self,*a): pass
    def do_GET(self):
        HITS.append(self.path)
        if self.path.startswith("/r/"):
            self.send_response(CFG["code"]); self.send_header("Location",TARGET["url"])
            if CFG["cc"]: self.send_header("Cache-Control",CFG["cc"])
            self.send_header("Content-Length","0"); self.end_headers(); return
        b=f"landed on {self.path}".encode(); self.send_response(200); self.send_header("Content-Length",str(len(b))); self.end_headers(); self.wfile.write(b)
class S(socketserver.ThreadingMixIn,http.server.HTTPServer): daemon_threads=True
srv=S(("127.0.0.1",8771),H); threading.Thread(target=srv.serve_forever,daemon=True).start()
print("status | Cache-Control | 2nd visit after the target moved | did the resolver see the 2nd visit?")
with sync_playwright() as p:
    br=p.chromium.launch()
    for code,cc in [(301,None),(308,None),(302,None),(307,None),(301,"no-store"),(301,"max-age=60"),(302,"max-age=60")]:
        ctx=br.new_context(); pg=ctx.new_page(); CFG.update(code=code,cc=cc)
        key=f"/r/{code}{cc or ''}".replace("=","").replace("-","")
        TARGET["url"]="/a"; HITS.clear(); pg.goto("http://127.0.0.1:8771"+key); first=pg.url.rsplit("/",1)[1]
        TARGET["url"]="/b"; HITS.clear(); pg.goto("http://127.0.0.1:8771"+key); second=pg.url.rsplit("/",1)[1]; seen=any(h==key for h in HITS)
        print(f"{code}    | {cc or '(none)':11} | went to /{second} (should be /b) | {'yes' if seen else 'NO, answered from the browser cache'}")
        ctx.close()
    br.close()

Sources

1. Martin Klein, Herbert Van de Sompel, Robert Sanderson and others, Scholarly Context Not Found: One in Five Articles Suffers from Reference Rot, PLOS ONE, 2014 (DOI 10.1371/journal.pone.0115253). The figures quoted are from its abstract.
2. Tim Berners-Lee, Cool URIs don't change, W3C, 1998.
3. IETF, RFC 3650: Handle System Overview, with RFC 3651 (namespace and service definition) and RFC 3652 (protocol). The DOI system is built on the Handle System.
4. ORCID, Format of the ORCID iD. The structure and the ISO 7064 MOD 11-2 check character; the code above implements its algorithm.
5. IETF, RFC 9110: HTTP Semantics (redirect status codes) and RFC 9111: HTTP Caching.
6. Crossref, Content negotiation (supported Accept values), and DOI content negotiation.
7. Crossref, DOI display guidelines (show DOIs as full https://doi.org/… links).

Practise this topic: Networking System Design

More guides