AIForum.

How to Detect Duplicate Titles Before Running a Bulk AI Batch

On this page

THE SHORT ANSWER

A title gate is a short similarity check that runs against your existing titles before the model is invoked, so a duplicate costs a directory read instead of a full generation. This guide covers normalization, Jaccard scoring, dual-surface comparison, threshold placement, and the cases where lexical overlap is the wrong tool entirely.
Quick facts

Duplicate titles in a bulk AI batch are cheaper to prevent than to repair. A title gate — a short similarity check that runs against your existing titles before the model is invoked — costs one directory read, while the generation it blocks can cost minutes of model time plus a full extraction, validation, and compile pass. This guide explains how to build that gate: what to normalize, which similarity score to use, where to put the thresholds, and where the approach stops working.

It assumes you run bulk content generation from a script into a CMS or static site, and that you already have a directory of published titles and slugs to compare against. The reference implementation is a Jaccard similarity gate, documented as 93 lines of dependency-free TypeScript in the Pickuma engineering write-up.

What a title gate has to catch

Duplicate detection is a spectrum, not a binary. As the Lingo4G documentation puts it, overlap can range from entire field values (exact duplicates) through almost all content (near duplicates) down to partial overlap, and the sensitivity of detection is a tunable parameter rather than a fixed property. A title gate sits at the cheap end of that spectrum: it compares short strings, not documents, and it must decide quickly.

Four collision types matter in practice:

  • Exact duplicates. The same title string, usually from a retry or a resumed batch.
  • Near-duplicates. One or two words differ — “Async Standup Tools” versus “Async Standup Tooling”.
  • Year-only variants. “The Best Async Standup Tools in 2025” versus the same title with 2026.
  • Paraphrases. Same intent, no shared nouns — “How to Run a Standup Asynchronously” versus “Async Daily Meetings Without a Live Call”.

A token-overlap score handles the first three well and the fourth badly. That distinction drives the rest of the design: use Jaccard for the cheap lexical cases, and know when to escalate.

How Jaccard similarity is computed

Jaccard similarity compares two sets of tokens. It is the size of the intersection divided by the size of the union, so a score of 1.00 means the two token sets are identical and 0.00 means they share nothing. It is symmetric, bounded between 0 and 1, and cheap enough to run against a few thousand existing titles on every batch item.

A worked example makes the year problem obvious. Compare “The Best Async Standup Tools in 2025” and “The Best Async Standup Tools in 2026” as raw strings. The token sets differ by exactly one token, so the score lands well below an abort threshold. Strip the year from both and the sets become identical: the score is 1.00, and the gate refuses the second title. That is the intended outcome, because a year-only difference calls for a changelog entry rather than a second URL.

Normalize before you compare

Three normalization steps do most of the work:

  1. Lowercase. Title case and sentence case should not read as different content.
  2. Split on non-alphanumeric characters. Punctuation, hyphens, and colons become separators rather than tokens.
  3. Strip year tokens. Tokens matching /^20dd$/ are removed before scoring.

The year rule is not cosmetic. In the documented corpus, 78 of 274 published titles carried a four-digit year, and leaving the year in the comparison raised the score of one unrelated pair from 0.09 to 0.17. That is noise sitting directly under the warning threshold, and it eats the headroom you need to separate a genuine near-duplicate from coincidence.

There is a tradeoff to inspect before you copy the filter. The common length > 2 rule that drops very short tokens also removes short but meaningful terms — two-letter product names, acronyms, and abbreviations that may be the whole subject of a title. If your niche depends on those, test what the filter discards on your own corpus rather than assuming it is harmless.

Score the title and the slug, then take the higher value

A single comparison surface is easy to defeat. A generator that rewords the title while keeping the same slug, or the reverse, slips past a title-only check. The documented gate therefore scores both and uses the higher of the title comparison and the slug comparison as the item’s score. Slugs are already lowercased and hyphen-delimited, so tokenizing them is trivial, and a slug collision often reveals a duplicate that a lightly reworded title conceals.

Choose a warning threshold and an abort threshold

The documented configuration aborts a topic when similarity against any existing title reaches 0.50 and prints a warning at 0.35. The two-level design matters more than the exact numbers. An abort stops the item outright; a warning surfaces a borderline pair for a human to judge. Collapsing both into a single threshold forces you to choose between blocking legitimate work and letting duplicates through.

Treat 0.50 and 0.35 as starting points, not defaults. They were selected for one corpus and one failure mode — a generator emitting near-identical titles within a single batch. A corpus with long, keyword-dense titles will produce a different score distribution than one with short titles, and the only reliable way to set a threshold is to measure your own pairs.

Run the check twice, ahead of the expensive call

The gate runs twice: once on the topic string before any prompt is assembled, and again on the title the model produced. Both checks sit ahead of the model invocation, the JSON extraction, the schema validation, and the MDX compile.

Ordering is the entire return on investment. In the documented setup the model call takes roughly six minutes; a duplicate caught at the topic stage never reaches the model, and a duplicate caught at the title stage never reaches the compile. The gate itself is a directory read against that six-minute generation, so it pays for itself on the first blocked item. Running the check only after generation throws away the cheaper of the two saves.

A minimal implementation

The following JavaScript is an illustrative implementation of the same rules — normalization, year stripping, dual-surface scoring, and two thresholds. It is written to be readable rather than to match any specific production file.

const YEAR = /^20dd$/;

function tokenize(text) {
  return text
    .toLowerCase()
    .split(/[^a-z0-9]+/)
    .filter((t) => t.length > 2 && !YEAR.test(t));
}

function jaccard(a, b) {
  const A = new Set(tokenize(a));
  const B = new Set(tokenize(b));
  if (A.size === 0 && B.size === 0) return 1;
  let shared = 0;
  for (const t of A) if (B.has(t)) shared++;
  const union = A.size + B.size - shared;
  return union === 0 ? 0 : shared / union;
}

function worstMatch(candidate, existing) {
  let worst = { score: 0, row: null };
  for (const row of existing) {
    const score = Math.max(
      jaccard(candidate.title, row.title),
      jaccard(candidate.slug, row.slug)
    );
    if (score > worst.score) worst = { score, row };
  }
  return worst;
}

const ABORT_AT = 0.50;
const WARN_AT = 0.35;

const { score, row } = worstMatch(candidate, existingTitles);
if (score >= ABORT_AT) throw new Error(`Duplicate title (${score.toFixed(2)}): ${row.title}`);
if (score >= WARN_AT) console.warn(`Near-duplicate title (${score.toFixed(2)}): ${row.title}`);

The function returns the worst match and its score. Wire it into the topic stage before prompt assembly and into the title stage after parsing the model’s JSON, and let both paths share the same existingTitles array so the two checks cannot drift apart.

Instrument the gate separately from everything else

You cannot tune a threshold you cannot measure. In the documented setup, duplicate aborts and MDX compile failures increment the same failed counter, which means the gate’s block rate and false-positive rate are unknown. That is a measurement problem before it is a tuning problem.

Add a distinct counter for gate rejections, split by stage (topic versus generated title) and by outcome (abort versus warning), and log the score and the matched title alongside it. Once the counters are separate, a rising abort rate tells you something about the generator, and a rising compile-failure rate tells you something about the output format. While they share a counter, both signals are lost.

Verify the gate on your own corpus before enforcing it

  1. Export every published title and slug into a single file.
  2. Run the scorer in report-only mode over the existing set, logging every pair above the warning threshold.
  3. Read the logged pairs and label each one as a true duplicate, an acceptable variation, or a false positive.
  4. Adjust the thresholds against those labels, then enable enforcement.
  5. Re-run the report after the first enforced batch and confirm the abort count matches what you expect from the labels.

Step three is the one people skip. Without labels, a threshold change is a guess dressed up as a configuration value.

When Jaccard is the wrong tool

Lexical overlap has a hard ceiling. Paraphrased titles that share no nouns score near zero, so a Jaccard gate will pass them no matter where you set the threshold. If your near-duplicates look like that, the gate is not under-tuned — it is the wrong instrument, and you need embeddings, which compare meaning rather than surface tokens.

A practical arrangement is to keep Jaccard as the fast first pass and add an embedding check for the band between the warning and abort thresholds, where lexical evidence is ambiguous. That keeps the cheap check in front of the expensive one, which is the same ordering logic that puts the gate ahead of the model call.

What to do next

Start in report-only mode with the 0.35 warning and 0.50 abort values as placeholders, log every pair above the warning, and label enough of them to see whether the distribution in your corpus resembles the one those numbers were chosen for. Separate your gate counter from your compile-failure counter in the same pass. If the labeled pairs show paraphrase collisions rather than token collisions, stop tuning and plan the embedding layer instead.

If you have run a title gate over a real batch, the community would benefit from the details: your corpus size, the thresholds you settled on, and the false positives you accepted.

FAQ

Does a Jaccard title gate catch paraphrased titles?

No. Jaccard is a lexical measure, so paraphrases with no shared nouns score near zero and pass the gate regardless of where you set the threshold. Those cases need embeddings instead.

Why strip year tokens before scoring?

Two reasons. Years add noise to unrelated comparisons — in the documented corpus, leaving one in raised an unrelated pair from 0.09 to 0.17 — and stripping them makes year-only variants score 1.00 and abort, which is the desired result since a year-only difference calls for a changelog entry rather than a second URL.

Where in the pipeline should the check run?

Twice: on the topic string before any prompt is assembled, and again on the title the model produced. Both checks should sit ahead of the model invocation, JSON extraction, schema validation, and compile step.

Can I tune the 0.50 and 0.35 thresholds immediately?

Not reliably. In the documented setup, duplicate aborts and MDX compile failures increment the same counter, so the gate's block and false-positive rates are unknown. Add a separate gate-rejection counter, then run in report-only mode and label real pairs before changing thresholds.

References

  1. https://pickuma.com/for-dev/jaccard-duplicate-title-gate-year-tokens/
  2. https://get.carrotsearch.com/lingo4g/latest/doc/v2/duplicate-detection/

Related terms

Jaccard similarity
A set-overlap measure: the size of the intersection of two token sets divided by the size of their union, giving a score from 0 to 1.
Token
A single unit produced by splitting a string — typically a word or number — that the similarity score compares.
Slug
The URL-friendly form of a title, usually lowercased and hyphen-delimited, used as a second comparison surface.
Near-duplicate
Two items whose content overlaps almost entirely but is not byte-identical, such as titles differing by one word.
Embedding
A numeric vector representation of text that allows comparison by meaning rather than by shared tokens, useful when paraphrases share no nouns.

Leave a Reply

Your email address will not be published. Required fields are marked *