ai-coding-minesIndexGitHub

Korean / CJK substring matching: short words swallow long ones

Python and databases

Symptom

A short Korean token registered as a filter swallows unrelated longer words.

마블 (Marvel) ⊂ 마블링 (marbling) · 쇼핑 (shopping) · 코브라 (cobra) ⊂ a toy product name.

Cause (a real trade-off)

Why this bites non-ASCII users

\b is an ASCII-era idea. In Korean, Japanese, and Chinese the space is optional, and product listings drop it aggressively to save characters. There is no boundary for the regex engine to find. Every off-the-shelf keyword filter assumes there is one.

Fix (exception list, not a rule change)

Leave the boundary rule alone and add true-positive protection exceptions: if every occurrence of the term is inside an exception word's span, ignore it; if any occurrence is outside, it's a real hit. Concatenated true positives (몽클레르패딩) survive.

Adding an exception is one line in a list; the matching rule itself is untouched.

In a first-match rule table, a generic pattern swallows compounds

A rule table mapped item words in English product names to Korean category words. pen → 만년필 (fountain pen) captured Tactical Pen, so a tactical pen became a fountain pen; mount → 거치대 (holder) captured Rail-mounted, so a rail-mounted product became a holder. Same swallowing, other direction: above, a short token swallows a long word; here a generic pattern swallows a compound.

Put specific patterns before generic ones. In a first-match table, order *is* priority. Before adding a row, ask "which existing rows does this pattern swallow?"

Why this bites non-ASCII users

The table exists because the source names are English and the buyers search in Korean — a translation table is what turns a Latin-script listing into something a CJK search index will hit. Any shop localising into a CJK market runs a table like this, and the first-match trap ships with it.