cjkfold repairs selected look-alike CJK code points that make extracted text fail ordinary comparisons. Our sandbox install succeeded in 19 seconds, and its build and test steps passed. Python ingestion maintainers should consider it when radical characters explain known search misses. Teams seeking PDF extraction, Chinese script conversion or a demonstrated ranking improvement should choose according to those separate requirements.
The project's color example explains the problem: extracted text can contain U+2F8A KANGXI RADICAL COLOR where a query contains U+8272 色. A reader may see the same shape, while a string comparison sees different code points. Repairing that discrepancy and deciding how strongly a search system should reward literal matches are separate decisions. The cjkfold README documents both the character operation and a retrieval helper.
What you can do with cjkfold
cjkfold folds selected radicals and strokes onto equivalent ideographs, while keeping compatibility normalization and invisible-character deletion separate. That division lets an ingestion maintainer choose a specific transformation instead of enabling a broad cleanup policy. The README documents fold for mapped look-alikes, normalize for compatibility normalization with punctuation preserved by default, and strip_invisible for an explicit deletion policy. These operations answer different questions about what a stored text derivative should contain.
The bamboo mapping shows why inspecting code points matters: ⺮ maps to 𥫗, rather than the familiar standalone 竹. Grace Morgan's audit explains that the package follows an equivalence table rather than guessing from character names. It also leaves ⺀, U+2E80 CJK RADICAL REPEAT, unchanged because it has no unified ideograph equivalent. A useful first evaluation is therefore a list of actual failed character comparisons, with the expected replacement recorded for each one. Morgan's mapping analysis supplies concrete examples.
What people use cjkfold for
Use cjkfold to repair known radical-code-point mismatches in extracted text; evaluate its retrieval helper separately against labeled queries. Grace Morgan audited cjkfold 0.4.0 under Python 3.11, and the project README uses a ranking inversion to explain its retrieval helper. These examples give an ingestion maintainer two practical starting points: examine the character transformations, then decide whether a separate retrieval adjustment belongs in the application.
- Morgan's independent audit exercised transformations, scanning, command-line behavior, retrieval helpers and framework adapters. Morgan reports 166 collected tests and 166 passes at the pinned commit. Those are the independent auditor's results.
- The README ranking example discussed by Morgan reports a relevant document scoring 0.29 against a short decoy's 0.36. The author uses that inversion to motivate an exact-match bonus.
For a character-repair evaluation, collect documents whose visible text fails a known comparison and inspect their code points. For a ranking evaluation, use queries with document relevance judgments, including examples where repeated query text appears in an irrelevant document. This separates the expected substitution from the decision about search order. The README's adapters for strings and document objects provide integration routes once the intended text policy is clear.
What happened when we ran it
Our sandbox installed cjkfold's recorded repository environment in 19 seconds, built it in 1 second and completed the test step successfully in 3 seconds. These measurements describe the recorded repository environment. Our run used commit 671ab91 on 15 November 2026, with 3 CPUs, 8 GB RAM and the image label ghcr.io/astral-sh/uv:python3.12-bookworm. The repository checkout contained 34 files, about 3,594 source lines and 0.2 MB of files.
| Recorded check | Our result | What the result describes |
|---|---|---|
| Install | Succeeded in 19 seconds | Repository-environment installation |
| Installed environment | 36 dependencies recorded, 39 MB on disk | Installed lab environment |
| Build | Succeeded in 1 second | Recorded build step |
| Tests | Succeeded in 3 seconds | Recorded test command |
| pip-audit | 0 known vulnerabilities | Advisory scan of the recorded environment |
| Repository scan | 34 files, about 3,594 source lines, 0.2 MB checked out | Source checkout |
| Project signals | One CI workflow, a tests directory, no Dockerfile | Repository structure |
A separate hands-on run used a fresh, unprivileged container with 2 CPUs, 4 GB RAM, no published ports and no secrets. Its record identifies commit 671ab914d507 and the same configured image label. The task was the README's 12-line Python Usage block, run unedited. The two runs used different environments, so their setup measurements are reported separately. The smoke-run record describes that scope.
| Hands-on step | Recorded outcome | Meaning |
|---|---|---|
| Clone | 1 second | Quickstart recipe clone step completed |
| Package installation | uv pip install cjkfold installed 0.4.0 in 1.2 seconds | Package installation into a fresh virtual environment |
| Installed size | 188 KB | Recorded virtual-environment site-packages size |
| Dependency count | 0 packages besides cjkfold | Package smoke environment |
| README snippet | Exit 0 in 0.3 seconds | The supplied Usage block completed |
| Teardown | ok | Recorded cleanup completed |
The clone record, installation record, size observation and dependency observation belong to that package run. Its snippet transcript records U+2F06 mapping to U+4E8C at position 2:2 and U+2EC6 mapping to U+89D2 at position 2:4. The supplied readable transcript displays 二 beside U+2F06, so the code-point labels provide the precise distinction here. Snippet output and teardown status preserve the recorded outcomes.
Where people get stuck: checks, imports and table generation
A clean check result can coexist with a zero-width space that changes a comparison. Morgan's audit demonstrates this command distinction: check looks for mapped look-alikes, while scan reports a broader set of character classes. The README also separates findings from input errors through exit codes. Choosing the command by its purpose prevents a narrow look-alike check from becoming an accidental claim about every character in a document.
| Symptom or task | Evidence-backed action | Recorded behavior |
|---|---|---|
| check says clean, but strings differ | Inspect the broader scan report | Audit fixture contains a zero-width space |
| m.fold raises AttributeError after the former module import | Use package-level imports or cjkfold.folding | 0.4.0 separates the colliding names |
| Direct import from cjkfold.normalize fails | Change the direct module path | Release notes identify ModuleNotFoundError |
| Regenerated astral mappings contain an extra character | Use the audited checked-in table and inspect the generator | Audit reports incorrect escape rendering |
| CLI receives binary or non-UTF-8 input | Extract text or convert encoding upstream | Input refusal is deliberate |
| Existing indexes retain old strings | Reprocess affected derived data | New ingestion behavior does not rewrite stored structures |
Version 0.4.0 moved the folding functions to cjkfold.folding and the writing functions to cjkfold.rewrite. The official release notes explain that the former normalize module collided with the normalize function. Documented package-level imports retain their behavior. Direct imports from the removed module path fail with ModuleNotFoundError, while the earlier name collision produced AttributeError when a caller tried m.fold on the function.
Morgan's audit identifies a separate generator defect at commit 671ab914d507304ad9b5adef41d6606a49d0d193. The checked-in table correctly represents astral targets with eight-digit escapes, and 23 mappings target code points above U+FFFF. The generator instead emits four-digit escape syntax: its representation for U+25AD7 becomes U+25AD followed by the literal digit 7. That is one intended character becoming two. The generator analysis recommends correcting escape formatting and adding a direct rendering test.
Keep raw text before enabling destructive rewrites
strip_invisible uses a selected safe deletion policy by default, while joiners, directional controls and variation selectors require manual review. Morgan lists soft hyphen, zero-width space, word joiner and byte order mark in the safe deletion group. Deleting U+200D can break an emoji sequence; deleting U+200C can alter Persian or Indic shaping. Removing bidi controls can change visual order and erase evidence relevant to analysis. The audit's rewrite discussion explains the consequences behind the classifications.
Preserve an immutable raw extraction alongside the rewritten derivative, record the package version and policy, and count changed code points. Morgan recommends that provenance trail because normalization can erase evidence. Start with scan as a report, inspect the dispositions, and select which transformations the corpus permits. This gives a maintainer a concrete answer when someone asks whether a disputed character came from extraction, folding, compatibility normalization or invisible-character deletion.
The README assigns exit 2 to input that is binary or not UTF-8, separating it from findings. scan ordinarily exits 0 because it is a report; check exits 1 when it finds look-alikes. A job runner should preserve those distinctions. A refused PDF needs extraction upstream, and a GBK or Shift-JIS file needs explicit encoding conversion. The command-line documentation describes the reporting and input-error behavior.
Existing indexes make this a migration decision
Existing vector indexes and caches need reprocessing when their source strings change, according to Morgan's audit. Changing a shared ingestion function affects newly processed material; rebuilding affected lexical structures, embeddings and caches is separate work. Put those derived artifacts into the adoption plan before switching the canonical text policy. A retained raw corpus and a separate repaired derivative make it easier to trace which representation each downstream system received. Morgan's operational analysis sets out that migration cost.
The exact-match helper folds both query and document text internally, then supplies a bonus for literal overlap. Morgan warns that adding 0.25 is especially unsafe when a store returns distances rather than higher-is-better similarities, or when indexes use different score scales. Select score direction, bonus scale and query-fragment size against labeled retrieval examples. Those are application choices: a correct character substitution tells you which strings match, while relevance judgments tell you which documents should come first.
English documentation exists alongside Chinese background material
The README explains installation, Python usage and command-line behavior in English, and includes a Chinese summary and a longer Chinese background link. An English-speaking developer can start with the main integration instructions and follow the background account when investigating the author's extraction decisions. The project README also links corpusfold as a broader PDF-to-knowledge-base pipeline built on this character layer, placing cjkfold within a larger ingestion workflow.
The README states Python 3.9+ and zero runtime dependencies. That package requirement differs from the 36 dependencies recorded in our repository environment and agrees with the separate smoke run's observation of 0 packages besides cjkfold. For a first integration, use the documented package-level imports and make the chosen text transformation visible in your ingestion configuration and retained metadata. This connects the installation instructions to the character policy selected for the corpus.
cjkfold vs Unicode NFKC, OpenCC and CJKFoldingFilter
Choose cjkfold for mapped radical repair, NFKC for compatibility normalization, OpenCC for Chinese script or regional wording conversion, and CJKFoldingFilter for its documented folds within a Solr analyzer. The useful comparison is the rewrite policy each permits and the integration point where it belongs. Start with the actual character mismatch or conversion requirement in the corpus, then select the operation that addresses it.
| Option | Documented transformation | Pick it when |
|---|---|---|
| cjkfold | Selected radical and stroke equivalence mappings | Extracted look-alikes cause failed comparisons |
| Unicode NFKC | Unicode compatibility normalization | Compatibility normalization is the intended policy |
| OpenCC | Traditional Chinese, Simplified Chinese, Japanese Shinjitai and regional wording conversion | Script or regional wording conversion is required |
| CJKFoldingFilter | Certain CJK folds in a Lucene/Solr analysis chain | Folding belongs in the Solr analyzer |
The cjkfold README reports that NFKC repairs all 214 Kangxi Radicals, but only 2 of the 115 named CJK Radicals Supplement characters. It reports cjkfold mappings for 114 of those 115 supplement characters, with the remaining radical intentionally unchanged. Those counts describe character coverage in the project's analysis. They help explain why a specific radical may survive an NFKC pass and why an additional targeted mapping can be useful. The README coverage comparison supplies the figures.
OpenCC supports conversion between Traditional Chinese, Simplified Chinese, Japanese Shinjitai and regional wording. Its documentation describes dictionaries, a reusable library and conversion tools. CJKFilterUtils provides a Lucene filter and filter factory for certain CJK folds, including modern Japanese kanji to traditional equivalents. Its README places that filter before Traditional-to-Simplified ICU transforms. OpenCC's introduction and CJKFilterUtils' README support these distinctions without treating the tools as interchangeable APIs.
What the other reviews get wrong
AISignal's description conflates targeted folding with broader normalization and Chinese script conversion. The cjkfold README documents three separate writing operations and explicitly excludes traditional-to-simplified conversion. AISignal also describes a custom curated mapping list, whereas the README calls the table generated and provides a generation command. These differences affect both adoption and maintenance: a character-equivalence table is a different commitment from a linguistic conversion dictionary. The supplied claim comparison can be checked against the project documentation.
The supplied package listing includes a cjkfold 0.4.0 wheel, contradicting AISignal's statement that official PyPI releases do not exist. Its October 2024 adoption snapshot also conflicts with the supplied repository chronology, which dates the GitHub presence to October 2026. The piwheels listing and XERJ chronology support those specific corrections. The separate hands-on installation record also names the installed package version as 0.4.0.
FAQ
cjkfold maps selected CJK radicals and strokes to equivalent ideographs so extracted text can match the intended characters. It is a Python package under the MIT license, and the README states Python 3.9+ as its requirement. Its inspection commands and writing operations have separate purposes: a report identifies characters for inspection, while a rewrite applies the selected character policy. The README documents installation, usage and licensing.
What is cjkfold?
cjkfold maps selected CJK radicals and strokes to equivalent ideographs so comparisons can use the intended characters. Its read-only inspection tools also report compatibility characters, invisible characters and damaged-text indicators. The distinction between reporting and rewriting matters: scan can identify a character that needs human review without choosing a replacement for it. The README's character discussion explains those dispositions.
Is cjkfold free and open source, and what license does it use?
cjkfold is free and open source under the MIT license, which the project README states explicitly. The supplied XERJ account also reports that GitHub's MIT classification and the LICENSE file at its pinned commit agree. Read the actual project license when incorporating or distributing the package. The README license statement and the pinned license account support this answer.
What language is cjkfold written in?
cjkfold is written in Python, and its README states Python 3.9+ as the requirement. The documented installation command is pip install cjkfold, followed by package-level imports for the desired operations. The README includes Python examples and a command-line interface, so a developer can choose either integration route while retaining the same distinction between reporting findings and rewriting mapped characters. The installation and usage documentation supplies that starting point.
What does the src/cjkfold directory contain?
src/cjkfold contains the Python package, including folding.py, rewrite.py, diagnose.py, inputs.py, cli.py, integrations.py, score.py and the generated _table.py. The supplied live directory listing shows those files. Version 0.4.0's release notes explain the folding and rewrite module names, making them useful context when tracing imports. The package source directory and 0.4.0 release notes support this layout.
What we have not tested yet
Our recorded runs cover repository setup and one README smoke example; production throughput, sustained memory use and comparative retrieval quality remain outside those observations. The supplied audit identifies no reproducible representative retrieval benchmark. AISignal's 98.7% accuracy figure lacks supporting evaluation in this evidence, and named production adoption and business outcomes are also unresolved. The recorded repository test summary supplies no test count; Morgan's 166 passes belong to the independent audit.
The generator reproduction comes from Morgan's pinned audit, and no supplied follow-up settles whether a later change fixes it. We have not rerun that reproduction or assessed cjkfold as a Unicode security boundary. The audit excludes general cross-script confusable detection and identifier validation, and the zero-advisory result is not a security certification. The configured container image label also supplies no independent proof of actual interpreter or operating-system versions, while the smoke record's latest-version installation describes that recorded run rather than a future installation.
The indexed XERJ code snapshot trails main by 17 commits and therefore supplies historical context rather than current API proof. A useful next evaluation is to retain raw text, apply the documented character policy to known failed comparisons, and assess retrieval changes against labeled queries. Python ingestion maintainers with those specific mismatches have a concrete reason to try cjkfold; teams whose primary requirement is extraction, script conversion or Unicode security should use tools designed for those jobs.


