mrkeyoor.comAsk

cdcailf/cjkfoldtested 15 Nov 2026commit 671ab91

cjkfold review

Use cjkfold when selected CJK radical code points explain failed comparisons in a Python ingestion pipeline.

  • README install 1.2 s
  • 188 KB on disk
19 sto install
  • 0 dependencies
  • Build 1 s
Dev Toolsevaluationupdated 15 Nov 2026
+0stars / 7d

cjkfold is a Python library that folds selected look-alike CJK radicals and strokes onto equivalent ideographs. It accepts extracted text and keeps compatibility normalization and invisible-character deletion as separate operations.

Verdict

Use cjkfold when selected CJK radical code points explain failed comparisons in a Python ingestion pipeline. Our recorded repository install, build and test steps passed, and the separate README smoke example completed successfully. Teams seeking PDF extraction, traditional-to-simplified conversion or general Unicode security checks should choose tools for those requirements. Keep table regeneration tied to the pinned audit's defect report and evaluate ranking policy separately.

cjkfold infographic: repair selected CJK code points in extracted text, plus our test result and who it suits
Tested 15 November 2026: the README snippet ran as written. Before that: source install 19 s, build 1 s; uv install took 1.2 s.

We ran it

Lab card: what happened when we ran cjkfold
Lab card from our sandbox run
Screenshot of cjkfold (github.com/cdcailf/cjkfold)
github.com/cdcailf/cjkfold
Install✓ · 19s36 packages · 39 MB
Build✓ · 1s
Tests✓ · 3sran, no count parsed
Known vulns0(pip-audit)
Repo34 files~3,594 lines of source · 0.2 MB · 1 CI workflows · tests dir

Answers from our run

Does cjkfold build from source?

Dependencies installed in 19 seconds (36 packages), and the build succeeded in 1 seconds. We cloned commit 671ab91 into a clean Debian container with 3 CPUs and no project-specific setup.

Do cjkfold's tests pass?

Yes. The project's own test command exited successfully in 3 seconds, although its output did not expose an individual test count.

Does cjkfold have known vulnerabilities in its dependencies?

pip-audit found none in the dependency tree at the time of our run.

Who should not use cjkfold?

Teams seeking PDF extraction or OCR.

What are the alternatives to cjkfold?

Unicode NFKC via Python unicodedata, OpenCC, CJKFoldingFilter. Use cjkfold when selected CJK radical code points explain failed comparisons in a Python ingestion pipeline.

Setup5/5In our 2026-11-15 run it installed as the README says in 1.2 s and the README’s “Usage” example ran unedited.Editor’s note: The recorded setup steps passed in one repository environment; broader platform setup coverage is absent.
Docs5/5The README has an install section (“Install”) and a usage section (“Usage”), and its example ran as written in our 2026-11-15 run.Editor’s note: The supplied README covers API and CLI workflows, but the record contains no complete documentation assessment.
Community2/5Last push on 2026-10-15 and 201 stars.Editor’s note: The supplied records do not measure support responsiveness, maintainer capacity or production adoption.
Maturity2/5Latest release v0.4.0 on 2026-10-15 (pre-1.0), repository created 2026-10-13 and its test suite passed in our sandbox.Editor’s note: Passing recorded steps and a pinned generator defect do not supply a complete maturity assessment.

How we score

Who it’s for

Python developers tracing failed text comparisons to CJK radical code points.
Ingestion maintainers who want character reports before selecting a rewrite policy.
Search teams evaluating character repair separately from ranking changes.

Who it’s NOT for

Teams seeking PDF extraction or OCR.
Developers seeking traditional-to-simplified Chinese conversion.
Security teams seeking general mixed-script confusable detection or identifier validation.

Setup reality

Our sandbox install succeeded in 19 seconds, with 36 dependencies recorded and 39 MB on disk. The build succeeded in 1 second and the test step succeeded in 3 seconds. pip-audit reported 0 known vulnerabilities. These measurements describe the recorded repository environment at commit 671ab91 on 15 November 2026.

The repository sandbox configuration recorded 3 CPUs, 8 GB RAM and image label ghcr.io/astral-sh/uv:python3.12-bookworm. A separate package smoke run used 2 CPUs and 4 GB RAM; its measurements appear separately in the run section.

The README documents pip installation and package-level imports. The core workflow takes extracted text, with extraction, OCR and retrieval infrastructure supplied by the surrounding application.

cjkfold repairs selected look-alike CJK code points that make extracted text fail ordinary comparisons. Our sandbox install succeeded in 19 seconds, and its build and test steps passed. Python ingestion maintainers should consider it when radical characters explain known search misses. Teams seeking PDF extraction, Chinese script conversion or a demonstrated ranking improvement should choose according to those separate requirements.

The project's color example explains the problem: extracted text can contain U+2F8A KANGXI RADICAL COLOR where a query contains U+8272 色. A reader may see the same shape, while a string comparison sees different code points. Repairing that discrepancy and deciding how strongly a search system should reward literal matches are separate decisions. The cjkfold README documents both the character operation and a retrieval helper.

What you can do with cjkfold

cjkfold folds selected radicals and strokes onto equivalent ideographs, while keeping compatibility normalization and invisible-character deletion separate. That division lets an ingestion maintainer choose a specific transformation instead of enabling a broad cleanup policy. The README documents fold for mapped look-alikes, normalize for compatibility normalization with punctuation preserved by default, and strip_invisible for an explicit deletion policy. These operations answer different questions about what a stored text derivative should contain.

The bamboo mapping shows why inspecting code points matters: ⺮ maps to 𥫗, rather than the familiar standalone 竹. Grace Morgan's audit explains that the package follows an equivalence table rather than guessing from character names. It also leaves ⺀, U+2E80 CJK RADICAL REPEAT, unchanged because it has no unified ideograph equivalent. A useful first evaluation is therefore a list of actual failed character comparisons, with the expected replacement recorded for each one. Morgan's mapping analysis supplies concrete examples.

What people use cjkfold for

Use cjkfold to repair known radical-code-point mismatches in extracted text; evaluate its retrieval helper separately against labeled queries. Grace Morgan audited cjkfold 0.4.0 under Python 3.11, and the project README uses a ranking inversion to explain its retrieval helper. These examples give an ingestion maintainer two practical starting points: examine the character transformations, then decide whether a separate retrieval adjustment belongs in the application.

  • Morgan's independent audit exercised transformations, scanning, command-line behavior, retrieval helpers and framework adapters. Morgan reports 166 collected tests and 166 passes at the pinned commit. Those are the independent auditor's results.
  • The README ranking example discussed by Morgan reports a relevant document scoring 0.29 against a short decoy's 0.36. The author uses that inversion to motivate an exact-match bonus.

For a character-repair evaluation, collect documents whose visible text fails a known comparison and inspect their code points. For a ranking evaluation, use queries with document relevance judgments, including examples where repeated query text appears in an irrelevant document. This separates the expected substitution from the decision about search order. The README's adapters for strings and document objects provide integration routes once the intended text policy is clear.

What happened when we ran it

Our sandbox installed cjkfold's recorded repository environment in 19 seconds, built it in 1 second and completed the test step successfully in 3 seconds. These measurements describe the recorded repository environment. Our run used commit 671ab91 on 15 November 2026, with 3 CPUs, 8 GB RAM and the image label ghcr.io/astral-sh/uv:python3.12-bookworm. The repository checkout contained 34 files, about 3,594 source lines and 0.2 MB of files.

Recorded check Our result What the result describes
Install Succeeded in 19 seconds Repository-environment installation
Installed environment 36 dependencies recorded, 39 MB on disk Installed lab environment
Build Succeeded in 1 second Recorded build step
Tests Succeeded in 3 seconds Recorded test command
pip-audit 0 known vulnerabilities Advisory scan of the recorded environment
Repository scan 34 files, about 3,594 source lines, 0.2 MB checked out Source checkout
Project signals One CI workflow, a tests directory, no Dockerfile Repository structure

A separate hands-on run used a fresh, unprivileged container with 2 CPUs, 4 GB RAM, no published ports and no secrets. Its record identifies commit 671ab914d507 and the same configured image label. The task was the README's 12-line Python Usage block, run unedited. The two runs used different environments, so their setup measurements are reported separately. The smoke-run record describes that scope.

Hands-on step Recorded outcome Meaning
Clone 1 second Quickstart recipe clone step completed
Package installation uv pip install cjkfold installed 0.4.0 in 1.2 seconds Package installation into a fresh virtual environment
Installed size 188 KB Recorded virtual-environment site-packages size
Dependency count 0 packages besides cjkfold Package smoke environment
README snippet Exit 0 in 0.3 seconds The supplied Usage block completed
Teardown ok Recorded cleanup completed

The clone record, installation record, size observation and dependency observation belong to that package run. Its snippet transcript records U+2F06 mapping to U+4E8C at position 2:2 and U+2EC6 mapping to U+89D2 at position 2:4. The supplied readable transcript displays 二 beside U+2F06, so the code-point labels provide the precise distinction here. Snippet output and teardown status preserve the recorded outcomes.

Where people get stuck: checks, imports and table generation

A clean check result can coexist with a zero-width space that changes a comparison. Morgan's audit demonstrates this command distinction: check looks for mapped look-alikes, while scan reports a broader set of character classes. The README also separates findings from input errors through exit codes. Choosing the command by its purpose prevents a narrow look-alike check from becoming an accidental claim about every character in a document.

Symptom or task Evidence-backed action Recorded behavior
check says clean, but strings differ Inspect the broader scan report Audit fixture contains a zero-width space
m.fold raises AttributeError after the former module import Use package-level imports or cjkfold.folding 0.4.0 separates the colliding names
Direct import from cjkfold.normalize fails Change the direct module path Release notes identify ModuleNotFoundError
Regenerated astral mappings contain an extra character Use the audited checked-in table and inspect the generator Audit reports incorrect escape rendering
CLI receives binary or non-UTF-8 input Extract text or convert encoding upstream Input refusal is deliberate
Existing indexes retain old strings Reprocess affected derived data New ingestion behavior does not rewrite stored structures

Version 0.4.0 moved the folding functions to cjkfold.folding and the writing functions to cjkfold.rewrite. The official release notes explain that the former normalize module collided with the normalize function. Documented package-level imports retain their behavior. Direct imports from the removed module path fail with ModuleNotFoundError, while the earlier name collision produced AttributeError when a caller tried m.fold on the function.

Morgan's audit identifies a separate generator defect at commit 671ab914d507304ad9b5adef41d6606a49d0d193. The checked-in table correctly represents astral targets with eight-digit escapes, and 23 mappings target code points above U+FFFF. The generator instead emits four-digit escape syntax: its representation for U+25AD7 becomes U+25AD followed by the literal digit 7. That is one intended character becoming two. The generator analysis recommends correcting escape formatting and adding a direct rendering test.

Keep raw text before enabling destructive rewrites

strip_invisible uses a selected safe deletion policy by default, while joiners, directional controls and variation selectors require manual review. Morgan lists soft hyphen, zero-width space, word joiner and byte order mark in the safe deletion group. Deleting U+200D can break an emoji sequence; deleting U+200C can alter Persian or Indic shaping. Removing bidi controls can change visual order and erase evidence relevant to analysis. The audit's rewrite discussion explains the consequences behind the classifications.

Preserve an immutable raw extraction alongside the rewritten derivative, record the package version and policy, and count changed code points. Morgan recommends that provenance trail because normalization can erase evidence. Start with scan as a report, inspect the dispositions, and select which transformations the corpus permits. This gives a maintainer a concrete answer when someone asks whether a disputed character came from extraction, folding, compatibility normalization or invisible-character deletion.

The README assigns exit 2 to input that is binary or not UTF-8, separating it from findings. scan ordinarily exits 0 because it is a report; check exits 1 when it finds look-alikes. A job runner should preserve those distinctions. A refused PDF needs extraction upstream, and a GBK or Shift-JIS file needs explicit encoding conversion. The command-line documentation describes the reporting and input-error behavior.

Existing indexes make this a migration decision

Existing vector indexes and caches need reprocessing when their source strings change, according to Morgan's audit. Changing a shared ingestion function affects newly processed material; rebuilding affected lexical structures, embeddings and caches is separate work. Put those derived artifacts into the adoption plan before switching the canonical text policy. A retained raw corpus and a separate repaired derivative make it easier to trace which representation each downstream system received. Morgan's operational analysis sets out that migration cost.

The exact-match helper folds both query and document text internally, then supplies a bonus for literal overlap. Morgan warns that adding 0.25 is especially unsafe when a store returns distances rather than higher-is-better similarities, or when indexes use different score scales. Select score direction, bonus scale and query-fragment size against labeled retrieval examples. Those are application choices: a correct character substitution tells you which strings match, while relevance judgments tell you which documents should come first.

English documentation exists alongside Chinese background material

The README explains installation, Python usage and command-line behavior in English, and includes a Chinese summary and a longer Chinese background link. An English-speaking developer can start with the main integration instructions and follow the background account when investigating the author's extraction decisions. The project README also links corpusfold as a broader PDF-to-knowledge-base pipeline built on this character layer, placing cjkfold within a larger ingestion workflow.

The README states Python 3.9+ and zero runtime dependencies. That package requirement differs from the 36 dependencies recorded in our repository environment and agrees with the separate smoke run's observation of 0 packages besides cjkfold. For a first integration, use the documented package-level imports and make the chosen text transformation visible in your ingestion configuration and retained metadata. This connects the installation instructions to the character policy selected for the corpus.

cjkfold vs Unicode NFKC, OpenCC and CJKFoldingFilter

Choose cjkfold for mapped radical repair, NFKC for compatibility normalization, OpenCC for Chinese script or regional wording conversion, and CJKFoldingFilter for its documented folds within a Solr analyzer. The useful comparison is the rewrite policy each permits and the integration point where it belongs. Start with the actual character mismatch or conversion requirement in the corpus, then select the operation that addresses it.

Option Documented transformation Pick it when
cjkfold Selected radical and stroke equivalence mappings Extracted look-alikes cause failed comparisons
Unicode NFKC Unicode compatibility normalization Compatibility normalization is the intended policy
OpenCC Traditional Chinese, Simplified Chinese, Japanese Shinjitai and regional wording conversion Script or regional wording conversion is required
CJKFoldingFilter Certain CJK folds in a Lucene/Solr analysis chain Folding belongs in the Solr analyzer

The cjkfold README reports that NFKC repairs all 214 Kangxi Radicals, but only 2 of the 115 named CJK Radicals Supplement characters. It reports cjkfold mappings for 114 of those 115 supplement characters, with the remaining radical intentionally unchanged. Those counts describe character coverage in the project's analysis. They help explain why a specific radical may survive an NFKC pass and why an additional targeted mapping can be useful. The README coverage comparison supplies the figures.

OpenCC supports conversion between Traditional Chinese, Simplified Chinese, Japanese Shinjitai and regional wording. Its documentation describes dictionaries, a reusable library and conversion tools. CJKFilterUtils provides a Lucene filter and filter factory for certain CJK folds, including modern Japanese kanji to traditional equivalents. Its README places that filter before Traditional-to-Simplified ICU transforms. OpenCC's introduction and CJKFilterUtils' README support these distinctions without treating the tools as interchangeable APIs.

What the other reviews get wrong

AISignal's description conflates targeted folding with broader normalization and Chinese script conversion. The cjkfold README documents three separate writing operations and explicitly excludes traditional-to-simplified conversion. AISignal also describes a custom curated mapping list, whereas the README calls the table generated and provides a generation command. These differences affect both adoption and maintenance: a character-equivalence table is a different commitment from a linguistic conversion dictionary. The supplied claim comparison can be checked against the project documentation.

The supplied package listing includes a cjkfold 0.4.0 wheel, contradicting AISignal's statement that official PyPI releases do not exist. Its October 2024 adoption snapshot also conflicts with the supplied repository chronology, which dates the GitHub presence to October 2026. The piwheels listing and XERJ chronology support those specific corrections. The separate hands-on installation record also names the installed package version as 0.4.0.

FAQ

cjkfold maps selected CJK radicals and strokes to equivalent ideographs so extracted text can match the intended characters. It is a Python package under the MIT license, and the README states Python 3.9+ as its requirement. Its inspection commands and writing operations have separate purposes: a report identifies characters for inspection, while a rewrite applies the selected character policy. The README documents installation, usage and licensing.

What is cjkfold?

cjkfold maps selected CJK radicals and strokes to equivalent ideographs so comparisons can use the intended characters. Its read-only inspection tools also report compatibility characters, invisible characters and damaged-text indicators. The distinction between reporting and rewriting matters: scan can identify a character that needs human review without choosing a replacement for it. The README's character discussion explains those dispositions.

Is cjkfold free and open source, and what license does it use?

cjkfold is free and open source under the MIT license, which the project README states explicitly. The supplied XERJ account also reports that GitHub's MIT classification and the LICENSE file at its pinned commit agree. Read the actual project license when incorporating or distributing the package. The README license statement and the pinned license account support this answer.

What language is cjkfold written in?

cjkfold is written in Python, and its README states Python 3.9+ as the requirement. The documented installation command is pip install cjkfold, followed by package-level imports for the desired operations. The README includes Python examples and a command-line interface, so a developer can choose either integration route while retaining the same distinction between reporting findings and rewriting mapped characters. The installation and usage documentation supplies that starting point.

What does the src/cjkfold directory contain?

src/cjkfold contains the Python package, including folding.py, rewrite.py, diagnose.py, inputs.py, cli.py, integrations.py, score.py and the generated _table.py. The supplied live directory listing shows those files. Version 0.4.0's release notes explain the folding and rewrite module names, making them useful context when tracing imports. The package source directory and 0.4.0 release notes support this layout.

What we have not tested yet

Our recorded runs cover repository setup and one README smoke example; production throughput, sustained memory use and comparative retrieval quality remain outside those observations. The supplied audit identifies no reproducible representative retrieval benchmark. AISignal's 98.7% accuracy figure lacks supporting evaluation in this evidence, and named production adoption and business outcomes are also unresolved. The recorded repository test summary supplies no test count; Morgan's 166 passes belong to the independent audit.

The generator reproduction comes from Morgan's pinned audit, and no supplied follow-up settles whether a later change fixes it. We have not rerun that reproduction or assessed cjkfold as a Unicode security boundary. The audit excludes general cross-script confusable detection and identifier validation, and the zero-advisory result is not a security certification. The configured container image label also supplies no independent proof of actual interpreter or operating-system versions, while the smoke record's latest-version installation describes that recorded run rather than a future installation.

The indexed XERJ code snapshot trails main by 17 commits and therefore supplies historical context rather than current API proof. A useful next evaluation is to retain raw text, apply the documented character policy to known failed comparisons, and assess retrieval changes against labeled queries. Python ingestion maintainers with those specific mismatches have a concrete reason to try cjkfold; teams whose primary requirement is extraction, script conversion or Unicode security should use tools designed for those jobs.

Alternatives

ProjectWhat it isPick it when
Unicode NFKC via Python unicodedataStandard-library compatibility normalization with a different rewrite scope from targeted CJK folding.Choose it when Unicode compatibility normalization is the intended text policy.
OpenCCDictionary-based Chinese script and regional wording conversion.Choose it when traditional/simplified or regional wording conversion is the requirement.
CJKFoldingFilterA Lucene/Solr filter for certain CJK character folds.Choose it when the required transformation belongs in a Solr analysis chain.

Sources

  1. cjkfold README
  2. cjkfold 0.4.0 release
  3. Grace Morgan independent audit
  4. piwheels package listing
  5. AISignal claims examined
  6. XERJ chronology and license account
  7. Supplied reader questions
  8. OpenCC README
  9. CJKFilterUtils README
  10. cjkfold source directory

More dev tools reviews