Unicode tables

rusty-js-ucd-tables holds the Unicode Character Database tables Cruft's text code looks up: identifier-property ranges, canonical and compatibility decomposition, combining classes, and normalization. It pins every table to one Unicode version (17.0.0) and checks at test time that the lookup arrays are sorted, so a code point cannot be silently misclassified.

Any package that touches text pulls in a chain of small crates with names like unicode-normalization and unicode-canonical-combining-class, plus a pile of property tables. They ship a snapshot of the Unicode Character Database (UCD): which code points start an identifier, how é decomposes into e plus a combining accent, what canonical combining class a mark carries. Two questions rarely get asked: which Unicode version those tables encode, and whether every table pulled in encodes the same one. If the normalizer is Unicode 15 and the identifier property table is Unicode 17, nothing errors. The answers quietly disagree with each other, and with the engine parsing the source. rusty-js-ucd-tables is Cruft's own UCD table layer, and it pins one version across every table it holds.

Alpha. rusty-js-ucd-tables (Cruft 0.0.10) stands in for the unicode-* family of UCD table crates (normalization, decomposition, combining-class, and identifier-property tables). It has not been externally audited. Its worst-case failure is a wrong classification, and one consumer, identifier and host-name validation, makes a wrong classification security-adjacent. Do not treat it as spec-conformant against the full UCD until the conformance-corpus gaps below are closed.

A table crate is data and lookups

The parser-and-TLS frame (attack surface, input validation, memory safety) mostly does not fit here. rusty-js-ucd-tables takes no untrusted bytes off a wire. It is arrays of code-point ranges and decomposition mappings, plus binary_search lookups over them. There is no unsafe, no panic!, no todo!(), and no unwrap on external input. Every lookup is total: a code point either hits a range or it does not, and the Hangul arithmetic path is bounds-guarded before it does any math.

So the failure mode is a silent wrong answer, a wrong membership verdict or a wrong decomposition mapping returned without a crash or an overflow. Two things a table crate can get wrong carry that risk: the data (is it the right Unicode version, from a real source) and the lookup precondition (are the arrays actually sorted the way binary_search demands).

What it does

The crate exposes a small, real surface:

  • Hangul algorithmic decomposition. UcdResolution::decompose(cp) routes a syllable through HangulResolver, which runs the pure LVT arithmetic. A default test walks all 11,172 syllables and asserts each one's algorithmic decomposition round-trips against an independently computed result.
  • Explicit decomposition tables (positional, orthographic, legacy-alias, cursive-shape, residual), looked up by binary_search. A default test walks every explicit row and asserts the round trip.
  • Normalization (normalize_str, NormalizationForm) over canonical and compatibility decomposition, combining-class (CCC), and composition tables. The code path is real; the conformance gap is in Limitations.
  • UAX #31 identifier properties. is_id_start / is_id_continue / is_xid_start / is_xid_continue over sorted range tables. This is the surface rusty-js-unicode-ident consumes; its provenance caveat is below.

Sortedness is a precondition the tests check

binary_search over an unsorted or overlapping array does not error. It returns a plausible-looking wrong answer. In a UCD table that means a code point gets misclassified: a character that should not start an identifier is accepted, or a decomposition points at the wrong mapping arm. Reading the data does not reveal this; the arrays are thousands of rows long and machine-generated.

The default test suite checks the precondition rather than trusting it:

  • Every identifier range table (ID_START, ID_CONTINUE, XID_START, XID_CONTINUE) is asserted sorted, non-overlapping, and non-inverted.
  • Every decomposition lookup table (POSITIONAL, ORTHOGRAPHIC, LEGACY, CURSIVE, RESIDUAL) is asserted sorted and bounded to an existing mapping arm.
  • The normalization decomposition, CCC, and composition tables are asserted sorted and mapping-bounded for their binary_search key.

One pinned Unicode version across every table

The unicode-* crates this replaces each pin their own version independently, and nothing checks that they agree. This crate once had the same problem: the identifier tables stated Unicode 17.0.0, the normalization tables carried no stated version, and their generator pointed one input file at UCD latest. Three tables, potentially three different snapshots of Unicode, with no error if they drifted apart.

That is now fixed structurally. The crate exposes one shared constant UNICODE_VERSION = "17.0.0", and every generated table module re-exports it as an alias rather than restating its own literal. Because each module points at the single source of truth, the modules cannot drift apart; the per-module assertion that the alias matches is a tautology that documents the invariant rather than a drift-catching test. The real pin is the one shared constant plus source-comment records of where each table came from. The same consolidation also caught and fixed a stale decomposition UNIQUE_MAPPING_COUNT pin.

Limitations

The version pin, the sortedness gate, and the closure laws are real and run by default. What is not yet in place is ingestion of the authoritative Unicode conformance corpora, and that is the crate's real maturity gap versus the crates.io originals.

  • The official NormalizationTest.txt check is opt-in, not default. The conformance walk is gated behind the RUSTY_JS_UCD_NORMALIZATION_TEST env var and skips entirely when it is unset. The default suite covers internal consistency (sorted, round-trip, version-agreed) but does not check the tables against Unicode's own normalization fixtures. The unicode-normalization crate this replaces has years of production exposure against exactly that corpus.
  • The identifier tables have no default test against DerivedCoreProperties.txt. Their provenance is test262's generated property-escape fixtures, a real and reasonable source but a second-order one, and there is no checked-in generator that re-derives them from primary UCD data. The trust sits on the fixture pipeline, not the UCD directly.
  • The closure tests cover the algebra, not the completeness. The Hangul and explicit-row laws show that whatever is in the tables decomposes and recomposes correctly. They do not show the tables contain every code point the full UCD assigns. Missing rows are silent (decompose returns None, treated as "no decomposition"), the same failure class as an out-of-date table.
  • No external audit, no interop fuzzing. Alpha means alpha.

Where the crates.io originals lead today: breadth of real-world exposure and default-wired authoritative conformance testing. Where this crate already leads: it makes the cross-table version invariant and the binary-search sortedness precondition into compile-and-test-time facts instead of unstated assumptions, which the fragmented unicode-* crates never do as a set.