Unicode identifiers

rusty-js-unicode-ident answers whether a Unicode code point may start or continue a JavaScript identifier. It is a thin adapter over the shared UCD tables, adding the ECMA-262 rules that differ from raw Unicode: the ASCII $ and _, the zero-width joiners as continue-only, and the ID-minus-XID residual set.

When const café = 1 is accepted and let \u{1F4A9} = 2 is rejected, something has to answer a yes/no question for every one of the 1.1 million Unicode code points: may this character begin an identifier, and may it continue one. In the Node world that answer is a crate, unicode-ident, a set of compressed lookup tables that Rust JS tooling (swc, oxc, biome) leans on to decide ID_Start and ID_Continue. It is a Unicode judgment installed with cargo add, and its Unicode version, its table sortedness, and its agreement with the parser that consumes it typically go unchecked. rusty-js-unicode-ident replaces that crate, and it is a thin adapter, not the table itself.

Alpha. rusty-js-unicode-ident (Cruft 0.0.10) is a stand-in for the crates.io unicode-ident crate. It is not externally audited and has near-zero interop testing against third-party parsers. Do not rely on it for adversarial input in production.

The adapter owns no table

rusty-js-unicode-ident holds no Unicode data. The broad Unicode property tables live in a sibling crate, rusty-js-ucd-tables, behind a module named identifier_tables. This crate is the thin adapter between that raw property surface and the two consumers that care about JS identifiers: the parser and the runtime. The whole dependency is use rusty_js_ucd_tables::identifier_tables;. Every non-ASCII membership question (is_id_start, is_id_continue, is_xid_start, is_xid_continue) delegates straight into that shared table.

In the Node world unicode-ident does both jobs, holding the data and answering the predicate. Here those are two crates. This one answers "is this a legal JS identifier character," and it answers by asking the table crate plus a small set of ECMA-262-specific corrections it applies itself. The correctness of the raw tables is the sibling's concern.

What ECMA-262 adds over raw UAX #31

If the crate merely forwarded to the table, it would not need to exist. What it adds is the gap between the Unicode UAX #31 definition of an identifier and what ECMA-262 specifies for JavaScript. Three corrections live here, and each is a real place the JS answer differs from the naive table answer:

  1. ASCII is decided locally, not by the table. For cp < 0x80 the crate never touches Unicode data; it answers from is_ascii_identifier_start / is_ascii_identifier_continue, which encode the ASCII letters plus _ and $. $ and _ as identifier characters are an ECMA-262 fact, not a general UAX #31 one.
  1. The zero-width joiners are Continue-only. ECMA-262 admits U+200C and U+200D (<ZWNJ> / <ZWJ>) in IdentifierPart but not IdentifierStart. The crate special-cases them in is_ecmascript_join_control and folds them into is_id_continue only. A test asserts they are rejected as start and accepted as continue.
  1. The ID-minus-XID residual. A handful of compatibility code points (U+037A, U+0E33, U+309B, the U+FC5E..U+FC63 block, and more) are in ID_Start but not XID_Start. is_id_xid_delta pins that exact set. The shared table now admits these directly, so the predicate is kept as a preservation pin for parser/runtime migration tests rather than as the live decision path. A test asserts each delta is ID but not XID.

The IdentifierDecision enum (Accept/Reject) and the identifier_start_decision / identifier_continue_decision entry points are the adapter's product: a two-valued verdict the parser can consume without knowing anything about Unicode ranges.

Delegation still leaves a security stake

Handing the hard part to another crate does not remove this crate from the security story. The adapter is total and memory-safe on its own terms: no unsafe, no unwrap on input, no panic!, and every u32 maps to a defined Accept/Reject. A malformed code point cannot crash it; it gets Reject.

But the correctness of the answer is only as good as the table it delegates to, and the answer feeds identifier parsing, which feeds (via IDNA and host-name handling elsewhere in the runtime) security-adjacent decisions. A table that was silently unsorted, or pinned to a different Unicode version than the parser expected, would misclassify characters rather than fail loudly. That was the exact concern raised against this crate and its table sibling.

The version-and-sortedness invariant, enforced

The concern was a verification gap with three parts: version pinning was inconsistent, the identifier tables stated Unicode 17.0.0 while the normalization tables carried no stated version and nothing asserted the two agreed; the authoritative conformance corpora were not run in a default test run; and nothing asserted the lookup tables were sorted and non-overlapping, so an unsorted row would silently misclassify instead of failing.

That is now closed, and the fix is enforced by default tests:

  • rusty-js-ucd-tables exposes one shared UNICODE_VERSION (17.0.0) and each generated table module asserts it matches.
  • Default tests prove every identifier range table is sorted, non-overlapping, and within the Unicode scalar range, and that every binary-search key is bounded.
  • This crate carries a default consumer test that its adapter uses the same pinned version: it asserts rusty_js_ucd_tables::UNICODE_VERSION == "17.0.0" and that identifier_tables::UNICODE_VERSION equals it.

So the silent-misclassify scenario is closed by an enforced cross-crate invariant. Full ingestion of the external DerivedCoreProperties.txt corpus remains useful future hardening, but the previously unasserted lookup-shape and version invariants are now checked by default.

Limitations

  • The tables are trusted, not re-derived here. This crate does not ingest DerivedCoreProperties.txt itself; its non-ASCII correctness is inherited from rusty-js-ucd-tables. Running the authoritative UCD corpus by default is not yet end-to-end conformance. What is in place is the structural guard (sorted, bounded, version-pinned) that would catch a corrupt table.
  • The non-ASCII membership tests are spot pins, not a sweep. They check specific code points (π, 中, U+2E2F rejected, the Other_ID_Start/Other_ID_Continue pins), not the full 1.1M-code-point space against an external oracle. They show the wiring is live and catch gross regressions; they do not show exhaustive agreement with Unicode 17.0.0.
  • unicode-ident is the maturity baseline, and it is more mature. The crates.io original is widely deployed under swc/oxc/biome, fuzzed in the wild, and its table generation is a public, reviewed pipeline. This crate is Cruft 0.0.10 alpha with no external interop testing. It is structurally sound and total, but it has not earned the deployment mileage the baseline has.
  • No interop testing against a third-party parser. Nothing here shows that a program Cruft's parser accepts is a program swc or V8 would also accept at the identifier boundary. That equivalence is unproven.