TextEncoder and TextDecoder

rusty_textencoder backs TextEncoder and TextDecoder. It is a UTF-8-only codec: it encodes strings to bytes, decodes bytes to text with replacement-character or fatal error handling, and supports streaming and BOM stripping. Every non-UTF-8 label (Shift_JIS, windows-1252, and the rest) is refused rather than decoded.

TextEncoder and TextDecoder run constantly: fetch(...).then(r => r.text()) runs one, new TextEncoder().encode(str) the other, and every CSV import, protobuf field, and WebSocket frame passes through them. In Node they are backed by encoding_rs, a Rust library that ships roughly 50 legacy encodings (Shift_JIS, GBK, windows-1252, the whole WHATWG registry) and has been fuzzed and shipped in Firefox for years. Cruft ships its own stand-in, rusty_textencoder. It wears the same name but is a much smaller thing, one that answers a smaller set of questions correctly.

Alpha. This page documents the rusty_textencoder crate (Cruft 0.0.10), the replacement for encoding_rs. It has not been externally audited. Its properties are unproven except where a specific test is cited below. Do not rely on it for production or adversarial input.

UTF-8 only

new TextDecoder('shift_jis') works in the browser and in Node because encoding_rs carries the full label registry. This crate is UTF-8 only. Every other label is refused outright, with no best-effort decode. resolve_label accepts exactly utf-8, utf8, unicode-1-1-utf-8, unicode11utf8, unicode20utf8, x-unicode20utf8, and returns DecoderError::UnknownEncoding for anything else. There is no Shift_JIS, no windows-1252, no fallback table.

That is the entire scope: a correct UTF-8 codec, and only that. Label canonicalization is tested (four spellings of UTF-8 all resolve to the canonical "utf-8"); non-UTF-8 refusal is the _ => Err(...) arm of resolve_label.

Encoding and the unsanitized bytes

Encoding a &str is straightforward: Rust strings are already valid UTF-8, so encode is s.as_bytes().to_vec(). The USVString surrogate replacement the spec describes happens at the JS/Rust boundary, not here. encode_into carries a real invariant: it writes whole code points only, never a truncated multi-byte sequence, and returns {read, written} where read counts UTF-16 code units (via char.len_utf16()) and written counts UTF-8 bytes. The no-split-at-buffer- boundary guarantee is pinned by a regression test (a 3-byte buffer takes h + é exactly and the result decodes cleanly), along with the read/written accounting.

Decoding is where untrusted input lives. Three behaviors to know:

  • Invalid bytes with the default fatal: false become U+FFFD (the replacement character), one per bad byte, without throwing (utf8_decode).
  • With fatal: true, the same invalid bytes return Err(DecoderError::InvalidSequence) instead.
  • A multi-byte code point split across two decode(..., {stream: true}) calls does not emit replacement characters: the trailing partial bytes are retained in pending and completed on the next call.

Memory safety

For a parser over untrusted bytes, the risk that matters is a panic or a memory-safety fault that takes the process down. A wrong character is survivable; a crash is not. The source contains no unsafe, and the decode loop advances by a computed need (1 to 4 bytes) with an explicit i + need > bytes.len() bounds check before it ever slices. Invalid lead bytes and invalid continuations both route to the U+FFFD-or-error arms rather than indexing past the end. std::str::from_utf8 on each candidate sequence is the final validator, so an ill-formed-but-length-correct sequence still cannot produce an invalid String. What is not proven is "no panic reachable on any adversarial input," because no fuzz target in this crate exercises the decode loop over the byte space at scale.

BOM handling

The one correctness bug once filed against this crate was in streamed BOM stripping, and it is fixed in the code you would run today. A UTF-8 BOM is EF BB BF. If those three bytes arrive split across two stream: true chunks (say EF then BB BF), an earlier version set bom_consumed = true on the short first chunk and then failed to strip the BOM, leaking a U+FEFF into the text.

The current decode checks, while the BOM is still undecided, whether the buffer so far is exactly EF or EF BB, and if so stows it in pending and returns an empty string rather than committing bom_consumed. The decoder waits until it has enough bytes to decide. Non-streamed BOM consumption and the ignore_bom: true preserve-the-BOM path are both pinned. The specific split-BOM-across-chunks case is fixed in code but not itself covered by a dedicated regression, so that path is present and correct by inspection but lacks its own test.

Limitations

What this crate does not do:

  • No legacy encodings. UTF-8 is the whole registry. If any consumer hands you a shift_jis, gbk, iso-8859-*, or windows-125x label, you get UnknownEncoding, not a best-effort decode. encoding_rs wins decisively on breadth, and it has years of fuzzing and Firefox shipping behind its UTF-8 path that this crate has not accumulated.
  • undefined versus absent argument is invisible in Rust. The WHATWG rule where encode(undefined) coerces to the 9-byte string "undefined" cannot be expressed against Option<&str>, so the JS-boundary layer must handle it. This crate models absent as None => 0 bytes only.
  • Test coverage is a consumer-regression suite, not a conformance suite. The pins are drawn from real npm consumers (undici, jsdom's whatwg-encoding, protobuf.js, PapaParse, node-mysql2) plus a handful of WPT-derived cases. That is meaningful grounding, but it is a small suite, not the WHATWG encoding WPT corpus and not a fuzz campaign. Interop against the full spec vector set is unproven.
  • No fuzzing of the decoder loop itself. Fuzz scaffolds exist for sibling codecs (basen, percent-decode, punycode, IDNA); the textencoder UTF-8 decode loop is not among them.