Lex: a literal ends where its own pattern says, not where a byte class does #220

Manually merged
tvidal merged 1 commit from feat/220-lex-a-literal-ends-where-its-own-pattern-says into master 2026-09-04 10:55:46 +02:00
Owner

Lex: a literal ends where its own pattern says, not where a byte class does

Along the way:

  • lex: a literal ends where its own pattern says, not where a byte class does

Touches 6 files: src/rust/crates/ck_driver/src/literal_lex.rs, src/rust/crates/ck_driver/tests/capability.rs, src/rust/crates/ck_driver/tests/typed_literal_lex.rs, tests/capability/geo_literal_parks_on_its_fold_body.cktest, tests/capability/money_eur_literal.cktest, tests/capability/money_usd_grouped_literal.cktest.

Lex: a literal ends where its own pattern says, not where a byte class does Along the way: - lex: a literal ends where its own pattern says, not where a byte class does Touches 6 files: `src/rust/crates/ck_driver/src/literal_lex.rs`, `src/rust/crates/ck_driver/tests/capability.rs`, `src/rust/crates/ck_driver/tests/typed_literal_lex.rs`, `tests/capability/geo_literal_parks_on_its_fold_body.cktest`, `tests/capability/money_eur_literal.cktest`, `tests/capability/money_usd_grouped_literal.cktest`.
tvidal approved these changes 2026-09-04 10:03:50 +02:00
tvidal left a comment

Reads well. One question on src/rust/crates/ck_driver/tests/typed_literal_lex.rs, nothing blocking.

Reads well. One question on `src/rust/crates/ck_driver/tests/typed_literal_lex.rs`, nothing blocking.
@ -137,0 +135,4 @@
let from = match marker.position.as_str() {
"Prefix" if src[start..].starts_with(&marker.text) => start + marker.text.len(),
"Embedded" | "Suffix" if numeric_led => start,
_ => continue,
Owner

The catch-all arm means a variant added later compiles and lands here without a word. Can the match on _ name every variant so the compiler tells us?, at src/rust/crates/ck_driver/src/literal_lex.rs:138.

The catch-all arm means a variant added later compiles and lands here without a word. Can the match on `_` name every variant so the compiler tells us?, at src/rust/crates/ck_driver/src/literal_lex.rs:138.
@ -73,0 +111,4 @@
let text = &app_file.src[rec.span.start.0 as usize..rec.span.end.0 as usize];
assert_eq!(text, spelling, "the TypedLit spans the WHOLE lexeme");
let claim = &table.claims()[rec.claim as usize];
Owner

The as cast truncates silently if claim is ever wider than the target. Is the bound guaranteed, or should this be try_from?

The `as` cast truncates silently if `claim` is ever wider than the target. Is the bound guaranteed, or should this be `try_from`?
Author
Owner

Not guaranteed. claim goes through try_from now and the overflow is a typed error in src/rust/crates/ck_driver/tests/typed_literal_lex.rs.

Not guaranteed. `claim` goes through try_from now and the overflow is a typed error in `src/rust/crates/ck_driver/tests/typed_literal_lex.rs`.
Author
Owner

Named them all. A new variant is a compile error in src/rust/crates/ck_driver/src/literal_lex.rs now, which is where we want to find out.

Named them all. A new variant is a compile error in `src/rust/crates/ck_driver/src/literal_lex.rs` now, which is where we want to find out.
lnguyen approved these changes 2026-09-04 10:39:47 +02:00
lnguyen left a comment

Read it too. No objection to src/rust/crates/ck_driver/tests/typed_literal_lex.rs.

Read it too. No objection to `src/rust/crates/ck_driver/tests/typed_literal_lex.rs`.
A claimed literal's scan window ended at an ordinary token boundary. Two
spellings in the roster name a character that boundary excludes, so they could
never be scanned as one lexeme at all: `Money<USD>`'s grouping `,` and
`Money<EUR>`'s decimal `,` — the whole EUR claim was unreachable, and the USD
grouping is the spelling `Types.ck` writes beside the claim.

The window is now widened by the characters the CLAIM'S OWN pattern spells
(every byte inside a quoted segment, `sep ','` included) and the pattern walks
the widened tail longest-first. One mechanism, read off the pattern SOURCE — no
branch keyed on a type, and a claim whose pattern names only ordinary body
characters (Color, every temporal claim) scans byte-for-byte as before. The
descending walk is what keeps `[$1.50, $2.50]` correct: the widened `1.50,` is
offered first, the pattern refuses it, `1.50` is next. That is the point — the
pattern decides where a literal ends, which is what it was always supposed to do.

  money_usd_grouped_literal_does_not_lex -> money_usd_grouped_literal   runs: 42
  money_eur_literal_does_not_lex         -> money_eur_literal           runs: 42
  BROKEN_CELL_COUNT 54 -> 52

`geo_literal_does_not_lex` moved with them — its pattern names `' '`, the one
space-bearing claim — but only one stage: it now lexes and typechecks and parks
in the fold, so it stays BROKEN and pays no ratchet. It is renamed
`geo_literal_parks_on_its_fold_body` and its declaration is `codegen:
Unsupported`. The park is NOT float arithmetic, as the old comment guessed:
`Types::signed` runs clean through the bounded evaluator; the fold body's
named-tuple construction is `NotEvaluable("expr")`.

Four cells' comments carried a cause that measurement contradicts, so they are
corrected in place (no outcome moves, none was BROKEN for the stated reason):

- `color_literal_*` / `timespan_*_parks` do not park on `decode_capture`'s
  missing optional arm; they park a step earlier. `tag_to_ty` types an optional
  capture `Ty::Unknown`, `Ty::named_tuple` then poisons the whole synthesized
  captures record, and `fold` has no field symbols to build anything from.
- the two `timespan_*_parks` cells are additionally hiding a WRONG VALUE:
  `LiteralDSL::inner_match` matches only the leading class-and-quantifier of a
  capture's inner pattern and drops the rest, so `<h: digit+ 'h'>` accepts bare
  digits and `24s` is captured as twenty-four HOURS. Closing debt #71 alone
  would flip those cells green over a wrong fold. Same defect, one stage
  earlier, is why `timespan_multi_group_literal_does_not_lex` fails outright.
- `datetime_literal_does_not_lex` fails in the LEXER, not the decode: `Date` and
  `Time` are other CLAIMS, and a capture's bareword roster is the three shipped
  `const Pattern`s. The fix is not a decision — `from_literal` reads `x.d.value`,
  so a capture naming a claim folds through that claim — only unbuilt.

The remaining literal-DSL work is CkLang content (`LiteralDSL.ck`'s
`inner_match`, its capture tags, `Types.ck`'s duration group order), not Rust,
and belongs to the sibling repo that ships the same files.

Verified: `ckc check` over all 437 capability cells, before and after, differs on
exactly the three cells above and nothing else; the 36 corpus apps are
byte-identical. `cargo fmt` clean, `cargo clippy --workspace --all-targets -D
warnings` 0/0, `--test architecture` 38/38. The folded VALUE is unobservable from
CkLang (`.value` on a tag-parameterised newtype does not lower), so
`typed_literal_lex` gains the two gates that can see it: `$1,000.50` and
`€1.000,50` both fold to 100050 minor units, and the aggregate spelling still
ends at its comma.
tvidal manually merged commit 5ad70ae134 into master 2026-09-04 10:55:46 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
northgale/cklang-bootstrap!220
No description provided.