Add fluxc compiler skeleton: token definitions and lexer

Introduces the fluxc Rust crate with the first two compiler stages:

- token.rs: define_tokens! macro generates TokenKind enum and its
  Display impl from a single table covering all Flux tokens
  (literals, keywords, operators, punctuation, Eof/Unknown).
  Span (half-open u32 byte range) and Token<'src> (kind + span +
  zero-copy text slice) round out the module.

- lexer.rs: Lexer<'src> produces Token<'src> from a source &str.
  Skips whitespace, // line comments, and /* */ block comments.
  Handles all integer bases (decimal, hex, octal, binary with _
  separators), floats (fractional + exponent), string/char literals
  with escape sequences, and Unicode identifiers via unicode-xid.
  Implements Iterator<Item = Token> and includes 17 unit tests.

Also adds .gitignore (ignores fluxc/target) and expands
examples/fibonacci.flx with an iterative variant.
This commit is contained in:
2026-03-10 17:20:17 +01:00
parent 0e08640f59
commit 4f80de51b2
7 changed files with 798 additions and 2 deletions

16
fluxc/Cargo.lock generated Normal file
View File

@@ -0,0 +1,16 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "fluxc"
version = "0.1.0"
dependencies = [
"unicode-xid",
]
[[package]]
name = "unicode-xid"
version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853"