Lexer and Parser

Lexer (flint-lexer)

The lexer converts raw UTF-8 source into a token stream.

Token Design

Every token carries:

• Kind: what kind of token it is (identifier, keyword, integer literal, operator, etc.)
• Span: the byte range (start, end) in the source file

Token kinds include:

• Keywords: fn, let, mut, if, else, match, for, while, loop, return, use, pub, struct, enum, impl, trait, const, static, defer, owned, as, in, and, or, not, break, continue, alias, never, false, true
• Literals: integer, float, string, character, byte string, byte character, raw string
• Identifiers
• Operators and delimiters
• Whitespace and newlines (preserved for the formatter pass)
• Comments (//, /* ... */, and ///)

Automatic Statement Termination

The lexer implements Flint’s semicolon-insertion rule. A synthetic statement terminator is inserted after a token if:

1. The token is an identifier, integer/float/string literal, true, false, never, break, continue, return, ), ], or }
2. The next real token is a newline
3. The parser is not inside an open (, [, or {

The lexer also does not insert terminators after infix operators, incomplete assignments, or ..

This keeps the rule simple, local, and deterministic, with no lookahead into expressions needed.

Position Preservation

The lexer is lossless. Every character in the input maps to exactly one token. Whitespace tokens are included in the token stream, separate from the “clean” stream seen by the parser. This dual-stream design lets the formatter work directly from the token stream without re-parsing.

Parser (flint-parser)

The parser builds a Concrete Syntax Tree (CST) from the clean token stream (whitespace excluded).

CST Design

The CST is:

• Lossless: all tokens from the original source appear in the tree, including whitespace and comments.
• Untyped at the syntax level: the CST represents structure, not semantics.
• Recoverable: parse errors are local; the parser continues after an error.

CST nodes are typed by their production rule: FunctionDecl, IfExpr, MatchArm, StructDecl, UseDecl, etc.

Error Recovery

When the parser encounters an unexpected token, it:

1. Emits a diagnostic with the expected and actual tokens.
2. Inserts an error node in the CST.
3. Attempts to continue by advancing to the next synchronization point (typically a statement boundary, closing brace, or keyword).

This lets a single flint check run report multiple errors rather than stopping at the first one.

AST Extraction

After CST construction, an AST is extracted by:

• Stripping whitespace and comment tokens.
• Extracting the semantic content from CST nodes into a typed AST representation.
• Preserving spans so every AST node knows its source location.

The AST is what subsequent passes operate on.

Formatter Integration

The formatter (flint fmt) operates on the CST, not the AST. It uses the CST’s whitespace tokens to understand the original layout, then replaces whitespace according to Flint’s formatting rules, producing a canonical output. The CST guarantees that the formatter never changes the semantic content; it only normalizes whitespace, ordering of use declarations, and blank lines.