Enum rustc_lexer::TokenKind

pub enum TokenKind {
    LineComment {
        doc_style: Option<DocStyle>,
    },
    BlockComment {
        doc_style: Option<DocStyle>,
        terminated: bool,
    },
    Whitespace,
    Ident,
    InvalidIdent,
    RawIdent,
    UnknownPrefix,
    Literal {
        kind: LiteralKind,
        suffix_start: u32,
    },
    Lifetime {
        starts_with_number: bool,
    },
    Semi,
    Comma,
    Dot,
    OpenParen,
    CloseParen,
    OpenBrace,
    CloseBrace,
    OpenBracket,
    CloseBracket,
    At,
    Pound,
    Tilde,
    Question,
    Colon,
    Dollar,
    Eq,
    Bang,
    Lt,
    Gt,
    Minus,
    And,
    Or,
    Plus,
    Star,
    Slash,
    Caret,
    Percent,
    Unknown,
}

Enum representing common lexeme types.

Variants

LineComment

Fields

doc_style: Option<DocStyle>

“// comment”

BlockComment

Fields

doc_style: Option<DocStyle>

terminated: bool

/* block comment */

Block comments can be recursive, so the sequence like /* /* */ will not be considered terminated and will result in a parsing error.

Whitespace

Any whitespace characters sequence.

Ident

“ident” or “continue” At this step keywords are also considered identifiers.

InvalidIdent

Like the above, but containing invalid unicode codepoints.

RawIdent

“r#ident”

UnknownPrefix

An unknown prefix like foo#, foo', foo". Note that only the prefix (foo) is included in the token, not the separator (which is lexed as its own distinct token). In Rust 2021 and later, reserved prefixes are reported as errors; in earlier editions, they result in a (allowed by default) lint, and are treated as regular identifier tokens.

Literal

Fields

kind: LiteralKind

suffix_start: u32

“12_u8”, “1.0e-40”, “b“123”“. See LiteralKind for more details.

Lifetime

Fields

starts_with_number: bool

“’a”

Semi

“;”

Comma

“,”

Dot

“.”

OpenParen

“(”

CloseParen

“)”

OpenBrace

“{”

CloseBrace

“}”

OpenBracket

“[”

CloseBracket

“]”

At

“@”

Pound

“#”

Tilde

“~”

Question

“?”

Colon

“:”

Dollar

“$”

Eq

“=”

Bang

“!”

Lt

“<”

Gt

“>”

Minus

“-”

And

“&”

Or

“|”

Plus

“+”

Star

“*”

Slash

“/”

Caret

“^”

Percent

“%”

Unknown

Unknown token, not expected by the lexer, e.g. “№”

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq<TokenKind> for TokenKind

fn eq(&self, other: &TokenKind) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenKind

impl Eq for TokenKind

impl StructuralEq for TokenKind

impl StructuralPartialEq for TokenKind

Layout

Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...) attributes. Please see the Rust Reference’s “Type Layout” chapter for details on type layout guarantees.

Size: 8 bytes

Size for each variant:

• LineComment: 1 byte
• BlockComment: 2 bytes
• Whitespace: 0 bytes
• Ident: 0 bytes
• InvalidIdent: 0 bytes
• RawIdent: 0 bytes
• UnknownPrefix: 0 bytes
• Literal: 7 bytes
• Lifetime: 1 byte
• Semi: 0 bytes
• Comma: 0 bytes
• Dot: 0 bytes
• OpenParen: 0 bytes
• CloseParen: 0 bytes
• OpenBrace: 0 bytes
• CloseBrace: 0 bytes
• OpenBracket: 0 bytes
• CloseBracket: 0 bytes
• At: 0 bytes
• Pound: 0 bytes
• Tilde: 0 bytes
• Question: 0 bytes
• Colon: 0 bytes
• Dollar: 0 bytes
• Eq: 0 bytes
• Bang: 0 bytes
• Lt: 0 bytes
• Gt: 0 bytes
• Minus: 0 bytes
• And: 0 bytes
• Or: 0 bytes
• Plus: 0 bytes
• Star: 0 bytes
• Slash: 0 bytes
• Caret: 0 bytes
• Percent: 0 bytes
• Unknown: 0 bytes