Enum TokenKind

pub enum TokenKind {
Show 42 variants    LineComment {
        doc_style: Option<DocStyle>,
    },
    BlockComment {
        doc_style: Option<DocStyle>,
        terminated: bool,
    },
    Whitespace,
    Ident,
    InvalidIdent,
    RawIdent,
    UnknownPrefix,
    UnknownPrefixLifetime,
    RawLifetime,
    InvalidPrefix,
    GuardedStrPrefix,
    Literal {
        kind: LiteralKind,
        suffix_start: u32,
    },
    Lifetime {
        starts_with_number: bool,
    },
    Semi,
    Comma,
    Dot,
    OpenParen,
    CloseParen,
    OpenBrace,
    CloseBrace,
    OpenBracket,
    CloseBracket,
    At,
    Pound,
    Tilde,
    Question,
    Colon,
    Dollar,
    Eq,
    Bang,
    Lt,
    Gt,
    Minus,
    And,
    Or,
    Plus,
    Star,
    Slash,
    Caret,
    Percent,
    Unknown,
    Eof,
}

Expand description

Enum representing common lexeme types.

Variants§

LineComment

“// comment”

Fields

§doc_style: Option<DocStyle>

BlockComment

/* block comment */

Block comments can be recursive, so a sequence like /* /* */ will not be considered terminated and will result in a parsing error.

Fields

§doc_style: Option<DocStyle>

§terminated: bool

Whitespace

Any whitespace character sequence.

Ident

“ident” or “continue”

At this step, keywords are also considered identifiers.

InvalidIdent

Like the above, but containing invalid unicode codepoints.

RawIdent

“r#ident”

UnknownPrefix

An unknown prefix, like foo#, foo', foo".

Note that only the prefix (foo) is included in the token, not the separator (which is lexed as its own distinct token). In Rust 2021 and later, reserved prefixes are reported as errors; in earlier editions, they result in a (allowed by default) lint, and are treated as regular identifier tokens.

UnknownPrefixLifetime

An unknown prefix in a lifetime, like 'foo#.

Note that like above, only the ' and prefix are included in the token and not the separator.

RawLifetime

'r#lt, which in edition < 2021 is split into several tokens: 'r # lt.

InvalidPrefix

Similar to the above, but always an error on every edition. This is used for emoji identifier recovery, as those are not meant to be ever accepted.

GuardedStrPrefix

Guarded string literal prefix: #" or ##.

Used for reserving “guarded strings” (RFC 3598) in edition 2024. Split into the component tokens on older editions.

Literal

Examples: 12u8, 1.0e-40, b"123". Note that _ is an invalid suffix, but may be present here on string and float literals. Users of this type will need to check for and reject that case.

See LiteralKind for more details.

Fields

§kind: LiteralKind

§suffix_start: u32

Lifetime

“’a”

Fields

§starts_with_number: bool

Semi

“;”

Comma

“,”

Dot

“.”

OpenParen

“(”

CloseParen

“)”

OpenBrace

“{”

CloseBrace

“}”

OpenBracket

“[”

CloseBracket

“]”

At

“@”

Pound

“#”

Tilde

“~”

Question

“?”

Colon

“:”

Dollar

“$”

Eq

“=”

Bang

“!”

Lt

“<”

Gt

“>”

Minus

“-”

And

“&”

Or

“|”

Plus

“+”

Star

“*”

Slash

“/”

Caret

“^”

Percent

“%”

Unknown

Unknown token, not expected by the lexer, e.g. “№”

Eof

End of input.

Trait Implementations§

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a copy of the value. Read more

1.0.0 · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.

1.0.0 · Source§

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenKind

impl Eq for TokenKind

impl StructuralPartialEq for TokenKind

Auto Trait Implementations§

impl Freeze for TokenKind

impl RefUnwindSafe for TokenKind

impl Send for TokenKind

impl Sync for TokenKind

impl Unpin for TokenKind

impl UnwindSafe for TokenKind

Blanket Implementations§

impl<T> Any for T
where T: 'static + ?Sized,

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more

impl<T> Borrow<T> for T
where T: ?Sized,

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more

impl<T> BorrowMut<T> for T
where T: ?Sized,

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more

impl<T> CloneToUninit for T
where T: Clone,

unsafe fn clone_to_uninit(&self, dst: *mut T)

🔬This is a nightly-only experimental API. (clone_to_uninit)

Performs copy-assignment from self to dst. Read more

impl<T> From<T> for T

fn from(t: T) -> T

Returns the argument unchanged.

impl<T, U> Into<U> for T
where U: From<T>,

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

impl<T> ToOwned for T
where T: Clone,

type Owned = T

The resulting type after obtaining ownership.

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more

impl<T, U> TryFrom<U> for T
where U: Into<T>,

type Error = Infallible

The type returned in the event of a conversion error.

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.

Layout§

Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...) attributes. Please see the Rust Reference's “Type Layout” chapter for details on type layout guarantees.

Size: 8 bytes

Size for each variant: