pub enum TokenKind {
Show 38 variants LineComment { doc_style: Option<DocStyle>, }, BlockComment { doc_style: Option<DocStyle>, terminated: bool, }, Whitespace, Ident, InvalidIdent, RawIdent, UnknownPrefix, Literal { kind: LiteralKind, suffix_start: u32, }, Lifetime { starts_with_number: bool, }, Semi, Comma, Dot, OpenParen, CloseParen, OpenBrace, CloseBrace, OpenBracket, CloseBracket, At, Pound, Tilde, Question, Colon, Dollar, Eq, Bang, Lt, Gt, Minus, And, Or, Plus, Star, Slash, Caret, Percent, Unknown, Eof,
}
Expand description

Enum representing common lexeme types.

Variants§

§

LineComment

Fields

§doc_style: Option<DocStyle>

“// comment”

§

BlockComment

Fields

§doc_style: Option<DocStyle>
§terminated: bool

/* block comment */

Block comments can be recursive, so a sequence like /* /* */ will not be considered terminated and will result in a parsing error.

§

Whitespace

Any whitespace character sequence.

§

Ident

“ident” or “continue”

At this step, keywords are also considered identifiers.

§

InvalidIdent

Like the above, but containing invalid unicode codepoints.

§

RawIdent

“r#ident”

§

UnknownPrefix

An unknown prefix, like foo#, foo', foo".

Note that only the prefix (foo) is included in the token, not the separator (which is lexed as its own distinct token). In Rust 2021 and later, reserved prefixes are reported as errors; in earlier editions, they result in a (allowed by default) lint, and are treated as regular identifier tokens.

§

Literal

Fields

§suffix_start: u32

Examples: 12u8, 1.0e-40, b"123". Note that _ is an invalid suffix, but may be present here on string and float literals. Users of this type will need to check for and reject that case.

See LiteralKind for more details.

§

Lifetime

Fields

§starts_with_number: bool

“’a”

§

Semi

“;”

§

Comma

“,”

§

Dot

“.”

§

OpenParen

“(”

§

CloseParen

“)”

§

OpenBrace

“{”

§

CloseBrace

“}”

§

OpenBracket

“[”

§

CloseBracket

“]”

§

At

“@”

§

Pound

“#”

§

Tilde

“~”

§

Question

“?”

§

Colon

“:”

§

Dollar

“$”

§

Eq

“=”

§

Bang

“!”

§

Lt

“<”

§

Gt

“>”

§

Minus

“-”

§

And

“&”

§

Or

“|”

§

Plus

“+”

§

Star

“*”

§

Slash

“/”

§

Caret

“^”

§

Percent

“%”

§

Unknown

Unknown token, not expected by the lexer, e.g. “№”

§

Eof

End of input.

Trait Implementations§

source§

impl Clone for TokenKind

source§

fn clone(&self) -> TokenKind

Returns a copy of the value. Read more
1.0.0 · source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
source§

impl Debug for TokenKind

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
source§

impl PartialEq<TokenKind> for TokenKind

source§

fn eq(&self, other: &TokenKind) -> bool

This method tests for self and other values to be equal, and is used by ==.
1.0.0 · source§

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.
source§

impl Copy for TokenKind

source§

impl Eq for TokenKind

source§

impl StructuralEq for TokenKind

source§

impl StructuralPartialEq for TokenKind

Auto Trait Implementations§

Blanket Implementations§

source§

impl<T> Any for Twhere T: 'static + ?Sized,

source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
source§

impl<T> Borrow<T> for Twhere T: ?Sized,

source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
source§

impl<T> BorrowMut<T> for Twhere T: ?Sized,

source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
source§

impl<T> From<T> for T

source§

fn from(t: T) -> T

Returns the argument unchanged.

source§

impl<T, U> Into<U> for Twhere U: From<T>,

source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

source§

impl<T> ToOwned for Twhere T: Clone,

§

type Owned = T

The resulting type after obtaining ownership.
source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
source§

impl<T, U> TryFrom<U> for Twhere U: Into<T>,

§

type Error = Infallible

The type returned in the event of a conversion error.
source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
source§

impl<T, U> TryInto<U> for Twhere U: TryFrom<T>,

§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.

Layout§

Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...) attributes. Please see the Rust Reference's “Type Layout” chapter for details on type layout guarantees.

Size: 8 bytes

Size for each variant:

  • LineComment: 1 byte
  • BlockComment: 2 bytes
  • Whitespace: 0 bytes
  • Ident: 0 bytes
  • InvalidIdent: 0 bytes
  • RawIdent: 0 bytes
  • UnknownPrefix: 0 bytes
  • Literal: 7 bytes
  • Lifetime: 1 byte
  • Semi: 0 bytes
  • Comma: 0 bytes
  • Dot: 0 bytes
  • OpenParen: 0 bytes
  • CloseParen: 0 bytes
  • OpenBrace: 0 bytes
  • CloseBrace: 0 bytes
  • OpenBracket: 0 bytes
  • CloseBracket: 0 bytes
  • At: 0 bytes
  • Pound: 0 bytes
  • Tilde: 0 bytes
  • Question: 0 bytes
  • Colon: 0 bytes
  • Dollar: 0 bytes
  • Eq: 0 bytes
  • Bang: 0 bytes
  • Lt: 0 bytes
  • Gt: 0 bytes
  • Minus: 0 bytes
  • And: 0 bytes
  • Or: 0 bytes
  • Plus: 0 bytes
  • Star: 0 bytes
  • Slash: 0 bytes
  • Caret: 0 bytes
  • Percent: 0 bytes
  • Unknown: 0 bytes
  • Eof: 0 bytes