Enum rustc_ast::token::TokenKind

pub enum TokenKind {
    Eq,
    Lt,
    Le,
    EqEq,
    Ne,
    Ge,
    Gt,
    AndAnd,
    OrOr,
    Not,
    Tilde,
    BinOp(BinOpToken),
    BinOpEq(BinOpToken),
    At,
    Dot,
    DotDot,
    DotDotDot,
    DotDotEq,
    Comma,
    Semi,
    Colon,
    ModSep,
    RArrow,
    LArrow,
    FatArrow,
    Pound,
    Dollar,
    Question,
    SingleQuote,
    OpenDelim(Delimiter),
    CloseDelim(Delimiter),
    Literal(Lit),
    Ident(Symbol, bool),
    Lifetime(Symbol),
    Interpolated(Lrc<Nonterminal>),
    DocComment(CommentKind, AttrStyle, Symbol),
    Eof,
}

Variants

Eq

Lt

Le

EqEq

Ne

Ge

Gt

AndAnd

OrOr

Not

Tilde

BinOp(BinOpToken)

BinOpEq(BinOpToken)

At

Dot

DotDot

DotDotDot

DotDotEq

Comma

Semi

Colon

ModSep

RArrow

LArrow

FatArrow

Pound

Dollar

Question

SingleQuote

Used by proc macros for representing lifetimes, not generated by lexer right now.

OpenDelim(Delimiter)

An opening delimiter (e.g., {).

CloseDelim(Delimiter)

A closing delimiter (e.g., }).

Literal(Lit)

Ident(Symbol, bool)

Identifier token. Do not forget about NtIdent when you want to match on identifiers. It’s recommended to use Token::(ident,uninterpolate,uninterpolated_span) to treat regular and interpolated identifiers in the same way.

Lifetime(Symbol)

Lifetime identifier token. Do not forget about NtLifetime when you want to match on lifetime identifiers. It’s recommended to use Token::(lifetime,uninterpolate,uninterpolated_span) to treat regular and interpolated lifetime identifiers in the same way.

Interpolated(Lrc<Nonterminal>)

An embedded AST node, as produced by a macro. This only exists for historical reasons. We’d like to get rid of it, for multiple reasons.

• It’s conceptually very strange. Saying a token can contain an AST node is like saying, in natural language, that a word can contain a sentence.
• It requires special handling in a bunch of places in the parser.
• It prevents Token from implementing Copy. It adds complexity and likely slows things down. Please don’t add new occurrences of this token kind!

DocComment(CommentKind, AttrStyle, Symbol)

A doc comment token. Symbol is the doc comment’s data excluding its “quotes” (///, /**, etc) similarly to symbols in string literal tokens.

Eof

Implementations

impl TokenKind

pub fn lit(kind: LitKind, symbol: Symbol, suffix: Option<Symbol>) -> TokenKind

pub fn break_two_token_op(&self) -> Option<(TokenKind, TokenKind)>

An approximation to proc-macro-style single-character operators used by rustc parser. If the operator token can be broken into two tokens, the first of which is single-character, then this function performs that operation, otherwise it returns None.

pub fn similar_tokens(&self) -> Option<Vec<TokenKind>>

Returns tokens that are likely to be typed accidentally instead of the current token. Enables better error recovery when the wrong token is found.

pub fn should_end_const_arg(&self) -> bool

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<__D: Decoder> Decodable<__D> for TokenKind

fn decode(__decoder: &mut __D) -> Self

impl<__E: Encoder> Encodable<__E> for TokenKind

fn encode(&self, __encoder: &mut __E)

impl<__CTX> HashStable<__CTX> for TokenKindwhere
    __CTX: HashStableContext,

fn hash_stable(&self, __hcx: &mut __CTX, __hasher: &mut StableHasher)

impl PartialEq<TokenKind> for Token

fn eq(&self, rhs: &TokenKind) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<TokenKind> for TokenKind

fn eq(&self, other: &TokenKind) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for TokenKind

Layout

Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...) attributes. Please see the Rust Reference’s “Type Layout” chapter for details on type layout guarantees.

Size: 16 bytes

Size for each variant:

• Eq: 0 bytes
• Lt: 0 bytes
• Le: 0 bytes
• EqEq: 0 bytes
• Ne: 0 bytes
• Ge: 0 bytes
• Gt: 0 bytes
• AndAnd: 0 bytes
• OrOr: 0 bytes
• Not: 0 bytes
• Tilde: 0 bytes
• BinOp: 1 byte
• BinOpEq: 1 byte
• At: 0 bytes
• Dot: 0 bytes
• DotDot: 0 bytes
• DotDotDot: 0 bytes
• DotDotEq: 0 bytes
• Comma: 0 bytes
• Semi: 0 bytes
• Colon: 0 bytes
• ModSep: 0 bytes
• RArrow: 0 bytes
• LArrow: 0 bytes
• FatArrow: 0 bytes
• Pound: 0 bytes
• Dollar: 0 bytes
• Question: 0 bytes
• SingleQuote: 0 bytes
• OpenDelim: 1 byte
• CloseDelim: 1 byte
• Literal: 15 bytes
• Ident: 7 bytes
• Lifetime: 7 bytes
• Interpolated: 15 bytes
• DocComment: 7 bytes
• Eof: 0 bytes