pub enum TokenKind {
Show 37 variants
Eq,
Lt,
Le,
EqEq,
Ne,
Ge,
Gt,
AndAnd,
OrOr,
Not,
Tilde,
BinOp(BinOpToken),
BinOpEq(BinOpToken),
At,
Dot,
DotDot,
DotDotDot,
DotDotEq,
Comma,
Semi,
Colon,
ModSep,
RArrow,
LArrow,
FatArrow,
Pound,
Dollar,
Question,
SingleQuote,
OpenDelim(Delimiter),
CloseDelim(Delimiter),
Literal(Lit),
Ident(Symbol, bool),
Lifetime(Symbol),
Interpolated(Lrc<Nonterminal>),
DocComment(CommentKind, AttrStyle, Symbol),
Eof,
}
Variants§
Eq
Lt
Le
EqEq
Ne
Ge
Gt
AndAnd
OrOr
Not
Tilde
BinOp(BinOpToken)
BinOpEq(BinOpToken)
At
Dot
DotDot
DotDotDot
DotDotEq
Comma
Semi
Colon
ModSep
RArrow
LArrow
FatArrow
Pound
Dollar
Question
SingleQuote
Used by proc macros for representing lifetimes, not generated by lexer right now.
OpenDelim(Delimiter)
An opening delimiter (e.g., {
).
CloseDelim(Delimiter)
A closing delimiter (e.g., }
).
Literal(Lit)
Ident(Symbol, bool)
Identifier token.
Do not forget about NtIdent
when you want to match on identifiers.
It’s recommended to use Token::(ident,uninterpolate,uninterpolated_span)
to
treat regular and interpolated identifiers in the same way.
Lifetime(Symbol)
Lifetime identifier token.
Do not forget about NtLifetime
when you want to match on lifetime identifiers.
It’s recommended to use Token::(lifetime,uninterpolate,uninterpolated_span)
to
treat regular and interpolated lifetime identifiers in the same way.
Interpolated(Lrc<Nonterminal>)
An embedded AST node, as produced by a macro. This only exists for historical reasons. We’d like to get rid of it, for multiple reasons.
- It’s conceptually very strange. Saying a token can contain an AST node is like saying, in natural language, that a word can contain a sentence.
- It requires special handling in a bunch of places in the parser.
- It prevents
Token
from implementingCopy
. It adds complexity and likely slows things down. Please don’t add new occurrences of this token kind!
DocComment(CommentKind, AttrStyle, Symbol)
A doc comment token.
Symbol
is the doc comment’s data excluding its “quotes” (///
, /**
, etc)
similarly to symbols in string literal tokens.
Eof
Implementations§
source§impl TokenKind
impl TokenKind
pub fn lit(kind: LitKind, symbol: Symbol, suffix: Option<Symbol>) -> TokenKind
sourcepub fn break_two_token_op(&self) -> Option<(TokenKind, TokenKind)>
pub fn break_two_token_op(&self) -> Option<(TokenKind, TokenKind)>
An approximation to proc-macro-style single-character operators used by rustc parser.
If the operator token can be broken into two tokens, the first of which is single-character,
then this function performs that operation, otherwise it returns None
.
sourcepub fn similar_tokens(&self) -> Option<Vec<TokenKind>>
pub fn similar_tokens(&self) -> Option<Vec<TokenKind>>
Returns tokens that are likely to be typed accidentally instead of the current token. Enables better error recovery when the wrong token is found.
pub fn should_end_const_arg(&self) -> bool
Trait Implementations§
source§impl<__CTX> HashStable<__CTX> for TokenKindwhere
__CTX: HashStableContext,
impl<__CTX> HashStable<__CTX> for TokenKindwhere __CTX: HashStableContext,
fn hash_stable(&self, __hcx: &mut __CTX, __hasher: &mut StableHasher)
source§impl PartialEq<TokenKind> for Token
impl PartialEq<TokenKind> for Token
source§impl PartialEq<TokenKind> for TokenKind
impl PartialEq<TokenKind> for TokenKind
impl StructuralPartialEq for TokenKind
Auto Trait Implementations§
impl !RefUnwindSafe for TokenKind
impl !Send for TokenKind
impl !Sync for TokenKind
impl Unpin for TokenKind
impl !UnwindSafe for TokenKind
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Layout§
Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...)
attributes. Please see the Rust Reference's “Type Layout” chapter for details on type layout guarantees.
Size: 16 bytes
Size for each variant:
Eq
: 0 bytesLt
: 0 bytesLe
: 0 bytesEqEq
: 0 bytesNe
: 0 bytesGe
: 0 bytesGt
: 0 bytesAndAnd
: 0 bytesOrOr
: 0 bytesNot
: 0 bytesTilde
: 0 bytesBinOp
: 1 byteBinOpEq
: 1 byteAt
: 0 bytesDot
: 0 bytesDotDot
: 0 bytesDotDotDot
: 0 bytesDotDotEq
: 0 bytesComma
: 0 bytesSemi
: 0 bytesColon
: 0 bytesModSep
: 0 bytesRArrow
: 0 bytesLArrow
: 0 bytesFatArrow
: 0 bytesPound
: 0 bytesDollar
: 0 bytesQuestion
: 0 bytesSingleQuote
: 0 bytesOpenDelim
: 1 byteCloseDelim
: 1 byteLiteral
: 15 bytesIdent
: 7 bytesLifetime
: 7 bytesInterpolated
: 15 bytesDocComment
: 7 bytesEof
: 0 bytes