Module rustc_ast::tokenstream
source · Expand description
Token Streams
TokenStream
s represent syntactic objects before they are converted into ASTs.
A TokenStream
is, roughly speaking, a sequence of TokenTree
s,
which are themselves a single Token
or a Delimited
subsequence of tokens.
Ownership
TokenStream
s are persistent data structures constructed as ropes with reference
counted-children. In general, this means that calling an operation on a TokenStream
(such as slice
) produces an entirely new TokenStream
from the borrowed reference to
the original. This essentially coerces TokenStream
s into “views” of their subparts,
and a borrowed TokenStream
is sufficient to build an owned TokenStream
without taking
ownership of the original.
Structs
- An
AttrTokenStream
is similar to aTokenStream
, but with extra information about the tokens for attribute targets. This is used during expansion to perform early cfg-expansion, and to process attributes during proc-macro invocations. - Stores the tokens for an attribute target, along with its attributes.
- A lazy version of
TokenStream
, which defers creation of an actualTokenStream
until it is needed.Box
is here only to reduce the structure size. - By-reference iterator over a
TokenStream
, that produces&TokenTree
items. - A
TokenStream
is an abstract sequence of tokens, organized intoTokenTree
s. - Owning by-value iterator over a
TokenStream
, that produces&TokenTree
items.
Enums
- Like
TokenTree
, but forAttrTokenStream
. - Similar to
proc_macro::Spacing
, but for tokens. - When the main Rust parser encounters a syntax-extension invocation, it parses the arguments to the invocation as a token tree. This is a very loose structure, such that all sorts of different AST fragments can be passed to syntax extensions using a uniform type.