Module rustc_ast::tokenstream
source · Expand description
Token Streams
TokenStreams represent syntactic objects before they are converted into ASTs.
A TokenStream is, roughly speaking, a sequence of TokenTrees,
which are themselves a single Token or a Delimited subsequence of tokens.
Ownership
TokenStreams are persistent data structures constructed as ropes with reference
counted-children. In general, this means that calling an operation on a TokenStream
(such as slice) produces an entirely new TokenStream from the borrowed reference to
the original. This essentially coerces TokenStreams into “views” of their subparts,
and a borrowed TokenStream is sufficient to build an owned TokenStream without taking
ownership of the original.
Structs
An
AttrTokenStream is similar to a TokenStream, but with extra
information about the tokens for attribute targets. This is used
during expansion to perform early cfg-expansion, and to process attributes
during proc-macro invocations.Stores the tokens for an attribute target, along
with its attributes.
Owning by-value iterator over a
TokenStream.By-reference iterator over a
TokenStream.A lazy version of
TokenStream, which defers creation
of an actual TokenStream until it is needed.
Box is here only to reduce the structure size.A
TokenStream is an abstract sequence of tokens, organized into TokenTrees.Enums
Like
TokenTree, but for AttrTokenStream.When the main Rust parser encounters a syntax-extension invocation, it
parses the arguments to the invocation as a token tree. This is a very
loose structure, such that all sorts of different AST fragments can
be passed to syntax extensions using a uniform type.