Struct rustc_ast::tokenstream::TokenStream

pub struct TokenStream(pub(crate) Lrc<Vec<TokenTree>>);

A TokenStream is an abstract sequence of tokens, organized into TokenTrees.

The goal is for procedural macros to work with TokenStreams and TokenTrees instead of a representation of the abstract syntax tree. Today’s TokenTrees can still contain AST via token::Interpolated for backwards compatibility.

Tuple Fields

0: Lrc<Vec<TokenTree>>

Implementations

impl TokenStream

pub fn add_comma(&self) -> Option<(TokenStream, Span)>

Given a TokenStream with a Stream of only two arguments, return a new TokenStream separating the two arguments with a comma for diagnostic suggestions.

impl TokenStream

pub fn new(streams: Vec<TokenTree>) -> TokenStream

pub fn is_empty(&self) -> bool

pub fn len(&self) -> usize

pub fn trees(&self) -> RefTokenTreeCursor<'_>

pub fn into_trees(self) -> TokenTreeCursor

pub fn eq_unspanned(&self, other: &TokenStream) -> bool

Compares two TokenStreams, checking equality without regarding span information.

pub fn map_enumerated_owned( self, f: impl FnMut(usize, TokenTree) -> TokenTree ) -> TokenStream

Applies the supplied function to each TokenTree and its index in self, returning a new TokenStream

It is equivalent to TokenStream::new(self.trees().cloned().enumerate().map(|(i, tt)| f(i, tt)).collect()).

pub fn token_alone(kind: TokenKind, span: Span) -> TokenStream

Create a token stream containing a single token with alone spacing.

pub fn token_joint(kind: TokenKind, span: Span) -> TokenStream

Create a token stream containing a single token with joint spacing.

pub fn delimited( span: DelimSpan, delim: Delimiter, tts: TokenStream ) -> TokenStream

Create a token stream containing a single Delimited.

pub fn from_ast( node: &(impl HasAttrs + HasSpan + HasTokens + Debug) ) -> TokenStream

pub fn from_nonterminal_ast(nt: &Nonterminal) -> TokenStream

fn flatten_token(token: &Token, spacing: Spacing) -> TokenTree

fn flatten_token_tree(tree: &TokenTree) -> TokenTree

pub fn flattened(&self) -> TokenStream

fn try_glue_to_last(vec: &mut Vec<TokenTree>, tt: &TokenTree) -> bool

pub fn push_tree(&mut self, tt: TokenTree)

Push tt onto the end of the stream, possibly gluing it to the last token. Uses make_mut to maximize efficiency.

pub fn push_stream(&mut self, stream: TokenStream)

Push stream onto the end of the stream, possibly gluing the first token tree to the last token. (No other token trees will be glued.) Uses make_mut to maximize efficiency.

pub fn chunks(&self, chunk_size: usize) -> Chunks<'_, TokenTree>

pub fn desugar_doc_comments(&mut self)

Desugar doc comments like /// foo in the stream into #[doc = r"foo"]. Modifies the TokenStream via Lrc::make_mut, but as little as possible.

Trait Implementations

impl Clone for TokenStream

fn clone(&self) -> TokenStream

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<__D: Decoder> Decodable<__D> for TokenStream

fn decode(__decoder: &mut __D) -> Self

impl Default for TokenStream

fn default() -> TokenStream

Returns the “default value” for a type.

impl<__E: Encoder> Encodable<__E> for TokenStream

fn encode(&self, __encoder: &mut __E)

impl FromIterator<TokenTree> for TokenStream

fn from_iter<I: IntoIterator<Item = TokenTree>>(iter: I) -> Self

Creates a value from an iterator.

impl<CTX> HashStable<CTX> for TokenStreamwhere CTX: HashStableContext,

fn hash_stable(&self, hcx: &mut CTX, hasher: &mut StableHasher)

impl PartialEq<TokenStream> for TokenStream

fn eq(&self, other: &TokenStream) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for TokenStream

Layout

Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...) attributes. Please see the Rust Reference's “Type Layout” chapter for details on type layout guarantees.

Size: 8 bytes