Syntax Reference

Main

CSTParser.to_codeobjectMethod
to_codeobject(x::EXPR)

Convert an EXPR into the object that Meta.parse would have produced from the original string, which could e.g. be an Expr, Symbol, or literal.

source
CSTParser.get_sigMethod
get_sig(x)

Returns the full signature of function, macro and datatype definitions. Should only be called when has_sig(x) == true.

source
CSTParser.read_ws_commentMethod
lex_ws_comment(l::Lexer, c)

Having hit an initial whitespace/comment/semicolon continues collecting similar Chars until they end. Returns a WS token with an indication of newlines/ semicolons. Indicating a semicolons takes precedence over line breaks as the former is equivalent to the former in most cases.

source
CSTParser.parseFunction
parse(str, cont = false)

Parses the passed string. If cont is true then will continue parsing until the end of the string returning the resulting expressions in a TOPLEVEL block.

source
CSTParser.parse_compoundMethod
parse_compound(ps::ParseState, ret::EXPR)

Attempts to parse a compound expression given the preceding expression ret.

source
CSTParser.parse_docMethod
parse_doc(ps::ParseState)

Used for top-level parsing - attaches documentation (such as this) to expressions.

source
CSTParser.parse_expressionFunction
parse_expression(ps)

Parses an expression until closer(ps) == true. Expects to enter the ParseState the token before the the beginning of the expression and ends on the last token.

Acceptable starting tokens are:

  • A keyword
  • An opening parentheses or brace.
  • An operator.
  • An instance (e.g. identifier, number, etc.)
  • An @.
source
CSTParser.EXPRType

EXPR represents Julia expressions overlaying a span of bytes in the source text. The full span starts at the first syntactically significant token and includes any trailing whitespace/comments.

Iterating or directly indexing EXPR results in a sequence of child EXPR in source order, including most syntax trivia but not including whitespace, comments and semicolons.

The fields of EXPR are:

  • head represents the type of the expression

    • For internal tree nodes it usually matches the associated Expr's head field. But not always because there's some additional heads, for example :brackets for grouping parentheses, :globalrefdoc, :quotenode, etc
    • For leaf nodes (ie, individual tokens), it's capitalized. Eg, :INTEGER for integer tokens, :END for end, :LPAREN for [, etc.
    • For syntactic operators such as = and <: (which have the operator itself as the expression head in normal Expr), the head is an EXPR.
  • args are the significant subexpressions, in the order used by Base.Expr. For leaf nodes, this is nothing.

  • trivia are any nontrivial tokens which are trivial after parsing.

    • This includes things like the parentheses in (1 + 2), and the keywords in begin x end
    • Whitespace and comments are not included in trivia
  • fullspan is the total number of bytes of text covered by this expression, including any trailing whitespace or comment trivia.

  • span is the number of bytes of text covered by the syntactically relevant part of this expression (ie, not including trailing whitespace or comment trivia).

  • val is the source text covered by span

  • parent is the parent node in the expression tree, or Nothing for the root.

  • meta contains metadata. This includes some ad-hoc information supplied by the parser. (But can also be used downstream in linting or static analysis.)

Whitespace, comments and semicolons are not represented explicitly. Rather, they're tacked onto the end of leaf tokens in args or trivia, in the last fullspan-span bytes of the token.

source
CSTParser._kw_convertMethod
_kw_convert(ps::ParseState, a::EXPR)

Converted an assignment expression to a keyword-argument expression.

source
CSTParser.check_spanFunction

check_span(x, neq = [])

Recursively checks whether the span of an expression equals the sum of the span of its components. Returns a vector of failing expressions.

source
CSTParser.closerMethod

closer(ps::ParseState)

A magical function determining whether the parsing of an expression should continue or stop.

source
CSTParser.compareMethod
compare(x,y)

Recursively checks whether two Base.Expr are the same. Returns unequal sub- expressions.

source
CSTParser.docableMethod
docable(head)

When parsing a block of expressions, can documentation be attached? Prefixed docs at the top-level are handled within parse(ps::ParseState, cont = false).

source
CSTParser.firstdiffMethod
firstdiff(s0::AbstractString, s1::AbstractString)

Returns the last byte index, i, for which s0 and s1 are the same such that: s0[1:i] == s1[1:i]

source
CSTParser.has_errorMethod
has_error(ps::ParseState)
has_error(x::EXPR)

Determine whether a parsing error occured while processing text with the given ParseState, or exists as a (sub) expression of x.

source
CSTParser.revfirstdiffMethod
revfirstdiff(s0::AbstractString, s1::AbstractString)

Reversed version of firstdiff but returns two indices, one for each string.

source
CSTParser.@precedenceMacro
@precedence ps prec body

Continues parsing binary operators until it hits a more loosely binding operator (with precdence lower than prec).

source

Components

CSTParser.parse_blockFunction

Continue parsing statements until an element of closers is hit (usually end). Statements are grouped in a Block EXPR.

source
CSTParser.parse_callFunction
parse_call(ps, ret)

Parses a function call. Expects to start before the opening parentheses and is passed the expression declaring the function name, ret.

source
CSTParser.parse_generatorMethod

parse_generator(ps)

Having hit for not at the beginning of an expression return a generator. Comprehensions are parsed as SQUAREs containing a generator.

source
CSTParser.parse_iteratorFunction

Parses an iterator, allowing for the preceding keyword outer. Returns an error expression if an invalid expression is parsed (anything other than =, in, ).

source
CSTParser.parse_iteratorsMethod
parse_iterators(ps::ParseState, allowfilter = false)

Parses a group of iterators e.g. used in a for loop or generator. Can allow for a succeeding Filter expression.

source
CSTParser.parse_parametersFunction
parse_parameters(ps::ParseState, args::Vector{EXPR}, args1::Vector{EXPR} = EXPR[]; usekw = true)

Parses parameter arguments for a function call (e.g. following a semicolon).

source
CSTParser.parse_blockexprMethod
parse_blockexpr(ps::ParseState, head)

General function for parsing block expressions comprised of a series of statements terminated by an end.

source
CSTParser.parse_blockexpr_sigMethod
parse_blockexpr_sig(ps::ParseState, head)

Utility function to parse the signature of a block statement (i.e. any statement preceding the main body of the block). Returns nothing in some cases (e.g. begin end)

source
CSTParser.parse_arrayFunction
parse_array(ps)

Having hit '[' return either:

  • A vect
  • A vcat
  • A ncat
  • A comprehension
  • An array (vcat of hcats)
source
CSTParser.parse_curlyMethod

parse_curly(ps, ret)

Parses the juxtaposition of ret with an opening brace. Parses a comma seperated list.

source
CSTParser.parse_refMethod
parse_ref(ps, ret)

Handles cases where an expression - ret - is followed by [. Parses the following bracketed expression and modifies it's .head appropriately.

source