parser-combinators documentation

The most common entry point for parsing is the parse-string* function. It is a specialized form of parse-string which will return a single parse result, which is the usual requirement. The more general form is useful mostly if the grammar is ambiguous and obtaining all possible parsers for external postprocessing is required.

The parse-string* function takes as an arguments a parser and a sequence. It also takes a &key argument :complete, which, if t, means that only a parse that consumes all the input is considered a success, and the parser will be backtracked until such a result is found or it fails. It returns multiple values. The primary is the parsing result, the secondary indicates whether it was incomplete, third if it was successful (in case NIL, which is normally returned on failure, can also be a result of the parse) and finally an object which registers additional state indicating where the parsing stopped if it was incomplete or failed.

A result of a parser can be an arbitrary object. But, since the parsing process involves backtracking and a certain degree of lazy evaluation, using mutation of objects or global environment is not reliable. If creation of more complex objects during parsing is desired a functional datastructres library is necessary. This is especially important when dealing with cross-cutting references, which might require holding a significant amount of transient state before they are merged at higher level.

The most basic parsers are literal parsers. A literal or a sequence, passed where a parser is expected¹, will create a parser which matches that object literally. This gives the most trivial example:

CL-USER> (parse-string* "ABC" "ABC")
"ABC"
NIL
T
NIL
CL-USER> (parse-string* "ABC" '(#\A #\B #\C))
"ABC"
NIL
T
NIL
CL-USER> (parse-string* "ABC" "ABD")
NIL
NIL
NIL
#<PARSER-COMBINATORS::CONTEXT-FRONT {AAAB041}>

The parser created from the string "ABC" matches a string "ABC", as well as a list of it's characters, but doesn't match a different string.

Most other parsers are obtained by calling parser generating functions². Conventionally most of these functions included in this library end either a question mark or a star. The former are fully backtracking, while the latter don't backtrack, in order to obtain additional performance if the grammar is known to be unambiguous. There are some exceptions, most notably sat, which takes a predicate and returns a parsers which accepts an item for which the predicate is true, and core combinators.

The most basic elements of any grammar are sequences and alternatives. While coming from Haskell/Parsec it might seem natural to use bind to express the former, it is usually an overkill and due to more verbose lambda syntax in Common Lisp it is fairly clunky. For this reason bind is not part of the exported interface.

With core combinators introduced in previous section, backtracking can now be explained in more detail. Backtracking is a way for parser combinators to deal with ambiguity resulting from necessity of considering context, in particular context resulting from parsing items which occur after an ambiguous pattern occurs. This method allows creation of parsers for more general grammars than many parser generators which are limited to, usually, one-character look-ahead. This of course comes with a time and memory cost, but on the other hand allows the parsers to be expressed more declaratively.

Consider an example:

CL-USER> (parse-string* (seq-list? (choice "aaa" "aa")
                                   "aaa")
                        "aaaaa")
("aa" "aaa")

Note that the first argument is ambiguous, since when looking at the input locally, both "aaa" and "aa" match the pattern. A way to make the correct choice is necessary.

What occurs here is that the first argument to seq-list? matches "aaa" first, then the second argument attempts to match and fails, since it runs out of letters "a". When that happens, since seq-list? is a backtracking parser, backtracking occurs, and other possibilities from previous parsers are considered. In this case, the first argument second choice is taken, "aa", and this allows the second argument to match, and the whole parser to succeed.

When ordering choices it is important to put the most likely possibility first, to save as much backtracking as possible. Occasionally, especially when matching literals, putting longest patterns first might be a good idea as well, if there is a possibility that some shorter ones can be a prefix of longer ones.

As mentioned above, sometimes a parser is just ambiguous and we are interested in all possible parsers. In this case the parse-string function is useful³. An example:

CL-USER> (parse-string (seq-list? (choice "aaa" "aa")
                                  (choice "aaa" "aa"))
                       "aaaaa")
#<PARSER-COMBINATORS::PARSE-RESULT {B1F1061}>
#<PARSER-COMBINATORS::CONTEXT-FRONT {B1EDFF1}>
CL-USER> (defparameter *results* (gather-results *))
*RESULTS*
CL-USER> (mapcar #'tree-of *results*)
(("aaa" "aa") ("aa" "aaa") ("aa" "aa"))
CL-USER> (mapcar #'suffix-of *results*)
(#<END-CONTEXT {BB2CC41}> #<END-CONTEXT {BB2CDE9}> #<VECTOR-CONTEXT {BB2CED1}>)

The function gather-results takes the parse-result object and generates all possible results. There are also current-result and next-result, which can be used to access result sequentially. Parsing occurs lazily, so every result requires more parsing, although usually partial. Note that this means that backtracking/parsing state are not released until the parse-result object is garbage collected.

In this case we see that there are three possible parses, two of them consume the whole input (suffix-of gives the remaining input from the result, and in this case first two give end-context), and one has some input remaining. Having multiple results is usually not useful, but occasionally it might be desired to pick one of them by external analysis. This is also useful for testing, since this way one can see all possible backtracks that can be made from a component parser on some test input.

From those core combinators more complex combinators can be constructed⁴. Most basic of those are repetition combinators, which take a parser and perhaps some additional information and return a sequence of matches. Most general repetition operators are between? and breadth?. They both take a parser, a minimal and maximal number of occurrences, either of which can be nil, and optionally a type of the result sequence (a list by default).

The difference between them is that between? will attempt to consume as many matches as possible, unless forced otherwise by backtracking, while breadth? will attempt to consume as few as possible, again, unless forced otherwise by backtracking. In most cases the former is more useful. Usually, more specific forms should be used, like opt?, many?, many1?, times?, atleast? and atmost?. See their docstrings for details, but they are fairly obvious specializations. Those have a non-backtracking versions, except obviously breadth?, but the same note as with sequence combinators matters, only the first result of the argument parser will ever be used.

There are some predefined parsers for common tokens. See their docstrings for details. The built in token parsers are: digit?, lower?, upper?, letter?, alphanum?, whitespace?, word?, nat?, int?, quoted?. Most of those have non-backtracking versions.

There are some built-in parsers which proved some common repetition patterns. If there are any other general and common patters, please submit them. The preexisting ones are sepby1?, sepby? and bracket?. Example:

CL-USER> (parse-string* (bracket? #\[ (sepby? (int?) #\,) #\])
               "[17,22,34]")
(17 22 34)
NIL
T
NIL

Sometimes it is desirable to skip part of the input string until a match can be found. The find? family of parser combinators achieves this. The most basic is find? itself, which skips input until a match can be found. The find-after? will only skip patterns given by its first argument, and the return the result of the second argument parser. The find-after-collect? will collect the skipped items and cons them to the result of the primary parser. The find-before? will collect the skipped items, and return them as a sequence, ignoring the second argument. That is useful if the terminator is part of some other pattern.

While in principle similar to non-backtracking find? versions (which also exists), there is a set of gather combinators, which are not only non-backtracking, but also specialized on input from. This makes them faster, but limited. The gather-before-token*, gather-if* and gather-if-not* operate on input sequence element level and so can traverse it without using the normally necessary context instrumentation. This can be a significant performance gain for recognizing bulk data delimited by single element terminator.

A more complex form of structured repetition are chains. Combinators chainl1? and chainr1? take an item parser, and an operator parser, which should return a function which will be used to reduce the sequence. The former applies the reduction with left associativity, and the latter with right associativity. The most basic application is to transform an infix operators to prefix operators. The file test-arithmetic shows how to use this to parse basic arithmetic expressions. This gist shows an example where the chainl1? operator is used to merge graphs representing molecule fragments in SMILES language.

The generalization of chains is expression? parser generator, which can create a parser for recursive expressions with multiple operators with different associativity and subexpressions. See the test-expression.lisp file for example of simple arithmetic parser.

The library attempts to initialize the parsers as much as possible when they are created. This includes constructing all subparsers. This is a problem for recursive parsers, since it will cause an infinite recursion in the parser construction stage. If no built-in structured combinators fits the problem, there are two ways to solve this.

One is to delay the construction of the parser until it is needed. This can also be useful if the parser requires significant precomputation and might not be used. This can achieved either by using delayed? macro, or as a non-first argument to mdo.

The other method introduces an indirection, which allows the recursive parser to be initialized once. The named? macro will give a name to its body, which then can be passed and called at some lower level. Once more, this gist shows an example. While this requires passing the recursive parser as an argument to the point where it is used, this saves the cost of recreating the parser multiple times and makes the recursion explicit, so it is a preferred approach.

(zero) is a parser generator for a parser which represents a parsing failure

(result v) is a parser generator for a parser which doesn't modify the input and returns v

(item) is a parser generator for a parser which consumes and returns one item from the input

The context protocol is used to abstract the input sequence. Vectors and lists are fully implemented. Unless handlers for new types of input are desired, this is not usually relevant to end users.

Error handling for parser combinators is generally hard, since failure to parse causes backtracking, and it is impossible in general to differentiate a proper backtracking from error in the parser or the input. Some basic ways to deal with are to factor the parser into small units and vigorously unit test them, and to make parsers which are liberal in what they accept as long as it is unambiguous. Using a hierarchical parser (at least separate lexing/parsing pass for languages with obvious tokens) might be helpful a well.

One way to approximately locate the place where the parsing failed is to examine the quaternary return value of the parse-string* function. It is a context-front object, and calling a position-of method on it will show the most advances position in the input which was touched during parsing. There is no guarantee that this will be the location of an actual error, but it often is at least near.