Overview

Miso.Util.Lexer is an internal lexer combinator library used by miso's JSON pipeline (Miso.JSON.Lexer) and URI/router tokeniser (Miso.Router). It is not designed for general application use, but is exposed for downstream code that needs to build custom lexers.

The central type is Lexer:

newtype Lexer token = Lexer
  { runLexer :: Stream -> Either LexerError (token, Stream) }

Lexer is a Monad, Alternative, and MonadFail. Its Alternative instance implements the maximal munch rule: when both branches succeed, the one that consumes the most input wins.

Key types

• Stream — the remaining input text together with a Location (line and column) cursor.
• Location — {line :: Int, column :: (Int, Int)}, used in error messages and Located tokens.
• Located token — a lexed value paired with the Location at which it was recognised.
• LexerError — either LexerError MisoString Location (unexpected input) or UnexpectedEOF Location (ran out of input).

Primitive combinators

satisfy  :: (Char -> Bool) -> Lexer Char   -- consume one matching char
char     :: Char -> Lexer Char              -- consume a specific char
string   :: MisoString -> Lexer MisoString   -- consume a literal prefix
string'  :: String -> Lexer String          -- same for String
peek     :: Lexer (Maybe Char)              -- look ahead without consuming
oops     :: Lexer token                     -- always fails

Stream and location access

getInput     :: Lexer Stream
putInput     :: Stream -> Lexer ()
modifyInput  :: (Stream -> Stream) -> Lexer ()
getLocation  :: Lexer Location
setLocation  :: Location -> Lexer ()

Running a lexer

runLexer :: Lexer token -> Stream -> Either LexerError (token, Stream)
mkStream :: MisoString -> Stream   -- create initial stream

See also

• Miso.Util.Parser — parser combinator library that consumes Lexer output
• Miso.JSON.Lexer — JSON tokeniser built on this module
• Miso.Router — URI tokeniser built on this module
• Miso.Util — higher-level sepBy, oneOf, …