5.13.5 Recognizers

The standard Forth text interpreter recognizes the following types of tokens: words in the dictionary, integer numbers, and floating point numbers. Defining new types of tokens isn’t yet standardized. Gforth provides recognizers to make the text interpreter extensible as well.

Recognizers take a string and return some data and a “table” for interpreting that data. Gforth implements that table as xt (which means any xt is a valid result of a recognizer), but other Forth systems can implement it as actual table, with three xts inside. The first xt is the interpretation/run-time xt, it performs the interpretation semantics on the data (usually, this means it just leaves the data on the stack). The second xt performs the compilation semantics, it gets the data and the run-time semantics xt. The third xt perfoms the postpone semantics, it also gets the data and the run-time semantics xt. You can use post, to postpone the run-time xt.

Recognizers are organized as stack, so you can arrange the sequence of recognizers in the same way as the vocabulary stack.

notfound ( ) unknown “notfound”

If a recognizer fails, it returns notfound

rec-nt ( addr u – nt recognized-nt | notfound  ) unknown “rec-nt”

recognize a name token

rec-num ( addr u – n/d table | notfound  ) unknown “rec-num”

converts a number to a single/double integer

rec-float ( addr u – r recognized-float | notfound  ) unknown “rec-float”

recognize floating point numbers

rec-string ( addr u – addr u’ r:string | rectype-null  ) unknown “rec-string”

Convert strings enclosed in double quotes into string literals, escapes are treated as in S\".

rec-to ( addr u – xt r:to | rectype-null  ) unknown “rec-to”

words prefixed with -> are treated as if preceeded by TO or IS, with +> as +TO and with '> as ADDR.

rec-tick ( addr u – xt rectype-num | rectype-null  ) unknown “rec-tick”

words prefixed with ` return their xt. Example: `dup gives the xt of dup

rec-dtick ( addr u – nt rectype-num | rectype-null  ) unknown “rec-dtick”

words prefixed with `` return their nt. Example: ``S" gives the nt of S"

rec-body ( addr u – xt recognized-tick | recognized-null  ) unknown “rec-body”

words bracketed with '<' '>' return their body. Example: <dup> gives the body of dup

get-recognizers ( – xt1 .. xtn n  ) unknown “get-recognizers”

push the content on the recognizer stack

set-recognizers ( xt1 .. xtn n  ) unknown “set-recognizers”

set the recognizer stack from content on the stack

recognize ( addr u rec-addr – ... rectype  ) unknown “recognize”

apply a recognizer stack to a string, delivering a token

rec-sequence: ( x1 .. xn n "name" –  ) unknown “rec-sequence:”

forth-recognize ( ) unknown “forth-recognize”