diff options
Diffstat (limited to 'doc/source/parsesteps.ht')
| -rw-r--r-- | doc/source/parsesteps.ht | 234 |
1 files changed, 234 insertions, 0 deletions
diff --git a/doc/source/parsesteps.ht b/doc/source/parsesteps.ht new file mode 100644 index 000000000000..ad083dce9d7a --- /dev/null +++ b/doc/source/parsesteps.ht @@ -0,0 +1,234 @@ +<? +ficlPageHeader("ficl parse steps") + +ficlAddToNavBarAs("Parse Steps") + +def entry(definition): + print "<dt>\n<code>" + definition + "</code>\n<dd>\n" + +?> + + +<? ficlHeader1("Parse Steps") ?> + +Unlike every other FORTH we know of, Ficl features an <i>extensible +parser chain</i>. The Ficl parser is not a monolithic function; instead, +it is comprised of a simple tokenizer and a series of <i>parse steps</i>. +A parse step is a step in the parser chain that handles a particular kind +of token, acting on the token as appropriate. Example parse steps, in +terms of traditional FORTH lore, would be the "number runner" and the +"colon compiler". +<p> + +The Ficl parser works like this: +<ol> + +<li> +Read in a new <i>token</i> (string of text with no internal whitespace). + +<li> +For each parse step in the chain, call the parse step, passing in the token. +If the parse step returns <code>FICL_TRUE</code>, that parse step must have +handled the token appropriately; move on to the next token. + +<li> +If the parser tries all the parse steps and none of them return +<code>FICL_TRUE</code>, the token is illegal—print an error +and reset the virtual machine. + +</ol> + +Parse steps can be written as native functions, or as Ficl script functions. +New parse steps can be appended to the chain at any time. + + +<? ficlHeader2("The Default Ficl Parse Chain") ?> + +These is the default Ficl parser chain, shown in order. + +<dl> + +<? entry("?word") ?> + +If compiling and local variable support is enabled, attempt to find the token in the local +variable dictionary. If found, execute the token's compilation semantics and return <code>FICL_TRUE</code>. +<p> + +Attempt to find the token in the system dictionary. If found, execute the token's semantics +(may be different when compiling than when interpreting) and return <code>FICL_TRUE</code>. + +<? entry("?prefix") ?> +This parse step is only active if prefix support is enabled, setting <code>FICL_WANT_PREFIX</code> +in <code>ficl.h</code> to a non-zero value. +Attempt to match the beginning of the token to the list of known prefixes. If there's a match, +execute the associated prefix method and return <code>FICL_TRUE</code>. + +<? entry("?number") ?> +Attempt to convert the token to a number in the present <code>BASE</code>. If successful, push the +value onto the stack if interpreting, otherwise compile it, then return <code>FICL_TRUE</code>. + +<? entry("?float") ?> +This parse step is only active if floating-point number support is enabled, +setting <code>FICL_WANT_FLOAT</code> in <code>ficl.h</code> to a non-zero value. +Attempt to convert the token to a floating-point number. If successful, push the +value onto the floating-point stack if interpreting, otherwise compile it, +then return <code>FICL_TRUE</code>. + +</dl> + + + +<? ficlHeader2("Adding A Parse Step From Within Ficl") ?> +<a name=ficlparsestep></a> + +You can add a parse step in two ways. The first is to write a Ficl word that +has the correct stack signature for a parse step: +<pre> +<i>MY-PARSE-STEP</i> ( c-addr u -- x*i flag ) +</pre> +where <code>c-addr u</code> are the address and length of the incoming token, +and <code>flag</code> is <code>FICL_TRUE</code> if the parse step processed +the token and <code>FICL_FALSE</code> otherwise. +<p> + +Install the parse step using <code>add-parse-step</code>. +A trivial example: +<pre> +: ?silly ( c-addr u -- flag ) + ." Oh no! Not another " type cr true ; +' ?silly add-parse-step +parse-order +</pre> + +<? ficlHeader2("Adding A Native Parse Step") ?> + +The other way to add a parse step is to write it in C and add it into the +parse chain with the following function: + +<pre> +void ficlSystemAddPrimitiveParseStep(ficlSystem *system, char *name, ficlParseStep step); +</pre> + +<code>name</code> is the display name of the parse step in the parse chain +(as displayed by the Ficl word <code>PARSE-ORDER</code>). <code>step</code> +is a pointer to the code for the parse step itself, +and must match the following declaration: +<pre> +typedef int (*ficlParseStep)(ficlVm *vm, ficlString s); +</pre> +<p> + +When a native parse step is run, <code>si</code> points to the incoming token. +The parse step must return <code>FICL_TRUE</code> if it succeeds in handling the +token, and <code>FICL_FALSE</code> otherwise. +See <code>ficlVmParseNumber()</code> in <code>system.c</code> for an example. + + +<? ficlHeader1("Prefixes") ?> + +What's a prefix, anyway? A prefix (contributed by Larry Hastings) is a token that's +recognized as the beginning of another token. Its presence modifies the semantics of +the rest of the token. An example is <code>0x</code>, which causes digits following +it to be converted to hex regardless of the current value of <code>BASE</code>. +<p> + +Caveat: Prefixes are matched in sequence, so the more of them there are, +the slower the interpreter gets. On the other hand, because the prefix +parse step occurs immediately after the dictionary lookup step, if you +have a prefix for a particular purpose, using it may save time since it +stops the parse process. Also, the Ficl interpreter is wonderfully fast, +and most interpretation only happens once, so it's likely you won't notice +any change in interpreter speed even if you make heavy use of prefixes. +<p> + +Each prefix is a Ficl word stored in a special wordlist called <code><PREFIXES></code>. When the +prefix parse step (<code>?prefix</code>, implemented in C as <code>ficlVmParsePrefix()</code>) is +executed, it searches each word in <code><PREFIXES></code> in turn, comparing it with the +initial characters of the incoming token. If a prefix matches, the parse step returns the remainder +of the token to the input stream and executes the code associated with the prefix. This code can be +anything you like, but it would typically do something with the remainder of the token. If the prefix +code does not consume the rest of the token, it will go through the parse process again (which may +be what you want). +<p> + +Prefixes are defined in <code>prefix.c</code> and in <code>softcore/prefix.fr</code>. +The best way to add prefixes is by defining them in your own code, bracketed with the special +words <code>START-PREFIXES</code> and <code>END-PREFIXES</code>. For example, the following +code would make <code>.(</code> a prefix. + +<pre> +start-prefixes +: .( .( ; +end-prefixes +</pre> +<p> + +The compile-time constant <code>FICL_EXTENDED_PREFIX</code> controls the inclusion of +several additional prefixes. This is turned off in the default build, since several +of these prefixes alter standard behavior, but you might like them. + + +<? ficlHeader1("Notes") ?> + +<ul> + +<li> +Prefixes and parser extensions are non-standard. However, with the exception of +prefix support, Ficl's default parse order follows the standard. +Inserting parse steps in some other order will almost certainly break standard behavior. +<p> + +<li> +The number of parse steps that can be added to the system is limited by the value of +<code>FICL_MAX_PARSE_STEPS</code> (defined in <code>sysdep.h</code>). The default +maximum number is 8. +<p> + +<li> +The compile-time constant <code>FICL_EXTENDED_PREFIX</code> controls the inclusion of +several additional prefixes. This is turned off in the default build, since several +of these prefixes alter standard behavior, but you might like them. +<p> + + +</ul> + +<? ficlHeader1("Parser Glossary") ?> + +<dl> + +<? entry("PARSE-ORDER ( -- )") ?> + +Prints the list of parse steps, in the order in which they are called. + +<? entry("ADD-PARSE-STEP ( xt -- )") ?> + +Appends a parse step to the parse chain. <code>xt</code> is the address +(execution token) of a Ficl word to use as the parse step. The word must be a +legal Ficl parse step (<a href=#ficlparsestep>see above</a>). + +<? entry("SHOW-PREFIXES ( -- )") ?> + +Prints the list of all prefixes. Each prefix is a Ficl word that is executed if its name +is found at the beginning of a token. + +<? entry("START-PREFIXES ( -- )") ?> + +Declares the beginning of a series of prefix definitions. +Should be followed, eventually, by <code>END-PREFIXES</code>. +(All <code>START-PREFIXES</code> does is tell the Ficl virtual machine +to compile into the <code><PREFIXES></code> wordlist.) + +<? entry("END-PREFIXES ( -- )") ?> + +Declares the end of a series of prefix definitions. +Should only be used after calling <code>START-PREFIXES</code>. +(All <code>END-PREFIXES</code> does is tell the Ficl virtual machine +to switch back to the wordlist that was in use before <code>START-PREFIXES</code> was called.) + +</dl> + + +<? +ficlPageFooter() +?>
\ No newline at end of file |