diff options
Diffstat (limited to 'usr.bin/lex/NEWS')
| -rw-r--r-- | usr.bin/lex/NEWS | 703 |
1 files changed, 703 insertions, 0 deletions
diff --git a/usr.bin/lex/NEWS b/usr.bin/lex/NEWS new file mode 100644 index 0000000000000..aff90a392a55a --- /dev/null +++ b/usr.bin/lex/NEWS @@ -0,0 +1,703 @@ +Changes between release 2.4.7 (03Aug94) and release 2.4.6: + + - Fixed serious bug in reading multiple files. + + - Fixed bug in scanning NUL's. + + - Fixed bug in input() returning 8-bit characters. + + - Fixed bug in matching text with embedded NUL's when + using %array or lex compatibility. + + - Fixed multiple invocations of YY_USER_ACTION when using '|' + continuation action. + + - Minor prototyping fixes. + +Changes between release 2.4.6 (04Jan94) and release 2.4.5: + + - Linking with -lfl no longer required if your program includes + its own yywrap() and main() functions. (This change will cause + problems if you have a non-ANSI compiler on a system for which + sizeof(int) != sizeof(void*) or sizeof(int) != sizeof(size_t).) + + - The use of 'extern "C++"' in FlexLexer.h has been modified to + get around an incompatibility with g++'s header files. + + +Changes between release 2.4.5 (11Dec93) and release 2.4.4: + + - Fixed bug breaking C++ scanners that use REJECT or variable + trailing context. + + - Fixed serious input problem for interactive scanners on + systems for which char is unsigned. + + - Fixed bug in incorrectly treating '$' operator as variable + trailing context. + + - Fixed bug in -CF table representation that could lead to + corrupt tables. + + - Fixed fairly benign memory leak. + + - Added `extern "C++"' wrapper to FlexLexer.h header. This + should overcome the g++ 2.5.X problems mentioned in the + NEWS for release 2.4.3. + + - Changed #include of FlexLexer.h to use <> instead of "". + + - Added feature to control whether the scanner attempts to + refill the input buffer once it's exhausted. This feature + will be documented in the 2.5 release. + + +Changes between release 2.4.4 (07Dec93) and release 2.4.3: + + - Fixed two serious bugs in scanning 8-bit characters. + + - Fixed bug in YY_USER_ACTION that caused it to be executed + inappropriately (on the scanner's own internal actions, and + with incorrect yytext/yyleng values). + + - Fixed bug in pointing yyin at a new file and resuming scanning. + + - Portability fix regarding min/max/abs macros conflicting with + function definitions in standard header files. + + - Added a virtual LexerError() method to the C++ yyFlexLexer class + for reporting error messages instead of always using cerr. + + - Added warning in flexdoc that the C++ scanning class is presently + experimental and subject to considerable change between major + releases. + + +Changes between release 2.4.3 (03Dec93) and release 2.4.2: + + - Fixed bug causing fatal scanner messages to fail to print. + + - Fixed things so FlexLexer.h can be included in other C++ + sources. One side-effect of this change is that -+ and -CF + are now incompatible. + + - libfl.a now supplies private versions of the the <string.h>/ + <strings.h> string routines needed by flex and the scanners + it generates, to enhance portability to some BSD systems. + + - More robust solution to 2.4.2's flexfatal() bug fix. + + - Added ranlib of installed libfl.a. + + - Some lint tweaks. + + - NOTE: problems have been encountered attempting to build flex + C++ scanners using g++ version 2.5.X. The problem is due to an + unfortunate heuristic in g++ 2.5.X that attempts to discern between + C and C++ headers. Because FlexLexer.h is installed (by default) + in /usr/local/include and not /usr/local/lib/g++-include, g++ 2.5.X + decides that it's a C header :-(. So if you have problems, install + the header in /usr/local/lib/g++-include instead. + + +Changes between release 2.4.2 (01Dec93) and release 2.4.1: + + - Fixed bug in libfl.a referring to non-existent "flexfatal" function. + + - Modified to produce both compress'd and gzip'd tar files for + distributions (you probably don't care about this change!). + + +Changes between release 2.4.1 (30Nov93) and release 2.3.8: + + - The new '-+' flag instructs flex to generate a C++ scanner class + (thanks to Kent Williams). flex writes an implementation of the + class defined in FlexLexer.h to lex.yy.cc. You may include + multiple scanner classes in your program using the -P flag. Note + that the scanner class also provides a mechanism for creating + reentrant scanners. The scanner class uses C++ streams for I/O + instead of FILE*'s (thanks to Tom Epperly). If the flex executable's + name ends in '+' then the '-+' flag is automatically on, so creating + a symlink or copy of "flex" to "flex++" results in a version of + flex that can be used exclusively for C++ scanners. + + Note that without the '-+' flag, flex-generated scanners can still + be compiled using C++ compilers, though they use FILE*'s for I/O + instead of streams. + + See the "GENERATING C++ SCANNERS" section of flexdoc for details. + + - The new '-l' flag turns on maximum AT&T lex compatibility. In + particular, -l includes support for "yylineno" and makes yytext + be an array instead of a pointer. It does not, however, do away + with all incompatibilities. See the "INCOMPATIBILITIES WITH LEX + AND POSIX" section of flexdoc for details. + + - The new '-P' option specifies a prefix to use other than "yy" + for the scanner's globally-visible variables, and for the + "lex.yy.c" filename. Using -P you can link together multiple + flex scanners in the same executable. + + - The distribution includes a "texinfo" version of flexdoc.1, + contributed by Roland Pesch (thanks also to Marq Kole, who + contributed another version). It has not been brought up to + date, but reflects version 2.3. See MISC/flex.texinfo. + + The flex distribution will soon include G.T. Nicol's flex + manual; he is presently bringing it up-to-date for version 2.4. + + - yywrap() is now a function, and you now *must* link flex scanners + with libfl.a. + + - Site-configuration is now done via an autoconf-generated + "configure" script contributed by Francois Pinard. + + - Scanners now use fread() (or getc(), if interactive) and not + read() for input. A new "table compression" option, -Cr, + overrides this change and causes the scanner to use read() + (because read() is a bit faster than fread()). -f and -F + are now equivalent to -Cfr and -CFr; i.e., they imply the + -Cr option. + + - In the blessed name of POSIX compliance, flex supports "%array" + and "%pointer" directives in the definitions (first) section of + the scanner specification. The former specifies that yytext + should be an array (of size YYLMAX), the latter, that it should + be a pointer. The array version of yytext is universally slower + than the pointer version, but has the advantage that its contents + remain unmodified across calls to input() and unput() (the pointer + version of yytext is, still, trashed by such calls). + + "%array" cannot be used with the '-+' C++ scanner class option. + + - The new '-Ca' option directs flex to trade off memory for + natural alignment when generating a scanner's tables. In + particular, table entries that would otherwise be "short" + become "long". + + - The new '-h' option produces a summary of the flex flags. + + - The new '-V' option reports the flex version number and exits. + + - The new scanner macro YY_START returns an integer value + corresponding to the current start condition. You can return + to that start condition by passing the value to a subsequent + "BEGIN" action. You also can implement "start condition stacks" + by storing the values in an integer stack. + + - You can now redefine macros such as YY_INPUT by just #define'ing + them to some other value in the first section of the flex input; + no need to first #undef them. + + - flex now generates warnings for rules that can't be matched. + These warnings can be turned off using the new '-w' flag. If + your scanner uses REJECT then you will not get these warnings. + + - If you specify the '-s' flag but the default rule can be matched, + flex now generates a warning. + + - "yyleng" is now a global, and may be modified by the user (though + doing so and then using yymore() will yield weird results). + + - Name definitions in the first section of a scanner specification + can now include a leading '^' or trailing '$' operator. In this + case, the definition is *not* pushed back inside of parentheses. + + - Scanners with compressed tables are now "interactive" (-I option) + by default. You can suppress this attribute (which makes them + run slightly slower) using the new '-B' flag. + + - Flex now generates 8-bit scanners by default, unless you use the + -Cf or -CF compression options (-Cfe and -CFe result in 8-bit + scanners). You can force it to generate a 7-bit scanner using + the new '-7' flag. You can build flex to generate 8-bit scanners + for -Cf and -CF, too, by adding -DDEFAULT_CSIZE=256 to CFLAGS + in the Makefile. + + - You no longer need to call the scanner routine yyrestart() to + inform the scanner that you have switched to a new file after + having seen an EOF on the current input file. Instead, just + point yyin at the new file and continue scanning. + + - You no longer need to invoke YY_NEW_FILE in an <<EOF>> action + to indicate you wish to continue scanning. Simply point yyin + at a new file. + + - A leading '#' no longer introduces a comment in a flex input. + + - flex no longer considers formfeed ('\f') a whitespace character. + + - %t, I'm happy to report, has been nuked. + + - The '-p' option may be given twice ('-pp') to instruct flex to + report minor performance problems as well as major ones. + + - The '-v' verbose output no longer includes start/finish time + information. + + - Newlines in flex inputs can optionally include leading or + trailing carriage-returns ('\r'), in support of several PC/Mac + run-time libraries that automatically include these. + + - A start condition of the form "<*>" makes the following rule + active in every start condition, whether exclusive or inclusive. + + - The following items have been corrected in the flex documentation: + + - '-C' table compression options *are* cumulative. + + - You may modify yytext but not lengthen it by appending + characters to the end. Modifying its final character + will affect '^' anchoring for the next rule matched + if the character is changed to or from a newline. + + - The term "backtracking" has been renamed "backing up", + since it is a one-time repositioning and not a repeated + search. What used to be the "lex.backtrack" file is now + "lex.backup". + + - Unindented "/* ... */" comments are allowed in the first + flex input section, but not in the second. + + - yyless() can only be used in the flex input source, not + externally. + + - You can use "yyrestart(yyin)" to throw away the + current contents of the input buffer. + + - To write high-speed scanners, attempt to match as much + text as possible with each rule. See MISC/fastwc/README + for more information. + + - Using the beginning-of-line operator ('^') is fairly + cheap. Using unput() is expensive. Using yyless() is + cheap. + + - An example of scanning strings with embedded escape + sequences has been added. + + - The example of backing-up in flexdoc was erroneous; it + has been corrected. + + - A flex scanner's internal buffer now dynamically grows if needed + to match large tokens. Note that growing the buffer presently + requires rescanning the (large) token, so consuming a lot of + text this way is a slow process. Also note that presently the + buffer does *not* grow if you unput() more text than can fit + into the buffer. + + - The MISC/ directory has been reorganized; see MISC/README for + details. + + - yyless() can now be used in the third (user action) section + of a scanner specification, thanks to Ceriel Jacobs. yyless() + remains a macro and cannot be used outside of the scanner source. + + - The skeleton file is no longer opened at run-time, but instead + compiled into a large string array (thanks to John Gilmore and + friends at Cygnus). You can still use the -S flag to point flex + at a different skeleton file, though if you use this option let + me know, as I plan to otherwise do away with -S in the near + future. + + - flex no longer uses a temporary file to store the scanner's + actions. + + - A number of changes have been made to decrease porting headaches. + In particular, flex no longer uses memset() or ctime(), and + provides a single simple mechanism for dealing with C compilers + that still define malloc() as returning char* instead of void*. + + - Flex now detects if the scanner specification requires the -8 flag + but the flag was not given or on by default. + + - A number of table-expansion fencepost bugs have been fixed, + making flex more robust for generating large scanners. + + - flex more consistently identifies the location of errors in + its input. + + - YY_USER_ACTION is now invoked only for "real" actions, not for + internal actions used by the scanner for things like filling + the buffer or handling EOF. + + - The rule "[^]]" now matches any character other than a ']'; + formerly it matched any character at all followed by a ']'. + This change was made for compatibility with AT&T lex. + + - A large number of miscellaneous bugs have been found and fixed + thanks to Gerhard Wilhelms. + + - The source code has been heavily reformatted, making patches + relative to previous flex releases no longer accurate. + + +Changes between 2.3 Patch #8 (21Feb93) and 2.3 Patch #7: + + - Fixed bugs in dynamic memory allocation leading to grievous + fencepost problems when generating large scanners. + - Fixed bug causing infinite loops on character classes with 8-bit + characters in them. + - Fixed bug in matching repetitions with a lower bound of 0. + - Fixed bug in scanning NUL characters using an "interactive" scanner. + - Fixed bug in using yymore() at the end of a file. + - Fixed bug in misrecognizing rules with variable trailing context. + - Fixed bug compiling flex on Suns using gcc 2. + - Fixed bug in not recognizing that input files with the character + ASCII 128 in them require the -8 flag. + - Fixed bug that could cause an infinite loop writing out + error messages. + - Fixed bug in not recognizing old-style lex % declarations if + followed by a tab instead of a space. + - Fixed potential crash when flex terminated early (usually due + to a bad flag) and the -v flag had been given. + - Added some missing declarations of void functions. + - Changed to only use '\a' for __STDC__ compilers. + - Updated mailing addresses. + + +Changes between 2.3 Patch #7 (28Mar91) and 2.3 Patch #6: + + - Fixed out-of-bounds array access that caused bad tables + to be produced on machines where the bad reference happened + to yield a 1. This caused problems installing or running + flex on some Suns, in particular. + + +Changes between 2.3 Patch #6 (29Aug90) and 2.3 Patch #5: + + - Fixed a serious bug in yymore() which basically made it + completely broken. Thanks goes to Jean Christophe of + the Nethack development team for finding the problem + and passing along the fix. + + +Changes between 2.3 Patch #5 (16Aug90) and 2.3 Patch #4: + + - An up-to-date version of initscan.c so "make test" will + work after applying the previous patches + + +Changes between 2.3 Patch #4 (14Aug90) and 2.3 Patch #3: + + - Fixed bug in hexadecimal escapes which allowed only digits, + not letters, in escapes + - Fixed bug in previous "Changes" file! + + +Changes between 2.3 Patch #3 (03Aug90) and 2.3 Patch #2: + + - Correction to patch #2 for gcc compilation; thanks goes to + Paul Eggert for catching this. + + +Changes between 2.3 Patch #2 (02Aug90) and original 2.3 release: + + - Fixed (hopefully) headaches involving declaring malloc() + and free() for gcc, which defines __STDC__ but (often) doesn't + come with the standard include files such as <stdlib.h>. + Reordered #ifdef maze in the scanner skeleton in the hope of + getting the declarations right for cfront and g++, too. + + - Note that this patch supercedes patch #1 for release 2.3, + which was never announced but was available briefly for + anonymous ftp. + + +Changes between 2.3 (full) release of 28Jun90 and 2.2 (alpha) release: + + User-visible: + + - A lone <<EOF>> rule (that is, one which is not qualified with + a list of start conditions) now specifies the EOF action for + *all* start conditions which haven't already had <<EOF>> actions + given. To specify an end-of-file action for just the initial + state, use <INITIAL><<EOF>>. + + - -d debug output is now contigent on the global yy_flex_debug + being set to a non-zero value, which it is by default. + + - A new macro, YY_USER_INIT, is provided for the user to specify + initialization action to be taken on the first call to the + scanner. This action is done before the scanner does its + own initialization. + + - yy_new_buffer() has been added as an alias for yy_create_buffer() + + - Comments beginning with '#' and extending to the end of the line + now work, but have been deprecated (in anticipation of making + flex recognize #line directives). + + - The funky restrictions on when semi-colons could follow the + YY_NEW_FILE and yyless macros have been removed. They now + behave identically to functions. + + - A bug in the sample redefinition of YY_INPUT in the documentation + has been corrected. + + - A bug in the sample simple tokener in the documentation has + been corrected. + + - The documentation on the incompatibilities between flex and + lex has been reordered so that the discussion of yylineno + and input() come first, as it's anticipated that these will + be the most common source of headaches. + + + Things which didn't used to be documented but now are: + + - flex interprets "^foo|bar" differently from lex. flex interprets + it as "match either a 'foo' or a 'bar', providing it comes at the + beginning of a line", whereas lex interprets it as "match either + a 'foo' at the beginning of a line, or a 'bar' anywhere". + + - flex initializes the global "yyin" on the first call to the + scanner, while lex initializes it at compile-time. + + - yy_switch_to_buffer() can be used in the yywrap() macro/routine. + + - flex scanners do not use stdio for their input, and hence when + writing an interactive scanner one must explictly call fflush() + after writing out a prompt. + + - flex scanner can be made reentrant (after a fashion) by using + "yyrestart( yyin );". This is useful for interactive scanners + which have interrupt handlers that long-jump out of the scanner. + + - a defense of why yylineno is not supported is included, along + with a suggestion on how to convert scanners which rely on it. + + + Other changes: + + - Prototypes and proper declarations of void routines have + been added to the flex source code, courtesy of Kevin B. Kenny. + + - Routines dealing with memory allocation now use void* pointers + instead of char* - see Makefile for porting implications. + + - Error-checking is now done when flex closes a file. + + - Various lint tweaks were added to reduce the number of gripes. + + - Makefile has been further parameterized to aid in porting. + + - Support for SCO Unix added. + + - Flex now sports the latest & greatest UC copyright notice + (which is only slightly different from the previous one). + + - A note has been added to flexdoc.1 mentioning work in progress + on modifying flex to generate straight C code rather than a + table-driven automaton, with an email address of whom to contact + if you are working along similar lines. + + +Changes between 2.2 Patch #3 (30Mar90) and 2.2 Patch #2: + + - fixed bug which caused -I scanners to bomb + + +Changes between 2.2 Patch #2 (27Mar90) and 2.2 Patch #1: + + - fixed bug writing past end of input buffer in yyunput() + - fixed bug detecting NUL's at the end of a buffer + + +Changes between 2.2 Patch #1 (23Mar90) and 2.2 (alpha) release: + + - Makefile fixes: definition of MAKE variable for systems + which don't have it; installation of flexdoc.1 along with + flex.1; fixed two bugs which could cause "bigtest" to fail. + + - flex.skel fix for compiling with g++. + + - README and flexdoc.1 no longer list an out-of-date BITNET address + for contacting me. + + - minor typos and formatting changes to flex.1 and flexdoc.1. + + +Changes between 2.2 (alpha) release of March '90 and previous release: + + User-visible: + + - Full user documentation now available. + + - Support for 8-bit scanners. + + - Scanners now accept NUL's. + + - A facility has been added for dealing with multiple + input buffers. + + - Two manual entries now. One which fully describes flex + (rather than just its differences from lex), and the + other for quick(er) reference. + + - A number of changes to bring flex closer into compliance + with the latest POSIX lex draft: + + %t support + flex now accepts multiple input files and concatenates + them together to form its input + previous -c (compress) flag renamed -C + do-nothing -c and -n flags added + Any indented code or code within %{}'s in section 2 is + now copied to the output + + - yyleng is now a bona fide global integer. + + - -d debug information now gives the line number of the + matched rule instead of which number rule it was from + the beginning of the file. + + - -v output now includes a summary of the flags used to generate + the scanner. + + - unput() and yyrestart() are now globally callable. + + - yyrestart() no longer closes the previous value of yyin. + + - C++ support; generated scanners can be compiled with C++ compiler. + + - Primitive -lfl library added, containing default main() + which calls yylex(). A number of routines currently living + in the scanner skeleton will probably migrate to here + in the future (in particular, yywrap() will probably cease + to be a macro and instead be a function in the -lfl library). + + - Hexadecimal (\x) escape sequences added. + + - Support for MS-DOS, VMS, and Turbo-C integrated. + + - The %used/%unused operators have been deprecated. They + may go away soon. + + + Other changes: + + - Makefile enhanced for easier testing and installation. + - The parser has been tweaked to detect some erroneous + constructions which previously were missed. + - Scanner input buffer overflow is now detected. + - Bugs with missing "const" declarations fixed. + - Out-of-date Minix/Atari patches provided. + - Scanners no longer require printf() unless FLEX_DEBUG is being used. + - A subtle input() bug has been fixed. + - Line numbers for "continued action" rules (those following + the special '|' action) are now correct. + - unput() bug fixed; had been causing problems porting flex to VMS. + - yymore() handling rewritten to fix bug with interaction + between yymore() and trailing context. + - EOF in actions now generates an error message. + - Bug involving -CFe and generating equivalence classes fixed. + - Bug which made -CF be treated as -Cf fixed. + - Support for SysV tmpnam() added. + - Unused #define's for scanner no longer generated. + - Error messages which are associated with a particular input + line are now all identified with their input line in standard + format. + - % directives which are valid to lex but not to flex are + now ignored instead of generating warnings. + - -DSYS_V flag can now also be specified -DUSG for System V + compilation. + + +Changes between 2.1 beta-test release of June '89 and previous release: + + User-visible: + + - -p flag generates a performance report to stderr. The report + consists of comments regarding features of the scanner rules + which result in slower scanners. + + - -b flag generates backtracking information to lex.backtrack. + This is a list of scanner states which require backtracking + and the characters on which they do so. By adding rules + one can remove backtracking states. If all backtracking states + are eliminated, the generated scanner will run faster. + Backtracking is not yet documented in the manual entry. + + - Variable trailing context now works, i.e., one can have + rules like "(foo)*/[ \t]*bletch". Some trailing context + patterns still cannot be properly matched and generate + error messages. These are patterns where the ending of the + first part of the rule matches the beginning of the second + part, such as "zx*/xy*", where the 'x*' matches the 'x' at + the beginning of the trailing context. Lex won't get these + patterns right either. + + - Faster scanners. + + - End-of-file rules. The special rule "<<EOF>>" indicates + actions which are to be taken when an end-of-file is + encountered and yywrap() returns non-zero (i.e., indicates + no further files to process). See manual entry for example. + + - The -r (reject used) flag is gone. flex now scans the input + for occurrences of the string "REJECT" to determine if the + action is needed. It tries to be intelligent about this but + can be fooled. One can force the presence or absence of + REJECT by adding a line in the first section of the form + "%used REJECT" or "%unused REJECT". + + - yymore() has been implemented. Similarly to REJECT, flex + detects the use of yymore(), which can be overridden using + "%used" or "%unused". + + - Patterns like "x{0,3}" now work (i.e., with lower-limit == 0). + + - Removed '\^x' for ctrl-x misfeature. + + - Added '\a' and '\v' escape sequences. + + - \<digits> now works for octal escape sequences; previously + \0<digits> was required. + + - Better error reporting; line numbers are associated with rules. + + - yyleng is a macro; it cannot be accessed outside of the + scanner source file. + + - yytext and yyleng should not be modified within a flex action. + + - Generated scanners #define the name FLEX_SCANNER. + + - Rules are internally separated by YY_BREAK in lex.yy.c rather + than break, to allow redefinition. + + - The macro YY_USER_ACTION can be redefined to provide an action + which is always executed prior to the matched rule's action. + + - yyrestart() is a new action which can be used to restart + the scanner after it has seen an end-of-file (a "real" one, + that is, one for which yywrap() returned non-zero). It takes + a FILE* argument indicating a new file to scan and sets + things up so that a subsequent call to yylex() will start + scanning that file. + + - Internal scanner names all preceded by "yy_" + + - lex.yy.c is deleted if errors are encountered during processing. + + - Comments may be put in the first section of the input by preceding + them with '#'. + + + + Other changes: + + - Some portability-related bugs fixed, in particular for machines + with unsigned characters or sizeof( int* ) != sizeof( int ). + Also, tweaks for VMS and Microsoft C (MS-DOS), and identifiers all + trimmed to be 31 or fewer characters. Shortened file names + for dinosaur OS's. Checks for allocating > 64K memory + on 16 bit'ers. Amiga tweaks. Compiles using gcc on a Sun-3. + - Compressed and fast scanner skeletons merged. + - Skeleton header files done away with. + - Generated scanner uses prototypes and "const" for __STDC__. + - -DSV flag is now -DSYS_V for System V compilation. + - Removed all references to FTL language. + - Software now covered by BSD Copyright. + - flex will replace lex in subsequent BSD releases. |