Tom Shields, March 17, 2014 PARKING LOT ISSUES: ------------------- - verify debian packaging still works? - there are no #line directives in y.tab.i, other than those that come from the input file and the skeleton file; to fix this, would need to count output lines in externs_file and add 'write_externs_lineno()' similar to 'write_code_lineno()' - if there are no defined symbols, the .tab.h file isn't empty (weird case, may not be worth fixing) - consider: treat []-actions identical to {}-actions if not processing a backtracking parser (avoids test case error)? BTYACC CHANGES CURRENTLY DEFERRED, BY FILE: ------------------------------------------- push.skel - skeleton for a 'push' parser - needs to be upgraded match the structure of yaccpar.skel defs.h - adopt '%include' changes - adopt '%define'/'%ifdef'/'%endif' - adopt -E flag to print preprocessed grammar to stdout error.c - adopt '%include' changes - NOTE: there is a btyacc change that might be worth adopting in byacc [FileError() refactoring to eliminate duplicated code in most of the error message functions] main.c - adopt '%define' changes - adopt '-DNAME' command line option to define preprocessor variable NAME - adopt -E flag to print preprocessed grammar to stdout - adopt '-S skeleton_file' command line option to select an alternate parser skeleton file - the skeleton file named by the -S flag is used as provided to open the file; consider a change to this behavior to check whether the named file has a path prefix, and if not, look in 'installation' directory if the file is not found in the working directory output.c - adopt '%include' changes reader.c - adopt '%include' changes - adopt '%define'/'%ifdef'/'%endif' changes - adopt -E flag to print preprocessed grammar to stdout - NOTE: there is a btyacc change that might be worth adopting in byacc [copy_string() & copy_comment() refactoring to eliminate duplicated code in copy_text() and copy_union()] warshall.c - NOTE: there is a btyacc change that might be worth adopting in byacc [shifting 'mask' incrementally rather than literal '1' by a variable amount each time thru the loop] ================================================================================ new files: ---------- skel2c - modified from btyacc distribution: don't generate #include defs.h - extended syntax recognized to include '%% insert VERSION here', generating the defines for YYMAJOR, YYMINOR and YYPATCH at that point - made generated tables type 'const char *const' to match skelton.c from byacc-20130925 baseline - added code to append text for write_section() to end of generated skeleton.c - remove conversion of tab to \t in generated skeleton.c - extended syntax recognized to include '%%ifdef', '%%ifndef', '%%else' and '%%endif'; used in yaccpar.skel to bracket code that is specific to backtracking yaccpar.skel.old - created from skeleton.c in byacc-20140101 baseline; use of this skeleton will create a version of skeleton.c that is close to that in the byacc-20140101 baseline - eliminated 'body_3' and 'trailer_2' skeleton segments - no need to generate yyerror() invocation dynamically; YYERROR_CALL() is already generated earlier, and so can be used in the skeleton to simplify - added 'const' to types in '%% tables' section to match what skel2c, start_int_table() and state_str_table() generate - added a few cosmetic changes (e.g., added some additional comments, reworded debugging output to match yaccpar.skel, changed yygrowstack() to return YYENOMEM for 'out of memory' error, rather than -1, to match yaccpar.skel; changed yyparse() return value from 1 to 2 for the 'out of memory' error to match yaccpar.skel) - added '#ifndef'/'#endif' around '#define YYINITSTACKSIZE 200' to allow the value to be changed at compile time - changed 'printf(' to 'fprintf(stderr, '; added stack depth (yydepth) to debugging output from yaccpar.skel - use 'YYINT' rather than 'short' for integer table types yaccpar.skel - renamed from btyaccpa.ske, merged with btyacc-c.ske - modified from btyacc distribution to match the latest byacc-20140101 skeleton structure & data structures - make local functions static - change "virtual memory exceeded" to "memory exhausted" for bison compatibility - change debug output generation from printf/puts/putc onto stdout to use fprintf/fputs/fputc onto stderr; include stack depth and whether or not in trial parsing - changed types of generated string tables to be 'const pointer to const char' - check all malloc()/realloc() return values, ensure return value of yyparse() = 2 if parsing failed due to memory exhaustion - change YYDBPR() macro to YYSTYPE_TOSTRING(); define semantics as delivering a char* value representing a semantic value (e.g., yylval or yyval, or the contents of an entry on the semantic stack); additional parameter passed: grammar symbol # (to assist interpretation of semantic value) - change YYPOSN to YYLTYPE and yyposn to yylloc (position corresponding to yylval) for bison compatibility; add yyloc (corresponding to yyval) - move default definition of YYLTYPE into output.c, generating a typedef - add '#if defined(YYLTYPE) || defined(YYLTYPE_IS_DECLARED)'/'#endif' around all lines specific to position processing - add '#if defined(YYDESTRUCT_CALL)'/'#endif' around all lines specific to semantic & position stack processing to reclaim memory associated with discarded symbols - add '%%ifdef YYBTYACC'/'%%endif' around all lines specific to backtrack parsing; converted by skel2c into '#if defined(YYBTYACC)'/'#endif' - distinguish between "yacc stack overflow" and "memory exhausted" situations - consolidated termination cleanup code; introduced yyreturn, set to 2 after labels yyoverflow/yyenomem, set to 1 after label yyabort, set to 0 after label yyaccept; all termination cases jump to label yyreturn, which does any cleanup then returns yyreturn value - replaced YYDELETEVAL & YYDELETEPOSN user-supplied macro capability by implementation of byacc-generated yydestruct() as defined by bison compatible %destructor mechanism - moved invocation of 'YYREDUCEPOSNFUNC' macro to immediately prior to, rather than after, execution of final rule action (so that, at some future date, implementation extensions can be added to enable custom calculation of locations associated with non-terminals within rule actions); deleted unnecessary flag 'reduce_posn'; deleted 'YYCALLREDUCEPOSN' macro; deleted C++ variant of 'YYREDUCEPOSNFUNC' invocation - adopt approach similar to bison for default computation of yyloc; change macro 'YYREDUCEPOSNFUNC' name to 'YYLLOC_DEFAULT' for bison compatibility; added 'yyerror_loc_range[2]' to hold start & end locations for error situations that pop the stack - use 'YYINT' rather than 'short' for integer table types, and for indexing parser tables readskel.c http://www.verisign.com/index.html- replaced error() with fprintf() mstring.h - moved contents of mstring.h to defs.h - mstring.h is obsolete mstring.c - replaced include of mstring.h with defs.h - changed 'START' to 'HEAD' to remove conflict with 'START' used for the start symbol defined in defs.h modified byacc files: --------------------- skeleton.c - skeleton.c is now generated from the appropriate skeleton file by 'skel2c' configure.in - added configuration for --enable-btyacc option; if 'yes' add '-DYYBTYACC' to DEFINES in makefile.in; --enable-btyacc defaults to 'no' - added configuration for --with-max-table-size option; if present, overrides the value of MAXTABLE defined in defs.h - regenerate configure using autoconf makefile.in - added mstring.c to C_FILES - added mstring$o to OBJS - added @DEFINES@ as value of DEFINES make variable - added new make variable SKELETON with value 'yaccpar.skel' - added rule to generate skeleton.c from $(SKELETON), depending on skel2c and makefile - added rm -f skeleton.c distclean rule - moved dependency on makefile from only main$o & skeleton$o to $(OBJS), since if ./configure is run changing, for example, from --enable-btyacc to --disable-btyacc, all files must be recompiled to ensure a clean executable - add @MAXTABLE@ for optional '-DMAXTABLE=nnn' if configured using --with-max-table-size=nnn - changed 'cd test && rn 0f test-*'to 'rm -f $(testdir)/test-*' test/run_test.sh - ??? test/run_make.sh - ??? defs.h - moved contents of mstring.h to defs.h - mstring.h is obsolete - added to get the various system defined machine limits; changed definitions of MAXCHAR, MAXSHORT, MINSHORT & BITS_PER_WORD to use defines from ; changed definitions of BIT and SETBIT to use value of BITS_PER_WORD - added typedef for __compar_fn_t, conditioned on _COMPAR_FN_T being undefined (at least for Mac OSX environment) - adopt new symbol class values ACTION and ARGUMENT - adopt changes/additions used by inherited attribute processing - clean up locations of extern function definitions to match where they actually live in source files - adopt error functions from inherited attribute processing; added new error functions - added keyword code LOCATIONS for %locations - added keyword code DESTRUCTOR for %destructor - added extern decl for 'int locations'; true if %locations present - added extern decl for 'int backtrack'; initialized to 0 (= false), set to 1 (= true) if -B flag is present - added extern decl for 'int destructor'; true if at least one %destructor present in grammar spec file - define 'YYINT' as the smallest C type that can be used to address a table of size 'MAXTABLE'; define 'YYINT' based on the value of 'MAXTABLE' using the standard system type size definitions from ; define 'MAXYYINT' and 'MINYYINT' accordingly - change 'Value_t' and 'Index_t' to 'YYINT' from 'short' - allow 'MAXTABLE' to be defined by '-DMAXTABLE=nnn' at compile-time closure.c - changed print_closure(), print_EFF() and print_first_derives() to 'static'; added fwd declarations - changed 'short' to 'Value_t' (in some instances, 'Value_t' was already used for variables/parameters that were related to variables/parameters declared as 'short' error.c - adopt error functions from inherited attribute processing; added a few additional inherited attribute error functions graph.c - changed 'short' to 'Value_t' (in some instances, 'Value_t' was already used for variables/parameters that were related to variables/parameters declared as 'short' lalr.c - changed MAXSHORT to MAXYYINT lr0.c - changed MAXSHORT to MAXYYINT - changed 'short' to 'Value_t' (in some instances, 'Value_t' was already used for variables/parameters that were related to variables/parameters declared as 'short' main.c - changed 'short' to 'Value_t' (in some instances, 'Value_t' was already used for variables/parameters that were related to variables/parameters declared as 'short' mkpar.c - backtracking attempts to resolve shift/reduce and reduce/reduce conflicts output.c - generate prefix & YYPREFIX defines into externs file (-i, .tab.i) and code file (-r, .code.c); generate into output file (.tab.c) only if not using -r option; eliminates doubled output of prefix aliases if -r with no -i in y.tab.c and y.code.c or if -r & -i in y.tab.i and y.code.c - changed types of generated string tables to be 'const pointer to const char' - adopt backtracking as an alternative in cases where otherwise we have a conflict in the parsing actions (3, rather than 2, choices) - wrap defines file with (where "yy" is value of 'symbol_prefix') #ifndef __yy_defines_h_ #define _yy_defines_h_ #endif - avoid writing %%xdecls skeleton section twice if -r used - eliminated 'body_3' and 'trailer_2' skeleton segments - no need to generate yyerror() invocation dynamically; YYERROR_CALL() is already generated earlier, and can be used in the pareser skeleton - if -P flag (pure_parser), add yylloc as 2nd parameter to yylex() (declaration & call) - change YYPOSN to YYLTYPE and yyposn to yylloc (position corresponding to yylval) for bison compatibility; add yyloc (corresponding to yyval) - generate yylloc parameters for yylex & yyerror if %locations present - add location as 1st parameter to declaraion & invocation of yyerror() if %locations present - output backtrack parsing tables if -B flag is present - added generation of yystos[] with output_accessing_symbols() to allow translation from a parser internal state number to the corresponding grammar symbol number [0 .. nsyms) of the accessing symbol of that parser state; used in the generated code for YYDESTRUCT_CALL() & YYSTYPE_TOSTRING() to enable the correct semantic value union tag to be determined when executing the implementation of YYDESTRUCT_CALL() or YYSTYPE_TOSTRING() (similar to yystos[] in bison) - added to output_prefix(): yystos; yycindex & yyctable if compiling backtracking; yyloc & yylloc if %locations used - extended yyname[] to include all grammar symbols, not just the terminal symbols: '$end', 'error', '$accept', all non-terminals, including internally generated non-terminals for embedded actions in rules, and 'illegal-symbol' (which bison spells '$undefined'); '$end' already defined as a symbol 0, rathern than adding 'end-of-file' as the name of symbol 0; added 'illegal-symbol' from byacc-20140101 (NOTE: the comment in the code that says byacc does not predefine '$end' and '$error' is incorrect; however, both bison and byacc spell '$error' as 'error') - added generation of #define YYTRANSLATE() from byacc-20140101, but changed the definition for the undefined symbol case because it is no longer in yyname[YYMAXTOKEN+1] but rather occurs after the last non-terminal symbol; added #define YYUNDFTOKEN to contain the index in yyname of 'illegal-symbol' - generate YYLTYPE in output_ltype() as a struct like for bison rather than using #define in yaccpar.skel - added 'write_code_lineno' invocation at start of 'output_prefix' - added 'write_code_lineno' invocation at start of 'output_pure_parser' - added 'write_code_lineno' invocation prior to generation of #include for externs file - added 'write_code_lineno' invocation after 1st 'write_section(fp, xdecls)' - added '++outline;' prior to output of '#define YYTRANSLATE' - this was actually causing almost all of the invocations of 'write_code_lineno' to put out the correct #line directive - corrected 'write_code_lineno' - the line number in a #line directive is the number of the next line, not the number of the #line line - changed MAXSHORT to MAXYYINT; changed 'high' local static from 'int' to 'long' so that it can get higher than 'MAXYYINT' without machine-dependent behavior; changed related formats from '%d' to '%ld' - generate 'YYINT' rather than 'short' for integer table types - generate YYDESTRUCT_DECL & YYDESTRUCT_CALL macros, similar to YYERROR_DECL and YYERROR_CALL macros, that can be redefined by user, if desired, to add additional parameters to yydestruct() (and even change the 'yydestruct' function name) - if at least one %destructor present, generate yydestruct(); 1st parameter is a string indicating the context in which yydestruct() is invoked (e.g., discarding input token, discarding state on stack, cleanup when aborting); 2nd parameter is the internal grammar symbol number [0..nsyms) of the accessing symbol of the parser state on the top of the stack; 3rd parameter is a pointer to the semantic value to be reclaimed associated with the grammar symbol in the 2nd parameter; if %locations is defined, the 4th parameter is a pointer to the position value to be reclaimed associated with the grammar symbol in the 2nd parameter reader.c - adopt []-actions, similar to {}-actions; {}-actions are only executed when not in trial mode, but []-actions are executed regardless of mode - adopt new symbol class values ACTION and ARGUMENT - adopt inherited attributes (syntax resembles arguments to non-terminal symbols) - adopt keyword table lookup from btyacc, modified to handle equivalence of '-' and '_' in spelling of keywords - adopt refactoring of tag table creation into cache_tag() for use in multiple locations - added new error functions in place of btyacc's generic error() function - changed '0' to 'NULL' for pointer initialization - reworked for-loop at end of get_line (part of DEFERRED '%ifdef/%endif' change) - added %locations directive for bison compatibility to enable position processing - added decl for 'int locations'; true if %locations present - added decl 'int backtrack'; initialized to 0 (= false), set to 1 (= true) if -B flag is present - process %locations if present, set location = 1 - only process []-actions and only generate 'if (!yytrial)' prefix for {}-actions if backtracking is enabled - add decl for 'int destructor'; true if at least one %destructor is present - add %destructor directive to enable semantic & position stack processing to reclaim memory associated with discarded symbols - process bison compatible %destructor (set destructor = 1); support @$ in %destructor code to reference the position value if %locations is defined - changed 'short' to 'Value_t' (in some instances, 'Value_t' was already used for variables/parameters that were related to variables/parameters declared as 'short' - if %locations present, support @N and @$ syntax as for bison to reference the locations associated with the N-th rhs symbol and the lhs symbol, respectively symtab.c - initialize fields added to 'struct bucket' for non-terminal symbol inherited attributes verbose.c - for parse states with conflicts, the contents of the y.output file include the trial shift and/or trial reduce actions - added output to the end of the verbose report showing the correspondance between grammar symbol #, internal parser symbol #, and grammar symbol name - changed 'short' to 'Value_t' (in some instances, 'Value_t' was already used for variables/parameters that were related to variables/parameters declared as 'short' yacc.1 - added options 'P', 'V', 'y' and '-o output_file' to the yacc command synopsis (already covered in the description section) - added options 'B', 'D' and 'L' to the yacc command synopsis; added text in the description section - added %locations description to the extensions section