berto -> AGElint: What's the point? MUST-READS & README (12/10/2011 2:06:50 AM)
|
AGElint MUST-READS Here are some must-read docs for installing and making best use of AGElint: AGElint README =============================================================================== INTRODUCTION Here, for your debugging and gaming pleasure, is AGElint. What is AGElint? Akin to the well-known C programming language utility lint. ("Lint: A Unix C language processor which carries out more thorough checks on the code than is usual with C compilers. Lint is named after the bits of fluff it supposedly picks from programs." See here.). Similarly, AGElint is a toolkit of programs designed to discover and report bugs, problems, glitches, anomalies in the AGEOD game data files. If anybody thinks that AGElint is poppycock, that I'm just making this up, please read this: http://en.wikipedia.org/wiki/Static_code_analysis http://en.wikipedia.org/wiki/Automated_code_review Proactive, pre-execution debugging (AGElint) vs. reactive, post-execution debugging (the traditional way). Static analysis (AGElint) vs. dynamic analysis (playtest the game, inspect the log files). Each approach has its strengths and weaknesses, but together they've got the bugs surrounded -- no escape! NOTE: I make no claim about the significance or insignificance of any discovered bug, problem, glitch, or anomaly. Whether or not it impacts game play, or goes entirely unnoticed. Whether in the larger scheme of things it's important, or unimportant. I just call them as I see them It's up for you to decide, and maybe for us as a community to determine. (If over time we agree that something is not worth reporting, or is not in fact any kind of problem, I will program it out of the toolkit.) FURTHER NOTE: I offer no judgment about AGEOD, or its games. Nothing asserted, nothing implied. AGElint is a tool, not a weapon. We use it to build better games, not to tear down any company or anybody. AGElint deals directly with the game data text files, and only peripherally with the AGEOD source .xls/DB files. So AGElint works outside "the system". It might be said that if you muck with the game data text files, you are on your own and should expect no official support from AGEOD. I might issue new releases of AGElint as circumstances -- and user enthusiasm -- warrant. AGElint is, and forever remains, a work-in-progress, improving over time. More layers of bug checking can be added, and its capabilities are far from exhausted. This is the current state of AGElint, which reflects my imperfect and evolving understanding of the AGE system and its scripting. I make my share of mistakes, but except for honest differences of opinion, I correct my goofs! In the spirit of sharing knowledge, empowering the players, and bettering the gaming experience, I give you AGElint. Players of the world unite! You have nothing to lose but your bugs! =============================================================================== BEFORE YOU BEGIN In order to install and run AGElint, you will need to use (a) Linux (or some other suitable Unix-like OS, such as FreeBSD); and/or (b) Cygwin for Windows (or some other Unix-like environment for Windows). AGElint, at least as currently constituted, is firmly rooted in Unix/Linux. (Note: after this, I will mostly cease referring to Unix/Linux, Cygwin, and qualifying the terms. Henceforth, I will mostly just use "Linux". You should take that to mean Linux, but also any of the alternative compatible host environments. I will only qualify as necessary.) I make no apologies for that. AGElint works on the vast amounts of AGEOD game text file data. And Linux, in countless ways, is a superior OS to Windows for raw data processing. (If you doubt that, just consider this one fact: Most of the world's financial exchanges are now Linux-based. But also: Linux is at the heart of many well known cutting edge technology companies, such as Google. And if not Linux, one form or another of Unix. Windows is okay as a consumer and gaming OS, but for back-end production work, Unix/Linux is king.) You will see much use of Linux commands, OS capabilities, and Unix philosophy coursing through the AGElint toolkit. AGElint as a native Windows application or toolkit? There's simply no way. To install and run AGElint, it's easiest simply to do it on Linux, with AGEOD game installations to process copied to your Linux filesystem (using rsync or scp/WinSCP over the network, for example), else installed on a USB stick (the method I preferred). If you are not prepared to run this in Linux, there is at least one Windows-based Unix environment alternative -- Cygwin for Windows. See http://cygwin.com/ From the Cygwin home page: Cygwin is: * a collection of tools which provide a Linux look and feel environment for Windows. * a DLL (cygwin1.dll) which acts as a Linux API layer providing substantial Linux API functionality. Cygwin is not: * a way to run native Linux apps on Windows. You must rebuild your application from source if you want it to run on Windows. * a way to magically make native Windows apps aware of UNIX® functionality like signals, ptys, etc. Again, you need to build your apps from source if you want to take advantage of Cygwin functionality. I have installed and used Cygwin for Windows over the years on multiple Windows systems. It is easy to install. It's a nice way to get your feet wet with Unix, short of diving right into Unix/Linux. If you are handy with Linux, indeed if you are "just" a Windows power user, installing Cygwin is not that hard. It's certainly no harder than (following the "official" AGEOD modding procedure): --installing OpenOffice --installing the AGE CSV splitter --editing the "official" AGEOD DB/.xls files --saving those files as .csv files --running those saved .csv files through the CSV splitter --placing the resulting game data files in their appropriate folders --running the AGEOD game in special edit mode --saving the game --etc. I will have nothing more to say about the "difficulty" of installing/running Linux/Cygwin. You decide how far your comfort zone extends. =============================================================================== INSTALLATION For Cygwin installation, please see the provided INSTALL_CYGWIN file. For AGElint installation, please see the provided INSTALL_AGELINT file. For more about the agelint compile process, read on. (If you have installed AGElint properly as described in the INSTALL_AGELINT and don't care to know the technical details, just skip the rest of this discussion and proceed to the OPERATION section.) If you use the recommended make command $ make clean; time make 2>&1 | egrep -iv "warning: conversion to" | tee make.log; echo; egrep -i "shift|reduce" make.log you should see something like: ------------------------------------------------------------------------------- [root@telemann agelint]# make clean; make 2>&1 | egrep -iv "warning: conversion to" | tee make.log; echo; egrep -i "shift|reduce" make.log rm -f agelint *.o lex.*.c *.tab.c *.tab.h *.output make.out core make -r agelint 2>&1 > make.out 2>&1; cat make.out | egrep -iv "with different width due to prototype|^lex.txt.c|useless"; date; ls -l agelint make[1]: Entering directory `/home/berto/games/ageod/agelint' gcc -O2 -Wall -Wshadow -Wconversion -Wpointer-arith -Wcast-qual -c agelint.c -o agelint.o ... bison -d -v -p txt -b txt txt.y txt.y: conflicts: 7 shift/reduce, 2 reduce/reduce gcc -O2 -Wall -Wshadow -Wconversion -Wpointer-arith -Wcast-qual -c txt.tab.c -o txt.tab.o txt.tab.c: In function ‘txtparse’: flex -B -Caem -i -Ptxt txt.l gcc -O2 -Wall -Wshadow -Wconversion -Wpointer-arith -Wcast-qual -c lex.txt.c -o lex.txt.o ... gcc -O2 -Wall -Wshadow -Wconversion -Wpointer-arith -Wcast-qual -o agelint agelint.o cmd.tab.o lex.cmd.o conf.tab.o lex.conf.o txt.tab.o lex.txt.o -ly -lm make[1]: Leaving directory `/home/berto/games/ageod/agelint' Tue Jan 17 01:42:21 CST 2012 -rwxr-xr-x. 1 root root 6900427 Jan 17 01:42 agelint agelint version 1.1.0.20120117 txt.y: conflicts: 7 shift/reduce, 2 reduce/reduce ------------------------------------------------------------------------------- What's going on here? The AGElint executable, agelint, is coded in C, lex, and yacc. The component agelint source files are: agelint.c agelint.h cmd.l cmd.y conf.l conf.y Makefile txt.l txt.y If you know C, the .c & .h & Makefile should be familiar. The .l are the flex (lex) lexer specifications, and the .y are the bison (yacc) parser (grammar) specifications. The .l and .y files are C-like. flex processes the .l files, and bison process the .y files, to generate C files -- of hideous complexity -- which are then compiled by gcc, then linked together with the compiled .c components into one single executable, agelint. If flex and bison are new to you, if you haven't programmed lexers and parsers before -- and if you wish to tweak or seriously rewrite the .l & .y files -- be prepared to have your mind blown. flex and bison programming can seem mighty weird. You will experience this if, say you make a "little" change here or there, and your conflicts line says something like: txt.y: conflicts: 768 shift/reduce, 523 reduce/reduce rather than the current txt.y: conflicts: 7 shift/reduce, 2 reduce/reduce These "conflicts" represent ambiguities in the lexer/parser specification. Ideally you want 0 conflicts of each type, but that is difficult to impossible, depending on the coding pains you take, and possibly the language (in this case, the AGE scripting language) that you are attempting to model. I will have (much) more to say about agelint's technical innards in forum postings and possibly added doc files at a later date. The AGElint toolkit includes a number of Bash & Perl scripts, some to invoke the agelint executable, but others to run independently. The Perl scripts fixlint.pl excludes.pl chklocals.pl chkimages.pl chkfiles.pl chkaliases.pl chksel.pl (with two links to that script, chkselfac.pl & chkselreg.pl) common.pl conf.pl chklint.pl (and perhaps others) and the Bash script dochk (and perhaps others) should run just fine with little or no tweaking required. =============================================================================== OPERATION A typical agelint command $ ./agelint -n -w +e +c5 +u +x -g rus "Events/GRNAI.sct" 2>&1 | less to check the RUS Events/GRNAI.sct file, not showing sources or using less, not showing notices or warnings, but showing errors, with 5 lines of context, with AGE Wiki URL, with syntax summary. (NOTE: You can take care of the pesky spaces in Windows file names either by enclosing your file specifications in "" as shown, else adding a \ before every space.) You can run AGElint again most or all of the game data text files in Events, but also Aliases, Includes, Scens, Scripts, and any of the subfolders in the GameData directory tree. Here is a usage summary for agelint: $ ./agelint +h Usage: agelint { +h | +V | {+|-}nwevsuxqXNLTG +c# | {+|-}{la|lf|li|ll}} [-S {ALL|<# ...>}] [-I {ALL|<# ...>}] [-g {acw|ncp|pon|rop|rus|wia|chk|fix} <data file>] +h show program help +V show program version +|-n show/don't show notices +|-w show/don't show warnings +|-e show/don't show errors +|-v show/don't show event +|-s show/don't show sources +|-u show/don't show AGEWIKI command URLs +|-x show/don't show AGE command syntax +|-q show/don't show squelches +|-X show/don't show text files +|-N show/don't show line numbers +|-L use/don't use less to display output +|-c[#] show/don't show context (# of lines, 1-9) +|-la list/don't list alias refs +|-lf list/don't list all file refs +|-li list/don't list image file refs +|-ll list/don't list localization refs -S squelch lines -I ignore lines +|-T run/don't run in test mode +|-G run/don't run in debug mode agelint checks just one file. chklint.pl is a wrapper script to process all files (with some exclusions that you can specify in the game excludes.???.dat file). Here is a typical chklint.pl operation: $ ./chklint.pl -n -w +e +c9 +u +x +q -g rus 2>&1 | tee ./chklint.rus.error.out The command-line options for chklint are much like those for agelint: $ ./chklint.pl -h Usage: chklint.pl {+|-}tnwevsuxqEFTG [+c#] [-S <# ...>] -g {acw|ncp|pon|rop|rus|wia|chk|fix} The Perl scripts, and support files: Here is a typical chkaliases.pl operation: $ ./chkaliases.pl -g ncp 2>&1 > chkaliases_ncp_TEST_20120117_rpt.txt Command lines for the other chk*.pl scripts are similar. fixlint.pl is the beginnings of a script to mechanistically autocorrect many of the simpler "bugs" in the AGE data text files (such as removing extraneous trailing |). It is designed for invoking within chklint.pl to fix hundreds and even thousands of problem across all the games files en masse, in one single operation. fixlint.pl, and its association with chklint.pl, is definitely a work-in-progress. Use at your own risk! dochk is the wrapper script for running chklint.pl and all the other chk*.pl scripts (and optionally preparing a .zip file with various report files). A typical operation: ------------------------------------------------------------------------------- [root@berto agelint]# time ./dochk acw TEST doing chklint... doing chkaliases... doing chklocals... doing chkimages... doing chkfiles... doing chkselfac... doing chkselreg... report files generated: chkaliases_acw_TEST_20120117_lst.txt chkaliases_acw_TEST_20120117_pat.txt chkaliases_acw_TEST_20120117_rpt.txt chkaliases_acw_TEST_20120117_sorted_rpt.txt chkfiles_acw_TEST_20120117_lst.txt chkfiles_acw_TEST_20120117_rpt.txt chkfiles_acw_TEST_20120117_sorted_rpt.txt chkimages_acw_TEST_20120117_lst.txt chkimages_acw_TEST_20120117_rpt.txt chkimages_acw_TEST_20120117_sorted_rpt.txt chklint_acw_TEST_20120117_error_rpt.txt chklint_acw_TEST_20120117_warning_rpt.txt chklocals_acw_TEST_20120117_lst.txt chklocals_acw_TEST_20120117_rpt.txt chklocals_acw_TEST_20120117_sorted_rpt.txt chkselfac_acw_TEST_20120117_lst.txt chkselfac_acw_TEST_20120117_rpt.txt chkselfac_acw_TEST_20120117_sorted_rpt.txt chkselreg_acw_TEST_20120117_lst.txt chkselreg_acw_TEST_20120117_rpt.txt chkselreg_acw_TEST_20120117_sorted_rpt.txt real 8m0.226s user 1m31.938s sys 4m22.007s ------------------------------------------------------------------------------- I could go on at great, great length; I could write a whole manual about how to use the AGElint toolkit. Instead, I will post usage examples at the forum in the weeks and months ahead. Whatever. Play around with agelint, experiment! =============================================================================== POSTSCRIPT At the moment... I have essentially finished the process of filling out the txt.y parser file. Most of the stanzas are "finished" to sufficient detail (basing off the "definitive" AGE Wiki pages), but a few are still written in more generic style. The txt.l lexer file is essentially complete, but I tweak it from time to time as necessary. The other *.[lych] files are generally pretty clean. (NOTE: I originally ported this over from my earlier Magna Mundi work, "EU3debug" -- a project internal to the EU3/Magna Mundi mod. There might be bits and pieces of EU3debug cruft remaining in the AGElint project files! And even a PIKT reference or two. [See my Open Source project: http://pikt.org.]) The various chk*.pl scripts are more or less done -- but there are always bugs, refinements, and extensions, right? At present, AGElint is more than a beginning, but far from an ending. The dream is to code a set of utilities and capabilities to check all manner of AGEOD bugs/errors/glitches/anomalies. In agelint itself, beyond the syntax checking already in place, there are the beginnings of semantic and logic checking. Outside of agelint, there are many more QA checks we can apply to the AGE game data files. There is a large component of the AGElint system I have omitted from this distribution: the AGE command/keyword inventory. I spent 1-1/2 days of computer processing time (no kidding!) surveying and compiling a "kwd" database of actual AGE command/keyword usage. When deciding the txt.y parser rules, I am guided primarily by the AGE Wiki documentation (where it exists, which is not always), and secondarily by actual usage as demonstrated by the game data files. They are not always in agreement! And there is much ambiguity and nuance. Bear that in mind when reviewing/critiquing the txt.y parser specification. The plan is to make the kwd database available for reference on the Web. Also to host an AGElint web services site, where you could upload files for checking without having to install and run AGElint yourself. (Other big plans, too.) There's still much more untapped potential, much more to be done... Good luck, and happy debugging! Robert Osterlund aka Berto 20120117 ===============================================================================
|
|
|
|