How to Test Assignments
515 Compiler Design
Overview:

For each assignment (p1, p3-p8) I have posted a set of tests and a bash script to run the tests.

There are two versions of this script.  One for p1, and one for p3-p8.  These scripts are very similar.

It is a good idea to understand how the testing script works.  If you fail a test you need to be able to find the problem.  The basic idea is that for each test there are three files:  input, output, error_output.  The tests are run like this

For p1:
$ parser tests/t001.in > results/t001.myout 2> results/t001.myerr

For p3-p5:
$ gpl tests/t001.gpl > results/t001.myout 2> results/t001.myerr

For p6-p8:
See below

If results/t001.myout is identical to tests/t001.out and results/t001.myerr is identical to tests/t001.err then the program passed this test.

You can use the diff or the vimdiff utility to see if the files are identical.

Sometimes I develop new and better tests after I announce the assignment.  I will announce new tests via e-mail and in lecture.  If you don't read e-mail and don't attend lecture, make sure you check for new tests.

If you develop a particularly good test, e-mail it to me and I will (1) tell you if I got the same output and (2) incorporate it in the test suite.

Running the Tests (p1, p3, p4, p5):

Replace the pN in the following instructions with the phase you are working on.  For example, if you are working on p1, replace pN with p1.

Step 1: Download the file pN_tests.tar.gz to the directory that contains your pN code (sftp should automatically be installed in Linux and OSX, for Cygwin if you followed my Cygwin instructions and checked the openssh button sftp should be installed):

$ sftp your_ecst_username@jaguar.ecst.csuchico.edu
sftp> cd /user/faculty/tyson/515/tests/pN
sftp> get pN_tests.tar.gz
sftp> exit
$

Step 2: Untar this file:

$ tar -xvf pN_tests.tar.gz

tar will create the directory tests which will contain all the pN tests

The file run_p1_tests or run_all_tests will be placed into the current directory.  They are bash scripts that execute all the tests in the tests directory.


Step 3: Run all the tests using the run_all_tests script (for p1 use the run_p1_tests script):

$ run_all_tests
Failed t000: stdout incorrect
Passed t001
Failed t002: stderr incorrect
Failed t005: stdout & stderr incorrect
...

If you get the error:

    bash: ./run_all_tests: Permission denied

you need to set the execute permission on this file

    $ chmod 700 run_all_tests
    $ run_all_tests
    ...

If you get a "command not found" error, you need to put "." in your PATH or you need to include the current directory in your command:

    $ ./run_all_tests

The run_all_tests script will execute your program (gpl in the current directory) for each of the tests in the tests directory.  It will put the results in the directory ./results and then compare your results to the results in the ./tests directory.  If these files differ you fail the test.

Running the tests for p6

In this phase you create graphics.  Thus you can't test your program without comparing the graphics your program generates with the graphics my program generates.

There is a new run_all_tests script for this phase.  Make sure you download this new script.  It works like the previous phases except it will compare the graphics your program generates (see below) in addition to the standard output and standard error.

If the program does not have any errors, a window will pop up running the gpl program specified in the test (e.g. tests/t001.gpl).  You will have to close the gpl program by typing a q into the window (don't close it by closing the window, gpl.cpp does some work when the 'q' is entered).

For programs that open windows, a file containing all the pixels in your window will be created (e.g. results/t001.pixels).  Your file (e.g. results/t001.pixels) must match my file (tests/t001.pixels) for you to pass the test.

I also provide a .jpg file (such as t001.jpg) for each test that opens a window.  Your window must look just like the posted .jpg file. 

NOTE: the window title is not in the .pixels file, so make sure you have the correct window title.

Running the tests for p7 & p8

In this phase you handle keyboard input.  gpl.cpp has a mechanism for reading keyboard input from a file in order to test how programs handle input. 

If there is a .keypresses file in the test directory, run_all_tests will redirect this file into gpl's standard input and pass the flag "-stdin" to gpl.  When gpl receives the "-stdin" flag, key presses are read from standard input instead of the keyboard (usually the Glut library reads the keypresses, this mechanism circumvents Glut's input mechanism).

Most tests include a 'q' in the .keypresses file.  That means the test will complete w/o any interaction from you.  However, some tests will require you to look at a .jpg file to make sure your graphics are correct.

When a test does not exit itself, you need to compare the current screen to the .jpg file corresponding to this tests.  After you make the comparison, press 'q' in the gpl window to exit gpl.  The run_all_tests script will start the next test as soon as you exit a test.

If a test does not exit itself and there is NO .jpg file, your program probably has an error.