diff options
author | Lorry Tar Creator <lorry-tar-importer@baserock.org> | 2013-03-14 05:42:27 +0000 |
---|---|---|
committer | <> | 2013-04-03 16:25:08 +0000 |
commit | c4dd7a1a684490673e25aaf4fabec5df138854c4 (patch) | |
tree | 4d57c44caae4480efff02b90b9be86f44bf25409 /README.TESTING | |
download | php2-master.tar.gz |
Imported from /home/lorry/working-area/delta_php2/php-5.4.13.tar.bz2.HEADphp-5.4.13master
Diffstat (limited to 'README.TESTING')
-rw-r--r-- | README.TESTING | 180 |
1 files changed, 180 insertions, 0 deletions
diff --git a/README.TESTING b/README.TESTING new file mode 100644 index 0000000..75141d5 --- /dev/null +++ b/README.TESTING @@ -0,0 +1,180 @@ +[IMPORTANT NOTICE] +------------------ + Failed tests usualy indicate a problem with your local system setup +and not within PHP itself (at least for official PHP release versions). +You may decide to automaticaly submit a test summary to our QA workflow +at the end of a test run. + Please do *not* submit a failed test as a bug or ask for help on why +it failed on your system without providing substantial backup information +on *why* the test failed on your special setup. Thank you :-) + + +[Testing Basics] +---------------- + The easiest way to test your PHP build is to run "make test" from the +command line after successfully compiling. This will run the complete +tests for all enabled functionalities and extensions using the PHP +CLI binary. + To execute test scripts, you must build PHP with some SAPI, then you +type "make test" to execute all or some test scripts saved under +"tests" directory under source root directory. + +Usage: +make test + + "make test" basically executes "run-tests.php" script +under the source root (parallel builds will not work). Therefore you +can execute the script as follows: + +TEST_PHP_EXECUTABLE=sapi/cli/php \ +sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB] + + +[Which "php" executable "make test" look for] +--------------------------------------------- +If you are running the run-tests.php script from the command line (as above) +you must set the TEST_PHP_EXECUTABLE environment variable to explicitly +select the PHP executable that is to be tested, that is, used to run the test scripts. + +If you run the tests using make test, the PHP CLI and CGI executables are +automatically set for you. "make test" executes "run-tests.php" script with the CLI binary. Some +test scripts such as session must be executed by CGI SAPI. Therefore, +you must build PHP with CGI SAPI to perform all tests. + +NOTE: PHP binary executing "run-tests.php" and php binary used for +executing test scripts may differ. If you use different PHP binary for +executing "run-tests.php" script, you may get errors. + + +[Which php.ini is used] +----------------------- + "make test" uses the same php.ini file as it would once installed. +The tests have been written to be independent of that php.ini file, +so if you find a test that is affected by a setting, please report +this, so we can address the issue. + + +[Which test scripts are executed] +--------------------------------- + "run-tests.php" ("make test"), without any arguments executes all +test scripts by extracting all directories named "tests" +from the source root and any subdirectories below. If there are files, +which have a "phpt" extension, "run-tests.php" looks at the sections +in these files, determines whether it should run it, by evaluating +the 'SKIP' section. If the test is eligible for execution, the 'FILE' +section is extracted into a ".php" file (with the same name besides +the extension) and gets executed. +When an argument is given or TESTS environment variable is set, the +GLOB is expanded by the shell and any file with extension "*.phpt" is +regarded as a test file. + + Tester can easily execute tests selectively with as follows. + +Examples: +./sapi/cli/php run-tests.php ext/mbstring/* +./sapi/cli/php run-tests.php ext/mbstring/020.phpt + + +[Test results] +-------------- + Test results are printed to standard output. If there is a failed test, +the "run-tests.php" script saves the result, the expected result and the +code executed to the test script directory. For example, if +ext/myext/tests/myext.phpt fails to pass, the following files are created: + +ext/myext/tests/myext.php - actual test file executed +ext/myext/tests/myext.log - log of test execution (L) +ext/myext/tests/myext.exp - expected output (E) +ext/myext/tests/myext.out - output from test script (O) +ext/myext/tests/myext.diff - diff of .out and .exp (D) + + Failed tests are always bugs. Either the test is bugged or not considering +factors applying to the tester's environment, or there is a bug in PHP. +If this is a known bug, we strive to provide bug numbers, in either the +test name or the file name. You can check the status of such a bug, by +going to: http://bugs.php.net/12345 where 12345 is the bug number. +For clarity and automated processing, bug numbers are prefixed by a hash +sign '#' in test names and/or test cases are named bug12345.phpt. + +NOTE: The files generated by tests can be selected by setting the +environment variable TEST_PHP_LOG_FORMAT. For each file you want to be +generated use the character in brackets as shown above (default is LEOD). +The php file will be generated always. + +NOTE: You can set environment variable TEST_PHP_DETAILED to enable +detailed test information. + +[Automated testing] + If you like to keep up to speed, with latest developments and quality +assurance, setting the environment variable NO_INTERACTION to 1, will not +prompt the tester for any user input. + +Normally, the exit status of "make test" is zero, regardless of the results +of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1, +and "make test" will set the exit status ("$?") to non-zero, when an +individual test has failed. + +Example script to be run by cron(1): +========== qa-test.sh ============= +#!/bin/sh + +CO_DIR=$HOME/cvs/php5 +MYMAIL=qa-test@domain.com +TMPDIR=/var/tmp +TODAY=`date +"%Y%m%d"` + +# Make sure compilation enviroment is correct +CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre' +export MAKE=gmake +export CC=gcc + +# Set test environment +export NO_INTERACTION=1 +export REPORT_EXIT_STATUS=1 + +cd $CO_DIR +cvs update . >>$TMPDIR/phpqatest.$TODAY +./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE +$MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1 +if test $? -gt 0 +then + cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL +fi +========== end of qa-test.sh ============= + +NOTE: the exit status of run-tests.php will be 1 when +REPORT_EXIT_STATUS is set. The result of "make test" may be higher +than that. At present, gmake 3.79.1 returns 2, so it is +advised to test for non-zero, rather then a specific value. + + +[Creating new test files] +------------------------- + Writing test file is very easy if you are used to PHP. +See the HOWTO at http://qa.php.net/write-test.php + + +[How to help us] +---------------- + If you find bug in PHP, you can submit bug report AND test script +for us. You don't have to write complete script, just give us test +script with following format. Please test the script and make sure +you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you +submit. + +<?php +/* +Bug #12345 +substr() bug. Do not return expected string. + +ACTUAL OUTPUT +XYXA + +EXPECTED OUTPUT +ABCD +*/ + +$str = "XYZABCD"; +echo substr($str,3,7); + +?> |