summaryrefslogtreecommitdiff
path: root/README.TESTING
diff options
context:
space:
mode:
authorLorry Tar Creator <lorry-tar-importer@baserock.org>2013-03-14 05:42:27 +0000
committer <>2013-04-03 16:25:08 +0000
commitc4dd7a1a684490673e25aaf4fabec5df138854c4 (patch)
tree4d57c44caae4480efff02b90b9be86f44bf25409 /README.TESTING
downloadphp2-master.tar.gz
Imported from /home/lorry/working-area/delta_php2/php-5.4.13.tar.bz2.HEADphp-5.4.13master
Diffstat (limited to 'README.TESTING')
-rw-r--r--README.TESTING180
1 files changed, 180 insertions, 0 deletions
diff --git a/README.TESTING b/README.TESTING
new file mode 100644
index 0000000..75141d5
--- /dev/null
+++ b/README.TESTING
@@ -0,0 +1,180 @@
+[IMPORTANT NOTICE]
+------------------
+ Failed tests usualy indicate a problem with your local system setup
+and not within PHP itself (at least for official PHP release versions).
+You may decide to automaticaly submit a test summary to our QA workflow
+at the end of a test run.
+ Please do *not* submit a failed test as a bug or ask for help on why
+it failed on your system without providing substantial backup information
+on *why* the test failed on your special setup. Thank you :-)
+
+
+[Testing Basics]
+----------------
+ The easiest way to test your PHP build is to run "make test" from the
+command line after successfully compiling. This will run the complete
+tests for all enabled functionalities and extensions using the PHP
+CLI binary.
+ To execute test scripts, you must build PHP with some SAPI, then you
+type "make test" to execute all or some test scripts saved under
+"tests" directory under source root directory.
+
+Usage:
+make test
+
+ "make test" basically executes "run-tests.php" script
+under the source root (parallel builds will not work). Therefore you
+can execute the script as follows:
+
+TEST_PHP_EXECUTABLE=sapi/cli/php \
+sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
+
+
+[Which "php" executable "make test" look for]
+---------------------------------------------
+If you are running the run-tests.php script from the command line (as above)
+you must set the TEST_PHP_EXECUTABLE environment variable to explicitly
+select the PHP executable that is to be tested, that is, used to run the test scripts.
+
+If you run the tests using make test, the PHP CLI and CGI executables are
+automatically set for you. "make test" executes "run-tests.php" script with the CLI binary. Some
+test scripts such as session must be executed by CGI SAPI. Therefore,
+you must build PHP with CGI SAPI to perform all tests.
+
+NOTE: PHP binary executing "run-tests.php" and php binary used for
+executing test scripts may differ. If you use different PHP binary for
+executing "run-tests.php" script, you may get errors.
+
+
+[Which php.ini is used]
+-----------------------
+ "make test" uses the same php.ini file as it would once installed.
+The tests have been written to be independent of that php.ini file,
+so if you find a test that is affected by a setting, please report
+this, so we can address the issue.
+
+
+[Which test scripts are executed]
+---------------------------------
+ "run-tests.php" ("make test"), without any arguments executes all
+test scripts by extracting all directories named "tests"
+from the source root and any subdirectories below. If there are files,
+which have a "phpt" extension, "run-tests.php" looks at the sections
+in these files, determines whether it should run it, by evaluating
+the 'SKIP' section. If the test is eligible for execution, the 'FILE'
+section is extracted into a ".php" file (with the same name besides
+the extension) and gets executed.
+When an argument is given or TESTS environment variable is set, the
+GLOB is expanded by the shell and any file with extension "*.phpt" is
+regarded as a test file.
+
+ Tester can easily execute tests selectively with as follows.
+
+Examples:
+./sapi/cli/php run-tests.php ext/mbstring/*
+./sapi/cli/php run-tests.php ext/mbstring/020.phpt
+
+
+[Test results]
+--------------
+ Test results are printed to standard output. If there is a failed test,
+the "run-tests.php" script saves the result, the expected result and the
+code executed to the test script directory. For example, if
+ext/myext/tests/myext.phpt fails to pass, the following files are created:
+
+ext/myext/tests/myext.php - actual test file executed
+ext/myext/tests/myext.log - log of test execution (L)
+ext/myext/tests/myext.exp - expected output (E)
+ext/myext/tests/myext.out - output from test script (O)
+ext/myext/tests/myext.diff - diff of .out and .exp (D)
+
+ Failed tests are always bugs. Either the test is bugged or not considering
+factors applying to the tester's environment, or there is a bug in PHP.
+If this is a known bug, we strive to provide bug numbers, in either the
+test name or the file name. You can check the status of such a bug, by
+going to: http://bugs.php.net/12345 where 12345 is the bug number.
+For clarity and automated processing, bug numbers are prefixed by a hash
+sign '#' in test names and/or test cases are named bug12345.phpt.
+
+NOTE: The files generated by tests can be selected by setting the
+environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
+generated use the character in brackets as shown above (default is LEOD).
+The php file will be generated always.
+
+NOTE: You can set environment variable TEST_PHP_DETAILED to enable
+detailed test information.
+
+[Automated testing]
+ If you like to keep up to speed, with latest developments and quality
+assurance, setting the environment variable NO_INTERACTION to 1, will not
+prompt the tester for any user input.
+
+Normally, the exit status of "make test" is zero, regardless of the results
+of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
+and "make test" will set the exit status ("$?") to non-zero, when an
+individual test has failed.
+
+Example script to be run by cron(1):
+========== qa-test.sh =============
+#!/bin/sh
+
+CO_DIR=$HOME/cvs/php5
+MYMAIL=qa-test@domain.com
+TMPDIR=/var/tmp
+TODAY=`date +"%Y%m%d"`
+
+# Make sure compilation enviroment is correct
+CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
+export MAKE=gmake
+export CC=gcc
+
+# Set test environment
+export NO_INTERACTION=1
+export REPORT_EXIT_STATUS=1
+
+cd $CO_DIR
+cvs update . >>$TMPDIR/phpqatest.$TODAY
+./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
+$MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
+if test $? -gt 0
+then
+ cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
+fi
+========== end of qa-test.sh =============
+
+NOTE: the exit status of run-tests.php will be 1 when
+REPORT_EXIT_STATUS is set. The result of "make test" may be higher
+than that. At present, gmake 3.79.1 returns 2, so it is
+advised to test for non-zero, rather then a specific value.
+
+
+[Creating new test files]
+-------------------------
+ Writing test file is very easy if you are used to PHP.
+See the HOWTO at http://qa.php.net/write-test.php
+
+
+[How to help us]
+----------------
+ If you find bug in PHP, you can submit bug report AND test script
+for us. You don't have to write complete script, just give us test
+script with following format. Please test the script and make sure
+you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
+submit.
+
+<?php
+/*
+Bug #12345
+substr() bug. Do not return expected string.
+
+ACTUAL OUTPUT
+XYXA
+
+EXPECTED OUTPUT
+ABCD
+*/
+
+$str = "XYZABCD";
+echo substr($str,3,7);
+
+?>