1 Major changes between releases Automated Testing Framework
2 ===========================================================================
5 Changes in version 0.20
6 ***********************
8 Experimental version released on February 7th, 2014.
10 This is the first release without the code for the deprecated tools. If
11 you require such code, please fetch a copy of the 0.19 release and extract
12 the 'tools' directory for your own consumption.
14 * Removed the deprecated tools. This includes atf-config, atf-report,
15 atf-run and atf-version.
17 * Issue #8: Fixed atf-c/macros_test:use test failures spotted by the clang
18 that ships with FreeBSD 11.0-CURRENT.
21 Changes in version 0.19
22 ***********************
24 Experimental version released on February 7th, 2014.
26 This is the last release to bundle the code for the deprecated tools.
27 The next release will drop their code and will stop worrying about
28 backwards compatibility between the ATF libraries and what the old tools
29 may or may not support.
31 If you still require the old tools for some reason, grab a copy of the
32 'tools' directory now. The code in this directory is standalone and
33 does not depend on any internal details of atf-c++ any longer.
35 * Various fixes and improvements to support running as part of the FreeBSD
38 * Project hosting moved from Google Code (as a subproject of Kyua) to
39 GitHub (as a first-class project). The main reason for the change is
40 the suppression of binary downloads in Google Code on Jan 15th, 2014.
41 See https://github.com/jmmv/atf/
43 * Removed builtin help from atf-sh(1) and atf-check(1) for simplicity
44 reasons. In other words, their -h option is gone.
46 * Moved the code of the deprecated tools into a 'tools' directory and
47 completely decoupled their code from the internals of atf-c++. The
48 reason for this is to painlessly allow a third-party to maintain a
49 copy of these tools after we delete them because upcoming changes to
50 atf-c++ would break the stale tools.
53 Changes in version 0.18
54 ***********************
56 Experimental version released on November 16th, 2013.
58 * Issue 45: Added require.memory support in atf-run for FreeBSD.
60 * Fixed an issue with the handling of cin with libc++.
62 * Issue 64: Fixed various mandoc formatting warnings.
64 * NetBSD PR bin/48284: Made atf-check flush its progress message to
65 stdout so that an interrupted test case always shows the last message
68 * NetBSD PR bin/48285: Fixed atf_check examples in atf-sh-api(3).
71 Changes in version 0.17
72 ***********************
74 Experimental version released on February 14th, 2013.
76 * Added the atf_utils_cat_file, atf_utils_compare_file,
77 atf_utils_copy_file, atf_utils_create_file, atf_utils_file_exists,
78 atf_utils_fork, atf_utils_grep_file, atf_utils_grep_string,
79 atf_utils_readline, atf_utils_redirect and atf_utils_wait utility
80 functions to atf-c-api. Documented the already-public
81 atf_utils_free_charpp function.
83 * Added the cat_file, compare_file, copy_file, create_file, file_exists,
84 fork, grep_collection, grep_file, grep_string, redirect and wait
85 functions to the atf::utils namespace of atf-c++-api. These are
86 wrappers around the same functions added to the atf-c-api library.
88 * Added the ATF_CHECK_MATCH, ATF_CHECK_MATCH_MSG, ATF_REQUIRE_MATCH and
89 ATF_REQUIRE_MATCH_MSG macros to atf-c to simplify the validation of a
90 string against a regular expression.
92 * Miscellaneous fixes for manpage typos and compilation problems with
95 * Added caching of the results of those configure tests that rely on
96 executing a test program. This should help crossbuild systems by
97 providing a mechanism to pre-specify what the results should be.
99 * PR bin/45690: Make atf-report convert any non-printable characters to
100 a plain-text representation (matching their corresponding hexadecimal
101 entities) in XML output files. This is to prevent the output of test
102 cases from breaking xsltproc later.
105 Changes in version 0.16
106 ***********************
108 Experimental version released on July 10th, 2012.
110 * Added a --enable-tools flag to configure to request the build of the
111 deprecated ATF tools, whose build is now disabled by default. In order
112 to continue running tests, you should migrate to Kyua instead of enabling
113 the build of the deprecated tools. The kyua-atf-compat package provides
114 transitional compatibility versions of atf-run and atf-report built on
117 * Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can
118 detect defined but unused test cases.
120 * PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and
121 tp-time XML tags leaking into the generated HTML file. Also improved
122 the CSS file slightly to correct alignment and color issues with the
125 * Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during
126 compilation with GNU G++.
128 * Flipped the default to building shared libraries for atf-c and atf-c++,
129 and started versioning them. As a side-effect, this removes the
130 --enable-unstable-shared flag from configure that appears to not work any
131 more (under NetBSD). Additionally, some distributions require the use of
132 shared libraries for proper dependency tracking (e.g. Fedora), so it is
133 better if we do the right versioning upstream.
135 * Project hosting moved from an adhoc solution (custom web site and
136 Monotone repository) to Google Code (standard wiki and Git). ATF now
137 lives in a subcomponent of the Kyua project.
140 Changes in version 0.15
141 ***********************
143 Experimental version released on January 16th, 2012.
145 * Respect stdin in atf-check. The previous release silenced stdin for any
146 processes spawned by atf, not only test programs, which caused breakage
147 in tests that pipe data through atf-check.
149 * Performance improvements to atf-sh.
151 * Enabled detection of unused parameters and variables in the code and
154 * Changed the behavior of "developer mode". Compiler warnings are now
155 enabled unconditionally regardless of whether we are in developer mode or
156 not; developer mode is now only used to perform strict warning checks and
157 to enable assertions. Additionally, developer mode is now only
158 automatically enabled when building from the repository, not for formal
161 * Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and
162 ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag
163 in configure scripts and detecting the presence of any of the ATF
164 bindings. Note that ATF_CHECK_SH was already introduced in 0.14, but it
165 has now been modified to also honor --with-atf if instantiated.
167 * Added timing support to atf-run / atf-report.
169 * Added support for a 'require.memory' property, to specify the minimum
170 amount of physical memory needed by the test case to yield valid results.
172 * PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by
173 atf-report so that invalid data in the output of test cases does not
177 Changes in version 0.14
178 ***********************
180 Experimental version released on June 14th, 2011.
182 * Added a pkg-config file for atf-sh and an aclocal file to ease the
183 detection of atf-sh from autoconf scripts.
185 * Made the default test case body defined by atf_sh fail. This is to
186 ensure that test cases are properly defined in test programs and helps
187 in catching typos in the names of the body functions.
189 * PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero.
190 This provides more consistent results with "normal" execution (in
191 particular, when tests are executed detached from a terminal).
193 * Made atf-run hardcode TZ=UTC for test cases. It used to undefine TZ, but
194 that does not take into account that libc determines the current timezone
195 from a configuration file.
197 * All test programs will now print a warning when they are not run through
198 atf-run(1) stating that this is unsupported and may deliver incorrect
201 * Added support for the 'require.files' test-case property. This allows
202 test cases to specify installed files that must be present for the test
206 Changes in version 0.13
207 ***********************
209 Experimental version released on March 31st, 2011.
211 This is the first release after the creation of the Kyua project, a more
212 modular and reliable replacement for ATF. From now on, ATF will change to
213 accomodate the transition to this new codebase, but ATF will still continue
214 to see development in the short/medium term. Check out the project page at
215 http://code.google.com/p/kyua/ for more details.
217 The changes in this release are:
219 * Added support to run the tests with the Kyua runtime engine (kyua-cli), a
220 new package that aims to replace atf-run and atf-report. The ATF tests
221 can be run with the new system by issuing a 'make installcheck-kyua' from
222 the top-level directory of the project (assuming the 'kyua' binary is
223 available during the configuration stage of ATF).
225 * atf-run and atf-report are now in maintenance mode (but *not* deprecated
226 yet!). Kyua already implements a new, much more reliable runtime engine
227 that provides similar features to these tools. That said, it is not
228 complete yet so all development efforts should go towards it.
230 * If GDB is installed, atf-run dumps the stack trace of crashing test
231 programs in an attempt to aid debugging. Contributed by Antti Kantee.
233 * Reverted default timeout change in previous release and reset its value
234 to 5 minutes. This was causing several issues, specially when running
235 the existing NetBSD test suite in qemu.
237 * Fixed the 'match' output checker in atf-check to properly validate the
238 last line of a file even if it does not have a newline.
240 * Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to
241 check for the presence (or lack thereof) of an element in a collection.
243 * PR bin/44176: Fixed a race condition in atf-run that would crash atf-run
244 when the cleanup of a test case triggered asynchronous modifications to
245 its work directory (e.g. killing a daemon process that cleans up a pid
246 file in the work directory).
248 * PR bin/44301: Fixed the sample XSLT file to report bogus test programs
249 instead of just listing them as having 0 test cases.
252 Changes in version 0.12
253 ***********************
255 Experimental version released on November 7th, 2010.
257 * Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as
258 ATF_REQUIRE_THROW but allows checking for the validity of the exception's
259 error message by means of a regular expression.
261 * Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a
262 regular expression match in a string.
264 * Changed the default timeout for test cases from 5 minutes to 30 seconds.
265 30 seconds is long enough for virtually all tests to complete, and 5
266 minutes is a way too long pause in a test suite where a single test case
269 * Deprecated the use.fs property. While this seemed like a good idea in
270 the first place to impose more control on what test cases can do, it
271 turns out to be bad. First, use.fs=false prevents bogus test cases
272 from dumping core so after-the-fact debugging is harder. Second,
273 supporting use.fs adds a lot of unnecessary complexity. atf-run will
274 now ignore any value provided to use.fs and will allow test cases to
275 freely access the file system if they wish to.
277 * Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c
278 library. The 'text' module became private in 0.11 but was being used
279 externally to simplify the parsing of configuration variables.
281 * Made atf-run recognize the 'unprivileged-user' configuration variable
282 and automatically drop root privileges when a test case sets
283 require.user=unprivileged. Note that this is, by no means, done for
284 security purposes; this is just for user convenience; tests should, in
285 general, not be blindly run as root in the first place.
288 Changes in version 0.11
289 ***********************
291 Experimental version released on October 20th, 2010.
293 * The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match
294 their counterparts in atf-c.
296 * Clearly separated the modules in atf-c that are supposed to be public
297 from those that are implementation details. The header files for the
298 internal modules are not installed any more.
300 * Made the atf-check tool private. It is only required by atf-sh and being
301 public has the danger of causing confusion. Also, making it private
302 simplifies the public API of atf.
304 * Changed atf-sh to enable per-command error checking (set -e) by default.
305 This catches many cases in which a test case is broken but it is not
306 reported as such because execution continues.
308 * Fixed the XSTL and CSS stylesheets to support expected failures.
311 Changes in version 0.10
312 ***********************
314 Experimental version released on July 2nd, 2010.
316 Miscellaneous features
318 * Added expected failures support to test cases and atf-run. These
319 include, for example, expected clean exits, expected reception of fatal
320 signals, expected timeouts and expected errors in condition checks.
321 These statuses can be used to denote test cases that are known to fail
322 due to a bug in the code they are testing. atf-report reports these
323 tests separately but they do not count towards the failed test cases
326 * Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to
327 allow easy checking of call failures that update errno.
329 * Added the has.cleanup meta-data property to test caes that specifies
330 whether the test case has a cleanup routine or not; its value is
331 automatically set. This property is read by atf-run to know if it has to
332 run the cleanup routine; skipping this run for every test case
333 significantly speeds up the run time of test suites.
335 * Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to
336 take the expected exception as the first argument and the statement to
337 execute as the second argument.
341 * Changed atf-check to support negating the status and output checks by
342 prefixing them with not- and added support to specify multiple checkers
343 for stdout and stderr, not only one.
345 * Added the match output checker to atf-check to look for regular
346 expressions in the stdout and stderr of commands.
348 * Modified the exit checks in atf-check to support checking for the
349 reception of signals.
351 Code simplifications and cleanups
353 * Removed usage messages from test programs to simplify the
354 implementation of every binding by a significant amount. They just now
355 refer the user to the appropriate manual page and do not attempt to wrap
356 lines on terminal boundaries. Test programs are not supposed to be run
357 by users directly so this minor interface regression is not important.
359 * Removed the atf-format internal utility, which is unused after the
360 change documented above.
362 * Removed the atf-cleanup internal utility. It has been unused since the
363 test case isolation was moved to atf-run in 0.8
365 * Splitted the Makefile.am into smaller files for easier maintenance and
366 dropped the use of M4. Only affects users building from the repository
369 * Intermixed tests with the source files in the source tree to provide
370 them more visibility and easier access. The tests directory is gone from
371 the source tree and tests are now suffixed by _test, not prefixed by t_.
373 * Simplifications to the atf-c library: removed the io, tcr and ui
374 modules as they had become unnecessary after all simplifications
375 introduced since the 0.8 release.
377 * Removed the application/X-atf-tcr format introduced in 0.8 release.
378 Tests now print a much simplified format that is easy to parse and nicer
379 to read by end users. As a side effect, the default for test cases is
380 now to print their results to stdout unless otherwise stated by providing
383 * Removed XML distribution documents and replaced them with plain-text
384 documents. They provided little value and introduced a lot of complexity
387 * Simplified the output of atf-version by not attempting to print a
388 revision number when building form a distfile. Makes the build system
392 Changes in version 0.9
393 **********************
395 Experimental version released on June 3rd, 2010.
397 * Added atf-sh, an interpreter to process test programs written using
398 the shell API. This is not really a shell interpreter by itself though:
399 it is just a wrapper around the system shell that eases the loading of
400 the necessary ATF libraries.
402 * Removed atf-compile in favour of atf-sh.
404 * Added the use.fs metadata property to test case, which is used to
405 specify which test cases require file system access. This is to
406 highlight dependencies on external resources more clearly and to speed up
407 the execution of test suites by skipping the creation of many unnecessary
410 * Fixed test programs to get a sane default value for their source
411 directory. This means that it should not be necessary any more to pass
412 -s when running test programs that do not live in the current directory.
414 * Defining test case headers became optional. This is trivial to achieve
415 in shell-based tests but a bit ugly in C and C++. In C, use the new
416 ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use
417 ATF_TEST_CASE_WITHOUT_HEAD.
420 Changes in version 0.8
421 **********************
423 Experimental version released on May 7th, 2010.
425 * Test programs no longer run several test cases in a row. The execution
426 of a test program now requires a test case name, and that single test
427 case is executed. To execute several test cases, use the atf-run utility
430 * Test programs no longer fork a subprocess to isolate the execution of
431 test cases. They run the test case code in-process, and a crash of the
432 test case will result in a crash of the test program. This is to ease
433 debugging of faulty test cases.
435 * Test programs no longer isolate their test cases. This means that they
436 will not create temporary directories nor sanitize the environment any
437 more. Yes: running a test case that depends on system state by hand will
438 most likely yield different results depending on where (machine,
439 directory, user environment, etc.) it is run. Isolation has been moved
442 * Test programs no longer print a cryptic format (application/X-atf-tcs)
443 on a special file channel. They can now print whatever they want on the
444 screen. Because test programs can now only run one test case every time,
445 providing controlled output is not necessary any more.
447 * Test programs no longer write their status into a special file
448 descriptor. Instead, they create a file with the results, which is later
449 parsed by atf-run. This changes the semantics of the -r flag.
451 * atf-run has been adjusted to perform the test case isolation. As a
452 result, there is now a single canonical place that implements the
453 isolation of test caes. In previous releases, the three language
454 bindings (C, C++ and shell) had to be kept in sync with each other (read:
455 not a nice thing to do at all). As a side effect of this change, writing
456 bindings for other languages will be much, much easier from now on.
458 * atf-run forks test programs on a test case basis, instead of on a test
459 program basis as it did before. This is to provide the test case
460 isolation that was before implemented by the test programs themselves.
462 * Removed the atf-exec tool. This was used to implement test case
463 isolation in atf-sh, but it is now unnecessary.
465 * It is now optional to define the descr meta-data property. It has been
466 proven to be mostly useless, because test cases often carry a descriptive
470 Changes in version 0.7
471 **********************
473 Experimental version released on December 22nd, 2009.
475 * Added build-time checks to atf-c and atf-c++. A binding for atf-sh
478 * Migrated all build-time checks for header files to proper ATF tests.
479 This demonstrates the use of the new feature described above.
481 * Added an internal API for child process management.
483 * Converted all plain-text distribution documents to a Docbook canonical
484 version, and include pre-generated plain text and HTML copies in the
487 * Simplified the contents of the Makefile.am by regenerating it from a
488 canonical Makefile.am.m4 source. As a side-effect, some dependency
489 specifications were fixed.
491 * Migrated all checks from the check target to installcheck, as these
492 require ATF to be installed.
494 * Fixed sign comparison mismatches triggered by the now-enabled
497 * Fixed many memory and object leaks.
500 Changes in version 0.6
501 **********************
503 Experimental version released on January 18th, 2009.
505 * Make atf-exec be able to kill its child process after a certain period
506 of time; this is controlled through the new -t option.
508 * Change atf-sh to use atf-exec's -t option to control the test case's
509 timeouts, instead of doing it internally. Same behavior as before, but
512 * atf-exec's -g option and atf-killpg are gone due to the previous
515 * Added the atf-check(1) tool, a program that executes a given command
516 and checks its exit code against a known value and allows the management
517 of stdout and stderr in multiple ways. This replaces the previous
518 atf_check function in the atf-sh library and exposes this functionality
519 to both atf-c and atf-c++.
521 * Added the ATF_REQUIRE family of macros to the C interface. These help
522 in checking for fatal test conditions. The old ATF_CHECK macros now
523 perform non-fatal checks only. I.e. by using ATF_CHECK, the test case
524 can now continue its execution and the failures will not be reported
525 until the end of the whole run.
527 * Extended the amount of ATF_CHECK_* C macros with new ones to provide
528 more features to the developer. These also have their corresponding
529 counterparts in the ATF_REQUIRE_* family. The new macros (listing the
530 suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and
534 Changes in version 0.5
535 **********************
537 Experimental version released on May 1st, 2008.
539 * Clauses 3 and 4 of the BSD license used by the project were dropped.
540 All the code is now under a 2-clause BSD license compatible with the GNU
541 General Public License (GPL).
543 * Added a C-only binding so that binary test programs do not need to be
544 tied to C++ at all. This binding is now known as the atf-c library.
546 * Renamed the C++ binding to atf-c++ for consistency with the new atf-c.
548 * Renamed the POSIX shell binding to atf-sh for consistency with the new
551 * Added a -w flag to test programs through which it is possible to
552 specify the work directory to be used. This was possible in prior
553 releases by defining the workdir configuration variable (-v workdir=...),
554 but was a conceptually incorrect mechanism.
556 * Test programs now preserve the execution order of test cases when they
557 are given in the command line. Even those mentioned more than once are
558 executed multiple times to comply with the user's requests.
561 Changes in version 0.4
562 **********************
564 Experimental version released on February 4th, 2008.
566 * Added two new manual pages, atf-c++-api and atf-sh-api, describing the
567 C++ and POSIX shell interfaces used to write test programs.
569 * Added a pkg-config file, useful to get the flags to build against the
570 C++ library or to easily detect the presence of ATF.
572 * Added a way for test cases to require a specific architecture and/or
573 machine type through the new 'require.arch' and 'require.machine'
574 meta-data properties, respectively.
576 * Added the 'timeout' property to test cases, useful to set an
577 upper-bound limit for the test's run time and thus prevent global test
578 program stalls due to the test case's misbehavior.
580 * Added the atf-exec(1) internal utility, used to execute a command
581 after changing the process group it belongs to.
583 * Added the atf-killpg(1) internal utility, used to kill process groups.
585 * Multiple portability fixes. Of special interest, full support for
586 SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12
589 * Fixed a serious bug that prevented atf-run(1) from working at all
590 under Fedora 8 x86_64. Due to the nature of the bug, other platforms
591 were likely affected too.
594 Changes in version 0.3
595 **********************
597 Experimental version released on November 11th, 2007.
599 * Added XML output support to atf-report. This is accompanied by a DTD
600 for the format's structure and sample XSLT/CSS files to post-process this
601 output and convert it to a plain HTML report.
603 * Changed atf-run to add system information to the report it generates.
604 This is currently used by atf-report's XML output only, and is later
605 printed in the HTML reports in a nice and useful summary table. The user
606 and system administrator are allowed to tune this feature by means of
609 * Removed the test cases' 'isolated' property. This was intended to
610 avoid touching the file system at all when running the related test case,
611 but this has not been true for a long while: some control files are
612 unconditionally required for several purposes, and we cannot easily get
613 rid of them. This way we remove several critical and delicate pieces of
616 * Improved atf-report's CSV output format to include information about
619 * Fixed the tests that used atf-compile to not require this tool as a
620 helper. Avoids systems without build-time utilities to skip many tests
621 that could otherwise be run. (E.g. NetBSD without the comp.tgz set
624 * Many general cleanups: Fixed many pieces of code marked as ugly and/or
628 Changes in version 0.2
629 **********************
631 Experimental version released on September 20th, 2007.
633 * Test cases now get a known umask on entry.
635 * atf-run now detects many unexpected failures caused by test programs and
636 reports them as bogus tests. atf-report is able to handle these new
637 errors and nicely reports them to the user.
639 * All the data formats read and written by the tools have been
640 documented and cleaned up. These include those grammars that define how
641 the different components communicate with each other as well as the
642 format of files written by the developers and users: the Atffiles and the
645 * Added the atf-version tool, a utility that displays information about
646 the currently installed version of ATF.
648 * Test cases can now define an optional cleanup routine to undo their
649 actions regardless of their exit status.
651 * atf-report now summarizes the list of failed (bogus) test programs
652 when using the ticker output format.
654 * Test programs now capture some termination signals and clean up any
655 temporary files before exiting the program.
657 * Multiple bug fixes and improvements all around.
660 Changes in version 0.1
661 **********************
663 Experimental version released on August 20th, 2007.
665 * First public version. This was released coinciding with the end of the
666 Google Summer of Code 2007 program.
669 ===========================================================================
670 vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2