1 Major changes between releases
2 ==============================
5 Changes in version 0.14
6 -----------------------
8 **NOT RELEASED YET; STILL UNDER DEVELOPMENT.**
10 * Explicitly require C++11 language features when compiling Kyua.
13 Changes in version 0.13
14 -----------------------
16 **Released on August 26th, 2016.**
18 * Fixed execution of test cases as an unprivileged user, at least under
19 NetBSD 7.0. Kyua-level failures were probably a regression introduced
20 in Kyua 0.12, but the underlying may have existed for much longer:
21 test cases might have previously failed for mysterious reasons when
22 running under an unprivileged user.
24 * Issue #134: Fixed metadata test broken on 32-bit platforms.
26 * Issue #139: Added per-test case start/end timestamps to all reports.
28 * Issue #156: Fixed crashes due to the invalid handling of cleanup
29 routine data and triggered by the reuse of PIDs in long-running Kyua
32 * Issue #159: Fixed TAP parser to ignore case while matching `TODO` and
33 `SKIP` directives, and to also recognize `Skipped`.
35 * Fixed potential crash due to a race condition in the unprogramming of
36 timers to control test deadlines.
39 Changes in version 0.12
40 -----------------------
42 **Released on November 22nd, 2015.**
44 This is a huge release and marks a major milestone for Kyua as it finally
45 implements a long-standing feature request: the ability to execute test
46 cases in parallel. This is a big deal because test cases are rarely
47 CPU-bound: running them in parallel yields much faster execution times for
48 large test suites, allowing faster iteration of changes during development.
50 As an example: the FreeBSD test suite as of this date contains 3285 test
51 cases. With sequential execution, a full test suite run takes around 12
52 minutes to complete, whereas on a 4-core machine with a high level of
53 parallelism it takes a little over 1 minute.
55 Implementing parallel execution required rewriting most of Kyua's core and
56 partly explains explains why there has not been a new release for over a
57 year. The current implementation is purely subprocess-based, which works
58 but has some limitations and has resulted in a core that is really complex
59 and difficult to understand. Future versions will investigate the use of
60 threads instead for a simplified programming model and additional
61 parallelization possibilities.
63 * Issue #2: Implemented support to execute test cases in parallel when
64 invoking `kyua test`. Parallel execution is *only* enabled when the new
65 `parallelism` configuration variable is set to a value greater than `1`.
66 The default behavior is still to run tests sequentially because some test
67 suites contain test cases with side-effects that might fail when run in
68 parallel. To resolve this, the new metadata property `is_exclusive` can
69 be set to `true` on a test basis to indicate that the test must be run on
72 * Known regression: Running `kyua debug` on a TAP-based test program does
73 not currently report the output in real time. The output will only be
74 displayed once the test program completes. This is a shortcoming of
75 the new parallel execution engine and will be resolved.
77 * Removed the external C-based testers code in favor of the new built-in
78 implementations. The new approach feels significantly faster than the
81 * Fixed the handling of relative paths in the `fs.*` functions available
82 in `Kyuafile`s. All paths are now resolved relative to the location of
83 the caller `Kyuafile`. `Kyuafile.top` has been updated with these
84 changes and you should update custom copies of this file with the new
87 * Changed temporary directory creation to always grant search
88 permissions on temporary directories. This is to prevent potential
89 problems when running Kyua as root and executing test cases that require
90 dropping privileges (as they may later be unable to use absolute paths
91 that point inside their work directory).
93 * The cleanup of work directories does not longer attempt to deal with
94 mount points. If a test case mounts a file system and forgets to unmount
95 it, the mount point will be left behind. It is now the responsibility of
96 the test case to clean after itself. The reasons for this change are
97 simplicity and clarity: there are many more things that a test case can
98 do that have side-effects on the system and Kyua cannot protect against
99 them all, so it is better to just have the test undo anything it might
102 * Improved `kyua report --verbose` to properly handle environment
103 variables with continuation lines in them, and fixed the integration
104 tests for this command to avoid false negatives.
106 * Changed the configuration file format to accept the definition of
107 unknown variables without declaring them local. The syntax version
108 number remains at 2. This is to allow configuration files for newer Kyua
109 versions to work on older Kyua versions, as there is no reason to forbid
112 * Fixed stacktrace gathering with FreeBSD's ancient version of GDB.
113 GDB 6.1.1 (circa 2004) does not have the `-ex` flag so we need to
114 generate a temporary GDB script and feed it to GDB with `-x` instead.
116 * Issue #136: Fixed the XML escaping in the JUnit output so that
117 non-printable characters are properly handled when they appear in the
118 process's stdout or stderr.
120 * Issue #141: Improved reporting of errors triggered by sqlite3. In
121 particular, all error messages are now tagged with their corresponding
122 database filename and, if they are API-level errors, the name of the
123 sqlite3 function that caused them.
125 * Issue #144: Improved documentation on the support for custom properties
126 in the test metadata.
128 * Converted the `INSTALL`, `NEWS`, and `README` distribution documents to
129 Markdown for better formatting online.
132 Changes in version 0.11
133 -----------------------
135 **Released on October 23rd, 2014.**
137 * Added support to print the details of all test cases (metadata and
138 their output) to `report`. This is via a new `--verbose` flag which
139 replaces the previous `--show-context`.
141 * Added support to specify the amount of physical disk space required
142 by a test case. This is in the form of a new `required_disk_space`
143 metadata property, which can also be provided by ATF test cases as
146 * Assimilated the contents of all the `kyua-*-tester(1)` and
147 `kyua-*-interface(7)` manual pages into more relevant places. In
148 particular, added more details on test program registration and their
149 metadata to `kyuafile(5)`, and added `kyua-test-isolation(7)`
150 describing the isolation features of the test execution.
152 * Assimilated the contents of all auxiliary manual pages, including
153 `kyua-build-root(7)`, `kyua-results-files(7)`, `kyua-test-filters(7)`
154 and `kyua-test-isolation(7)`, into the relevant command-specific
155 manual pages. This is for easier discoverability of relevant
156 information when reading how specific Kyua commands work.
158 * Issue #30: Plumbed through support to query configuration variables
159 from ATF's test case heads. This resolves the confusing situation
160 where test cases could only do this from their body and cleanup
163 * Issue #49: Extended `report` to support test case filters as
164 command-line arguments. Combined with `--verbose`, this allows
165 inspecting the details of a test case failure after execution.
167 * Issue #55: Deprecated support for specifying `test_suite` overrides on
168 a test program basis. This idiom should not be used but support for
171 * Issue #72: Added caching support to the `getcwd(3)` test in configure
172 so that the result can be overriden for cross-compilation purposes.
174 * Issue #83: Changed manual page headings to include a `kyua` prefix in
175 their name. This prevents some possible confusion when displaying,
176 for example, the `kyua-test` manual page with a plain name of `test`.
178 * Issue #84: Started passing test-suite configuration variables to plain
179 and TAP test programs via the environment. The name of the
180 environment variables set this way is prefixed by `TEST_ENV_`, so a
181 configuration variable of the form
182 `test_suites.some_name.allow_unsafe_ops=yes` in `kyua.conf` becomes
183 `TEST_ENV_allow_unsafe_ops=YES` in the environment.
185 * Issues #97 and #116: Fixed the build on Illumos.
187 * Issue #102: Set `TMPDIR` to the test case's work directory when running
188 the test case. If the test case happens to use the `mktemp(3)` family
189 of functions (due to misunderstandings on how Kyua works or due to
190 the reuse of legacy test code), we don't want it to easily escape the
191 automanaged work directory.
193 * Issue #103: Started being more liberal in the parsing of TAP test
194 results by treating the number in `ok` and `not ok` lines as optional.
196 * Issue #105: Started using tmpfs instead of md as a temporary file
197 system for tests in FreeBSD so that we do not leak `md(4)` devices.
199 * Issue #109: Changed the privilege dropping code to start properly
200 dropping group privileges when `unprivileged_user` is set. Also fixes
201 `testers/run_test:fork_wait__unprivileged_group`.
203 * Issue #110: Changed `help` to display version information and clarified
204 the purpose of the `about` command in its documentation.
206 * Issue #111: Fixed crash when defining a test program in a `Kyuafile`
207 that has not yet specified the test suite name.
209 * Issue #114: Improved the `kyuafile(5)` manual page by clarifying the
210 restrictions of the `include()` directive and by adding abundant
214 Changes in version 0.10
215 -----------------------
217 **Experimental version released on August 14th, 2014.**
219 * Merged `kyua-cli` and `kyua-testers` into a single `kyua` package.
221 * Dropped the `kyua-atf-compat` package.
223 * Issue #100: Do not try to drop privileges to `unprivileged_user` when we
224 are already running as an unprivileged user. Doing so is not possible
225 and thus causes spurious test failures when the current user is not
226 root and the current user and `unprivileged_user` do not match.
228 * Issue #79: Mention `kyua.conf(5)` in the *See also* section of `kyua(1)`.
230 * Issue #75: Change the `rewrite__expected_signal__bad_arg` test in
231 `testers/atf_result_test` to use a different signal value. This is to
232 prevent triggering a core dump that made the test fail in some platforms.
235 Changes in kyua-cli version 0.9
236 -------------------------------
238 **Experimental version released on August 8th, 2014.**
242 The internal architecture of Kyua to record the results of test suite
243 runs has completely changed in this release. Kyua no longer stores all
244 the different test suite run results as different "actions" within the
245 single `store.db` database. Instead, Kyua now generates a separate
246 results file inside `~/.kyua/store/` for every test suite run.
248 Due to the complexity involved in the migration process and the little
249 need for it, this is probably going to be the only release where the
250 `db-migrate` command is able to convert an old `store.db` file to the
253 Changes in more detail:
255 * Added the `report-junit` command to generate JUnit XML result files.
256 The output has been verified to work within Jenkins.
258 * Switched to results files specific to their corresponding test suite
259 run. The unified `store.db` file is now gone: `kyua test` creates a
260 new results file for every invocation under `~/.kyua/store/` and the
261 `kyua report*` commands are able to locate the latest file for a
262 corresponding test suite automatically.
264 * The `db-migrate` command takes an old `store.db` file and generates
265 one results file for every previously-recorded action, later deleting
268 * The `--action` flag has been removed from all commands that accepted
269 it. This has been superseded by the tests results files.
271 * The `--store` flag that many commands took has been renamed to
272 `--results-file` in line with the semantical changes.
274 * The `db-exec` command no longer creates an empty database when none
275 is found. This command is now intended to run only over existing
279 Changes in kyua-testers version 0.3
280 -----------------------------------
282 **Experimental version released on August 8th, 2014.**
284 * Made the testers set a "sanitized" value for the `HOME` environment
285 variable where, for example, consecutive and trailing slashes have
286 been cleared. Mac OS X has a tendency to append a trailing slash to
287 the value of `TMPDIR`, which can cause third-party tests to fail if
288 they compare `${HOME}` with `$(pwd)`.
290 * Issues #85, #86, #90 and #92: Made the TAP parser more complete: mark
291 test cases reported as `TODO` or `SKIP` as passed; handle skip plans;
292 ignore lines that look like `ok` and `not ok` but aren't results; and
293 handle test programs that report a pass but exit with a non-zero code.
296 Changes in kyua-cli version 0.8
297 -------------------------------
299 **Experimental version released on December 7th, 2013.**
301 * Added support for Lutok 0.4.
303 * Issue #24: Plug the bootstrap tests back into the test suite. Fixes
304 in `kyua-testers` 0.2 to isolate test cases into their own sessions
305 should allow these to run fine.
307 * Issue #74: Changed the `kyuafile(5)` parser to automatically discover
308 existing tester interfaces. The various `*_test_program()` functions
309 will now exist (or not) based on tester availability, which simplifies
310 the addition of new testers or the selective installation of them.
313 Changes in kyua-testers version 0.2
314 -----------------------------------
316 **Experimental version released on December 7th, 2013.**
318 * Issue #74: Added the `kyua-tap-tester`, a new backend to interact with
319 test programs that comply with the Test Anything Protocol.
321 * Issue #69: Cope with the lack of `AM_PROG_AR` in `configure.ac`, which
322 first appeared in Automake 1.11.2. Fixes a problem in Ubuntu 10.04
323 LTS, which appears stuck in 1.11.1.
325 * Issue #24: Improve test case isolation by confining the tests to their
326 own session instead of just to their own process group.
329 Changes in kyua-cli version 0.7
330 -------------------------------
332 **Experimental version released on October 18th, 2013.**
334 * Made failures from testers more resilent. If a tester fails, the
335 corresponding test case will be marked as broken instead of causing
338 * Added the `--results-filter` option to the `report-html` command and
339 set its default value to skip passed results from HTML reports. This
340 is to keep these reports more succint and to avoid generating tons of
341 detail files that will be, in general, useless.
343 * Switched to use Lutok 0.3 to gain compatibility with Lua 5.2.
345 * Issue #69: Cope with the lack of `AM_PROG_AR` in `configure.ac`, which
346 first appeared in Automake 1.11.2. Fixes a problem in Ubuntu 10.04
347 LTS, which appears stuck in 1.11.1.
350 Changes in kyua-cli version 0.6
351 -------------------------------
353 **Experimental version released on February 22nd, 2013.**
355 * Issue #36: Changed `kyua help` to not fail when the configuration file
356 is bogus. Help should always work.
358 * Issue #37: Simplified the `syntax()` calls in configuration and
359 `Kyuafile` files to only specify the requested version instead of also
360 the format name. The format name is implied by the file being loaded, so
361 there is no use in the caller having to specify it. The version number
362 of these file formats has been bumped to 2.
364 * Issue #39: Added per-test-case metadata values to the HTML reports.
366 * Issue #40: Rewrote the documentation as manual pages and removed the
367 previous GNU Info document.
369 * Issue #47: Started using the independent testers in the `kyua-testers`
370 package to run the test cases. Kyua does not implement the logic to
371 invoke test cases any more, which provides for better modularity,
372 extensibility and robustness.
374 * Issue #57: Added support to specify arbitrary metadata properties for
375 test programs right from the `Kyuafile`. This is to make plain test
376 programs more versatile, by allowing them to specify any of the
377 requirements (allowed architectures, required files, etc.) supported
380 * Reduced automatic screen line wrapping of messages to the `help`
381 command and the output of tables by `db-exec`. Wrapping any other
382 messages (specially anything going to stderr) was very annoying
383 because it prevented natural copy/pasting of text.
385 * Increased the granularity of the error codes returned by `kyua(1)` to
386 denote different error conditions. This avoids the overload of `1` to
387 indicate both "expected" errors from specific subcommands and
388 unexpected errors caused by the internals of the code. The manual now
389 correctly explain how the exit codes behave on a command basis.
391 * Optimized the database schema to make report generation almost
394 * Bumped the database schema to 2. The database now records the
395 metadata of both test programs and test cases generically, without
396 knowledge of their interface.
398 * Added the `db-migrate` command to provide a mechanism to upgrade a
399 database with an old schema to the current schema.
401 * Removed the GDB build-time configuration variable. This is now part
402 of the `kyua-testers` package.
404 * Issue #31: Rewrote the `Kyuafile` parsing code in C++, which results in
405 a much simpler implementation. As a side-effect, this gets rid of the
406 external Lua files required by `kyua`, which in turn make the tool
409 * Added caching of various configure test results (particularly in those
410 tests that need to execute a test program) so that cross-compilers can
411 predefine the results of the tests without having to run the
415 Changes in kyua-testers version 0.1
416 -----------------------------------
418 **Experimental version released on February 19th, 2013.**
420 This is the first public release of the `kyua-testers` package.
422 The goal of this first release is to adopt all the test case execution
423 code of `kyua-cli` 0.5 and ship it as a collection of independent tester
424 binaries. The `kyua-cli` package will rely on these binaries to run the
425 tests, which provides better modularity and simplicity to the
426 architecture of Kyua.
428 The code in this package is all C as opposed to the current C++ codebase
429 of `kyua-cli`, which means that the overall build times of Kyua are now
433 Changes in kyua-cli version 0.5
434 -------------------------------
436 **Experimental version released on July 10th, 2012.**
438 * Issue #15: Added automatic stacktrace gathering of crashing test cases.
439 This relies on GDB and is a best-effort operation.
441 * Issue #32: Added the `--build-root` option to the debug, list and test
442 commands. This allows executing test programs from a different
443 directory than where the `Kyuafile` scripts live. See the *Build roots*
444 section in the manual for more details.
446 * Issue #33: Removed the `kyuaify.sh` script. This has been renamed to
447 atf2kyua and moved to the `kyua-atf-compat` module, where it ships as a
448 first-class utility (with a manual page and tests).
450 * Issue #34: Changed the HTML reports to include the stdout and stderr of
453 * Fixed the build when using a "build directory" and a clean source tree
457 Changes in kyua-cli version 0.4
458 -------------------------------
460 **Experimental version released on June 6th, 2012.**
462 * Added the `report-html` command to generate HTML reports of the
463 execution of any recorded action.
465 * Changed the `--output` flag of the `report` command to only take a
466 path to the target file, not its format. Different formats are better
467 supported by implementing different subcommands, as the options they
468 may receive will vary from format to format.
470 * Added a `--with-atf` flag to the configure script to control whether
471 the ATF tests get built or not. May be useful for packaging systems
472 that do not have ATF in them yet. Disabling ATF also cuts down the
473 build time of Kyua significantly, but with the obvious drawbacks.
475 * Grouped `kyua` subcommands by topic both in the output of `help` and
476 in the documentation. In general, the user needs to be aware of
477 commands that rely on a current project and those commands that rely
478 purely on the database to generate reports.
480 * Made `help` print the descriptions of options and commands properly
483 * Changed most informational messages to automatically wrap on screen
486 * Rewrote the configuration file parsing module for extensibility. This
487 will allow future versions of Kyua to provide additional user-facing
488 options in the configuration file.
490 No syntax changes have been made, so existing configuration files
491 (version 1) will continue to be parsed without problems. There is one
492 little exception though: all variables under the top-level
493 `test_suites` tree must be declared as strings.
495 Similarly, the `-v` and `--variable` flags to the command line must
496 now carry a `test_suites.` prefix when referencing any variables under
500 Changes in kyua-cli version 0.3
501 -------------------------------
503 **Experimental version released on February 24th, 2012.**
505 * Made the `test` command record the results of the executed test
506 cases into a SQLite database. As a side effect, `test` now supports a
507 `--store` option to indicate where the database lives.
509 * Added the `report` command to generate plain-text reports of the
510 test results stored in the database. The interface of this command is
511 certainly subject to change at this point.
513 * Added the `db-exec` command to directly interact with the store
516 * Issue #28: Added support for the `require.memory` test case property
517 introduced in ATF 0.15.
519 * Renamed the user-specific configuration file from `~/.kyuarc` to
520 `~/.kyua/kyua.conf` for consistency with other files stored in the
521 `~/.kyua/` subdirectory.
523 * Switched to use Lutok instead of our own wrappers over the Lua C
524 library. Lutok is just what used to be our own utils::lua module, but
525 is now distributed separately.
527 * Removed the `Atffile`s from the source tree. Kyua is stable enough
528 to generate trustworthy reports, and we do not want to give the
529 impression that atf-run / atf-report are still supported.
531 * Enabled logging to stderr for our own test programs. This makes it
532 slightly easier to debug problems in our own code when we get a
536 Changes in kyua-cli version 0.2
537 -------------------------------
539 **Experimental version released on August 24th, 2011.**
541 The biggest change in this release is the ability for Kyua to run test
542 programs implemented using different frameworks. What this means is
543 that, now, a Kyua test suite can include not only ATF-based test
544 programs, but also "legacy" (aka plain) test programs that do not use
545 any framework. I.e. if you have tests that are simple programs that
546 exit with 0 on success and 1 on failure, you can plug them in into a
549 Other than this, there have been several user-visible changes. The most
550 important are the addition of the new `config` and `debug` subcommands
551 to the `kyua` binary. The former can be used to inspect the runtime
552 configuration of Kyua after parsing, and the latter is useful to
553 interact with failing tests cases in order to get more data about the
556 Without further ado, here comes the itemized list of changes:
558 * Generalized the run-time engine to support executing test programs
559 that implement different interfaces. Test programs that use the ATF
560 libraries are just a special case of this. (Issue #18.)
562 * Added support to the engine to run `plain` test programs: i.e. test
563 programs that do not use any framework and report their pass/fail
564 status as an exit code. This is to simplify the integration of legacy
565 test programs into a test suite, and also to demonstrate that the
566 run-time engine is generic enough to support different test
567 interfaces. (Issue #18.)
569 * Added the `debug` subcommand. This command allows end users to tweak
570 the execution of a specific test case and to poke into the behavior of
571 its execution. At the moment, all this command allows is to view the
572 stdout and stderr of the command in real time (which the `test`
573 command currently completely hides).
575 * Added the `config` subcommand. This command allows the end user to
576 inspect the current configuration variables after evaluation, without
577 having to read through configuration files. (Issue #11.)
579 * Removed the `test_suites_var` function from configuration files. This
580 was used to set the value of test-suite-sepecific variables, but it
581 was ugly-looking. It is now possible to use the more natural syntax
582 `test_suites.<test-suite-name>.<variable> = <value>`. (Issue #11.)
584 * Added a mechanism to disable the loading of configuration files
585 altogether. Needed for testing purposes and for scriptability.
586 Available by passing the `--config=none` flag.
588 * Enabled detection of unused parameters and variables in the code and
589 fixed all warnings. (Issue #23.)
591 * Changed the behavior of "developer mode". Compiler warnings are now
592 enabled unconditionally regardless of whether we are in developer mode
593 or not; developer mode is now only used to perform strict warning
594 checks and to enable assertions. Additionally, developer mode is now
595 only automatically enabled when building from the repository, not for
596 formal releases. (Issue #22.)
598 * Fixed many build and portability problems to Debian sid with GCC 4.6.3
599 and Ubuntu 10.04.1 LTS. (Issues #20, #21, #26.)
602 Changes in kyua-cli version 0.1
603 -------------------------------
605 **Experimental version released on June 23rd, 2011.**
607 This is the first public release of the `kyua-cli` package.
609 The scope of this release is to provide functional replacement for the
610 `atf-run` utility included in the atf package. At this point, `kyua`
611 can reliably run the NetBSD 5.99.53 test suite delivering the same
612 results as `atf-run`.
614 The reporting facilities of this release are quite limited. There is
615 no replacement for `atf-report` yet, and there is no easy way of
616 debugging failing test programs other than running them by hand. These
617 features will mark future milestones and therefore be part of other
620 Be aware that this release has suffered very limited field testing.
621 The test suite for `kyua-cli` is quite comprehensive, but some bugs may
622 be left in any place.