diff options
Diffstat (limited to 'contrib/atf/TODO')
| -rw-r--r-- | contrib/atf/TODO | 184 |
1 files changed, 0 insertions, 184 deletions
diff --git a/contrib/atf/TODO b/contrib/atf/TODO deleted file mode 100644 index 556baba6ec91..000000000000 --- a/contrib/atf/TODO +++ /dev/null @@ -1,184 +0,0 @@ -Things to do Automated Testing Framework -=========================================================================== - - -Last revised: November 30th, 2010 - - -This document includes the list of things that need to be done in ATF that -are most requested by the users. This information used to be available in -an ad-hoc bug tracker but that proved to be a bad idea. I have collected -all worthy comments in here. - -Please note that most work these days is going into Kyua (see -http://code.google.com/p/kyua/). The ideas listed here apply to the -components of ATF that have *not* been migrated to the new codebase yet. -For bug reports or ideas that apply to the components that already have -been migrated, please use the bug tracker in the URL above. Similarly, -whenever a component is migrated, the ideas in this file should be revised -and migrated to the new bug tracker where appropriate. - - ---------------------------------------------------------------------------- -Add build-time checks to atf-sh - -The 0.7 release introduced build-time tests to atf-c and atf-c++, but not -to atf-sh. Expose the functionality to the shell interface. - -This will probably require writing an atf-build utility that exposes the C -code and can be called from the shell. - ---------------------------------------------------------------------------- -Revisit what to do when an Atffile lists a non-existent file - ---------------------------------------------------------------------------- -Add ATF_CHECK* versions to atf-c++ to support non-fatal tests - ---------------------------------------------------------------------------- -Implement race-condition tests - -gcooper: - -I would think that stress/negative tests would be of more value than race -condition tests (they're similar, but not exactly the same in my mind). - -In particular, - -1. Feed through as much data as possible to determine where reporting - breaks down. -2. Feed through data quickly and terminate ASAP. The data should be - captured. Terminate child applications with unexpected exit codes and - signals (in particular, SIGCHLD, SIGPIPE, exit codes that terminate, - etc). -3. Open up a file descriptor in the test application, don't close the file - descriptor. -4. fork(2) a process; don't wait(2) for the application to complete. - -There are other scenarios that could be exercised, but these are the ones -I could think of off the topic of my head. - --- - -jmmv: - -1. The thing is: how do you express any of this in a portable/abstract - interface? How do you express that a test case "receives data"? What - does that exactly mean? I don't think the framework should care about - this: each test should be free to decide where its data is and how to - deal with it. - -2. Ditto. - -3. Not sure I understand your request, but testing for "unexpected exit - codes" is already supported. See wiki:DesignXFail for the feature - design details. - -4. What's the problem with this case? The test case exits right away after - terminating the execution of its body; any open file descriptors, - leaked memory, etc. die with it. - -5. forking and not waiting for a subprocess was a problem already - addressed. - -I kinda have an idea of what Antti means with "race condition tests", but -every time I have tried to describe my understanding of matters I seem to -be wrong. Would be nice to have a clear description of what this involves; -in particular, what are the expectations from the framework and how should -the feature be exposed. - -As of now, what I understand by "race condition test" is: a test case that -exercises a race condition. The test case may finish without triggering -the race, in which case it just exists with a successful status. -Otherwise, if the race triggers, the test case gets stuck and times out. -The result should be reported as an "expected failure" different from -timeout. - --- - -pooka: - -Yup. Plus some atf-wide mechanism for the operator to supply some kind of -guideline if the test should try to trigger the race for a second or for -an hour. - --- - -jmmv: - -Alright. While mocking up some code for this, I think that your two -requests are complementary. - -On the one hand, when you are talking about a "race condition" test you -really mean an "expected race condition" test. Correct? If so, we need to -extend the xfail mechanism to add one more case, which is to report any -failures as a race condition error and, if there is no failure, report the -test as successful. - -On the other hand, the atf-wide mechanism to support how long the test -should run for can be thought as a "stress test" mechanism. I.e. run this -test for X time / iterations and report its results regularly without -involving xfail at all. - -So, with this in mind: - -* For a test that triggers an unfixed race condition, you set xfail to - race mode and define the test as a stress test. Any failures are - reported as expected failures. - -* For a test that verifies a supposedly-fixed race condition, you do *not* - set xfail to race mode, and only set the test to stress test. Any - failures are reported as real failures. - -These stress test cases implement a single iteration of the test and -atf-run is in charge of running the test several times, stopping on the -first failure. - -Does that make sense? - ---------------------------------------------------------------------------- -Implement ATF_REQUIRE_ERRNO - -pooka: - -Most of the lines in tests against system functionality are: - -if (syscall(args) == -1) - atf_tc_fail_errno("flop") - -Some shorthand would be helpful, like ATF_REQUIRE_ERRNO(syscall(args)) -Also, a variant which allows arbitrary return value checks (e.g. "!= 0" or -"< 124" or "!= size") would be nice. - --- - -gcooper: - -There's a problem with this request; not all functions fail in the same -way ... in particular compare the pthread family of functions (which -return errno) vs many native syscalls. Furthermore, compare some -fcntl-like syscalls vs other syscalls. One size fits all solutions may not -be a wise idea in this case, so I think that the problem statement needs -to be better defined, because the above request is too loose. - -FWIW, there's also a TEST macro in LTP, which tests for non-zero status, -and sets an appropriate set of global variables for errnos and return -codes, respectively. It was a good idea, but has been mostly abandoned -because it's too difficult to define a success and failure in a universal -manner, so I think that we need to be careful with what's implemented in -ATF to not repeat the mistakes that others have made. - --- - -jmmv: - -I think you've got a good point. - -This was mostly intended to simplify the handling of the stupid errno -global variable. I think this is valuable to have, but maybe the -macro/function name should be different because _ERRNO can be confusing. -Probably something like an ATF_CHECK_LIBC / ATF_CHECK_PTHREAD approach -would be more flexible and simple. - - -=========================================================================== -vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2 |