diff options
Diffstat (limited to 'TODO')
| -rw-r--r-- | TODO | 184 |
1 files changed, 184 insertions, 0 deletions
@@ -0,0 +1,184 @@ +Things to do Automated Testing Framework +=========================================================================== + + +Last revised: November 30th, 2010 + + +This document includes the list of things that need to be done in ATF that +are most requested by the users. This information used to be available in +an ad-hoc bug tracker but that proved to be a bad idea. I have collected +all worthy comments in here. + +Please note that most work these days is going into Kyua (see +http://code.google.com/p/kyua/). The ideas listed here apply to the +components of ATF that have *not* been migrated to the new codebase yet. +For bug reports or ideas that apply to the components that already have +been migrated, please use the bug tracker in the URL above. Similarly, +whenever a component is migrated, the ideas in this file should be revised +and migrated to the new bug tracker where appropriate. + + +--------------------------------------------------------------------------- +Add build-time checks to atf-sh + +The 0.7 release introduced build-time tests to atf-c and atf-c++, but not +to atf-sh. Expose the functionality to the shell interface. + +This will probably require writing an atf-build utility that exposes the C +code and can be called from the shell. + +--------------------------------------------------------------------------- +Revisit what to do when an Atffile lists a non-existent file + +--------------------------------------------------------------------------- +Add ATF_CHECK* versions to atf-c++ to support non-fatal tests + +--------------------------------------------------------------------------- +Implement race-condition tests + +gcooper: + +I would think that stress/negative tests would be of more value than race +condition tests (they're similar, but not exactly the same in my mind). + +In particular, + +1. Feed through as much data as possible to determine where reporting + breaks down. +2. Feed through data quickly and terminate ASAP. The data should be + captured. Terminate child applications with unexpected exit codes and + signals (in particular, SIGCHLD, SIGPIPE, exit codes that terminate, + etc). +3. Open up a file descriptor in the test application, don't close the file + descriptor. +4. fork(2) a process; don't wait(2) for the application to complete. + +There are other scenarios that could be exercised, but these are the ones +I could think of off the topic of my head. + +-- + +jmmv: + +1. The thing is: how do you express any of this in a portable/abstract + interface? How do you express that a test case "receives data"? What + does that exactly mean? I don't think the framework should care about + this: each test should be free to decide where its data is and how to + deal with it. + +2. Ditto. + +3. Not sure I understand your request, but testing for "unexpected exit + codes" is already supported. See wiki:DesignXFail for the feature + design details. + +4. What's the problem with this case? The test case exits right away after + terminating the execution of its body; any open file descriptors, + leaked memory, etc. die with it. + +5. forking and not waiting for a subprocess was a problem already + addressed. + +I kinda have an idea of what Antti means with "race condition tests", but +every time I have tried to describe my understanding of matters I seem to +be wrong. Would be nice to have a clear description of what this involves; +in particular, what are the expectations from the framework and how should +the feature be exposed. + +As of now, what I understand by "race condition test" is: a test case that +exercises a race condition. The test case may finish without triggering +the race, in which case it just exists with a successful status. +Otherwise, if the race triggers, the test case gets stuck and times out. +The result should be reported as an "expected failure" different from +timeout. + +-- + +pooka: + +Yup. Plus some atf-wide mechanism for the operator to supply some kind of +guideline if the test should try to trigger the race for a second or for +an hour. + +-- + +jmmv: + +Alright. While mocking up some code for this, I think that your two +requests are complementary. + +On the one hand, when you are talking about a "race condition" test you +really mean an "expected race condition" test. Correct? If so, we need to +extend the xfail mechanism to add one more case, which is to report any +failures as a race condition error and, if there is no failure, report the +test as successful. + +On the other hand, the atf-wide mechanism to support how long the test +should run for can be thought as a "stress test" mechanism. I.e. run this +test for X time / iterations and report its results regularly without +involving xfail at all. + +So, with this in mind: + +* For a test that triggers an unfixed race condition, you set xfail to + race mode and define the test as a stress test. Any failures are + reported as expected failures. + +* For a test that verifies a supposedly-fixed race condition, you do *not* + set xfail to race mode, and only set the test to stress test. Any + failures are reported as real failures. + +These stress test cases implement a single iteration of the test and +atf-run is in charge of running the test several times, stopping on the +first failure. + +Does that make sense? + +--------------------------------------------------------------------------- +Implement ATF_REQUIRE_ERRNO + +pooka: + +Most of the lines in tests against system functionality are: + +if (syscall(args) == -1) + atf_tc_fail_errno("flop") + +Some shorthand would be helpful, like ATF_REQUIRE_ERRNO(syscall(args)) +Also, a variant which allows arbitrary return value checks (e.g. "!= 0" or +"< 124" or "!= size") would be nice. + +-- + +gcooper: + +There's a problem with this request; not all functions fail in the same +way ... in particular compare the pthread family of functions (which +return errno) vs many native syscalls. Furthermore, compare some +fcntl-like syscalls vs other syscalls. One size fits all solutions may not +be a wise idea in this case, so I think that the problem statement needs +to be better defined, because the above request is too loose. + +FWIW, there's also a TEST macro in LTP, which tests for non-zero status, +and sets an appropriate set of global variables for errnos and return +codes, respectively. It was a good idea, but has been mostly abandoned +because it's too difficult to define a success and failure in a universal +manner, so I think that we need to be careful with what's implemented in +ATF to not repeat the mistakes that others have made. + +-- + +jmmv: + +I think you've got a good point. + +This was mostly intended to simplify the handling of the stupid errno +global variable. I think this is valuable to have, but maybe the +macro/function name should be different because _ERRNO can be confusing. +Probably something like an ATF_CHECK_LIBC / ATF_CHECK_PTHREAD approach +would be more flexible and simple. + + +=========================================================================== +vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2 |