diff options
| author | Ed Maste <emaste@FreeBSD.org> | 2018-02-21 15:54:23 +0000 |
|---|---|---|
| committer | Ed Maste <emaste@FreeBSD.org> | 2018-02-21 15:54:23 +0000 |
| commit | a9cf54b0c985f6749b4f055145fbb01254553a8b (patch) | |
| tree | 2e1e51c527039404cb0c523bfc622b7421c12c13 /usr.bin/ministat | |
| parent | cb4985fbf6fb5ec48566799853a5f9a46f2c5e02 (diff) | |
| download | src-a9cf54b0c985f6749b4f055145fbb01254553a8b.tar.gz src-a9cf54b0c985f6749b4f055145fbb01254553a8b.zip | |
ministat: disallow negative variance / nan Stddev
With all values identical it was possible for Var() to return a negative
value due to limited floating point precision, resulting in "nan"
reported as Stddev.
Variance cannot actually be negative, so just return 0. We can later
investigate alternate algorithms for calculating variance to reduce the
effect of catastrophic cancellation here.
Reported by: Arshan Khanifar <arshankhanifar_gmail.com>
Approved by: phk
Sponsored by: The FreeBSD Foundation
Notes
Notes:
svn path=/head/; revision=329723
Diffstat (limited to 'usr.bin/ministat')
| -rw-r--r-- | usr.bin/ministat/ministat.c | 6 |
1 files changed, 6 insertions, 0 deletions
diff --git a/usr.bin/ministat/ministat.c b/usr.bin/ministat/ministat.c index 625fb329e2b8..150593e5be95 100644 --- a/usr.bin/ministat/ministat.c +++ b/usr.bin/ministat/ministat.c @@ -208,6 +208,12 @@ static double Var(struct dataset *ds) { + /* + * Due to limited precision it is possible that sy^2/n > syy, + * but variance cannot actually be negative. + */ + if (ds->syy <= ds->sy * ds->sy / ds->n) + return (0); return (ds->syy - ds->sy * ds->sy / ds->n) / (ds->n - 1.0); } |