usr.bin/ministat/README

   1
   2 A small tool to do the statistics legwork on benchmarks etc.
   3
   4 Prepare your data into two files, one number per line
   5 run
   6         ./ministat data_before data_after
   7
   8 and see what it says.
   9
  10 You need at least three data points in each data set, but the more
  11 you have the better your result generally gets.
  12
  13 Here are two typical outputs:
  14
  15 x _1
  16 + _2
  17 +--------------------------------------------------------------------------+
  18 |x            +    x+      x            x   x             +           ++   |
  19 |        |_________|______AM_______________|__A___________M_______________||
  20 +--------------------------------------------------------------------------+
  21     N           Min           Max        Median           Avg        Stddev
  22 x   5         36060         36138         36107       36105.6     31.165686
  23 +   5         36084         36187         36163       36142.6     49.952978
  24 No difference proven at 95.0% confidence
  25
  26 Here nothing can be concluded from the numbers.  It _may_ be possible to
  27 prove something if many more measurements are made, but with only five
  28 measurements, nothing is proven.
  29
  30
  31 x _1
  32 + _2
  33 +--------------------------------------------------------------------------+
  34 |                                                               +          |
  35 |                               x                               +         +|
  36 |x                    x         x          x                    +         +|
  37 |         |_______________A_____M_________|                   |_M___A____| |
  38 +--------------------------------------------------------------------------+
  39     N           Min           Max        Median           Avg        Stddev
  40 x   5         0.133         0.137         0.136        0.1354  0.0015165751
  41 +   5         0.139          0.14         0.139        0.1394 0.00054772256
  42 Difference at 95.0% confidence
  43         0.004 +/- 0.00166288
  44         2.95421% +/- 1.22812%
  45         (Student's t, pooled s = 0.00114018)
  46
  47 Here we have a clearcut difference, not very big, but clear and unambiguous.
  48
  49