5987 zfs prefetch code needs work

5987 zfs prefetch code needs work

illumos/illumos-gate@cf6106c8a0d6598b045811f9650d66e07eb332af

https://www.illumos.org/issues/5987
  The existing ZFS prefetch code (dmu_zfetch.c) has some problems:
  1. It's nearly impossible to understand. e.g. there are an abundance of kstats
  but it's hard to know what they mean (see below).
  2. For some workloads, it detects patterns that aren't really there (e.g.
  strided patterns, backwards scans), and generates needless i/os prefetching
  blocks that will never be referenced.
  3. It has lock contention issues. These are caused primarily by
  dmu_zfetch_colinear() calling dmu_zfetch_dofetch() (which can block waiting for
  i/o) with the zf_rwlock held for writer, thus blocking all other threads
  accessing this file.
  I suggest that we rewrite this code to detect only forward, sequential streams.
  [... truncated ...]

Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Approved by: Gordon Ross <gordon.ross@nexenta.com>
Author: Matthew Ahrens <mahrens@delphix.com>