We have a Sequent S81 which nfs mounts /usr/spool/mail from a much
speedier Sun 670.  Under even moderate loads on the Sequent, we get
nfs server not responding timeouts, and we strongly suspect that the
Sequent is losing response packets from the Sun.

We have tried some of the standard network performance tests.  For
example, spray from the fast server to the slow Sequent produces
the following:

   /usr/etc/spray uicsl
   sending 1162 packets of lnth 86 to uicsl ...
           in 10.5 seconds elapsed time,
           635 packets (54.65%) dropped
   Sent:   110 packets/sec, 9.3K bytes/sec
   Rcvd:   50 packets/sec, 4.2K bytes/sec

By adding a 1 us delay between packets, we can virtually eliminate
lost packets:

   <eagle.csl.uiuc.edu> 6: /usr/etc/spray -d 1 uicsl
   sending 1162 packets of lnth 86 to uicsl ...
           in 11.7 seconds elapsed time,
           1 packets (0.09%) dropped
   Sent:   99 packets/sec, 8.4K bytes/sec
   Rcvd:   99 packets/sec, 8.4K bytes/sec

This indicates to us that the Sequent cannot deal with the bandwidth
the Sun can generate.

The very useful NFS and NIS book from O'Reilly gives helpful advice
about how to deal with the combination of slow server/fast client,
but doesn't have anything to say about our situation.  Can anyone
offer suggestions of how we can make our Sun server more considerate
of it's slower client.  (Unfortunately, not mounting /usr/spool/mail
is not an option, because the users are already {*filter*}ed.)

--
*  Jean Ezell                 *  University of Illinois   *
*  ez...@eagle.csl.uiuc.edu   *  Coordinated Science Lab  *
*  voice:  (217) 244-5536     *  1308 W. Main St.         *
*    fax:  (217) 244-1642     *  Urbana, IL  61801        *

I suggest two things.  

First, in the client's mount, set the "rsize" (read request size)
option to 1 KB from the default of 8 KB.  (See mount(8) for details on
this.)  This should help because the most closely packed packets from
the server are probably from NFS read requests where the Sun takes one
8 KB NFS request and splits it into six 1.5 KB IP packets which it
pumps onto the net in quick succession.  With 1 KB reads, each NFS
request fits into one IP packet, and the Sun will tend not to produce
them so quickly because it take more time to process a full NFS request
than to fragment one request into multiple IP packets.

This also helps because if a 1 KB read packet is lost, only that one
packet needs to be resent.  If one of the six 1.5 KB packets from an 8
KB read are lost, all six packets need to be resent.  Thus, loosing one
out of twelve read requests will tend to double read reply traffic with
8 KB reads, but add less than 10% to read reply traffic with 1 KB
reads.

Second, if things are still slow with the first fix, try reducing the
"timeo" (NFS timeout) option to .1 seconds from the default of .7.
Using a longer timeout prevents the client from sending retries too
soon in case the server really has the packet but just hasn't gotten
around to processing it.  But in your case, packets really *are*
getting lost, so you might as well retry them as soon as possible.
Since you've got a fast server, chances are that it really is
completing most operations within .1 second.  (Be careful with this,
because if the server gets slow, or if the client is on the other side
of a slow router, then this could add a lot of bogus traffic to your
net.)

Dave Hitz                                       h...@netapp.com
Network Appliance Corporation                   (408) 562-1906

   We have a Sequent S81 which nfs mounts /usr/spool/mail from a much
   speedier Sun 670.  Under even moderate loads on the Sequent, we get
   nfs server not responding timeouts, and we strongly suspect that the
   Sequent is losing response packets from the Sun.

Indeed. The problem is the ethernet controller on the S81's SCED card.
It cannot handle packet trains - a handful of packets back to back on
the wire. Check the number of receiver overflows - packets for the S81
seen by the SCED but dropped - reported by sestat.

Your Sun will reply to an NFS read request from the S81 by sending an
8K (say) block as 5 or 6 ethernet packets. These packets go out on the
ethernet one after the other, separated by the minimum inter-packet
delay given in the Blue Book. The Sequent drops one of these packets
now and again: its controller can't keep up with this rate of traffic.
The S81 kernel tries to re-assemble the NFS reply, finds a packet
missing and waits for a while in case it turns up. After a while, it
discards the reply - what the hell, it was a datagram anyway - and
sends the NFS read request again. When the Sun sees this, it fires off
the same 8K block as 5 or 6 packets and the cycle repeats itself.

   The very useful NFS and NIS book from O'Reilly gives helpful advice
   about how to deal with the combination of slow server/fast client,
   but doesn't have anything to say about our situation.  Can anyone
   offer suggestions of how we can make our Sun server more considerate
   of it's slower client.

Experiment with some of the NFS mount options such as rsize and wsize
(the NFS read and write block sizes) and timeo (NFS timeout) and retrans
(retransmission count). The default values for these will probably
have been tuned for a Sun with a decent ethernet controller. Dropping
the read and write block size to 4K for this NFS mounted filesystem
might do the trick. There are no hard and fast rules: you'll need to
experiment to see what works best in your environment.

   (Unfortunately, not mounting /usr/spool/mail
   is not an option, because the users are already {*filter*}ed.)

This should be an option: your argument is not a good excuse. NFS
mounting the mailbox partition is a Very Bad Idea. First of all, you
may run into protection problems because user and delivery agents may
be running as root when accessing a mailbox. NFS requests made as root
normally gets converted to the null UID nobody. The server will fail
the request which makes the user or delivery agent break. Secondly,
mailbox locking is a Big Problem. It is all too easy for two processes
running on different hosts to simultaneously access an NFS mounted
mailbox and trash it. Consider someone writing back their mailbox as
the mail system is delivering mail to it....

The lock protocols used by UNIX mail systems can easily get upset by
NFS. Creating a lockfile or renaming the mailbox can be fooled because
of NFS directory cacheing. There is a window when NFS clients and
their servers can see a different picture of the same directory. Thus
a mail program may not see a lock file which really does exist.... The
other option is to use the NFS lock daemon to "lock" the mailbox, but
this is not exactly a reliable piece of software.

What you can do is arrange remote access to mailboxes using the Post
Office Protocol (POP). This allows users to read and send mail without
having the mailbox located on the host they are using (or NFS
mounted). Some user agents like MH can be configured to support POP
transparently: the users don't even need to know where their mailbox
lives or that POP is being used.

Hope this helps

                Jim

Somebody says:

To which Jim replies:

Hmm.. I find this curious -- since I've been here, most parts of campus
have NFS mounted the mail directory from a central location.  First of
all, root email is read only from the server (and trusted hosts - no
big deal), and locking does not appear to be a problem whatsoever... thousands
of mailboxes, and nobody's gets munched.  Most programs are pretty intelligent
about this sort of thing.