[nfsv4] stable storage for server restart

rick@snowhite.cis.uoguelph.ca

From: rick@snowhite.cis.uoguelph.ca
Date: Sat, 19 Feb 2005 18:49:16 -0500 (EST)
Message-Id: <200502192349.SAA64363@snowhite.cis.uoguelph.ca>
Subject: [nfsv4] stable storage for server restart

Here are a few recollections w.r.t. stable storage for server restart, from
when I did it for my BSD server, in case it is of interest.
(I cc'd nfsv4@ietf.org, just in case anyone not on the linux list is
interested. Apologies in advance for cluttering up your email. Also, I'd be
interested in hearing any thoughts others have on the design.)

Here is a brief description of my design:
- I used a single file written as an append log as follows:
  - at server startup (before any Compounds are performed), the log is
    read and an in-memory structure is created, indicating what clients
    can reclaim state
  - during reclaim, the in-memory structure is used to check for Grace and
    is marked for successful reclaims, per client
* - at the end of the grace period, the file is truncated to 0 length and
    a new append log is written from the in-memory structure, with one record
    for each client that successfully reclaimed some state
  - then normal, non-grace operation starts...
    - records are appended to the log when a client acquires the first state
      (first Open) after a SetClientID and when state is revoked for a
      client (I do not support revocation of only some state for a client)
      (nb: The first Open records are only done for clients that didn't
       successfully reclaim during grace.)

Notes:
- I had to lock the other nfsd threads out when updating stable storage. The
  reason was:
  - I needed to record the revocation before issuing conflicting lock state,
    and I wanted to avoid races between multiple clients trying to acquire
    conflicting locks while the write(s) to disk were in progress. Since
    revocation is a rare event, I didn't see this as a serious performance
    hit.
  - For the case of first Open, the record indicates successful lock
    state acquisition. If another client acquires a conflicting lock
    while the disk write(s) for the log are in progress, there would be
    a record indicating that the client had successfully acquired state
    although the lock failed, due to a conflict. Is this actually a
    problem? I'm not convinced it is, but my code "plays it safe" for now.
    I could see this being a significant performance hit, if lots of new
    clients did SetClientIDs followed by Opens at the same time. (Ones
    that haven't already reclaimed locks at server restart.)
  - The other two cases (when server first starts up and at end-of-grace)
    only occur once per server reboot and only add a little time to the
    grace period.

I think the weakest part of this design is that, if the server crashes
again while at "*", the append only log is not complete (possibly empty).
This will result in clients not being allowed to reclaim, that otherwise
should be able to (ie. no entry->no reclaim->NFS4ERR_NOGRACE for all reclaim
requests).

The append log will grow, but I only see a problem if clients go hogwild
with SetClientIDs. It does get truncated when the server restarts, so a
sysadmin can just reboot when it gets too big:-)

It also doesn't support the notion of only some state for a client being
revoked. (I've looked at that one a bit and it seems to get quite
challenging. Maybe someday I'll come up with a simple scheme I'm convinced
works for that case.)

rick

_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4