PG,OSD: delay ops for map prior to queueing in the OpWQ

Previously, we simply queued ops in the OpWQ without checking.  The PG
would then check in do_request whether the message should wait for a new
map.  Unfortunately, this has the side effect that any op requeued for
any reason must also requeue the waiting_for_map queue.

Now, we will check before queueing the op whether it must wait on a map.
To avoid contention, there is now a map_lock which must be held along
with the PG lock in order to update the osdmap_ref.  The map_lock also
protects the waiting_for_map list and queueing PG ops at the back of
the OpWQ.  A few details:
1) It is no longer necessary to requeue waiting_for_map in on_change()
since the other ops are queued at the front.
2) Once waiting_for_map is non-empty, all ops are delayed to simplify
ordering.
3) waiting_for_map may now be non-empty during split, so we must split
waiting_for_map along with waiting_for_active.  This must be done
under the map_lock.

The bug which uncovered this involved an out of order op as follows:

client.4208.0:2378 (e252) arrives, object is degraded
client.4208.0:2379 (e253) arrives, waits for map
client.4208.0:2378 (e252) is requeued after recovery
client.4208.0:2379 (e253) is requeued on map arrival
client.4208.0:2379 is processed
client.4208.0:2378 is processed

Fixes: #4955
Signed-off-by: Samuel Just <sam.just@inktank.com>

1 files changed, 1 insertions, 1 deletions

diff --git a/src/osd/OSD.cc b/src/osd/OSD.cc
index f9f2b46a221..26dacf577af 100644
--- a/src/osd/OSD.cc
+++ b/src/osd/OSD.cc
@@ -6416,7 +6416,7 @@ void OSD::enqueue_op(PG *pg, OpRequestRef op)
 	   << " cost " << op->request->get_cost()
 	   << " latency " << latency
 	   << " " << *(op->request) << dendl;
-  op_wq.queue(make_pair(PGRef(pg), op));
+  pg->queue_op(op);
 }
 
 void OSD::OpWQ::_enqueue(pair<PGRef, OpRequestRef> item)