Identity plugin thread safety

A common case is for Nova (or other service) to create a service
authentication plugin from a configuration file and then have many
greenlet threads that want to reuse that authentication. If a token
expires then many threads all try and fetch a new token to use and can
step over each other.

I was hoping for a way to put a lock in so that all plugins were thread
safe however fixing it for identity plugins solves almost all real world
situations and anyone doing non-identity plugins will have to manage
threads themselves.

Change-Id: Ib6487de7de638abc69660c851bd048a8ec177109
Closes-Bug: #1493835

2 files changed, 17 insertions, 2 deletions

diff --git a/doc/source/authentication-plugins.rst b/doc/source/authentication-plugins.rst
index 5afc0bc..89384a5 100644
--- a/doc/source/authentication-plugins.rst
+++ b/doc/source/authentication-plugins.rst
@@ -235,3 +235,10 @@ can be retrieved.
 
 The most simple example of a plugin is the
 :py:class:`keystoneclient.auth.token_endpoint.Token` plugin.
+
+When writing a plugin you should ensure that any fetch operation is thread
+safe. A common pattern is for a service to hold a single service authentication
+plugin globally and re-use that between all threads. This means that when a
+token expires there may be multiple threads that all try to fetch a new plugin
+at the same time. It is the responsibility of the plugin to ensure that this
+case is handled in a way that still results in correct reauthentication.
diff --git a/keystoneclient/auth/identity/base.py b/keystoneclient/auth/identity/base.py
index 57e1723..02c4fe6 100644
--- a/keystoneclient/auth/identity/base.py
+++ b/keystoneclient/auth/identity/base.py
@@ -12,6 +12,7 @@
 
 import abc
 import logging
+import threading
 import warnings
 
 from oslo_config import cfg
@@ -54,6 +55,7 @@ class BaseIdentityPlugin(base.BaseAuthPlugin):
         self.reauthenticate = reauthenticate
 
         self._endpoint_cache = {}
+        self._lock = threading.Lock()
 
         self._username = username
         self._password = password
@@ -236,8 +238,14 @@ class BaseIdentityPlugin(base.BaseAuthPlugin):
         :returns: Valid AccessInfo
         :rtype: :py:class:`keystoneclient.access.AccessInfo`
         """
-        if self._needs_reauthenticate():
-            self.auth_ref = self.get_auth_ref(session)
+        # Hey Kids! Thread safety is important particularly in the case where
+        # a service is creating an admin style plugin that will then proceed
+        # to make calls from many threads. As a token expires all the threads
+        # will try and fetch a new token at once, so we want to ensure that
+        # only one thread tries to actually fetch from keystone at once.
+        with self._lock:
+            if self._needs_reauthenticate():
+                self.auth_ref = self.get_auth_ref(session)
 
         return self.auth_ref