Documentation updates for 1.4

* major additions to 1.4 migration doc; removed additional
  verbosity regarding caching methodology and reorganized the
  doc to present itself more as a "what's changed" guide

* as we now have a path for asyncio, update that doc so that
  we aren't spreading obsolete information

* updates to the 2.0 migration guide with latest info, however
  this is still an architecture doc and not a migration guide
  yet, will need further rework.

* start really talking about 1.x vs. 2.0 style everywhere.  Querying
  is most of the docs so this is going to be a prominent
  theme, start getting it to fit in

* Add introductory documentation for ORM example sections as these
  are too sparse

* new documentation for do_orm_execute(), many separate sections,
  adding deprecation notes to before_compile() and similar

* new example suites to illustrate do_orm_execute(),
  with_loader_criteria()

* modernized horizontal sharding examples and added a separate
  example to distinguish between multiple databases and single
  database w/ multiple tables use case

* introducing DEEP ALCHEMY, will use zzzeeksphinx 1.1.6

* no name for the alchemist yet however the dragon's name
  is Flambé

Change-Id: Id6b5c03b1ce9ddb7b280f66792212a0ef0a1c541

12 files changed, 361 insertions, 29 deletions

diff --git a/doc/build/orm/events.rst b/doc/build/orm/events.rst
index ecf0cc65b..1db1137e0 100644
--- a/doc/build/orm/events.rst
+++ b/doc/build/orm/events.rst
@@ -10,36 +10,101 @@ For an introduction to the most commonly used ORM events, see the section
 at :ref:`event_toplevel`.  Non-ORM events such as those regarding connections
 and low-level statement execution are described in :ref:`core_event_toplevel`.
 
-.. _orm_attribute_events:
+Session Events
+--------------
 
-Attribute Events
-----------------
+The most basic event hooks are available at the level of the ORM
+:class:`_orm.Session` object.   The types of things that are intercepted
+here include:
+
+* **Persistence Operations** - the ORM flush process that sends changes to the
+  database can be extended using events that fire off at different parts of the
+  flush, to augment or modify the data being sent to the database or to allow
+  other things to happen when persistence occurs.   Read more about persistence
+  events at :ref:`session_persistence_events`.
 
-.. autoclass:: sqlalchemy.orm.events.AttributeEvents
+* **Object lifecycle events** - hooks when objects are added, persisted,
+  deleted from sessions.   Read more about these at
+  :ref:`session_lifecycle_events`.
+
+* **Execution Events** - Part of the :term:`2.0 style` execution model, all
+  SELECT statements against ORM entities emitted, as well as bulk UPDATE
+  and DELETE statements outside of the flush process, are intercepted
+  from the :meth:`_orm.Session.execute` method using the
+  :meth:`_orm.SessionEvents.do_orm_execute` method.  Read more about this
+  event at :ref:`session_execute_events`.
+
+Be sure to read the :ref:`session_events_toplevel` chapter for context
+on these events.
+
+.. autoclass:: sqlalchemy.orm.SessionEvents
    :members:
 
 Mapper Events
 -------------
 
-.. autoclass:: sqlalchemy.orm.events.MapperEvents
+Mapper event hooks encompass things that happen as related to individual
+or multiple :class:`_orm.Mapper` objects, which are the central configurational
+object that maps a user-defined class to a :class:`_schema.Table` object.
+Types of things which occur at the :class:`_orm.Mapper` level include:
+
+* **Per-object persistence operations** - the most popular mapper hooks are the
+  unit-of-work hooks such as :meth:`_orm.MapperEvents.before_insert`,
+  :meth:`_orm.MapperEvents.after_update`, etc.  These events are contrasted to
+  the more coarse grained session-level events such as
+  :meth:`_orm.SessionEvents.before_flush` in that they occur within the flush
+  process on a per-object basis; while finer grained activity on an object is
+  more straightforward, availability of :class:`_orm.Session` features is
+  limited.
+
+* **Mapper configuration events** - the other major class of mapper hooks are
+  those which occur as a class is mapped, as a mapper is finalized, and when
+  sets of mappers are configured to refer to each other.  These events include
+  :meth:`_orm.MapperEvents.instrument_class`,
+  :meth:`_orm.MapperEvents.before_mapper_configured` and
+  :meth:`_orm.MapperEvents.mapper_configured` at the individual
+  :class:`_orm.Mapper` level, and  :meth:`_orm.MapperEvents.before_configured`
+  and :meth:`_orm.MapperEvents.after_configured` at the level of collections of
+  :class:`_orm.Mapper` objects.
+
+.. autoclass:: sqlalchemy.orm.MapperEvents
    :members:
 
 Instance Events
 ---------------
 
-.. autoclass:: sqlalchemy.orm.events.InstanceEvents
+Instance events are focused on the construction of ORM mapped instances,
+including when they are instantiated as :term:`transient` objects,
+when they are loaded from the database and become :term:`persistent` objects,
+as well as when database refresh or expiration operations occur on the object.
+
+.. autoclass:: sqlalchemy.orm.InstanceEvents
    :members:
 
-Session Events
---------------
 
-.. autoclass:: sqlalchemy.orm.events.SessionEvents
+
+.. _orm_attribute_events:
+
+Attribute Events
+----------------
+
+Attribute events are triggered as things occur on individual attributes of
+ORM mapped objects.  These events form the basis for things like
+:ref:`custom validation functions <simple_validators>` as well as
+:ref:`backref handlers <relationships_backref>`.
+
+.. seealso::
+
+  :ref:`mapping_attributes_toplevel`
+
+.. autoclass:: sqlalchemy.orm.AttributeEvents
    :members:
 
+
 Query Events
 ------------
 
-.. autoclass:: sqlalchemy.orm.events.QueryEvents
+.. autoclass:: sqlalchemy.orm.QueryEvents
    :members:
 
 Instrumentation Events
@@ -47,6 +112,6 @@ Instrumentation Events
 
 .. automodule:: sqlalchemy.orm.instrumentation
 
-.. autoclass:: sqlalchemy.orm.events.InstrumentationEvents
+.. autoclass:: sqlalchemy.orm.InstrumentationEvents
    :members:
 
diff --git a/doc/build/orm/examples.rst b/doc/build/orm/examples.rst
index e8bb894fd..7a79104b9 100644
--- a/doc/build/orm/examples.rst
+++ b/doc/build/orm/examples.rst
@@ -147,6 +147,13 @@ Horizontal Sharding
 Extending the ORM
 =================
 
+.. _examples_session_orm_events:
+
+ORM Query Events
+-----------------
+
+.. automodule:: examples.extending_query
+
 .. _examples_caching:
 
 Dogpile Caching
diff --git a/doc/build/orm/extending.rst b/doc/build/orm/extending.rst
index 31e543a85..04800ffc0 100644
--- a/doc/build/orm/extending.rst
+++ b/doc/build/orm/extending.rst
@@ -2,6 +2,11 @@
 Events and Internals
 ====================
 
+The SQLAlchemy ORM as well as Core are extended generally through the use
+of event hooks.  Be sure to review the use of the event system in general
+at :ref:`event_toplevel`.
+
+
 .. toctree::
     :maxdepth: 2
 
diff --git a/doc/build/orm/extensions/baked.rst b/doc/build/orm/extensions/baked.rst
index e8651dbaa..4751fef36 100644
--- a/doc/build/orm/extensions/baked.rst
+++ b/doc/build/orm/extensions/baked.rst
@@ -26,7 +26,7 @@ the caching of the SQL calls and result sets themselves is available in
    action taken by the user, using the system described at :ref:`sql_caching`.
 
 
-.. note::
+.. deepalchemy::
 
     The :mod:`sqlalchemy.ext.baked` extension is **not for beginners**.  Using
     it correctly requires a good high level understanding of how SQLAlchemy, the
diff --git a/doc/build/orm/internals.rst b/doc/build/orm/internals.rst
index 08434d3bb..1a06b73b8 100644
--- a/doc/build/orm/internals.rst
+++ b/doc/build/orm/internals.rst
@@ -85,6 +85,10 @@ sections, are listed here.
 
 .. autodata:: sqlalchemy.orm.interfaces.NOT_EXTENSION
 
+.. autofunction:: sqlalchemy.orm.loading.merge_result
+
+.. autofunction:: sqlalchemy.orm.loading.merge_frozen_result
+
 
 .. autodata:: sqlalchemy.orm.interfaces.ONETOMANY
 
diff --git a/doc/build/orm/mapped_attributes.rst b/doc/build/orm/mapped_attributes.rst
index b8a0f89c9..a8711d2e6 100644
--- a/doc/build/orm/mapped_attributes.rst
+++ b/doc/build/orm/mapped_attributes.rst
@@ -1,3 +1,5 @@
+.. _mapping_attributes_toplevel:
+
 .. currentmodule:: sqlalchemy.orm
 
 Changing Attribute Behavior
diff --git a/doc/build/orm/query.rst b/doc/build/orm/query.rst
index ed45a65e7..592004e86 100644
--- a/doc/build/orm/query.rst
+++ b/doc/build/orm/query.rst
@@ -18,16 +18,16 @@ The Query Object
 
 Following is the full interface for the :class:`_query.Query` object.
 
-.. autoclass:: sqlalchemy.orm.query.Query
+.. autoclass:: sqlalchemy.orm.Query
    :members:
 
-   .. automethod:: sqlalchemy.orm.query.Query.prefix_with
+   .. automethod:: sqlalchemy.orm.Query.prefix_with
 
-   .. automethod:: sqlalchemy.orm.query.Query.suffix_with
+   .. automethod:: sqlalchemy.orm.Query.suffix_with
 
-   .. automethod:: sqlalchemy.orm.query.Query.with_hint
+   .. automethod:: sqlalchemy.orm.Query.with_hint
 
-   .. automethod:: sqlalchemy.orm.query.Query.with_statement_hint
+   .. automethod:: sqlalchemy.orm.Query.with_statement_hint
 
 ORM-Specific Query Constructs
 =============================
diff --git a/doc/build/orm/session_api.rst b/doc/build/orm/session_api.rst
index bad816967..ada035e95 100644
--- a/doc/build/orm/session_api.rst
+++ b/doc/build/orm/session_api.rst
@@ -35,6 +35,21 @@ Session and sessionmaker()
         arguments that will assist in determining amongst a set of database
         connections which one should be used to invoke this statement.
 
+    .. attribute:: local_execution_options
+
+        Dictionary view of the execution options passed to the
+        :meth:`.Session.execute` method.  This does not include options
+        that may be associated with the statement being invoked.
+
+        .. seealso::
+
+            :attr:`_orm.ORMExecuteState.execution_options`
+
+    .. attribute::  execution_options
+        The complete dictionary of current execution options.
+
+        This is a merge of the statement level options with the
+        locally passed execution options.
 
 .. autoclass:: Session
    :members:
diff --git a/doc/build/orm/session_basics.rst b/doc/build/orm/session_basics.rst
index afa9ae23d..f63b7abd0 100644
--- a/doc/build/orm/session_basics.rst
+++ b/doc/build/orm/session_basics.rst
@@ -41,7 +41,6 @@ another :class:`.Session` when you want to work with them again, so that they
 can resume their normal task of representing database state.
 
 
-
 Basics of Using a Session
 =========================
 
@@ -49,8 +48,8 @@ The most basic :class:`.Session` use patterns are presented here.
 
 .. _session_getting:
 
-Instantiating
--------------
+Opening and Closing a Session
+-----------------------------
 
 The :class:`_orm.Session` may be constructed on its own or by using the
 :class:`_orm.sessionmaker` class.    It typically is passed a single
@@ -119,6 +118,8 @@ can be used by any number of functions and threads simultaenously.
 
     :class:`_orm.Session`
 
+.. _session_querying_1x:
+
 Querying (1.x Style)
 --------------------
 
@@ -159,6 +160,8 @@ The :class:`_query.Query` object is introduced in great detail in
 
     :ref:`query_api_toplevel`
 
+.. _session_querying_20:
+
 Querying (2.0 style)
 --------------------
 
@@ -166,7 +169,7 @@ Querying (2.0 style)
 
 SQLAlchemy 2.0 will standardize the production of SELECT statements across both
 Core and ORM by making direct use of the :class:`_sql.Select` object within the
-ORM, removing the need for there to be a separate :class:`_orm.query.Query`
+ORM, removing the need for there to be a separate :class:`_orm.Query`
 object.    This mode of operation is available in SQLAlchemy 1.4 right now to
 support applications that will be migrating to 2.0.   The :class:`_orm.Session`
 must be instantiated with the
diff --git a/doc/build/orm/session_events.rst b/doc/build/orm/session_events.rst
index 066fe7c24..0040f6f47 100644
--- a/doc/build/orm/session_events.rst
+++ b/doc/build/orm/session_events.rst
@@ -1,7 +1,7 @@
 .. _session_events_toplevel:
 
-Tracking Object and Session Changes with Events
-===============================================
+Tracking queries, object and Session Changes with Events
+=========================================================
 
 SQLAlchemy features an extensive :ref:`Event Listening <event_toplevel>`
 system used throughout the Core and ORM.   Within the ORM, there are a
@@ -12,6 +12,236 @@ as some older events that aren't as relevant as they once were.  This
 section will attempt to introduce the major event hooks and when they
 might be used.
 
+.. _session_execute_events:
+
+Execute Events
+---------------
+
+.. versionadded:: 1.4  The :class:`_orm.Session` now features a single
+   comprehensive hook designed to intercept all SELECT statements made
+   on behalf of the ORM as well as bulk UPDATE and DELETE statements.
+   This hook supersedes the previous :meth:`_orm.QueryEvents.before_compile`
+   event as well :meth:`_orm.QueryEvents.before_compile_update` and
+   :meth:`_orm.QueryEvents.before_compile_delete`.
+
+:class:`_orm.Session` features a comprehensive system by which all queries
+invoked via the :meth:`_orm.Session.execute` method, which includes all
+SELECT statements emitted by :class:`_orm.Query` as well as all SELECT
+statements emitted on behalf of column and relationship loaders, may
+be intercepted and modified.   The system makes use of the
+:meth:`_orm.SessionEvents.do_orm_execute` event hook as well as the
+:class:`_orm.ORMExecuteState` object to represent the event state.
+
+
+Basic Query Interception
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:meth:`_orm.SessionEvents.do_orm_execute` is firstly useful for any kind of
+interception of a query, which includes those emitted by
+:class:`_orm.Query` with :term:`1.x style` as well as when an ORM-enabled
+:term:`2.0 style` :func:`_sql.select`,
+:func:`_sql.update` or :func:`_sql.delete` construct is delivered to
+:meth:`_orm.Session.execute`.   The :class:`_orm.ORMExecuteState` construct
+provides accessors to allow modifications to statements, parameters, and
+options::
+
+    Session = sessionmaker(engine, future=True)
+
+    @event.listens_for(Session, "do_orm_execute")
+    def _do_orm_execute(orm_execute_state):
+        if orm_execute_state.is_select:
+            # add populate_existing for all SELECT statements
+
+            orm_execute_state.update_execution_options(populate_existing=True)
+
+            # check if the SELECT is against a certain entity and add an
+            # ORDER BY if so
+            col_descriptions = orm_execute_state.statement.column_descriptions
+
+            if col_descriptions[0]['entity'] is MyEntity:
+                orm_execute_state.statement = statement.order_by(MyEntity.name)
+
+The above example illustrates some simple modifications to SELECT statements.
+At this level, the :meth:`_orm.SessionEvents.do_orm_execute` event hook intends
+to replace the previous use of the :meth:`_orm.QueryEvents.before_compile` event,
+which was not fired off consistently for various kinds of loaders; additionally,
+the :meth:`_orm.QueryEvents.before_compile` only applies to :term:`1.x style`
+use with :class:`_orm.Query` and not with :term:`2.0 style` use of
+:meth:`_orm.Session.execute`.
+
+
+.. _do_orm_execute_global_criteria:
+
+Adding global WHERE / ON criteria
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+One of the most requested query-extension features is the ability to add WHERE
+criteria to all occurrences of an entity in all queries.   This is achievable
+by making use of the :func:`_orm.with_loader_criteria` query option, which
+may be used on its own, or is ideally suited to be used within the
+:meth:`_orm.SessionEvents.do_orm_execute` event::
+
+    from sqlalchemy.orm import with_loader_criteria
+
+    Session = sessionmaker(engine, future=True)
+
+    @event.listens_for(Session, "do_orm_execute")
+    def _do_orm_execute(orm_execute_state):
+        if orm_execute_state.is_select:
+            orm_execute_state.statement = orm_execute_state.statement.options(
+                with_loader_criteria(MyEntity.public == True)
+            )
+
+Above, an option is added to all SELECT statements that will limit all queries
+against ``MyEntity`` to filter on ``public == True``.   The criteria
+will be applied to **all** loads of that class within the scope of the
+immediate query as well as subsequent relationship loads, which includes
+lazy loads, selectinloads, etc.
+
+For a series of classes that all feature some common column structure,
+if the classes are composed using a :ref:`declarative mixin <declarative_mixins>`,
+the mixin class itself may be used in conjunction with the :func:`_orm.with_loader_criteria`
+option by making use of a Python lambda.  The Python lambda will be invoked at
+query compilation time against the specific entities which match the criteria.
+Given a series of classes based on a mixin called ``HasTimestamp``::
+
+    import datetime
+
+    class HasTimestamp(object):
+        timestamp = Column(DateTime, default=datetime.datetime.now)
+
+
+    class SomeEntity(HasTimestamp, Base):
+        __tablename__ = "some_entity"
+        id = Column(Integer, primary_key=True)
+
+    class SomeOtherEntity(HasTimestamp, Base):
+        __tablename__ = "some_entity"
+        id = Column(Integer, primary_key=True)
+
+
+The above classes ``SomeEntity`` and ``SomeOtherEntity`` will each have a column
+``timestamp`` that defaults to the current date and time.   An event may be used
+to intercept all objects that extend from ``HasTimestamp`` and filter their
+``timestamp`` column on a date that is no older than one month ago::
+
+    @event.listens_for(Session, "do_orm_execute")
+    def _do_orm_execute(orm_execute_state):
+        if orm_execute_state.is_select:
+            one_month_ago = datetime.datetime.today() - datetime.timedelta(months=1)
+
+            orm_execute_state.statement = orm_execute_state.statement.options(
+                with_loader_criteria(
+                    HasTimestamp,
+                    lambda cls: cls.timestamp >= one_month_ago,
+                    include_aliases=True
+                )
+            )
+
+.. seealso::
+
+    :ref:`examples_session_orm_events` - includes working examples of the
+    above :func:`_orm.with_loader_criteria` recipes.
+
+.. _do_orm_execute_re_executing:
+
+Re-Executing Statements
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. deepalchemy:: the statement re-execution feature involves a slightly
+   intricate recursive sequence, and is intended to solve the fairly hard
+   problem of being able to re-route the execution of a SQL statement into
+   various non-SQL contexts.    The twin examples of "dogpile caching" and
+   "horizontal sharding", linked below, should be used as a guide for when this
+   rather advanced feature is appropriate to be used.
+
+The :class:`_orm.ORMExecuteState` is capable of controlling the execution of
+the given statement; this includes the ability to either not invoke the
+statement at all, allowing a pre-constructed result set retrieved from a cache to
+be returned instead, as well as the ability to invoke the same statement
+repeatedly with different state, such as invoking it against multiple database
+connections and then merging the results together in memory.   Both of these
+advanced patterns are demonstrated in SQLAlchemy's example suite as detailed
+below.
+
+When inside the :meth:`_orm.SessionEvents.do_orm_execute` event hook, the
+:meth:`_orm.ORMExecuteState.invoke_statement` method may be used to invoke
+the statement using a new nested invocation of :meth:`_orm.Session.execute`,
+which will then preempt the subsequent handling of the current execution
+in progress and instead return the :class:`_engine.Result` returned by the
+inner execution.   The event handlers thus far invoked for the
+:meth:`_orm.SessionEvents.do_orm_execute` hook within this process will
+be skipped within this nested call as well.
+
+The :meth:`_orm.ORMExecuteState.invoke_statement` method returns a
+:class:`_engine.Result` object; this object then features the ability for it to
+be "frozen" into a cacheable format and "unfrozen" into a new
+:class:`_engine.Result` object, as well as for its data to be merged with
+that of other :class:`_engine.Result` objects.
+
+E.g., using :meth:`_orm.SessionEvents.do_orm_execute` to implement a cache::
+
+    from sqlalchemy.orm import loading
+
+    cache = {}
+
+    @event.listens_for(Session, "do_orm_execute")
+    def _do_orm_execute(orm_execute_state):
+        if "my_cache_key" in orm_execute_state.execution_options:
+            cache_key = orm_execute_state.execution_options["my_cache_key"]
+
+            if cache_key in cache:
+                frozen_result = cache[cache_key]
+            else:
+                frozen_result = orm_execute_state.invoke_statement().freeze()
+                cache[cache_key] = frozen_result
+
+            return loading.merge_frozen_result(
+                orm_execute_state.session,
+                orm_execute_state.statement,
+                frozen_result,
+                load=False,
+            )
+
+With the above hook in place, an example of using the cache would look like::
+
+    stmt = select(User).where(User.name == 'sandy').execution_options(my_cache_key="key_sandy")
+
+    result = session.execute(stmt)
+
+Above, a custom execution option is passed to
+:meth:`_sql.Select.execution_options` in order to establish a "cache key" that
+will then be intercepted by the :meth:`_orm.SessionEvents.do_orm_execute` hook.  This
+cache key is then matched to a :class:`_engine.FrozenResult` object that may be
+present in the cache, and if present, the object is re-used.  The recipe makes
+use of the :meth:`_engine.Result.freeze` method to "freeze" a
+:class:`_engine.Result` object, which above will contain ORM results, such that
+it can be stored in a cache and used multiple times. In order to return a live
+result from the "frozen" result, the :func:`_orm.loading.merge_frozen_result`
+function is used to merge the "frozen" data from the result object into the
+current session.
+
+The above example is implemented as a complete example in :ref:`examples_caching`.
+
+The :meth:`_orm.ORMExecuteState.invoke_statement` method may also be called
+multiple times, passing along different information to the
+:paramref:`_orm.ORMExecuteState.invoke_statement.bind_arguments` parameter such
+that the :class:`_orm.Session` will make use of different
+:class:`_engine.Engine` objects each time.  This will return a different
+:class:`_engine.Result` object each time; these results can be merged together
+using the :meth:`_engine.Result.merge` method.  This is the technique employed
+by the :ref:`horizontal_sharding_toplevel` extension; see the source code to
+familiarize.
+
+.. seealso::
+
+    :ref:`examples_caching`
+
+    :ref:`examples_sharding`
+
+
+
+
 .. _session_persistence_events:
 
 Persistence Events
diff --git a/doc/build/orm/session_transaction.rst b/doc/build/orm/session_transaction.rst
index c5f47697b..6bef0cee6 100644
--- a/doc/build/orm/session_transaction.rst
+++ b/doc/build/orm/session_transaction.rst
@@ -654,9 +654,9 @@ entire database interaction is rolled back.
 
 .. versionchanged:: 1.4  This section introduces a new version of the
    "join into an external transaction" recipe that will work equally well
-   for both "future" and "non-future" engines and sessions.   The recipe
-   here from previous versions such as 1.3 will also continue to work for
-   "non-future" engines and sessions.
+   for both :term:`2.0 style` and :term:`1.x style`engines and sessions.
+   The recipe here from previous versions such as 1.3 will also continue to
+   work for 1.x engines and sessions.
 
 
 The recipe works by establishing a :class:`_engine.Connection` within a
diff --git a/doc/build/orm/tutorial.rst b/doc/build/orm/tutorial.rst
index a6ef57bed..dbad10b6f 100644
--- a/doc/build/orm/tutorial.rst
+++ b/doc/build/orm/tutorial.rst
@@ -1375,9 +1375,10 @@ and ``Address`` because there's only one foreign key between them. If there
 were no foreign keys, or several, :meth:`_query.Query.join`
 works better when one of the following forms are used::
 
-    query.join(Address, User.id==Address.user_id)    # explicit condition
-    query.join(User.addresses)                       # specify relationship from left to right
-    query.join(Address, User.addresses)              # same, with explicit target
+    query.join(Address, User.id==Address.user_id)          # explicit condition
+    query.join(User.addresses)                             # specify relationship from left to right
+    query.join(Address, User.addresses)                    # same, with explicit target
+    query.join(User.addresses.and_(Address.name != 'foo')) # use relationship + additional ON criteria
 
 As you would expect, the same idea is used for "outer" joins, using the
 :meth:`_query.Query.outerjoin` function::