Add multivalued insert context for defaultsreview/mike_bayer/ticket_4075

Added a new method :class:`.DefaultExecutionContext.current_parameters`
which is used within a function-based default value generator in
order to retrieve the current parameters being passed to the statement.
The new function differs from the ``.current_parameters`` attribute in
that it also provides for optional grouping of parameters that
correspond to a multi-valued "insert" construct.  Previously it was not
possible to identify the subset of parameters that were relevant to
the function call.

Change-Id: I6894c7b4a2bce3e83c3ade8af0e5b2f8df37b785
Fixes: #4075

8 files changed, 250 insertions, 26 deletions

diff --git a/doc/build/changelog/migration_12.rst b/doc/build/changelog/migration_12.rst
index e9deb75c4..4ded828c4 100644
--- a/doc/build/changelog/migration_12.rst
+++ b/doc/build/changelog/migration_12.rst
@@ -812,7 +812,6 @@ if the application is working with plain floats.
 
 :ticket:`4020`
 
-
 .. change_3249:
 
 Support for GROUPING SETS, CUBE, ROLLUP
@@ -840,6 +839,43 @@ are named in the documentation now::
 
 :ticket:`3429`
 
+.. _change_4075:
+
+Parameter helper for multi-valued INSERT with contextual default generator
+--------------------------------------------------------------------------
+
+A default generation function, e.g. that described at
+:ref:`context_default_functions`, can look at the current parameters relevant
+to the statment via the :attr:`.DefaultExecutionContext.current_parameters`
+attribute.  However, in the case of a :class:`.Insert` construct that specifies
+multiple VALUES clauses via the :meth:`.Insert.values` method, the user-defined
+function is called multiple times, once for each parameter set, however there
+was no way to know which subset of keys in
+:attr:`.DefaultExecutionContext.current_parameters` apply to that column.  A
+new function :meth:`.DefaultExecutionContext.get_current_parameters` is added,
+which includes a keyword argument
+:paramref:`.DefaultExecutionContext.get_current_parameters.isolate_multiinsert_groups`
+defaulting to ``True``, which performs the extra work of delivering a sub-dictionary of
+:attr:`.DefaultExecutionContext.current_parameters` which has the names
+localized to the current VALUES clause being processed::
+
+
+    def mydefault(context):
+        return context.get_current_parameters()['counter'] + 12
+
+    mytable = Table('mytable', meta,
+        Column('counter', Integer),
+        Column('counter_plus_twelve',
+               Integer, default=mydefault, onupdate=mydefault)
+    )
+
+    stmt = mytable.insert().values(
+        [{"counter": 5}, {"counter": 18}, {"counter": 20}])
+
+    conn.execute(stmt)
+
+:ticket:`4075`
+
 Key Behavioral Changes - ORM
 ============================
 
diff --git a/doc/build/changelog/unreleased_12/4075.rst b/doc/build/changelog/unreleased_12/4075.rst
new file mode 100644
index 000000000..deacc077b
--- /dev/null
+++ b/doc/build/changelog/unreleased_12/4075.rst
@@ -0,0 +1,19 @@
+.. change::
+    :tags: bug, sql
+    :tickets: 4075
+
+    Added a new method :meth:`.DefaultExecutionContext.get_current_parameters`
+    which is used within a function-based default value generator in
+    order to retrieve the current parameters being passed to the statement.
+    The new function differs from the
+    :attr:`.DefaultExecutionContext.current_parameters` attribute in
+    that it also provides for optional grouping of parameters that
+    correspond to a multi-valued "insert" construct.  Previously it was not
+    possible to identify the subset of parameters that were relevant to
+    the function call.
+
+    .. seealso::
+
+        :ref:`change_4075`
+
+        :ref:`context_default_functions`
\ No newline at end of file
diff --git a/doc/build/core/defaults.rst b/doc/build/core/defaults.rst
index f6cf4f4b3..d65668527 100644
--- a/doc/build/core/defaults.rst
+++ b/doc/build/core/defaults.rst
@@ -90,38 +90,57 @@ as the function itself without calling it (i.e. there are no parenthesis
 following) - SQLAlchemy will execute the function at the time the statement
 executes.
 
+.. _context_default_functions:
+
 Context-Sensitive Default Functions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The Python functions used by :paramref:`.Column.default` and :paramref:`.Column.onupdate` may also make use of
-the current statement's context in order to determine a value. The `context`
-of a statement is an internal SQLAlchemy object which contains all information
-about the statement being executed, including its source expression, the
-parameters associated with it and the cursor. The typical use case for this
-context with regards to default generation is to have access to the other
-values being inserted or updated on the row. To access the context, provide a
-function that accepts a single ``context`` argument::
+The Python functions used by :paramref:`.Column.default` and
+:paramref:`.Column.onupdate` may also make use of the current statement's
+context in order to determine a value. The `context` of a statement is an
+internal SQLAlchemy object which contains all information about the statement
+being executed, including its source expression, the parameters associated with
+it and the cursor. The typical use case for this context with regards to
+default generation is to have access to the other values being inserted or
+updated on the row. To access the context, provide a function that accepts a
+single ``context`` argument::
 
     def mydefault(context):
-        return context.current_parameters['counter'] + 12
+        return context.get_current_parameters()['counter'] + 12
 
     t = Table('mytable', meta,
         Column('counter', Integer),
         Column('counter_plus_twelve', Integer, default=mydefault, onupdate=mydefault)
     )
 
-Above we illustrate a default function which will execute for all INSERT and
-UPDATE statements where a value for ``counter_plus_twelve`` was otherwise not
-provided, and the value will be that of whatever value is present in the
-execution for the ``counter`` column, plus the number 12.
-
-While the context object passed to the default function has many attributes,
-the ``current_parameters`` member is a special member provided only during the
-execution of a default function for the purposes of deriving defaults from its
-existing values. For a single statement that is executing many sets of bind
-parameters, the user-defined function is called for each set of parameters,
-and ``current_parameters`` will be provided with each individual parameter set
-for each execution.
+The above default generation function is applied so that it will execute for
+all INSERT and UPDATE statements where a value for ``counter_plus_twelve`` was
+otherwise not provided, and the value will be that of whatever value is present
+in the execution for the ``counter`` column, plus the number 12.
+
+For a single statement that is being executed using "executemany" style, e.g.
+with multiple parameter sets passed to :meth:`.Connection.execute`, the user-
+defined function is called once for each set of parameters. For the use case of
+a multi-valued :class:`.Insert` construct (e.g. with more than one VALUES
+clause set up via the :meth:`.Insert.values` method), the user-defined function
+is also called once for each set of parameters.
+
+When the function is invoked, the special method
+:meth:`.DefaultExecutionContext.get_current_parameters` is available from
+the context object (an subclass of :class:`.DefaultExecutionContext`).  This
+method returns a dictionary of column-key to values that represents the
+full set of values for the INSERT or UPDATE statement.   In the case of a
+multi-valued INSERT construct, the subset of parameters that corresponds to
+the individual VALUES clause is isolated from the full parameter dictionary
+and returned alone.
+
+.. versionadded:: 1.2
+
+    Added :meth:`.DefaultExecutionContext.get_current_parameters` method,
+    which improves upon the still-present
+    :attr:`.DefaultExecutionContext.current_parameters` attribute
+    by offering the service of organizing multiple VALUES clauses
+    into individual parameter dictionaries.
 
 SQL Expressions
 ---------------
diff --git a/lib/sqlalchemy/engine/base.py b/lib/sqlalchemy/engine/base.py
index b5c95cb17..1719de516 100644
--- a/lib/sqlalchemy/engine/base.py
+++ b/lib/sqlalchemy/engine/base.py
@@ -977,7 +977,7 @@ class Connection(Connectable):
         except BaseException as e:
             self._handle_dbapi_exception(e, None, None, None, None)
 
-        ret = ctx._exec_default(default, None)
+        ret = ctx._exec_default(None, default, None)
         if self.should_close_with_result:
             self.close()
 
diff --git a/lib/sqlalchemy/engine/default.py b/lib/sqlalchemy/engine/default.py
index 227ff0845..4b9aa9493 100644
--- a/lib/sqlalchemy/engine/default.py
+++ b/lib/sqlalchemy/engine/default.py
@@ -1177,10 +1177,11 @@ class DefaultExecutionContext(interfaces.ExecutionContext):
                 self.root_connection._handle_dbapi_exception(
                     e, None, None, None, self)
 
-    def _exec_default(self, default, type_):
+    def _exec_default(self, column, default, type_):
         if default.is_sequence:
             return self.fire_sequence(default, type_)
         elif default.is_callable:
+            self.current_column = column
             return default.arg(self)
         elif default.is_clause_element:
             # TODO: expensive branching here should be
@@ -1195,17 +1196,97 @@ class DefaultExecutionContext(interfaces.ExecutionContext):
         else:
             return default.arg
 
+    current_parameters = None
+    """A dictionary of parameters applied to the current row.
+
+    This attribute is only available in the context of a user-defined default
+    generation function, e.g. as described at :ref:`context_default_functions`.
+    It consists of a dictionary which includes entries for each column/value
+    pair that is to be part of the INSERT or UPDATE statement. The keys of the
+    dictionary will be the key value of each :class:`.Column`, which is usually
+    synonymous with the name.
+
+    Note that the :attr:`.DefaultExecutionContext.current_parameters` attribute
+    does not accommodate for the "multi-values" feature of the
+    :meth:`.Insert.values` method.  The
+    :meth:`.DefaultExecutionContext.get_current_parameters` method should be
+    preferred.
+
+    .. seealso::
+
+        :meth:`.DefaultExecutionContext.get_current_parameters`
+
+        :ref:`context_default_functions`
+
+    """
+
+    def get_current_parameters(self, isolate_multiinsert_groups=True):
+        """Return a dictionary of parameters applied to the current row.
+
+        This method can only be used in the context of a user-defined default
+        generation function, e.g. as described at
+        :ref:`context_default_functions`. When invoked, a dictionary is
+        returned which includes entries for each column/value pair that is part
+        of the INSERT or UPDATE statement. The keys of the dictionary will be
+        the key value of each :class:`.Column`, which is usually synonymous
+        with the name.
+
+        :param isolate_multiinsert_groups=True: indicates that multi-valued
+         INSERT contructs created using :meth:`.Insert.values` should be
+         handled by returning only the subset of parameters that are local
+         to the current column default invocation.   When ``False``, the
+         raw parameters of the statement are returned including the
+         naming convention used in the case of multi-valued INSERT.
+
+        .. versionadded:: 1.2  added
+           :meth:`.DefaultExecutionContext.get_current_parameters`
+           which provides more functionality over the existing
+           :attr:`.DefaultExecutionContext.current_parameters`
+           attribute.
+
+        .. seealso::
+
+            :attr:`.DefaultExecutionContext.current_parameters`
+
+            :ref:`context_default_functions`
+
+        """
+        try:
+            parameters = self.current_parameters
+            column = self.current_column
+        except AttributeError:
+            raise exc.InvalidRequestError(
+                "get_current_parameters() can only be invoked in the "
+                "context of a Python side column default function")
+        if isolate_multiinsert_groups and \
+            self.isinsert and \
+                self.compiled.statement._has_multi_parameters:
+            if column._is_multiparam_column:
+                index = column.index + 1
+                d = {column.original.key: parameters[column.key]}
+            else:
+                d = {column.key: parameters[column.key]}
+                index = 0
+            keys = self.compiled.statement.parameters[0].keys()
+            d.update(
+                (key, parameters["%s_m%d" % (key, index)])
+                for key in keys
+            )
+            return d
+        else:
+            return parameters
+
     def get_insert_default(self, column):
         if column.default is None:
             return None
         else:
-            return self._exec_default(column.default, column.type)
+            return self._exec_default(column, column.default, column.type)
 
     def get_update_default(self, column):
         if column.onupdate is None:
             return None
         else:
-            return self._exec_default(column.onupdate, column.type)
+            return self._exec_default(column, column.onupdate, column.type)
 
     def _process_executemany_defaults(self):
         key_getter = self.compiled._key_getters_for_crud_column[2]
diff --git a/lib/sqlalchemy/sql/crud.py b/lib/sqlalchemy/sql/crud.py
index 5739c22f9..8421b1e66 100644
--- a/lib/sqlalchemy/sql/crud.py
+++ b/lib/sqlalchemy/sql/crud.py
@@ -395,7 +395,10 @@ def _create_update_prefetch_bind_param(compiler, c, process=True, name=None):
 
 
 class _multiparam_column(elements.ColumnElement):
+    _is_multiparam_column = True
+
     def __init__(self, original, index):
+        self.index = index
         self.key = "%s_m%d" % (original.key, index + 1)
         self.original = original
         self.default = original.default
diff --git a/lib/sqlalchemy/sql/elements.py b/lib/sqlalchemy/sql/elements.py
index 9213d616c..3b2bcb4ff 100644
--- a/lib/sqlalchemy/sql/elements.py
+++ b/lib/sqlalchemy/sql/elements.py
@@ -3656,6 +3656,8 @@ class ColumnClause(Immutable, ColumnElement):
 
     onupdate = default = server_default = server_onupdate = None
 
+    _is_multiparam_column = False
+
     _memoized_property = util.group_expirable_memoized_property()
 
     def __init__(self, text, type_=None, is_literal=False, _selectable=None):
diff --git a/test/sql/test_defaults.py b/test/sql/test_defaults.py
index 1ef49bf04..fc42d420f 100644
--- a/test/sql/test_defaults.py
+++ b/test/sql/test_defaults.py
@@ -15,6 +15,7 @@ from sqlalchemy.dialects import sqlite
 from sqlalchemy.testing import fixtures
 from sqlalchemy.util import u, b
 from sqlalchemy import util
+from sqlalchemy.testing import mock
 import itertools
 
 t = f = f2 = ts = currenttime = metadata = default_generator = None
@@ -1630,3 +1631,66 @@ class InsertFromSelectTest(fixtures.TestBase):
             testing.db.execute(table.select().order_by(table.c.x)).fetchall(),
             [(2, 1, 5), (7, 1, 12)]
         )
+
+class CurrentParametersTest(fixtures.TablesTest):
+    __backend__ = True
+
+    @classmethod
+    def define_tables(cls, metadata):
+        def gen_default(context):
+            pass
+
+        Table(
+            "some_table", metadata,
+            Column('x', String(50), default=gen_default),
+            Column('y', String(50)),
+        )
+
+    def _fixture(self, fn):
+
+        def gen_default(context):
+            fn(context)
+        some_table = self.tables.some_table
+        some_table.c.x.default.arg = gen_default
+        return fn
+
+    def _test(self, exec_type, usemethod):
+        collect = mock.Mock()
+
+        @self._fixture
+        def fn(context):
+            collect(context.get_current_parameters())
+
+        table = self.tables.some_table
+        if exec_type in ('multivalues', 'executemany'):
+            parameters = [{"y": "h1"}, {"y": "h2"}]
+        else:
+            parameters = [{"y": "hello"}]
+
+        if exec_type == 'multivalues':
+            stmt, params = table.insert().values(parameters), {}
+        else:
+            stmt, params = table.insert(), parameters
+
+        with testing.db.connect() as conn:
+            conn.execute(stmt, params)
+        eq_(
+            collect.mock_calls,
+            [mock.call({"y": param['y'], "x": None}) for param in parameters]
+        )
+
+    def test_single_w_attribute(self):
+        self._test("single", "attribute")
+
+    def test_single_w_method(self):
+        self._test("single", "method")
+
+    def test_executemany_w_attribute(self):
+        self._test("executemany", "attribute")
+
+    def test_executemany_w_method(self):
+        self._test("executemany", "method")
+
+    @testing.requires.multivalues_inserts
+    def test_multivalued_w_method(self):
+        self._test("multivalues", "method")