fix a handful of warnings that were emitting but not raising, usually because they were inside an "expect_warnings" block. modify "expect_warnings" to always use "raise_on_any_unexpected" behavior; remove this parameter. Fixed issue in semi-private ``await_only()`` and ``await_fallback()`` concurrency functions where the given awaitable would remain un-awaited if the function threw a ``GreenletError``, which could cause "was not awaited" warnings later on if the program continued. In this case, the given awaitable is now cancelled before the exception is thrown. Change-Id: I33668c5e8c670454a3d879e559096fb873b57244
Fixed a bug that prevented use of :attr:`_engine.URL.normalized_query` in SQLAlchemy v2. Fixes: #9682 Change-Id: I2704154af34f438b4cbb290602fc936c1184c074
Repaired a major shortcoming which was identified in the :ref:`engine_insertmanyvalues` performance optimization feature first introduced in the 2.0 series. This was a continuation of the change in 2.0.9 which disabled the SQL Server version of the feature due to a reliance in the ORM on apparent row ordering that is not guaranteed to take place. The fix applies new logic to all "insertmanyvalues" operations, which takes effect when a new parameter :paramref:`_dml.Insert.returning.sort_by_parameter_order` on the :meth:`_dml.Insert.returning` or :meth:`_dml.UpdateBase.return_defaults` methods, that through a combination of alternate SQL forms, direct correspondence of client side parameters, and in some cases downgrading to running row-at-a-time, will apply sorting to each batch of returned rows using correspondence to primary key or other unique values in each row which can be correlated to the input data. Performance impact is expected to be minimal as nearly all common primary key scenarios are suitable for parameter-ordered batching to be achieved for all backends other than SQLite, while "row-at-a-time" mode operates with a bare minimum of Python overhead compared to the very heavyweight approaches used in the 1.x series. For SQLite, there is no difference in performance when "row-at-a-time" mode is used. It's anticipated that with an efficient "row-at-a-time" INSERT with RETURNING batching capability, the "insertmanyvalues" feature can be later be more easily generalized to third party backends that include RETURNING support but not necessarily easy ways to guarantee a correspondence with parameter order. Fixes: #9618 References: #9603 Change-Id: I1d79353f5f19638f752936ba1c35e4dc235a8b7c
Added :func:`_sa.create_pool_from_url` and :func:`_asyncio.create_async_pool_from_url` to create a :class:`_pool.Pool` instance from an input url passed as string or :class:`_sa.URL`. Fixes: #9613 Change-Id: Icd8aa3f2849e6fd1bc5341114f3ef8d216a2c543
The support for pool ping listeners to receive exception events via the :meth:`.ConnectionEvents.handle_error` event added in 2.0.0b1 for :ticket:`5648` failed to take into account dialect-specific ping routines such as that of MySQL and PostgreSQL. The dialect feature has been reworked so that all dialects participate within event handling. Additionally, a new boolean element :attr:`.ExceptionContext.is_pre_ping` is added which identifies if this operation is occurring within the pre-ping operation. For this release, third party dialects which implement a custom :meth:`_engine.Dialect.do_ping` method can opt in to the newly improved behavior by having their method no longer catch exceptions or check exceptions for "is_disconnect", instead just propagating all exceptions outwards. Checking the exception for "is_disconnect" is now done by an enclosing method on the default dialect, which ensures that the event hook is invoked for all exception scenarios before testing the exception as a "disconnect" exception. If an existing ``do_ping()`` method continues to catch exceptions and check "is_disconnect", it will continue to work as it did previously, but ``handle_error`` hooks will not have access to the exception if it isn't propagated outwards. Fixes: #5648 Change-Id: I6535d5cb389e1a761aad8c37cfeb332c548b876d
Repaired a regression caused by the fix for :ticket:`8419` which caused asyncpg connections to be reset (i.e. transaction ``rollback()`` called) and returned to the pool normally in the case that the connection were not explicitly returned to the connection pool and was instead being intercepted by Python garbage collection, which would fail if the garbage collection operation were being called outside of the asyncio event loop, leading to a large amount of stack trace activity dumped into logging and standard output. The correct behavior is restored, which is that all asyncio connections that are garbage collected due to not being explicitly returned to the connection pool are detached from the pool and discarded, along with a warning, rather than being returned the pool, as they cannot be reliably reset. In the case of asyncpg connections, the asyncpg-specific ``terminate()`` method will be used to end the connection more gracefully within this process as opposed to just dropping it. This change includes a small behavioral change that is hoped to be useful for debugging asyncio applications, where the warning that's emitted in the case of asyncio connections being unexpectedly garbage collected has been made slightly more aggressive by moving it outside of a ``try/except`` block and into a ``finally:`` block, where it will emit unconditionally regardless of whether the detach/termination operation succeeded or not. It will also have the effect that applications or test suites which promote Python warnings to exceptions will see this as a full exception raise, whereas previously it was not possible for this warning to actually propagate as an exception. Applications and test suites which need to tolerate this warning in the interim should adjust the Python warnings filter to allow these warnings to not raise. The behavior for traditional sync connections remains unchanged, that garbage collected connections continue to be returned to the pool normally without emitting a warning. This will likely be changed in a future major release to at least emit a similar warning as is emitted for asyncio drivers, as it is a usage error for pooled connections to be intercepted by garbage collection without being properly returned to the pool. Fixes: #9237 Change-Id: Ib35cfb2e628f2eb2da6d2b65674702556f55603a
Fixed a long-standing race condition in the connection pool which could occur under eventlet/gevent monkeypatching schemes in conjunction with the use of eventlet/gevent ``Timeout`` conditions, where a connection pool checkout that's interrupted due to the timeout would fail to clean up the failed state, causing the underlying connection record and sometimes the database connection itself to "leak", leaving the pool in an invalid state with unreachable entries. This issue was first identified and fixed in SQLAlchemy 1.2 for :ticket:`4225`, however the failure modes detected in that fix failed to accommodate for ``BaseException``, rather than ``Exception``, which prevented eventlet/gevent ``Timeout`` from being caught. In addition, a block within initial pool connect has also been identified and hardened with a ``BaseException`` -> "clean failed connect" block to accommodate for the same condition in this location. Big thanks to Github user @niklaus for their tenacious efforts in identifying and describing this intricate issue. Fixes: #8974 Change-Id: I95a0e1f080d0cee6f1a66977432a586fdf87f686
Changed how the positional compilation is performed. It's rendered by the compiler the same as the pyformat compilation. The string is then processed to replace the placeholders with the correct ones, and to obtain the correct order of the parameters. This vastly simplifies the computation of the order of the parameters, that in case of nested CTE is very hard to compute correctly. Reworked how numeric paramstyle behavers: - added support for repeated parameter, without duplicating them like in normal positional dialects - implement insertmany support. This requires that the dialect supports out of order placehoders, since all parameters that are not part of the VALUES clauses are placed at the beginning of the parameter tuple - support for different identifiers for a numeric parameter. It's for example possible to use postgresql style placeholder $1, $2, etc Added two new dialect based on sqlite to test "numeric" fully using both :1 style and $1 style. Includes a workaround for SQLite's not-really-correct numeric implementation. Changed parmstyle of asyncpg dialect to use numeric, rendering with its native $ identifiers Fixes: #8926 Fixes: #8849 Change-Id: I7c640467d49adfe6d795cc84296fc7403dcad4d6
Fixed issue where table reflection using :paramref:`.Table.extend_existing` would fail to deduplicate a same-named column if the existing :class:`.Table` used a separate key. The :paramref:`.Table.autoload_replace` parameter would allow the column to be skipped but under no circumstances should a :class:`.Table` ever have the same-named column twice. Additionally, changed deprecation warnings to exceptions as were implemented in I1d58c8ebe081079cb669e7ead60886ffc1b1a7f5 . Fixes: #8925 Change-Id: I83d0f8658177a7ffbb06e01dbca91377d1a98d49
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>" pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not exists in sqlalchemy fixtures Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55