summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Also fix asyncio exception imports.fix_asyncio_run_py36Federico Caselli2021-01-241-10/+4
| | | | | | | | The python documentation imports everything directly from the asyncio package, and it seems that py < 3.8 does not have the asyncio.exception module Change-Id: I4719185f02f3c8d030ff24601e15f81767ba7d5a
* Limit AsyncAdaptedQueue to Python 3.7Mike Bayer2021-01-231-0/+4
| | | | | | | | | Tests here are failing for python 3.6 due to the lack of asyncio.run(). It seems to be non-trivial to vendor a working version of this in Python 3.6 as the tests here are running it in alternate threads. Change-Id: I9398c9fb2aa87f3228ce2f59277de732091bd541
* Merge "remove __allowlist__"mike bayer2021-01-231-4/+0
|\
| * remove __allowlist__Mike Bayer2021-01-231-4/+0
| | | | | | | | | | | | this testing element is not used Change-Id: I484c9a9f070122545fcdabe5a7f13b0bfca17023
* | Merge "Improve error message when await_ call errors"mike bayer2021-01-234-7/+34
|\ \ | |/ |/|
| * Improve error message when await_ call errorsFederico Caselli2021-01-214-7/+34
| | | | | | | | | | Fixes: #5832 Change-Id: Ia2ed8f1d1ec54e5f6e1a8f817a69446fdb3b7f6d
* | Merge "Removed some legacy terms in favor of modern equivalents. (D&I)"mike bayer2021-01-234-5/+5
|\ \
| * | Removed some legacy terms in favor of modern equivalents. (D&I)jonathan vanasco2021-01-214-5/+5
| |/ | | | | | | | | | | | | | | | | | | | | | | | | Migrated testing fixture: `TestBase.__whitelist__` -> `TestBase.__allowlist__` Migrated tox commands from deprecated to current: `whitelist_externals` > `allowlist_externals` Migrated test_session: `blacklist` -> `blocklist` Change-Id: I395d5ee977ff22fa703276b9b873cc96c59b9a35
* | Fix a couple of bugs in the asyncio implementationFederico Caselli2021-01-218-29/+146
|/ | | | | | | | | | | | | Log an informative message if a connection is not closed and the gc is reclaiming it when using an async dpapi, that does not support running IO at that stage. The ``AsyncAdaptedQueue`` used by default on async dpapis should instantiate a queue only when it's first used to avoid binding it to a possibly wrong event loop. Fixes: #5823 Change-Id: Ibfc50e209b1937ae3d6599ae7997f028c7a92c33
* Revert "Implement support for functions as FROM with columns clause support"Mike Bayer2021-01-2117-1667/+75
| | | | | | | This reverts commit 05a31f2708590161d4b3b4c7ff65196c99b4a22b. Atom has this little button called "push" and just pushes to master, I wasn't even *on* master. oops
* Implement support for functions as FROM with columns clause supportMike Bayer2021-01-2017-75/+1667
| | | | | | | WIP Fixes: #3566 Change-Id: I5b093b72533ef695293e737eb75850b9713e5e03
* Merge "chain joins from SelectState context, not Select"mike bayer2021-01-203-1/+84
|\
| * chain joins from SelectState context, not SelectMike Bayer2021-01-203-1/+84
| | | | | | | | | | | | | | | | | | | | | | | | | | Fixed issue in new :meth:`_sql.Select.join` method where chaining from the current JOIN wasn't looking at the right state, causing an expression like "FROM a JOIN b <onclause>, b JOIN c <onclause>" rather than "FROM a JOIN b <onclause> JOIN c <onclause>". Added :meth:`_sql.Select.outerjoin_from` method to complement :meth:`_sql.Select.join_from`. Fixes: #5858 Change-Id: I1346ebe0963bbd1e4bf868650e3ee1d6d3072f04
* | Add missing changelog fileMike Bayer2021-01-201-0/+13
|/ | | | | | | In 8860117c9655a4bdeafeba we forgot to add the changelog file. Fixes: #5169 Change-Id: Ia7a106e9e49aacbfd788eaec46b7826e8e4e065a
* Merge "Document Table/Column accessors"mike bayer2021-01-202-2/+112
|\
| * Document Table/Column accessorsMike Bayer2021-01-192-2/+112
| | | | | | | | | | | | | | | | | | | | | | | | | | As Sphinx will not allow us to add attributes to the .rst file while maintaining order, these have to be added as class-level attributes. Inlcude notes that "index" and "unique" parameters, while indicated by Column.index / Column.unique, do not actually indicate if the column is part of an index. Fixes: #5851 Change-Id: I18fbaf6c504c4b1005b4c51057f80397fb48b387
* | Merge "Disallow non-native psycopg2 Unicode in Python 3; update docs"mike bayer2021-01-193-56/+92
|\ \ | |/ |/|
| * Disallow non-native psycopg2 Unicode in Python 3; update docsFederico Caselli2021-01-193-56/+92
| | | | | | | | | | | | | | | | | | | | | | | | | | Fixed issue where the psycopg2 dialect would silently pass the ``use_native_unicode=False`` flag without actually having any effect under Python 3, as the psycopg2 DBAPI uses Unicode unconditionally under Python 3. This usage now raises an :class:`_exc.ArgumentError` when used under Python 3. Added test support for Python 2. Additionally, added documentation for client_encoding parameter that may be passed to libpq directly via psycopg2. Change-Id: I40ddf6382c157fa9399c21f0e01064197ea100f8
* | Merge "More replacements of mapper to map_imperatively"mike bayer2021-01-187-22/+32
|\ \
| * | More replacements of mapper to map_imperativelyFederico Caselli2021-01-157-22/+32
| |/ | | | | | | | | Ref: #5829 Change-Id: I6778fde8d2af66011c7a98beeb6d1b690c748afc
* | restore greenlet reqMike Bayer2021-01-181-0/+1
| | | | | | | | | | | | | | | | | | greenlet 1.0 is now released, so as openstack builds are up again and the upstream pip bug is fixed, let's now try to restore the requirements we had as we'd rather not have users confused about why asyncio doesn't work without additional packages. Change-Id: I33fe4eb754f3043f28f37be036df7a602917bac2
* | Merge "``Identity`` implies ``nullable=False``."mike bayer2021-01-178-24/+142
|\ \
| * | ``Identity`` implies ``nullable=False``.Federico Caselli2021-01-168-24/+142
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Altered the behavior of the :class:`_schema.Identity` construct such that when applied to a :class:`_schema.Column`, it will automatically imply that the value of :paramref:`_sql.Column.nullable` should default to ``False``, in a similar manner as when the :paramref:`_sql.Column.primary_key` parameter is set to ``True``. This matches the default behavior of all supporting databases where ``IDENTITY`` implies ``NOT NULL``. The PostgreSQL backend is the only one that supports adding ``NULL`` to an ``IDENTITY`` column, which is here supported by passing a ``True`` value for the :paramref:`_sql.Column.nullable` parameter at the same time. Fixes: #5775 Change-Id: I0516d506ff327cff35cda605e8897a27440e0373
* | | introduce generalized decorator to prevent invalid method callsMike Bayer2021-01-169-14/+131
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This introduces the ``_exclusive_against()`` utility decorator that can be used to prevent repeated invocations of methods that typically should only be called once. An informative error message is now raised for a selected set of DML methods (currently all part of :class:`_dml.Insert` constructs) if they are called a second time, which would implicitly cancel out the previous setting. The methods altered include: :class:`_sqlite.Insert.on_conflict_do_update`, :class:`_sqlite.Insert.on_conflict_do_nothing` (SQLite), :class:`_postgresql.Insert.on_conflict_do_update`, :class:`_postgresql.Insert.on_conflict_do_nothing` (PostgreSQL), :class:`_mysql.Insert.on_duplicate_key_update` (MySQL) Fixes: #5169 Change-Id: I9278fa87cd3470dcf296ff96bb0fb17a3236d49d
* | Merge "update aiomysql extra to match asyncpg, also dedupe asyncio extras"mike bayer2021-01-161-3/+5
|\ \
| * | update aiomysql extra to match asyncpg, also dedupe asyncio extrasThomas Grainger2021-01-131-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | <!-- Provide a general summary of your proposed changes in the Title field above --> ### Description currently the asyncio extra-deps are duplicated between every async driver (or it seems they are supposed to be). This updates all async drivers to depend on the `sqlalchemy[asyncio]` extra using the `%(asyncio)s` shorthand ### Checklist <!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once) --> This pull request is: - [ ] A documentation / typographical error fix - Good to go, no issue or tests are needed - [ ] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. - [ ] A new feature implementation - please include the issue number, and create an issue if none exists, which must include a complete example of how the feature would look. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. **Have a nice day!** Closes: #5830 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5830 Pull-request-sha: 9146e5f1391015b284fd96def8c04e36a7c985d2 Change-Id: I1865cc67b4cdf9f73a7effbfcb86cf2c1f824644
* | | Merge "Create explicit GC ordering between ConnectionFairy/ConnectionRecord"mike bayer2021-01-1610-17/+102
|\ \ \
| * | | Create explicit GC ordering between ConnectionFairy/ConnectionRecordMike Bayer2021-01-1511-25/+106
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed issue where connection pool would not return connections to the pool or otherwise be finalized upon garbage collection under pypy if the checked out connection fell out of scope without being closed. This is a long standing issue due to pypy's difference in GC behavior that does not call weakref finalizers if they are relative to another object that is also being garbage collected. A strong reference to the related record is now maintained so that the weakref has a strong-referenced "base" to trigger off of. Fixes: #5842 Change-Id: Id5448fdacb6cceaac1ea40b2fbc851f052ed8e86
* | | | Merge "Guard against re-entrant autobegin in Core, ORM"mike bayer2021-01-167-6/+100
|\ \ \ \
| * | | | Guard against re-entrant autobegin in Core, ORMMike Bayer2021-01-157-6/+100
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed bug in "future" version of :class:`.Engine` where emitting SQL during the :meth:`.EngineEvents.do_begin` event hook would cause a re-entrant condition due to autobegin, including the recipe documented for SQLite to allow for savepoints and serializable isolation support. Fixed issue in new :class:`_orm.Session` similar to that of the :class:`_engine.Connection` where the new "autobegin" logic could be tripped into a re-entrant state if SQL were executed within the :meth:`.SessionEvents.after_transaction_create` event hook. Also repair the new "testing_engine" pytest fixture to set up for "future" engine appropriately, which wasn't working leading to the test_execute.py tests not using the future engine since recent f1e96cb0874927a475d0c11139. Fixes: #5845 Change-Id: Ib2432d8c8bd753e24be60720ec47affb2df15a4a
* | | | | run handle error for commit/rollback fail and cancel transactionMike Bayer2021-01-155-45/+141
|/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed bug in asyncpg dialect where a failure during a "commit" or less likely a "rollback" should cancel the entire transaction; it's no longer possible to emit rollback. Previously the connection would continue to await a rollback that could not succeed as asyncpg would reject it. Fixes: #5824 Change-Id: I5a4916740c269b410f4d1a78ed25191de344b9d0
* | | | Merge "Add missing requirements to suite tests"mike bayer2021-01-151-2/+8
|\ \ \ \ | |_|_|/ |/| | |
| * | | Add missing requirements to suite testsGord Thompson2021-01-151-2/+8
| | | | | | | | | | | | | | | | | | | | Fixes: #5841 Change-Id: I53e8405e6ca8c3fd6909744632b76db724d9eb1f
* | | | Merge "update execute() arg formats in modules and tests"mike bayer2021-01-1537-419/+592
|\ \ \ \
| * | | | update execute() arg formats in modules and testsMike Bayer2021-01-1537-419/+592
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | continuing with producing a SQLAlchemy 1.4.0b2 that internally does not emit any of its own 2.0 deprecation warnings, migrate the *args and **kwargs passed to execute() methods that now must be a single list or dictionary. Alembic 1.5 is again waiting on this internal consistency to be present so that it can pass all tests with no 2.0 deprecation warnings. Change-Id: If6b792e57c8c5dff205419644ab68e631575a2fa
* | | | | Merge "Replace mapper with map_imperatively in mapping styles example"mike bayer2021-01-151-6/+8
|\ \ \ \ \ | |_|/ / / |/| | | |
| * | | | Replace mapper with map_imperatively in mapping styles exampleFederico Caselli2021-01-121-6/+8
| | | | | | | | | | | | | | | | | | | | | | | | | Fixes: #5829 Change-Id: Iad49b038a6061c1b699b31400a199bf65ced6156
* | | | | Merge "Add mssql+turbodbc to external dialect list"mike bayer2021-01-152-0/+13
|\ \ \ \ \
| * | | | | Add mssql+turbodbc to external dialect listGord Thompson2021-01-132-0/+13
| | |_|_|/ | |/| | | | | | | | | | | | | | | | | | Fixes: #5834 Change-Id: I1f207b84751e7e3425aa9e8e393787eeb9b595b7
* | | | | allow Executable to be accepted by Session.execute()Mike Bayer2021-01-144-2/+84
| |_|/ / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed an issue where the API to create a custom executable SQL construct using the ``sqlalchemy.ext.compiles`` extension according to the documentation that's been up for many years would no longer function if only ``Executable, ClauseElement`` were used as the base classes, additional classes were needed if wanting to use :meth:`_orm.Session.execute`. This has been resolved so that those extra classes aren't needed. Change-Id: I99b8acd88515c2a52842d62974199121e64c0381
* | | | fix double is_none()Mike Bayer2021-01-141-8/+4
| |_|/ |/| | | | | | | | | | | | | | | | | | | | I added an extra is_none() by mistake the other day and for some reason it sneaked past flake8. it's breaking all the builds so get it back in Change-Id: I17b311341169571efa856e062c6be7e8f362618f
* | | Merge "Use UnsupportedCompilationError for no default compiler"mike bayer2021-01-144-11/+49
|\ \ \
| * | | Use UnsupportedCompilationError for no default compilerMike Bayer2021-01-144-11/+49
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed issue where the stringification that is sometimes called when attempting to generate the "key" for the ``.c`` collection on a selectable would fail if the column were an unlabeled custom SQL construct using the ``sqlalchemy.ext.compiler`` extension, and did not provide a default compilation form; while this seems like an unusual case, it can get invoked for some ORM scenarios such as when the expression is used in an "order by" in combination with joined eager loading. The issue is that the lack of a default compiler function was raising :class:`.CompileError` and not :class:`.UnsupportedCompilationError`. Fixes: #5836 Change-Id: I5af243b2c70c7dcca4b212a3869c3017a50c132b
* | | Fix docs typo: subuqery -> subqueryEvan Moore2021-01-143-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | <!-- Provide a general summary of your proposed changes in the Title field above --> ### Description Fixes #5839 (sorry, realized just now that an issue wasn't required) ### Checklist <!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once) --> This pull request is: - [x] A documentation / typographical error fix - Good to go, no issue or tests are needed - [ ] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. - [ ] A new feature implementation - please include the issue number, and create an issue if none exists, which must include a complete example of how the feature would look. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. **Have a nice day!** Closes: #5840 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5840 Pull-request-sha: b6f6af929fab4dab771565f6373b0275acbca793 Change-Id: I0b4c9b2d893e8141ee1cd21742a9b111fe371bac
* | | Merge "reinvent xdist hooks in terms of pytest fixtures"mike bayer2021-01-14115-2092/+2637
|\ \ \ | |/ / |/| |
| * | reinvent xdist hooks in terms of pytest fixturesMike Bayer2021-01-13115-2092/+2638
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To allow the "connection" pytest fixture and others work correctly in conjunction with setup/teardown that expects to be external to the transaction, remove and prevent any usage of "xdist" style names that are hardcoded by pytest to run inside of fixtures, even function level ones. Instead use pytest autouse fixtures to implement our own r"setup|teardown_test(?:_class)?" methods so that we can ensure function-scoped fixtures are run within them. A new more explicit flow is set up within plugin_base and pytestplugin such that the order of setup/teardown steps, which there are now many, is fully documented and controllable. New granularity has been added to the test teardown phase to distinguish between "end of the test" when lock-holding structures on connections should be released to allow for table drops, vs. "end of the test plus its teardown steps" when we can perform final cleanup on connections and run assertions that everything is closed out. From there we can remove most of the defensive "tear down everything" logic inside of engines which for many years would frequently dispose of pools over and over again, creating for a broken and expensive connection flow. A quick test shows that running test/sql/ against a single Postgresql engine with the new approach uses 75% fewer new connections, creating 42 new connections total, vs. 164 new connections total with the previous system. As part of this, the new fixtures metadata/connection/future_connection have been integrated such that they can be combined together effectively. The fixture_session(), provide_metadata() fixtures have been improved, including that fixture_session() now strongly references sessions which are explicitly torn down before table drops occur afer a test. Major changes have been made to the ConnectionKiller such that it now features different "scopes" for testing engines and will limit its cleanup to those testing engines corresponding to end of test, end of test class, or end of test session. The system by which it tracks DBAPI connections has been reworked, is ultimately somewhat similar to how it worked before but is organized more clearly along with the proxy-tracking logic. A "testing_engine" fixture is also added that works as a pytest fixture rather than a standalone function. The connection cleanup logic should now be very robust, as we now can use the same global connection pools for the whole suite without ever disposing them, while also running a query for PostgreSQL locks remaining after every test and assert there are no open transactions leaking between tests at all. Additional steps are added that also accommodate for asyncio connections not explicitly closed, as is the case for legacy sync-style tests as well as the async tests themselves. As always, hundreds of tests are further refined to use the new fixtures where problems with loose connections were identified, largely as a result of the new PostgreSQL assertions, many more tests have moved from legacy patterns into the newest. An unfortunate discovery during the creation of this system is that autouse fixtures (as well as if they are set up by @pytest.mark.usefixtures) are not usable at our current scale with pytest 4.6.11 running under Python 2. It's unclear if this is due to the older version of pytest or how it implements itself for Python 2, as well as if the issue is CPU slowness or just large memory use, but collecting the full span of tests takes over a minute for a single process when any autouse fixtures are in place and on CI the jobs just time out after ten minutes. So at the moment this patch also reinvents a small version of "autouse" fixtures when py2k is running, which skips generating the real fixture and instead uses two global pytest fixtures (which don't seem to impact performance) to invoke the "autouse" fixtures ourselves outside of pytest. This will limit our ability to do more with fixtures until we can remove py2k support. py.test is still observed to be much slower in collection in the 4.6.11 version compared to modern 6.2 versions, so add support for new TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that will run the suite for fewer backends under Python 2. For Python 3 pin pytest to modern 6.2 versions where performance for collection has been improved greatly. Includes the following improvements: Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would be raised rather than :class:`.exc.TimeoutError`. Also repaired the :paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using the async engine, which previously would ignore the timeout and block rather than timing out immediately as is the behavior with regular :class:`.QueuePool`. For asyncio the connection pool will now also not interact at all with an asyncio connection whose ConnectionFairy is being garbage collected; a warning that the connection was not properly closed is emitted and the connection is discarded. Within the test suite the ConnectionKiller is now maintaining strong references to all DBAPI connections and ensuring they are released when tests end, including those whose ConnectionFairy proxies are GCed. Identified cx_Oracle.stmtcachesize as a major factor in Oracle test scalability issues, this can be reset on a per-test basis rather than setting it to zero across the board. the addition of this flag has resolved the long-standing oracle "two task" error problem. For SQL Server, changed the temp table style used by the "suite" tests to be the double-pound-sign, i.e. global, variety, which is much easier to test generically. There are already reflection tests that are more finely tuned to both styles of temp table within the mssql test suite. Additionally, added an extra step to the "dropfirst" mechanism for SQL Server that will remove all foreign key constraints first as some issues were observed when using this flag when multiple schemas had not been torn down. Identified and fixed two subtle failure modes in the engine, when commit/rollback fails in a begin() context manager, the connection is explicitly closed, and when "initialize()" fails on the first new connection of a dialect, the transactional state on that connection is still rolled back. Fixes: #5826 Fixes: #5827 Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
* | | provide FAQ entry for percent sign escaping in SQL compilerMike Bayer2021-01-131-0/+55
| | | | | | | | | | | | | | | | | | this is coming up more than once so provide a document Change-Id: I23dcd4c7a6527b2f33502e67ffad4335f895e6f9
* | | remove greenlet from default requiresMike Bayer2021-01-121-2/+1
| |/ |/| | | | | | | | | | | | | | | | | openstack builds are broken right now and the greenlet != 4.1.17 is blocking me from getting them to work, as greenlet has not released yet. greenlet is set up for the "asyncio" dependency and for tox, need to get openstack working for now. Change-Id: Idc7057e93f97cd6e606bb32a9d2b046f157bb4b9
* | Merge remote-tracking branch 'origin/pr/5809' into masterMike Bayer2021-01-111-1/+1
|\ \ | | | | | | | | | Change-Id: I08b2ba039ed44147cf2dafc91d569d646138e697
| * | Remove the extra parenthesis.Kazantcev Andrey2020-12-311-1/+1
| | |