summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/sql/base.py
Commit message (Collapse)AuthorAgeFilesLines
...
* | implement cython for cache_anon_map, prefix_anon_mapMike Bayer2021-12-211-21/+6
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These are small bits where cache_anon_map in particular is part of the cache key generation scheme which is a key target for cython. changing such a tiny element of the cache key gen is doing basically nothing yet, as the cython impl is mostly the exact same speed as the python one. I guess for cython to be effective we'd need to redo the whole cache key generation and possibly not use the same kinds of structures, which might not be very easy to do. Additionally, some cython runtime import errors are being observed on jenkins, add an upfront check to the test suite to indicate if the expected build succeeded when REQUIRE_SQLALCHEMY_CEXT is set. Running case CacheAnonMap Running python .... Done Running cython .... Done | python | cython | cy / py | test_get_anon_non_present| 0.301266758 | 0.231203834 | 0.767438915 | test_get_anon_present| 0.300919362 | 0.227336695 | 0.755473803 | test_has_key_non_present| 0.152725077 | 0.133191719 | 0.872101171 | test_has_key_present| 0.152689778 | 0.133673095 | 0.875455428 | Running case PrefixAnonMap Running python .. Done Running cython .. Done | python | cython | cy / py | test_apply_non_present| 0.358715744 | 0.335245703 | 0.934572034 | test_apply_present | 0.354434996 | 0.338579782 | 0.955266229 | Change-Id: I0d3f1dd285c044afc234479141d831b2ee0455be
* Warn when caching is disabled / documentMike Bayer2021-12-061-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds new warnings for all elements that don't indicate their caching behavior, including user-defined ClauseElement subclasses and third party dialects. it additionally adds new documentation to discuss an apparent performance degradation in 1.4 when caching is disabled as a result in the significant expense incurred by ORM lazy loaders, which in 1.3 used BakedQuery so were actually cached. As a result of adding the warnings, a fair degree of lesser used SQL expression objects identified that they did not define caching behavior so would have been producing ``[no key]``, including PostgreSQL constructs ``hstore`` and ``array``. These have been amended to use inherit cache where appropriate. "on conflict" constructs in PostgreSQL, MySQL, SQLite still explicitly don't generate a cache key at this time. The change also adds a test for all constructs via assert_compile() to assert they will not generate cache warnings. Fixes: #7394 Change-Id: I85958affbb99bfad0f5efa21bc8f2a95e7e46981
* Removals: MetaData.bind, Table.bind, all other .bindMike Bayer2021-12-021-54/+0
| | | | | Change-Id: I1ef2eb2018f4b68825fe40a2a8d99084cf217b35 References: #7257
* Clean up most py3k compatFederico Caselli2021-11-241-11/+13
| | | | Change-Id: I8172fdcc3103ff92aa049827728484c8779af6b7
* Remove object in class definitionFederico Caselli2021-11-221-5/+5
| | | | | References: #4600 Change-Id: I2a62ddfe00bc562720f0eae700a497495d7a987a
* fully implement future engine and remove legacyMike Bayer2021-11-071-20/+89
| | | | | | | | | | | | | | | | | | | | | | | | | | | The major action here is to lift and move future.Connection and future.Engine fully into sqlalchemy.engine.base. This removes lots of engine concepts, including: * autocommit * Connection running without a transaction, autobegin is now present in all cases * most "autorollback" is obsolete * Core-level subtransactions (i.e. MarkerTransaction) * "branched" connections, copies of connections * execution_options() returns self, not a new connection * old argument formats, distill_params(), simplifies calling scheme between engine methods * before/after_execute() events (oriented towards compiled constructs) don't emit for exec_driver_sql(). before/after_cursor_execute() is still included for this * old helper methods superseded by context managers, connection.transaction(), engine.transaction() engine.run_callable() * ancient engine-level reflection methods has_table(), table_names() * sqlalchemy.testing.engines.proxying_engine References: #7257 Change-Id: Ib20ed816642d873b84221378a9ec34480e01e82c
* 2.0 removals: LegacyRow, connectionless execution, close_with_resultMike Bayer2021-10-311-40/+0
| | | | | | | | | | | | | | | | | | | in order to remove LegacyRow / LegacyResult, we have to also lose close_with_result, which connectionless execution relies upon. also includes a new profiles.txt file that's all against py310, as that's what CI is on now. some result counts changed by one function call which was enough to fail the low-count result tests. Replaces Connectable as the common interface between Connection and Engine with EngineEventsTarget. Engine is no longer Connectable. Connection and MockConnection still are. References: #7257 Change-Id: Iad5eba0313836d347e65490349a22b061356896a
* process bulk_update_tuples before cache key or compilationMike Bayer2021-10-191-6/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed regression where the use of a :class:`_orm.hybrid_property` attribute or a mapped :func:`_orm.composite` attribute as a key passed to the :meth:`_dml.Update.values` method for an ORM-enabled :class:`_dml.Update` statement, as well as when using it via the legacy :meth:`_orm.Query.update` method, would be processed for incoming ORM/hybrid/composite values within the compilation stage of the UPDATE statement, which meant that in those cases where caching occurred, subsequent invocations of the same statement would no longer receive the correct values. This would include not only hybrids that use the :meth:`_orm.hybrid_property.update_expression` method, but any use of a plain hybrid attribute as well. For composites, the issue instead caused a non-repeatable cache key to be generated, which would break caching and could fill up the statement cache with repeated statements. The :class:`_dml.Update` construct now handles the processing of key/value pairs passed to :meth:`_dml.Update.values` and :meth:`_dml.Update.ordered_values` up front when the construct is first generated, before the cache key has been generated so that the key/value pairs are processed each time, and so that the cache key is generated against the individual column/value pairs that will ultimately be used in the statement. Fixes: #7209 Change-Id: I08f248d1d60ea9690b014c21439b775d951fb9e5
* fixes for usage of the null() and similar constantsMike Bayer2021-10-081-7/+13
| | | | | | | | | | | | | | | | | | | | | | | | Adjusted the "column disambiguation" logic that's new in 1.4, where the same expression repeated gets an "extra anonymous" label, so that the logic more aggressively deduplicates those labels when the repeated element is the same Python expression object each time, as occurs in cases like when using "singleton" values like :func:`_sql.null`. This is based on the observation that at least some databases (e.g. MySQL, but not SQLite) will raise an error if the same label is repeated inside of a subquery. Related to :ticket:`7153`, fixed an issue where result column lookups would fail for "adapted" SELECT statements that selected for "constant" value expressions most typically the NULL expression, as would occur in such places as joined eager loading in conjunction with limit/offset. This was overall a regression due to issue :ticket:`6259` which removed all "adaption" for constants like NULL, "true", and "false", but this broke the case where the same adaption logic were used to match the constant to a labeled expression referring to the constant in a subquery. Fixes: #7153 Fixes: #7154 Change-Id: I43823343721b9e70524ea3f5e8f39dd543a3e92b
* coerce for multivalues keysMike Bayer2021-09-211-1/+5
| | | | | | | | | Fixed issue where using ORM column expressions as keys in the list of dictionaries passed to :meth:`_sql.Insert.values` for "multi-valued insert" would not be processed correctly into the correct column expressions. Fixes: #7060 Change-Id: I1c4c286c33ea6eeaafba617996828f5c88ff0a1c
* Replace all http:// links to https://Federico Caselli2021-07-041-1/+1
| | | | | | Also replace http://pypi.python.org/pypi with https://pypi.org/project Change-Id: I84b5005c39969a82140706472989f2a30b0c7685
* document missing ColumnCollection methodsMike Bayer2021-06-131-25/+86
| | | | | | | | add additional notes regarding the methods that are on this object. Fixes: #6629 Change-Id: I68446cb16cbfcef9af9c2998687b3b536607cf0d
* Remove pep484 type comments from the codeFederico Caselli2021-05-161-5/+3
| | | | | | | | | | | | | Current effort is around the stub package, and having typing in two places makes thing worse, since the types here are usually outdated compared to the version in the stubs. Once v2 gets under way we can start consolidating the types here. Fixes: #6461 Change-Id: I7132a444bd7138123074bf5bc664b4bb119a85ce
* Correct cache key for proxy_owner, with_context_optionsMike Bayer2021-05-101-3/+6
| | | | | | | | | | | | | | | | | | Fixed issue in subquery loader strategy which prevented caching from working correctly. This would have been seen in the logs as a "generated" message instead of "cached" for all subqueryload SQL emitted, which by saturating the cache with new keys would degrade overall performance; it also would produce "LRU size alert" warnings. In this issue we also observe that the local LRU cache for lazyloader and selectinloader will get used for all subsequent loads as well, which makes it more likely to hit the limit of 30. However rather than trying to work this out, it would be better if we removed the loader-local LRU caches altogether once we are confident these are working well. Fixes: #6459 Change-Id: Id953e8f75536bb87f7e3315929cebcd8f84a5a50
* Support filter_by() from columns, functions, Core SQLMike Bayer2021-05-011-1/+19
| | | | | | | | | | | | Fixed regression where :meth:`_orm.Query.filter_by` would not work if the lead entity were a SQL function or other expression derived from the primary entity in question, rather than a simple entity or column of that entity. Additionally, improved the behavior of :meth:`_sql.Select.filter_by` overall to work with column expressions even in a non-ORM context. Fixes: #6414 Change-Id: I316b5bf98293bec1ede08787f6181dd14be85419
* Fix with_expression() cache leak; don't adapt singletonsMike Bayer2021-04-141-0/+9
| | | | | | | | | | | | | | | | | | | | Fixed a cache leak involving the :func:`_orm.with_expression` loader option, where the given SQL expression would not be correctly considered as part of the cache key. Additionally, fixed regression involving the corresponding :func:`_orm.query_expression` feature. While the bug technically exists in 1.3 as well, it was not exposed until 1.4. The "default expr" value of ``null()`` would be rendered when not needed, and additionally was also not adapted correctly when the ORM rewrites statements such as when using joined eager loading. The fix ensures "singleton" expressions like ``NULL`` and ``true`` aren't "adapted" to refer to columns in ORM statements, and additionally ensures that a :func:`_orm.query_expression` with no default expression doesn't render in the statement if a :func:`_orm.with_expression` isn't used. Fixes: #6259 Change-Id: I5a70bc12dadad125bbc4324b64048c8d4a18916c
* Disallow AliasedReturnsRows from executionMike Bayer2021-04-051-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Executing a :class:`_sql.Subquery` using :meth:`_engine.Connection.execute` is deprecated and will emit a deprecation warning; this use case was an oversight that should have been removed from 1.4. The operation will now execute the underlying :class:`_sql.Select` object directly for backwards compatibility. Similarly, the :class:`_sql.CTE` class is also not appropriate for execution. In 1.3, attempting to execute a CTE would result in an invalid "blank" SQL statement being executed; since this use case was not working it now raises :class:`_exc.ObjectNotExecutableError`. Previously, 1.4 was attempting to execute the CTE as a statement however it was working only erratically. The change also breaks out StatementRole from ReturnsRowsRole, as these roles should not be in the same lineage (some statements don't return rows, the whole class of ReturnsRows that are from clauses are not statements). Consolidate StatementRole and CoerceTextStatementRole as there's no usage difference between these. Simplify some old tests that were trying to make sure that "execution options" didn't transmit from a cte/subquery out to a select; as cte/subuqery() aren't executable in any case the options are removed. Fixes: #6204 Change-Id: I62613b7ab418afdd22c409eae75659e3f52fb65f
* Adapt loader_criteria params for current queryMike Bayer2021-03-261-3/+3
| | | | | | | | | | | | | | | | | | Fixed critical issue in the new :meth:`_orm.PropComparator.and_` feature where loader strategies that emit secondary SELECT statements such as :func:`_orm.selectinload` and :func:`_orm.lazyload` would fail to accommodate for bound parameters in the user-defined criteria in terms of the current statement being executed, as opposed to the cached statement, causing stale bound values to be used. This also adds a warning for the case where an object that uses :func:`_orm.lazyload` in conjunction with :meth:`_orm.PropComparator.and_` is attempted to be serialized; the loader criteria cannot reliably be serialized and deserialized and eager loading should be used for this case. Fixes: #6139 Change-Id: I5a638bbecb7b583db2d3c0b76469f5a25c13dd3b
* Added back ``items`` and ``values`` to ``ColumnCollection`` class.Federico Caselli2021-03-171-0/+6
| | | | | Fixes: #6068 Change-Id: Idb42c864e17c02d7b89cffa667dcf853ec93e5c2
* Implement support for functions as FROM with columns clause supportMike Bayer2021-02-031-6/+4
| | | | | | | | | | | | | | | | Implemented support for "table valued functions" along with additional syntaxes supported by PostgreSQL, one of the most commonly requested features. Table valued functions are SQL functions that return lists of values or rows, and are prevalent in PostgreSQL in the area of JSON functions, where the "table value" is commonly referred towards as the "record" datatype. Table valued functions are also supported by Oracle and SQL Server. Moved from I5b093b72533ef695293e737eb75850b9713e5e03 due to accidental push Fixes: #3566 Change-Id: Iea36d04c80a5ed3509dcdd9ebf0701687143fef5
* Fix many spell glitches in docstrings and commentsLele Gaifax2021-01-241-2/+2
| | | | | | | | | | These were revealed by running `pylint --disable all --enable spelling --spelling-dict en_US` over all sources. Closes: #5868 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5868 Pull-request-sha: bb249195d92e3b806e81ecf1192d5a1b3cd5db48 Change-Id: I96080ec93a9fbd20ce21e9e16265b3c77f22bb14
* introduce generalized decorator to prevent invalid method callsMike Bayer2021-01-161-0/+25
| | | | | | | | | | | | | | | | | | | This introduces the ``_exclusive_against()`` utility decorator that can be used to prevent repeated invocations of methods that typically should only be called once. An informative error message is now raised for a selected set of DML methods (currently all part of :class:`_dml.Insert` constructs) if they are called a second time, which would implicitly cancel out the previous setting. The methods altered include: :class:`_sqlite.Insert.on_conflict_do_update`, :class:`_sqlite.Insert.on_conflict_do_nothing` (SQLite), :class:`_postgresql.Insert.on_conflict_do_update`, :class:`_postgresql.Insert.on_conflict_do_nothing` (PostgreSQL), :class:`_mysql.Insert.on_duplicate_key_update` (MySQL) Fixes: #5169 Change-Id: I9278fa87cd3470dcf296ff96bb0fb17a3236d49d
* allow Executable to be accepted by Session.execute()Mike Bayer2021-01-141-1/+1
| | | | | | | | | | | | Fixed an issue where the API to create a custom executable SQL construct using the ``sqlalchemy.ext.compiles`` extension according to the documentation that's been up for many years would no longer function if only ``Executable, ClauseElement`` were used as the base classes, additional classes were needed if wanting to use :meth:`_orm.Session.execute`. This has been resolved so that those extra classes aren't needed. Change-Id: I99b8acd88515c2a52842d62974199121e64c0381
* happy new yearMike Bayer2021-01-041-1/+1
| | | | Change-Id: Ic5bb19ca8be3cb47c95a0d3315d84cb484bac47c
* Fixes for lambda expressions and relationship loadersMike Bayer2020-12-111-0/+20
| | | | | | | | | | | | | | | | | | Fixed bug in lambda SQL feature, used by ORM :meth:`_orm.with_loader_criteria` as well as available generally in the SQL expression language, where assigning a boolean value True/False to a variable would cause the query-time expression calculation to fail, as it would produce a SQL expression not compatible with a bound value. Fixed issue where the :attr:`_orm.ORMExecuteState.is_relationship_load` parameter would not be set correctly for many lazy loads, all selectinloads, etc. The flag is essential in order to test if options should be added to statements or if they would already have been propagated via relationship loads. Fixes: #5763 Fixes: #5764 Change-Id: I66aafbef193f892ff75ede0670698647b7475482
* Don't emit warnings on descriptor accessMike Bayer2020-11-201-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit is revising 5162f2bc5fc0ac239f26a76fc9f0c2, which when I did it felt a little rushed but I couldn't find anything wrong. Well here we are :). Fixed issue where a :class:`.RemovedIn20Warning` would erroneously emit when the ``.bind`` attribute were accessed internally on objects, particularly when stringifying a SQL construct. Alter the deprecated() decorator so that we can use it just to add docstring warnings but not actually warn when the function is accessed, adding new argument enable_warnings that can be set to False. Added a safety feature to deprecated_20() that will disallow an ":attr:" from proceeding if enable_warnings=False isn't present, unless there's an extra flag warn_on_attribute_access, since we want Session.transaction to emit a deprecation warning. This is a little hacky but it's essentially modifying the decorator to require a positive assertion that a deprecation decorator on a descriptor should actually warn on access. Remove the warning filter for session.transaction and get tests to pass to ensure this is not also being called internally. Added tests to ensure that common places .bind can be passed as a parameter definitely warn as I was not able to find this otherwise. Fixes: #5717 Change-Id: Ia586b4f9ee6b212f3a71104b1caf40b5edd399e2
* Convert to autoload_with internallyMike Bayer2020-11-071-1/+3
| | | | | | | | | Fixed bug where the now-deprecated ``autoload`` parameter was being called internally within the reflection routines when a related table were reflected. Fixes: #5684 Change-Id: I6ab439a2f49ff1ae2d3c7a15b531cbafbc3cf594
* Add deprecation for base Executable.bindMike Bayer2020-10-161-0/+4
| | | | | | | | | | These attributes will be removed in SQLAlchemy 2.0. Also alters the deprecation message to qualify the type of object correctly. this in turn requires changes in the warnings filter and deprecation tests. Change-Id: I5779d9813e88f42e5db0c7b5e3ffff1d1535c203
* Merge "Deprecate bound metadata"mike bayer2020-10-121-0/+5
|\
| * Deprecate bound metadataMike Bayer2020-10-121-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The :paramref:`_schema.MetaData.bind` argument as well as the overall concept of "bound metadata" is deprecated in SQLAlchemy 1.4 and will be removed in SQLAlchemy 2.0. The parameter as well as related functions now emit a :class:`_exc.RemovedIn20Warning` when :ref:`deprecation_20_mode` is in use. Added new parameter :paramref:`_automap.AutomapBase.prepare.autoload_with` which supersedes :paramref:`_automap.AutomapBase.prepare.reflect` and :paramref:`_automap.AutomapBase.prepare.engine`. Fixes: #4634 Fixes: #5142 Change-Id: Iaabf9b481931e2fb68b97b5954c32e65772a298e
* | Deprecate duplicated column names in Table definitionMike Bayer2020-10-121-4/+14
|/ | | | | | | | | | | | | | The :class:`_schema.Table` class now raises a deprecation warning when columns with the same name are defined. To replace a column a new parameter :paramref:`_schema.Table.append_column.replace_existing` was added to the :meth:`_schema.Table.append_column` method. The :meth:`_expression.ColumnCollection.contains_column` will now raises an error when called with a string, suggesting the caller to use ``in`` instead. Co-authored-by: Federico Caselli <cfederico87@gmail.com> Change-Id: I1d58c8ebe081079cb669e7ead60886ffc1b1a7f5
* upgrade to black 20.8b1Mike Bayer2020-09-281-21/+14
| | | | | | | It's better, the majority of these changes look more readable to me. also found some docstrings that had formatting / quoting issues. Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
* Update select usage to use the new 1.4 formatFederico Caselli2020-09-081-1/+1
| | | | | | | | | | | | | | | | This change includes mainly that the bracketed use within select() is moved to positional, and keyword arguments are removed from calls to the select() function. it does not yet fully address other issues such as keyword arguments passed to the table.select(). Additionally, allows False / None to both be considered as "disable" for all of select.correlate(), select.correlate_except(), query.correlate(), which establishes consistency with passing of ``False`` for the legact select(correlate=False) argument. Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
* Convert lazy loader, selectinload, load_on_ident to lambda statementsMike Bayer2020-08-051-1/+39
| | | | | | | | | Building on newly robust lambdas in I29a513c98917b1d503abfdd61e6b6e8800851aa8, convert key loading off of the "baked" system so that baked is no longer used by the ORM. Change-Id: I3abfb45dd6e50f84f29d39434caa0b550ce27864
* Convert remaining ORM APIs to support 2.0 styleMike Bayer2020-07-111-0/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is kind of a mixed bag of all kinds to help get us to 1.4 betas. The documentation stuff is a work in progress. Lots of other relatively small changes to APIs and things. More commits will follow to continue improving the documentation and transitioning to the 1.4/2.0 hybrid documentation. In particular some refinements to Session usage models so that it can match Engine's scoping / transactional patterns, and a decision to start moving away from "subtransactions" completely. * add select().from_statement() to produce FromStatement in an ORM context * begin referring to select() that has "plugins" for the few edge cases where select() will have ORM-only behaviors * convert dynamic.AppenderQuery to its own object that can use select(), though at the moment it uses Query to support legacy join calling forms. * custom query classes for AppenderQuery are replaced by do_orm_execute() hooks for custom actions, a separate gerrit will document this * add Session.get() to replace query.get() * Deprecate session.begin->subtransaction. propose within the test suite a hypothetical recipe for apps that rely on this pattern * introduce Session construction level context manager, sessionmaker context manager, rewrite the whole top of the session_transaction.rst documentation. Establish context manager patterns for Session that are identical to engine * ensure same begin_nested() / commit() behavior as engine * devise all new "join into an external transaction" recipe, add test support for it, add rules into Session so it just works, write new docs. need to ensure this doesn't break anything * vastly reduce the verbosity of lots of session docs as I dont think people read this stuff and it's difficult to keep current in any case * constructs like case(), with_only_columns() really need to move to *columns, add a coercion rule to just change these. * docs need changes everywhere I look. in_() is not in the Core tutorial? how do people even know about it? Remove tons of cruft from Select docs, etc. * build a system for common ORM options like populate_existing and autoflush to populate from execution options. * others? Change-Id: Ia4bea0f804250e54d90b3884cf8aab8b66b82ecf
* Add future=True to create_engine/Session; unify select()Mike Bayer2020-07-081-0/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several weeks of using the future_select() construct has led to the proposal there be just one select() construct again which features the new join() method, and otherwise accepts both the 1.x and 2.x argument styles. This would make migration simpler and reduce confusion. However, confusion may be increased by the fact that select().join() is different Current thinking is we may be better off with a few hard behavioral changes to old and relatively unknown APIs rather than trying to play both sides within two extremely similar but subtly different APIs. At the moment, the .join() thing seems to be the only behavioral change that occurs without the user taking any explicit steps. Session.execute() will still behave the old way as we are adding a future flag. This change also adds the "future" flag to Session() and session.execute(), so that interpretation of the incoming statement, as well as that the new style result is returned, does not occur for existing applications unless they add the use of this flag. The change in general is moving the "removed in 2.0" system further along where we want the test suite to fully pass even if the SQLALCHEMY_WARN_20 flag is set. Get many tests to pass when SQLALCHEMY_WARN_20 is set; this should be ongoing after this patch merges. Improve the RemovedIn20 warning; these are all deprecated "since" 1.4, so ensure that's what the messages read. Make sure the inforamtion link is on all warnings. Add deprecation warnings for parameters present and add warnings to all FromClause.select() types of methods. Fixes: #5379 Fixes: #5284 Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51 References: #5159
* introduce deferred lambdasMike Bayer2020-07-031-6/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The coercions system allows us to add in lambdas as arguments to Core and ORM elements without changing them at all. By allowing the lambda to produce a deterministic cache key where we can also cheat and yank out literal parameters means we can move towards having 90% of "baked" functionality in a clearer way right in Core / ORM. As a second step, we can have whole statements inside the lambda, and can then add generation with __add__(), so then we have 100% of "baked" functionality with full support of ad-hoc literal values. Adds some more short_selects tests for the moment for comparison. Other tweaks inside cache key generation as we're trying to approach a certain level of performance such that we can remove the use of "baked" from the loader strategies. As we have not yet closed #4639, however the caching feature has been fully integrated as of b0cfa7379cf8513a821a3dbe3028c4965d9f85bd, we will also add complete caching documentation here and close that issue as well. Closes: #4639 Fixes: #5380 Change-Id: If91f61527236fd4d7ae3cad1f24c38be921c90ba
* Fix a wide variety of typos and broken linksaplatkouski2020-06-251-9/+9
| | | | | | | | | | | | Note the PR has a few remaining doc linking issues listed in the comment that must be addressed separately. Signed-off-by: aplatkouski <5857672+aplatkouski@users.noreply.github.com> Closes: #5371 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5371 Pull-request-sha: 7e7d233cf3a0c66980c27db0fcdb3c7d93bc2510 Change-Id: I9c36e8d8804483950db4b42c38ee456e384c59e3
* perf tweaksMike Bayer2020-06-191-2/+8
| | | | | | | | | | | | | | - avoid abc checks in distill_20 - ColumnEntity subclasses are unique to their compile state and have no querycontext specific state. They can do a simple memoize of their fetch_column without using attributes, and they can memoize their _getter() too so that it goes into the cache, just like instance_processor() does. - unify ORMColumnEntity and RawColumnEntity for the row processor part, add some test coverage for the case where it is used in a from_statement - do a faster generate if there are no memoized entries - query._params is always immutabledict Change-Id: I1e2dfe607a1749b5b434fc11f9348ee631501dfa
* Turn on caching everywhere, add loggingMike Bayer2020-06-101-6/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* Convert bulk update/delete to new execution modelMike Bayer2020-06-061-3/+7
| | | | | | | | | | | | | | | This reorganizes the BulkUD model in sqlalchemy.orm.persistence to be based on the CompileState concept and to allow plain update() / delete() to be passed to session.execute() where the ORM synchronize session logic will take place. Also gets "synchronize_session='fetch'" working with horizontal sharding. Adding a few more result.scalar_one() types of methods as scalar_one() seems like what is normally desired. Fixes: #5160 Change-Id: I8001ebdad089da34119eb459709731ba6c0ba975
* Improve rendering of core statements w/ ORM elementsMike Bayer2020-05-311-1/+31
| | | | | | | | | | | | | | | | | | | | This patch contains a variety of ORM and expression layer tweaks to support ORM constructs in select() statements, without the 1.3.x requiremnt in Query that a full _compile_context() + new select() is needed in order to get a working statement object. Includes such tweaks as the ability to implement aliased class of an aliased class, as we are looking to fully support ACs against subqueries, as well as the ability to access anonymously-labeled ColumnProperty expressions within subqueries by naming the ".key" of the label after the property key. Some tuning to query.join() as well as ORMJoin internals to allow things to work more smoothly. Change-Id: Id810f485c5f7ed971529489b84694e02a3356d6d
* callcount reductions and refinement for cached queriesMike Bayer2020-05-281-10/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit includes that we've removed the "_orm_query" attribute from compile state as well as query context. The attribute created reference cycles and also added method call overhead. As part of this change, the interface for ORMExecuteState changes a bit, as well as the interface for the horizontal sharding extension which now deprecates the "query_chooser" callable in favor of "execute_chooser", which receives the contextual object. This will also work more nicely when we implement the new execution path for bulk updates and deletes. Pre-merge execution options for statement, connection, arguments all up front in Connection. that way they can be passed to the before_execute / after_execute events, and the ExecutionContext doesn't have to merge as second time. Core execute is pretty close to 1.3 now. baked wasn't using the new one()/first()/one_or_none() methods, fixed that. Convert non-buffered cursor strategy to be a stateless singleton. inline all the paths by which the strategy gets chosen, oracle and SQL Server dialects make use of the already-invoked post_exec() hook to establish the alternate strategies, and this is actually much nicer than it was before. Add caching to mapper instance processor for getters. Identified a reference cycle per query that was showing up as a lot of gc cleanup, fixed that. After all that, performance not budging much. Even test_baked_query now runs with significantly fewer function calls than 1.3, still 40% slower. Basically something about the new patterns just makes this slower and while I've walked a whole bunch of them back, it hardly makes a dent. that said, the performance issues are relatively small, in the 20-40% time increase range, and the new caching feature does provide for regular ORM and Core queries that are cached, and they are faster than non-cached. Change-Id: I7b0b0d8ca550c05f79e82f75cd8eff0bbfade053
* Convert execution to move through SessionMike Bayer2020-05-251-30/+50
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-241-8/+120
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* Create initial 2.0 engine implementationMike Bayer2020-04-161-0/+11
| | | | | | | | | | | | | | | | | | | Implemented the SQLAlchemy 2 :func:`.future.create_engine` function which is used for forwards compatibility with SQLAlchemy 2. This engine features always-transactional behavior with autobegin. Allow execution options per statement execution. This includes that the before_execute() and after_execute() events now accept an additional dictionary with these options, empty if not passed; a legacy event decorator is added for backwards compatibility which now also emits a deprecation warning. Add some basic tests for execution, transactions, and the new result object. Build out on a new testing fixture that swaps in the future engine completely to start with. Change-Id: I70e7338bb3f0ce22d2f702537d94bb249bd9fb0a Fixes: #4644
* Set up absolute references for create_engine and relatedMike Bayer2020-04-141-1/+2
| | | | | | | includes more replacements for create_engine(), Connection, disambiguation of Result from future/baked Change-Id: Icb60a79ee7a6c45ea9056c211ffd1be110da3b5e
* Run search and replace of symbolic module namesMike Bayer2020-04-141-31/+52
| | | | | | | | Replaces a wide array of Sphinx-relative doc references with an abbreviated absolute form now supported by zzzeeksphinx. Change-Id: I94bffcc3f37885ffdde6238767224296339698a2
* Fix almost all read-level sphinx warningsMike Bayer2020-04-111-1/+3
| | | | | | | | There are some related to changelog that I can't figure out and are likely due to something in the changelog extension. also one thing with a "collection" I can't figure out. Change-Id: I0a9e6f4291c3589aa19a4abcb9245cd22a266fe0
* Try to measure new style caching in the ORM, take twoMike Bayer2020-04-011-14/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf Trying to see if by making the cache key memoized, we still can have the older "identity" form of caching which is the cheapest of all, at the same time as the newer "cache key each time" version that is not nearly as cheap; but still much cheaper than no caching at all. Also needed is a per-execution update of _keymap when we invoke from a cached select, so that Column objects that are anonymous or otherwise adapted will match up. this is analogous to the adaption of bound parameters from the cache key. Adds test coverage for the keymap / construct_params() changes related to caching. Also hones performance to a large extent for statement construction and cache key generation. Also includes a new memoized attribute approach that vastly simplifies the previous approach of "group_expirable_memoized_property" and finally integrates cleanly with _clone(), _generate(), etc. no more hardcoding of attributes is needed, as well as that most _reset_memoization() calls are no longer needed as the reset is inherent in a _generate() call; this also has dramatic performance improvements. Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
View Lorry Controller Status