summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/sql/elements.py
Commit message (Collapse)AuthorAgeFilesLines
* Ensure bindparam key escaping applied in all casesMike Bayer2021-04-121-5/+11
| | | | | | | | | | | Fixed regression where the :class:`_sql.BindParameter` object would not properly render for an IN expression (i.e. using the "post compile" feature in 1.4) if the object were copied from either an internal cloning operation, or from a pickle operation, and the parameter name contained spaces or other special characters. Fixes: #6249 Change-Id: Icd0d4096c8fa4eb1a1d4c20f8a96d8b1ae439f0a
* Infer types in BindParameter when expanding=TrueMike Bayer2021-04-081-2/+13
| | | | | | | | | | | | | Enhanced the "expanding" feature used for :meth:`_sql.ColumnOperators.in_` operations to infer the type of expression from the right hand list of elements, if the left hand side does not have any explicit type set up. This allows the expression to support stringification among other things. In 1.3, "expanding" was not automatically used for :meth:`_sql.ColumnOperators.in_` expressions, so in that sense this change fixes a behavioral regression. Fixes: #6222 Change-Id: Icdfda1e2c226a21896cafd6d8f251547794451c2
* Disallow AliasedReturnsRows from executionMike Bayer2021-04-051-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Executing a :class:`_sql.Subquery` using :meth:`_engine.Connection.execute` is deprecated and will emit a deprecation warning; this use case was an oversight that should have been removed from 1.4. The operation will now execute the underlying :class:`_sql.Select` object directly for backwards compatibility. Similarly, the :class:`_sql.CTE` class is also not appropriate for execution. In 1.3, attempting to execute a CTE would result in an invalid "blank" SQL statement being executed; since this use case was not working it now raises :class:`_exc.ObjectNotExecutableError`. Previously, 1.4 was attempting to execute the CTE as a statement however it was working only erratically. The change also breaks out StatementRole from ReturnsRowsRole, as these roles should not be in the same lineage (some statements don't return rows, the whole class of ReturnsRows that are from clauses are not statements). Consolidate StatementRole and CoerceTextStatementRole as there's no usage difference between these. Simplify some old tests that were trying to make sure that "execution options" didn't transmit from a cte/subquery out to a select; as cte/subuqery() aren't executable in any case the options are removed. Fixes: #6204 Change-Id: I62613b7ab418afdd22c409eae75659e3f52fb65f
* Ensure Grouping._with_binary_element_type() maintains classMike Bayer2021-04-021-1/+1
| | | | | | | | | | Fixed regression where use of the :meth:`.Operators.in_` method with a :class:`_sql.Select` object against a non-table-bound column would produce an ``AttributeError``, or more generally using a :class:`_sql.ScalarSelect` that has no datatype in a binary expression would produce invalid state. Fixes: #6181 Change-Id: I1ddea433b3603fdab8f489bff571b512a6ffc66b
* Default caching to opt-out for 3rd party dialectsMike Bayer2021-04-011-6/+36
| | | | | | | | | | | | | | | | | | | Added a new flag to the :class:`_engine.Dialect` class called :attr:`_engine.Dialect.supports_statement_cache`. This flag now needs to be present directly on a dialect class in order for SQLAlchemy's :ref:`query cache <sql_caching>` to take effect for that dialect. The rationale is based on discovered issues such as :ticket:`6173` revealing that dialects which hardcode literal values from the compiled statement, often the numerical parameters used for LIMIT / OFFSET, will not be compatible with caching until these dialects are revised to use the parameters present in the statement only. For third party dialects where this flag is not applied, the SQL logging will show the message "dialect does not support caching", indicating the dialect should seek to apply this flag once they have verified that no per-statement literal values are being rendered within the compilation phase. Fixes: #6184 Change-Id: I6fd5b5d94200458d4cb0e14f2f556dbc25e27e22
* Adapt loader_criteria params for current queryMike Bayer2021-03-261-4/+6
| | | | | | | | | | | | | | | | | | Fixed critical issue in the new :meth:`_orm.PropComparator.and_` feature where loader strategies that emit secondary SELECT statements such as :func:`_orm.selectinload` and :func:`_orm.lazyload` would fail to accommodate for bound parameters in the user-defined criteria in terms of the current statement being executed, as opposed to the cached statement, causing stale bound values to be used. This also adds a warning for the case where an object that uses :func:`_orm.lazyload` in conjunction with :meth:`_orm.PropComparator.and_` is attempted to be serialized; the loader criteria cannot reliably be serialized and deserialized and eager loading should be used for this case. Fixes: #6139 Change-Id: I5a638bbecb7b583db2d3c0b76469f5a25c13dd3b
* Accept **kw in annotated._clone() methodMike Bayer2021-03-251-1/+2
| | | | | | | | | | Fixed bug where combinations of the new "relationship with criteria" feature could fail in conjunction with features that make use of the new "lambda SQL" feature, including loader strategies such as selectinload and lazyload, for more complicated scenarios such as polymorphic loading. Fixes: #6131 Change-Id: I915dead6596866ae5fd1a7f593a90bce4b61d1af
* open up window functions (ha ha) for mysql / mariadbMike Bayer2021-03-241-2/+1
| | | | | | | MySQL 8 and MariaDB 10.2 support window functions so this should be turned on. Change-Id: Ic1fdd328f1f0def7dfc8451385c16ea81902074c
* Correct for coercion from list args to positional for caseMike Bayer2021-03-191-2/+2
| | | | | | | | | Fixed regression in the :func:`_sql.case` construct, where the "dictionary" form of argument specification failed to work correctly if it were passed positionally, rather than as a "whens" keyword argument. Fixes: #6097 Change-Id: I4138f54309a08c8e4e63cfafc211176e0b9a76c7
* Raise at Core / ORM concrete inh level for label overlapMike Bayer2021-03-181-6/+15
| | | | | | | | | | | | | | | | | | | Fixed regression where the :class:`.ConcreteBase` would fail to map at all when a mapped column name overlapped with the discriminator column name, producing an assertion error. The use case here did not function correctly in 1.3 as the polymorphic union would produce a query that ignored the discriminator column entirely, while emitting duplicate column warnings. As 1.4's architecture cannot easily reproduce this essentially broken behavior of 1.3 at the ``select()`` level right now, the use case now raises an informative error message instructing the user to use the ``.ConcreteBase._concrete_discriminator_name`` attribute to resolve the conflict. To assist with this configuration, ``.ConcreteBase._concrete_discriminator_name`` may be placed on the base class only where it will be automatically used by subclasses; previously this was not the case. Fixes: #6090 Change-Id: I8b7d01e4c9ea0dc97f30b8cd658b3505b24312a7
* Use explicit names for mapper _get_clause parametersMike Bayer2021-03-171-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | Fixed a critical regression in the relationship lazy loader where the SQL criteria used to fetch a related many-to-one object could go stale in relation to other memoized structures within the loader if the mapper had configuration changes, such as can occur when mappers are late configured or configured on demand, producing a comparison to None and returning no object. Huge thanks to Alan Hamlett for their help tracking this down late into the night. The primary change is that mapper._get_clause() uses a fixed name for its bound parameters, which is memoized under a lambda statement in the case of many-to-one lazy loading. This has implications for some other logic namely the .compare() used by loader strategies to determine use_get needed to be adjusted. This change also repairs the lambda module's behavior of removing the "required" flag from bound parameters, which caused this issue to also fail silently rather than issuing an error for a required bind parameter. Fixes: #6055 Change-Id: I19e1aba9207a049873e0f13c19bad7541e223cfd
* Fix named CHECK constraint name omitted on repeated createsGord Thompson2021-03-061-27/+2
| | | | | | | | | | | | | Fixed issue where the CHECK constraint generated by :class:`_types.Boolean` or :class:`_types.Enum` would fail to render the naming convention correctly after the first compilation, due to an unintended change of state within the name given to the constraint. This issue was first introduced in 0.9 in the fix for issue #3067, and the fix revises the approach taken at that time which appears to have been more involved than what was needed. Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #6007 Change-Id: I7ecff0a9d86191520f6841b3922a5af5a6971fba
* Further refine labeling for renamed columnsMike Bayer2021-02-121-8/+28
| | | | | | | | | | | | | | | Forked from I22f6cf0f0b3360e55299cdcb2452cead2b2458ea we are attempting to decide the case for columns mapped under a different name. since the .key feature of Column seems to support this fully, see if an annotation can be used to indicate an effective .key for a column. The effective change is that the labeling of column expressions in rows has been improved to retain the original name of the ORM attribute even if used in a subquery. References: #5933 Change-Id: If251f556f7d723f50d349f765f1690d6c679d2ef
* Apply consistent labeling for all future style ORM queriesMike Bayer2021-02-111-0/+3
| | | | | | | | | | | | | | | | Fixed issue in new 1.4/2.0 style ORM queries where a statement-level label style would not be preserved in the keys used by result rows; this has been applied to all combinations of Core/ORM columns / session vs. connection etc. so that the linkage from statement to result row is the same in all cases. also repairs a cache key bug where query.from_statement() vs. select().from_statement() would not be disambiguated; the compile options were not included in the cache key for FromStatement. Fixes: #5933 Change-Id: I22f6cf0f0b3360e55299cdcb2452cead2b2458ea
* Merge "Implement support for functions as FROM with columns clause support"mike bayer2021-02-031-52/+123
|\
| * Implement support for functions as FROM with columns clause supportMike Bayer2021-02-031-52/+123
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implemented support for "table valued functions" along with additional syntaxes supported by PostgreSQL, one of the most commonly requested features. Table valued functions are SQL functions that return lists of values or rows, and are prevalent in PostgreSQL in the area of JSON functions, where the "table value" is commonly referred towards as the "record" datatype. Table valued functions are also supported by Oracle and SQL Server. Moved from I5b093b72533ef695293e737eb75850b9713e5e03 due to accidental push Fixes: #3566 Change-Id: Iea36d04c80a5ed3509dcdd9ebf0701687143fef5
* | Add coercions to literal()Mike Bayer2021-02-031-1/+1
|/ | | | | | | | | | To prevent literal() from receiving a ClauseElement which then leads to confusing results, add a new LiteralValueRole coercion that does an _is_literal() check and implement for literal(). Fixes: #5639 Change-Id: Ibd686544af2d7c013765278f984baf237de88caf
* Fix many spell glitches in docstrings and commentsLele Gaifax2021-01-241-2/+2
| | | | | | | | | | These were revealed by running `pylint --disable all --enable spelling --spelling-dict en_US` over all sources. Closes: #5868 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5868 Pull-request-sha: bb249195d92e3b806e81ecf1192d5a1b3cd5db48 Change-Id: I96080ec93a9fbd20ce21e9e16265b3c77f22bb14
* Revert "Implement support for functions as FROM with columns clause support"Mike Bayer2021-01-211-136/+51
| | | | | | | This reverts commit 05a31f2708590161d4b3b4c7ff65196c99b4a22b. Atom has this little button called "push" and just pushes to master, I wasn't even *on* master. oops
* Implement support for functions as FROM with columns clause supportMike Bayer2021-01-201-51/+136
| | | | | | | WIP Fixes: #3566 Change-Id: I5b093b72533ef695293e737eb75850b9713e5e03
* allow Executable to be accepted by Session.execute()Mike Bayer2021-01-141-1/+0
| | | | | | | | | | | | Fixed an issue where the API to create a custom executable SQL construct using the ``sqlalchemy.ext.compiles`` extension according to the documentation that's been up for many years would no longer function if only ``Executable, ClauseElement`` were used as the base classes, additional classes were needed if wanting to use :meth:`_orm.Session.execute`. This has been resolved so that those extra classes aren't needed. Change-Id: I99b8acd88515c2a52842d62974199121e64c0381
* happy new yearMike Bayer2021-01-041-1/+1
| | | | Change-Id: Ic5bb19ca8be3cb47c95a0d3315d84cb484bac47c
* Improve type detection for Values / TupleMike Bayer2020-12-181-4/+21
| | | | | | | | | | | | | | | | | Fixed issue in new :class:`_sql.Values` construct where passing tuples of objects would fall back to per-value type detection rather than making use of the :class:`_schema.Column` objects passed directly to :class:`_sql.Values` that tells SQLAlchemy what the expected type is. This would lead to issues for objects such as enumerations and numpy strings that are not actually necessary since the expected type is given. note this changes NullType() to raise CompileError for literal_processor; NullType() does not imply the actual value NULL as much as it does "unknown type" so this should make failure modes more clear. Fixes: #5785 Change-Id: Ifbf5e78373102380b301098f30e15011efa98b5e
* Major revisals to lambdasMike Bayer2020-12-161-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Improve coercions._deep_is_literal to check sequences for clause elements, thus allowing a phrase like lambda: col.in_([literal("x"), literal("y")]) to be handled 2. revise closure variable caching completely. All variables entering must be part of a closure cache key or rejected. only objects that can be resolved to HasCacheKey or FunctionType are accepted; all other types are rejected. This adds a high degree of strictness to lambdas and will make them a little more awkward to use in some cases, however prevents several classes of critical issues: a. previously, a lambda that had an expression derived from some kind of state, like "self.x", or "execution_context.session.foo" would produce a closure cache key from "self" or "execution_context", objects that can very well be per-execution and would therefore cause a AnalyzedFunction objects to overflow. (memory won't leak as it looks like an LRUCache is already used for these) b. a lambda, such as one used within DeferredLamdaElement, that produces different SQL expressions based on the arguments (which is in fact what it's supposed to do), however it would through the use of conditionals produce different bound parameter combinations, leading to literal parameters not tracked properly. These are now rejected as uncacheable whereas previously they would again be part of the closure cache key, causing an overflow of AnalyizedFunction objects. 3. Ensure non-mapped mixins are handled correctly by with_loader_criteria(). 4. Fixed bug in lambda SQL system where we are not supposed to allow a Python function to be embedded in the lambda, since we can't predict a bound value from it. While there was an error condition added for this, it was not tested and wasn't working; an informative error is now raised. 5. new docs for lambdas 6. consolidated changelog for all of these Fixes: #5760 Fixes: #5765 Fixes: #5766 Fixes: #5768 Fixes: #5770 Change-Id: Iedaa636c3225fad496df23b612c516c8ab247ab7
* Allow dialect-specific stringificationMike Bayer2020-11-081-2/+12
| | | | | | | | | | | | | | | | | | Dialect-specific constructs such as :meth:`_postgresql.Insert.on_conflict_do_update` can now stringify in-place without the need to specify an explicit dialect object. The constructs, when called upon for ``str()``, ``print()``, etc. now have internal direction to call upon their appropriate dialect rather than the "default"dialect which doesn't know how to stringify these. The approach is also adapted to generic schema-level create/drop such as :class:`_schema.AddConstraint`, which will adapt its stringify dialect to one indicated by the element within it, such as the :class:`_postgresql.ExcludeConstraint` object. mostly towards being able to provide doctest-style examples for "on conflict" constructs using print statements. Change-Id: I4b855516fe6dee2df77744c1bb21a373d7fbab93
* Merge "tutorial 2.0 WIP"mike bayer2020-10-311-1/+0
|\
| * tutorial 2.0 WIPreview/mike_bayer/tutorial20Mike Bayer2020-10-311-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Add SelectBase.exists() method as it seems strange this is not available already. The Exists construct itself does not provide full SELECT-building capabilities so it makes sense this should be used more like a scalar_subquery. Make sure stream_results is getting set up when yield_per is used, for 2.0 style statements as well. this was hardcoded inside of Query.yield_per() and is now moved to take place within QueryContext. Change-Id: Icafcd4fd9b708772343d56edf40995c9e8f835d6
* | Merge "Deprecate bind args, execute() methods that were missed"mike bayer2020-10-311-0/+7
|\ \ | |/ |/|
| * Deprecate bind args, execute() methods that were missedMike Bayer2020-10-301-0/+7
| | | | | | | | | | | | in particular text(bind), DDL.execute(). Change-Id: Ie85ae9f61219182f5649f68e5f52b4923843199c
* | Apply underscore naming to several more operatorsjonathan vanasco2020-10-301-25/+33
|/ | | | | | | | | | | | | | | | | | | | | | | | | | The operator changes are: * `isfalse` is now `is_false` * `isnot_distinct_from` is now `is_not_distinct_from` * `istrue` is now `is_true` * `notbetween` is now `not_between` * `notcontains` is now `not_contains` * `notendswith` is now `not_endswith` * `notilike` is now `not_ilike` * `notlike` is now `not_like` * `notmatch` is now `not_match` * `notstartswith` is now `not_startswith` * `nullsfirst` is now `nulls_first` * `nullslast` is now `nulls_last` Because these are core operators, the internal migration strategy for this change is to support legacy terms for an extended period of time -- if not indefinitely -- but update all documentation, tutorials, and internal usage to the new terms. The new terms are used to define the functions, and the legacy terms have been deprecated into aliases of the new terms. Fixes: #5435 Change-Id: Ifbd7cb1cdda5981990243c4fc4b4ff467dc132ac
* Ensure escaping of percent signs in columns, parametersMike Bayer2020-10-171-25/+44
| | | | | | | | | | | | | | | | | | | | | | | | Improved support for column names that contain percent signs in the string, including repaired issues involving anoymous labels that also embedded a column name with a percent sign in it, as well as re-established support for bound parameter names with percent signs embedded on the psycopg2 dialect, using a late-escaping process similar to that used by the cx_Oracle dialect. * Added new constructor for _anonymous_label() that ensures incoming string tokens based on column or table names will have percent signs escaped; abstracts away the format of the label. * generalized cx_Oracle's quoted_bind_names facility into the compiler itself, and leveraged this for the psycopg2 dialect's issue with percent signs in names as well. the parameter substitution is now integrated with compiler.construct_parameters() as well as the recently reworked set_input_sizes(), reducing verbosity in the cx_Oracle dialect. Fixes: #5653 Change-Id: Ia2ad13ea68b4b0558d410026e5a33f5cb3fbab2c
* Repair reduction in OverMike Bayer2020-10-121-0/+9
| | | | | | | | Fixed issue where a plain pickle dumps call of the :class:`_sql.Over` construct didn't work. Fixes: #5644 Change-Id: I4b07f74ecd5d52f0794128585367012200a38a36
* upgrade to black 20.8b1Mike Bayer2020-09-281-3/+6
| | | | | | | It's better, the majority of these changes look more readable to me. also found some docstrings that had formatting / quoting issues. Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
* new docs WIPMike Bayer2020-09-241-8/+17
| | | | | | | | | | | | | | | This WIP is part of the final push for 1.4's docs to fully "2.0-ize" what we can, and have it all ready. So far this includes a rewrite of the 2.0 migration, set up for the 1.4 /2.0 docs style, and a total redesign of the index page using a new flex layout in zzzeeksphinx. It also reworks some of the API reference sections to have more subheaders. zzzeeksphinx is also enhanced to provide automatic summaries for all api doc section. Change-Id: I01d360cb9c8749520246b96ee6496143c6037918
* Update select usage to use the new 1.4 formatFederico Caselli2020-09-081-53/+49
| | | | | | | | | | | | | | | | This change includes mainly that the bracketed use within select() is moved to positional, and keyword arguments are removed from calls to the select() function. it does not yet fully address other issues such as keyword arguments passed to the table.select(). Additionally, allows False / None to both be considered as "disable" for all of select.correlate(), select.correlate_except(), query.correlate(), which establishes consistency with passing of ``False`` for the legact select(correlate=False) argument. Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
* Create a real type for Tuple() and handle appropriately in compilerMike Bayer2020-08-171-22/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | Improved the :func:`_sql.tuple_` construct such that it behaves predictably when used in a columns-clause context. The SQL tuple is not supported as a "SELECT" columns clause element on most backends; on those that do (PostgreSQL, not surprisingly), the Python DBAPI does not have a "nested type" concept so there are still challenges in fetching rows for such an object. Use of :func:`_sql.tuple_` in a :func:`_sql.select` or :class:`_orm.Query` will now raise a :class:`_exc.CompileError` at the point at which the :func:`_sql.tuple_` object is seen as presenting itself for fetching rows (i.e., if the tuple is in the columns clause of a subquery, no error is raised). For ORM use,the :class:`_orm.Bundle` object is an explicit directive that a series of columns should be returned as a sub-tuple per row and is suggested by the error message. Additionally ,the tuple will now render with parenthesis in all contexts. Previously, the parenthesization would not render in a columns context leading to non-defined behavior. As part of this change, Tuple receives a dedicated datatype which appears to allow us the very desirable change of removing the bindparam._expanding_in_types attribute as well as ClauseList._tuple_values (which might already have not been needed due to #4645). Fixes: #5127 Change-Id: Iecafa0e0aac2f1f37ec8d0e1631d562611c90200
* Robustness for lambdas, lambda statementsMike Bayer2020-08-051-37/+12
| | | | | | | | | | | | | | | | | | | | in order to accommodate relationship loaders with lambda caching, a lot more is needed. This is a full refactor of the lambda system such that it now has two levels of caching; the first level caches what can be known from the __code__ element, then the next level of caching is against the lambda itself and the contents of __closure__. This allows for the elements inside the lambdas, like columns and entities, to change and then be part of the cache key. Lazy/selectinloads' use of baked queries had to add distinct cache key elements, which was attempted here but overall things needed to be more robust than that. This commit is broken out from the very long and sprawling commit at Id6b5c03b1ce9ddb7b280f66792212a0ef0a1c541 . Change-Id: I29a513c98917b1d503abfdd61e6b6e8800851aa8
* Allow Grouping to pass along proxy_set of elementMike Bayer2020-07-211-0/+7
| | | | | | | | | | Repaired an issue where the "ORDER BY" clause rendering a label name rather than a complete expression, which is particularly important for SQL Server, would fail to occur if the expression were enclosed in a parenthesized grouping in some cases. This case has been added to test support. Fixes: #5470 Change-Id: Ie0e27c39e5d53be78b32f7810f93d2d0536375e7
* Convert remaining ORM APIs to support 2.0 styleMike Bayer2020-07-111-22/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is kind of a mixed bag of all kinds to help get us to 1.4 betas. The documentation stuff is a work in progress. Lots of other relatively small changes to APIs and things. More commits will follow to continue improving the documentation and transitioning to the 1.4/2.0 hybrid documentation. In particular some refinements to Session usage models so that it can match Engine's scoping / transactional patterns, and a decision to start moving away from "subtransactions" completely. * add select().from_statement() to produce FromStatement in an ORM context * begin referring to select() that has "plugins" for the few edge cases where select() will have ORM-only behaviors * convert dynamic.AppenderQuery to its own object that can use select(), though at the moment it uses Query to support legacy join calling forms. * custom query classes for AppenderQuery are replaced by do_orm_execute() hooks for custom actions, a separate gerrit will document this * add Session.get() to replace query.get() * Deprecate session.begin->subtransaction. propose within the test suite a hypothetical recipe for apps that rely on this pattern * introduce Session construction level context manager, sessionmaker context manager, rewrite the whole top of the session_transaction.rst documentation. Establish context manager patterns for Session that are identical to engine * ensure same begin_nested() / commit() behavior as engine * devise all new "join into an external transaction" recipe, add test support for it, add rules into Session so it just works, write new docs. need to ensure this doesn't break anything * vastly reduce the verbosity of lots of session docs as I dont think people read this stuff and it's difficult to keep current in any case * constructs like case(), with_only_columns() really need to move to *columns, add a coercion rule to just change these. * docs need changes everywhere I look. in_() is not in the Core tutorial? how do people even know about it? Remove tons of cruft from Select docs, etc. * build a system for common ORM options like populate_existing and autoflush to populate from execution options. * others? Change-Id: Ia4bea0f804250e54d90b3884cf8aab8b66b82ecf
* introduce deferred lambdasMike Bayer2020-07-031-8/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The coercions system allows us to add in lambdas as arguments to Core and ORM elements without changing them at all. By allowing the lambda to produce a deterministic cache key where we can also cheat and yank out literal parameters means we can move towards having 90% of "baked" functionality in a clearer way right in Core / ORM. As a second step, we can have whole statements inside the lambda, and can then add generation with __add__(), so then we have 100% of "baked" functionality with full support of ad-hoc literal values. Adds some more short_selects tests for the moment for comparison. Other tweaks inside cache key generation as we're trying to approach a certain level of performance such that we can remove the use of "baked" from the loader strategies. As we have not yet closed #4639, however the caching feature has been fully integrated as of b0cfa7379cf8513a821a3dbe3028c4965d9f85bd, we will also add complete caching documentation here and close that issue as well. Closes: #4639 Fixes: #5380 Change-Id: If91f61527236fd4d7ae3cad1f24c38be921c90ba
* Merge "Fix a wide variety of typos and broken links"mike bayer2020-06-261-37/+41
|\
| * Fix a wide variety of typos and broken linksaplatkouski2020-06-251-37/+41
| | | | | | | | | | | | | | | | | | | | | | | | Note the PR has a few remaining doc linking issues listed in the comment that must be addressed separately. Signed-off-by: aplatkouski <5857672+aplatkouski@users.noreply.github.com> Closes: #5371 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5371 Pull-request-sha: 7e7d233cf3a0c66980c27db0fcdb3c7d93bc2510 Change-Id: I9c36e8d8804483950db4b42c38ee456e384c59e3
* | Default psycopg2 executemany mode to "values_only"Mike Bayer2020-06-251-12/+4
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The psycopg2 dialect now defaults to using the very performant ``execute_values()`` psycopg2 extension for compiled INSERT statements, and also impements RETURNING support when this extension is used. This allows INSERT statements that even include an autoincremented SERIAL or IDENTITY value to run very fast while still being able to return the newly generated primary key values. The ORM will then integrate this new feature in a separate change. Implements RETURNING for insert with executemany Adds support to return_defaults() mode and inserted_primary_key to support mutiple INSERTed rows, via return_defauls_rows and inserted_primary_key_rows accessors. within default execution context, new cached compiler getters are used to fetch primary keys from rows inserted_primary_key now returns a plain tuple. this is not yet a row-like object however this can be added. Adds distinct "values_only" and "batch" modes, as "values" has a lot of benefits but "batch" breaks cursor.rowcount psycopg2 minimum version 2.7 so we can remove the large number of checks for very old versions of psycopg2 simplify tests to no longer distinguish between native and non-native json Fixes: #5401 Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
* Turn on caching everywhere, add loggingMike Bayer2020-06-101-44/+94
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* Document that type_coerce does not currently imply parenthesizationMike Bayer2020-06-041-11/+25
| | | | | | | | | | | | We've had a few issues where the current solution is to use the self_group() method, so document that as the current approach for the parenthesization use case. Whether or not type_coerce() is changed later, this is how it works at the moment. Fixes: #5375 Change-Id: I97414762a87ec8f1fd1adc1b6be5a52e576814ca References: #5344
* Inline a few ORM arguments, othersMike Bayer2020-06-031-7/+13
| | | | | | small changes Change-Id: Id89a0651196c431d0aaf6935f5a4e7b12dd70c6c
* Improve rendering of core statements w/ ORM elementsMike Bayer2020-05-311-5/+18
| | | | | | | | | | | | | | | | | | | | This patch contains a variety of ORM and expression layer tweaks to support ORM constructs in select() statements, without the 1.3.x requiremnt in Query that a full _compile_context() + new select() is needed in order to get a working statement object. Includes such tweaks as the ability to implement aliased class of an aliased class, as we are looking to fully support ACs against subqueries, as well as the ability to access anonymously-labeled ColumnProperty expressions within subqueries by naming the ".key" of the label after the property key. Some tuning to query.join() as well as ORMJoin internals to allow things to work more smoothly. Change-Id: Id810f485c5f7ed971529489b84694e02a3356d6d
* Convert execution to move through SessionMike Bayer2020-05-251-23/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-241-7/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* Correctly apply self_group in type_coerce element.Federico Caselli2020-05-221-6/+10
| | | | | | | | The type coerce element did not correctly apply grouping rules when using in an expression Fixes: #5344 Change-Id: Id67b0e60ac54f8992f931aaed62731672f60c96c
View Lorry Controller Status