summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/sql/compiler.py
Commit message (Collapse)AuthorAgeFilesLines
...
* Create initial 2.0 engine implementationMike Bayer2020-04-161-2/+6
| | | | | | | | | | | | | | | | | | | Implemented the SQLAlchemy 2 :func:`.future.create_engine` function which is used for forwards compatibility with SQLAlchemy 2. This engine features always-transactional behavior with autobegin. Allow execution options per statement execution. This includes that the before_execute() and after_execute() events now accept an additional dictionary with these options, empty if not passed; a legacy event decorator is added for backwards compatibility which now also emits a deprecation warning. Add some basic tests for execution, transactions, and the new result object. Build out on a new testing fixture that swaps in the future engine completely to start with. Change-Id: I70e7338bb3f0ce22d2f702537d94bb249bd9fb0a Fixes: #4644
* Run search and replace of symbolic module namesMike Bayer2020-04-141-7/+10
| | | | | | | | Replaces a wide array of Sphinx-relative doc references with an abbreviated absolute form now supported by zzzeeksphinx. Change-Id: I94bffcc3f37885ffdde6238767224296339698a2
* Remove code deprecated before version 1.1Federico Caselli2020-04-091-14/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Remove deprecated method ``get_primary_keys` in the :class:`.Dialect` and :class:`.Inspector` classes. - Remove deprecated event ``dbapi_error`` and the method ``ConnectionEvents.dbapi_error`. - Remove support for deprecated engine URLs of the form ``postgres://``. - Remove deprecated dialect ``mysql+gaerdbms``. - Remove deprecated parameter ``quoting`` from :class:`.mysql.ENUM` and :class:`.mysql.SET` in the ``mysql`` dialect. - Remove deprecated function ``comparable_property``. and function ``comparable_using`` in the declarative extension. - Remove deprecated function ``compile_mappers``. - Remove deprecated method ``collection.linker``. - Remove deprecated method ``Session.prune`` and parameter ``Session.weak_identity_map``. This change also removes the class ``StrongInstanceDict``. - Remove deprecated parameter ``mapper.order_by``. - Remove deprecated parameter ``Session._enable_transaction_accounting`. - Remove deprecated parameter ``Session.is_modified.passive``. - Remove deprecated class ``Binary``. Please use :class:`.LargeBinary`. - Remove deprecated methods ``Compiled.compile``, ``ClauseElement.__and__`` and ``ClauseElement.__or__`` and attribute ``Over.func``. - Remove deprecated ``FromClause.count`` method. - Remove deprecated parameter ``Table.useexisting``. - Remove deprecated parameters ``text.bindparams`` and ``text.typemap``. - Remove boolean support for the ``passive`` parameter in ``get_history``. - Remove deprecated ``adapt_operator`` in ``UserDefinedType.Comparator``. Fixes: #4643 Change-Id: Idcd390c77bf7b0e9957907716993bdaa3f1a1763
* Repair caching / traversals for valuesMike Bayer2020-04-011-2/+7
| | | | | | | | | | | | | | The test suite wasn't running the copy_internals most fixtures, enable that and try to get all cases working. Set up selectable.values to do tuple conversion within compilation step. at the same time, disable caching for selectable.values for the moment and make it equivalent to dml_multi_values. fix cache / compare / copy cases for dml_values and dml_multi_values which weren't fully tested or covered. Change-Id: I484ca6e9cb2b66c2e6a321698f2abc0838db1460
* Try to measure new style caching in the ORM, take twoMike Bayer2020-04-011-10/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf Trying to see if by making the cache key memoized, we still can have the older "identity" form of caching which is the cheapest of all, at the same time as the newer "cache key each time" version that is not nearly as cheap; but still much cheaper than no caching at all. Also needed is a per-execution update of _keymap when we invoke from a cached select, so that Column objects that are anonymous or otherwise adapted will match up. this is analogous to the adaption of bound parameters from the cache key. Adds test coverage for the keymap / construct_params() changes related to caching. Also hones performance to a large extent for statement construction and cache key generation. Also includes a new memoized attribute approach that vastly simplifies the previous approach of "group_expirable_memoized_property" and finally integrates cleanly with _clone(), _generate(), etc. no more hardcoding of attributes is needed, as well as that most _reset_memoization() calls are no longer needed as the reset is inherent in a _generate() call; this also has dramatic performance improvements. Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
* Add a third labeling mode for SELECT statementsMike Bayer2020-03-291-21/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Enhanced the disambiguating labels feature of the :func:`~.sql.expression.select` construct such that when a select statement is used in a subquery, repeated column names from different tables are now automatically labeled with a unique label name, without the need to use the full "apply_labels()" feature that conbines tablename plus column name. The disambigated labels are available as plain string keys in the .c collection of the subquery, and most importantly the feature allows an ORM :func:`.orm.aliased` construct against the combination of an entity and an arbitrary subquery to work correctly, targeting the correct columns despite same-named columns in the source tables, without the need for an "apply labels" warning. The existing labeling style is now called LABEL_STYLE_TABLENAME_PLUS_COL. This labeling style will remain used throughout the ORM as has been the case for over a decade, however, the new disambiguation scheme could theoretically replace this scheme entirely. The new scheme would dramatically alter how SQL looks when rendered from the ORM to be more succinct but arguably harder to read. The tablename_columnname scheme used by Join.c is unaffected here, as that's still hardcoded to that scheme. Fixes: #5221 Change-Id: Ib47d9e0f35046b3afc77bef6e65709b93d0c3026
* Correct ambiguous func / class linksMike Bayer2020-03-251-5/+6
| | | | | | | | | :func:`.sql.expression.select`, :func:`.sql.expression.insert` and :class:`.sql.expression.Insert` were hitting many ambiguous symbol errors, due to future.select, as well as the PG/MySQL variants of Insert. Change-Id: Iac862bfc172a7f7f0cbba5353a83dc203bed376c
* Merge "Implement SQL VALUES in core."mike bayer2020-03-241-0/+40
|\
| * Implement SQL VALUES in core.Gord Thompson2020-03-241-0/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | Added a core :class:`Values` object that enables a VALUES construct to be used in the FROM clause of an SQL statement for databases that support it (mainly PostgreSQL and SQL Server). Fixes: #4868 Closes: #5030 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5030 Pull-request-sha: 84684038a8efa93b460318e0db53f6c644554588 Change-Id: Ib8109b63bc1a9dc04ab987c5322ca3375f7e824d
* | Convert schema_translate to a post compileMike Bayer2020-03-241-25/+75
|/ | | | | | | | | | | | | | | | Revised the :paramref:`.Connection.execution_options.schema_translate_map` feature such that the processing of the SQL statement to receive a specific schema name occurs within the execution phase of the statement, rather than at the compile phase. This is to support the statement being efficiently cached. Previously, the current schema being rendered into the statement for a particular run would be considered as part of the cache key itself, meaning that for a run against hundreds of schemas, there would be hundreds of cache keys, rendering the cache much less performant. The new behavior is that the rendering is done in a similar manner as the "post compile" rendering added in 1.4 as part of :ticket:`4645`, :ticket:`4808`. Fixes: #5004 Change-Id: Ia5c89eb27cc8dc2c5b8e76d6c07c46290a7901b6
* Merge "Rework select(), CompoundSelect() in terms of CompileState"mike bayer2020-03-111-52/+131
|\
| * Rework select(), CompoundSelect() in terms of CompileStateMike Bayer2020-03-101-52/+131
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Continuation of I408e0b8be91fddd77cf279da97f55020871f75a9 - add an options() method to the base Generative construct. this will be where ORM options can go - Change Null, False_, True_ to be singletons, so that we aren't instantiating them and having to use isinstance. The previous issue with this was that they would produce dupe labels in SELECT statements. Apply the duplicate column logic, newly added in 1.4, to these objects as well as to non-apply-labels SELECT statements in general as a means of improving this. - create a revised system for generating ClauseList compilation constructs that simplfies up front creation to not actually use ClauseList; a simple tuple is rendered by the compiler using the same constrcution rules as what are used for ClauseList but without creating the actual object. Apply to Select, CompoundSelect, revise Update, Delete - Select, CompoundSelect get an initial CompileState implementation. All methods used only within compilation are moved here - refine update/insert/delete compile state to not require an outside boolean - refine and simplify Select._copy_internals - rework bind(), which is going away, to not use some of the internal traversal stuff - remove "autocommit", "for_update" parameters from Select, references #4643 - remove "autocommit" parameter from TextClause , references #4643 - add deprecation warnings for statement.execute(), engine.execute(), statement.scalar(), engine.scalar(). Fixes: #5193 Change-Id: I04ca0152b046fd42c5054ba10f37e43fc6e5a57b
* | Simplified module pre-loading strategy and made it linter friendlyFederico Caselli2020-03-071-4/+7
|/ | | | | | | | | | | | | | | | | Introduced a modules registry to register modules that should be lazily loaded in the package init. This ensures that they are in the system module cache, avoiding potential thread safety issues as when importing them directly in the function that uses them. The module registry is used to obtain these modules directly, ensuring that the all the lazily loaded modules are resolved at the proper time This replaces dependency_for decorator and the dependencies decorator logic, removing the need to pass the resolved modules as arguments of the decodated functions and removes possible errors caused by linters. Fixes: #4689 Fixes: #4656 Change-Id: I2e291eba4297867fc0ddb5d875b9f7af34751d01
* Decouple compiler state from DML objects; make cacheableMike Bayer2020-03-061-25/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Targeting select / insert / update / delete, the goal is to minimize overhead of construction and generative methods so that only the raw arguments passed are handled. An interim stage that converts the raw state into more compiler-ready state is added, which is analogous to the ORM QueryContext which will also be rolled in to be a similar concept, as is currently being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10. the ORM update/delete BulkUD concept is also going to be rolled onto this idea. So while the compiler-ready state object, here called DMLState, looks a little thin, it's the base of a bigger pattern that will allow for ORM functionality to embed itself directly into the compiler, execution context, and result set objects. This change targets the DML objects, primarily focused on the values() method which is the most complex process. The work done by values() is minimized as much as possible while still being able to create a cache key. Additional computation is then offloaded to a new object ValuesState that is handled by the compiler. Architecturally, a big change here is that insert.values() and update.values() will generate BindParameter objects for the values now, which are then carefully received by crud.py so that they generate the expected names. This is so that the values() portion of these constructs is cacheable. for the "multi-values" version of Insert, this is all skipped and the plan right now is that a multi-values insert is not worth caching (can always be revisited). Using the coercions system in values() also gets us nicer validation for free, we can remove the NotAClauseElement thing from schema, and we also now require scalar_subquery() is called for an insert/update that uses a SELECT as a column value, 1.x deprecation path is added. The traversal system is then applied to the DML objects including tests so that they have traversal, cloning, and cache key support. cloning is not a use case for DML however having it present allows better validation of the structure within the tests. Special per-dialect DML is explicitly not cacheable at the moment, more as a proof of concept that third party DML constructs can exist as gracefully not-cacheable rather than producing an incomplete cache key. A few selected performance improvements have been added as well, simplifying the immutabledict.union() method and adding a new SQLCompiler function that can generate delimeter-separated clauses like WHERE and ORDER BY without having to build a ClauseList object at all. The use of ClauseList will be removed from Select in an upcoming commit. Overall, ClaustList is unnecessary for internal use and only adds overhead to statement construction and will likely be removed as much as possible except for explcit use of conjunctions like and_() and or_(). Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
* Restore crud flags if visiting_cte is setMike Bayer2020-03-031-0/+1
| | | | | | | | | | Fixed bug where a CTE of an INSERT/UPDATE/DELETE that also uses RETURNING could then not be SELECTed from directly, as the internal state of the compiler would try to treat the outer SELECT as a DELETE statement itself and access nonexistent state. Fixes: #5181 Change-Id: Icba76f2148c8344baa1c04bac4ab6c6d24f23072
* Ensure all nested exception throws have a causeMike Bayer2020-03-021-5/+10
| | | | | | | | | | | | | | | Applied an explicit "cause" to most if not all internally raised exceptions that are raised from within an internal exception catch, to avoid misleading stacktraces that suggest an error within the handling of an exception. While it would be preferable to suppress the internally caught exception in the way that the ``__suppress_context__`` attribute would, there does not as yet seem to be a way to do this without suppressing an enclosing user constructed context, so for now it exposes the internally caught exception as the cause so that full information about the context of the error is maintained. Fixes: #4849 Change-Id: I55a86b29023675d9e5e49bc7edc5a2dc0bcd4751
* Discontinue dynamic __visit_name__Mike Bayer2020-03-011-0/+6
| | | | | | | | | Removed very antiquated logic that checks if __visit_name__ is a property. There's no need for this as the compiler can handle switching between implementations. Convert _compile_dispatch() to be fully inlined. Change-Id: Ic0c7247c2d7dfed93a27f09250a8ed6352370764
* Repair inline flagMike Bayer2020-02-221-1/+1
| | | | | | | | | | | | | | | In 9fca5d827d we attempted to deprecate the "inline=True" flag and add a generative inline() method, however failed to include any tests and the method was implemented incorrectly such that it would get overwritten with the boolean flag immediately. Rename the internal "inline" flag to "_inline" and add test support both for the method as well as deprecated support for the flag, including a fixture addition to assert the expected value of the flag as it generally does not affect the actual compiled SQL string. Change-Id: I0450049f17f1f0d91e22d27f1a973a2b6c0e59f7
* Result initial introductionMike Bayer2020-02-211-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This builds on cc718cccc0bf8a01abdf4068c7ea4f3 which moved RowProxy to Row, allowing Row to be more like a named tuple. - KeyedTuple in ORM is replaced with Row - ResultSetMetaData broken out into "simple" and "cursor" versions for ORM and Core, as well as LegacyCursor version. - Row now has _mapping attribute that supplies full mapping behavior. Row and SimpleRow both have named tuple behavior otherwise. LegacyRow has some mapping features on the tuple which emit deprecation warnings (e.g. keys(), values(), etc). the biggest change for mapping->tuple is the behavior of __contains__ which moves from testing of "key in row" to "value in row". - ResultProxy breaks into ResultProxy and FutureResult (interim), the latter has the newer APIs. Made available to dialects using execution options. - internal reflection methods and most tests move off of implicit Row mapping behavior and move to row._mapping, result.mappings() method using future result - a new strategy system for cursor handling replaces the various subclasses of RowProxy - some execution context adjustments. We will leave EC in but refined things like get_result_proxy() and out parameter handling. Dialects for 1.4 will need to adjust from get_result_proxy() to get_result_cursor_strategy(), if they are using this method - out parameter handling now accommodated by get_out_parameter_values() EC method. Oracle changes for this. external dialect for DB2 for example will also need to adjust for this. - deprecate case_insensitive flag for engine / result, this feature is not used mapping-methods on Row are deprecated, and replaced with Row._mapping.<meth>, including: row.keys() -> use row._mapping.keys() row.items() -> use row._mapping.items() row.values() -> use row._mapping.values() key in row -> use key in row._mapping int in row -> use int < len(row) Fixes: #4710 Fixes: #4878 Change-Id: Ieb9085e9bcff564359095b754da9ae0af55679f0
* Query linter optionAlessio Bogon2020-01-221-13/+163
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added "from linting" as a built-in feature to the SQL compiler. This allows the compiler to maintain graph of all the FROM clauses in a particular SELECT statement, linked by criteria in either the WHERE or in JOIN clauses that link these FROM clauses together. If any two FROM clauses have no path between them, a warning is emitted that the query may be producing a cartesian product. As the Core expression language as well as the ORM are built on an "implicit FROMs" model where a particular FROM clause is automatically added if any part of the query refers to it, it is easy for this to happen inadvertently and it is hoped that the new feature helps with this issue. The original recipe is from: https://github.com/sqlalchemy/sqlalchemy/wiki/FromLinter The linter is now enabled for all tests in the test suite as well. This has necessitated that a lot of the queries be adjusted to not include cartesian products. Part of the rationale for the linter to not be enabled for statement compilation only was to reduce the need for adjustment for the many test case statements throughout the test suite that are not real-world statements. This gerrit is adapted from Ib5946e57c9dba6da428c4d1dee6760b3e978dda0. Fixes: #4737 Change-Id: Ic91fd9774379f895d021c3ad564db6062299211c Closes: #4830 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4830 Pull-request-sha: f8a21aa6262d1bcc9ff0d11a2616e41fba97a47a
* Support GenericFunction.name passed as a quoted_nameMike Bayer2020-01-061-0/+2
| | | | | | | | | | | | | | | A function created using :class:`.GenericFunction` can now specify that the name of the function should be rendered with or without quotes by assigning the :class:`.quoted_name` construct to the .name element of the object. Prior to 1.3.4, quoting was never applied to function names, and some quoting was introduced in :ticket:`4467` but no means to force quoting for a mixed case name was available. Additionally, the :class:`.quoted_name` construct when used as the name will properly register its lowercase name in the function registry so that the name continues to be available via the ``func.`` registry. Fixes: #5079 Change-Id: I0653ab8b16e75e628ce82dbbc3d0f77f8336c407
* happy new yearMike Bayer2020-01-011-1/+1
| | | | Change-Id: I08440dc25e40ea1ccea1778f6ee9e28a00808235
* Use expanding IN for all literal value IN expressionsMike Bayer2019-12-221-33/+189
| | | | | | | | | | | | | | | | | | | | | | | | The "expanding IN" feature, which generates IN expressions at query execution time which are based on the particular parameters associated with the statement execution, is now used for all IN expressions made against lists of literal values. This allows IN expressions to be fully cacheable independently of the list of values being passed, and also includes support for empty lists. For any scenario where the IN expression contains non-literal SQL expressions, the old behavior of pre-rendering for each position in the IN is maintained. The change also completes support for expanding IN with tuples, where previously type-specific bind processors weren't taking effect. As part of this change, a more explicit separation between "literal execute" and "post compile" bound parameters is being made; as the "ansi bind rules" feature is rendering bound parameters inline, as we now support "postcompile" generically, these should be used here, however we have to render literal values at execution time even for "expanding" parameters. new test fixtures etc. are added to assert everything goes to the right place. Fixes: #4645 Change-Id: Iaa2b7bfbfaaf5b80799ee17c9b8507293cba6ed1
* Add CTE prefixesMarat Sharafutdinov2019-12-181-1/+2
| | | | | | | | | | | | | Added support for prefixes to the :class:`.CTE` construct, to allow support for Postgresql 12 "MATERIALIZED" and "NOT MATERIALIZED" phrases. Pull request courtesy Marat Sharafutdinov. Fixes: #5040 Closes: #5043 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5043 Pull-request-sha: d1b9059a0b6dae8dc2479ac670999b4af07908e0 Change-Id: I2e9cb5d7f85961ec98ee51965de5b3ec4a97be2f
* Do the CompoundSelect check for number of columns in the compile phaseMike Bayer2019-12-161-2/+27
| | | | | | | | Starting to go forward with the general idea of moving more of Core / ORM construction into the compile phase. Bigger initiatives like the refactor of Query will follow onto this. Change-Id: I0f364d3182e21e32ed85ef34cfd11fd9d11cf653
* Include DISTINCT in error message for label referenceMike Bayer2019-12-061-1/+4
| | | | | | | | Needed to add tests to ensure this label reference is handled correctly, so also modified the exception message to be more clear if someone has this error within distinct(). Change-Id: I6e685e46ae336596272d14366445ac224c18d92c
* Support for generated columnsCaselIT2019-11-081-0/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | Added DDL support for "computed columns"; these are DDL column specifications for columns that have a server-computed value, either upon SELECT (known as "virtual") or at the point of which they are INSERTed or UPDATEd (known as "stored"). Support is established for Postgresql, MySQL, Oracle SQL Server and Firebird. Thanks to Federico Caselli for lots of work on this one. ORM round trip tests included. The ORM makes use of existing FetchedValue support and no additional ORM logic is present for the basic feature. It has been observed that Oracle RETURNING does not return the new value of a computed column upon UPDATE; it returns the prior value. As this is very dangerous, a warning is emitted if a computed column is rendered into the RETURNING clause of an UPDATE statement. Fixes: #4894 Closes: #4928 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4928 Pull-request-sha: d39c521d5ac6ebfb4fb5b53846451de79752e64c Change-Id: I2610b2999a5b1b127ed927dcdaeee98b769643ce
* Add anonymizing context to cache keys, comparison; convert traversalMike Bayer2019-11-041-7/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Created new visitor system called "internal traversal" that applies a data driven approach to the concept of a class that defines its own traversal steps, in contrast to the existing style of traversal now known as "external traversal" where the visitor class defines the traversal, i.e. the SQLCompiler. The internal traversal system now implements get_children(), _copy_internals(), compare() and _cache_key() for most Core elements. Core elements with special needs like Select still implement some of these methods directly however most of these methods are no longer explicitly implemented. The data-driven system is also applied to ORM elements that take part in SQL expressions so that these objects, like mappers, aliasedclass, query options, etc. can all participate in the cache key process. Still not considered is that this approach to defining traversibility will be used to create some kind of generic introspection system that works across Core / ORM. It's also not clear if real statement caching using the _cache_key() method is feasible, if it is shown that running _cache_key() is nearly as expensive as compiling in any case. Because it is data driven, it is more straightforward to optimize using inlined code, as is the case now, as well as potentially using C code to speed it up. In addition, the caching sytem now accommodates for anonymous name labels, which is essential so that constructs which have anonymous labels can be cacheable, that is, their position within a statement in relation to other anonymous names causes them to generate an integer counter relative to that construct which will be the same every time. Gathering of bound parameters from any cache key generation is also now required as there is no use case for a cache key that does not extract bound parameter values. Applies-to: #4639 Change-Id: I0660584def8627cad566719ee98d3be045db4b8d
* Use separate label generator for column_label naming conventionMike Bayer2019-10-141-1/+11
| | | | | | | | | | | | | | | | | | | | Fixed bug where a table that would have a column label overlap with a plain column name, such as "foo.id AS foo_id" vs. "foo.foo_id", would prematurely generate the ``._label`` attribute for a column before this overlap could be detected due to the use of the ``index=True`` or ``unique=True`` flag on the column in conjunction with the default naming convention of ``"column_0_label"``. This would then lead to failures when ``._label`` were used later to generate a bound parameter name, in particular those used by the ORM when generating the WHERE clause for an UPDATE statement. The issue has been fixed by using an alternate ``._label`` accessor for DDL generation that does not affect the state of the :class:`.Column`. The accessor also bypasses the key-deduplication step as it is not necessary for DDL, the naming is now consistently ``"<tablename>_<columnname>"`` without any subsequent numeric symbols when used in DDL. Fixes: #4911 Change-Id: Iabf5fd3250738d800d6e41a2a3a27a7ce2405e7d
* Add _alembic_quote method to format_constraint()Mike Bayer2019-10-131-2/+5
| | | | | | | | | | | | | | Alembic needs a portable way of getting at the name of an index without quoting being applied. As we would like the indexes created by the Column index=True flag to support deferred index names, supply a function that delivers this for Alembic without it having to dig too deeply into the internals. the _alembic_quote flag may be made public at a later time, however as we've been through many quoting flags that are difficult to get rid of, try to be conservative to start. Change-Id: I184adaeae26c2e75093aaea5ebe01a3815cadb08
* Add result map targeting for custom compiled, text objectsMike Bayer2019-10-071-20/+52
| | | | | | | | | | | | | | | | In order for text(), custom compiled objects, etc. to be usable by Query(), they are all targeted by object key in the result map. As we no longer want Query to implicitly label these, as well as that text() has no label feature, support adding entries to the result map that have no name, key, or type, only the object itself, and then ensure that the compiler sets up for positional targeting when this condition is detected. Allows for more flexible ORM query usage with custom expressions and text() while having less special logic in query itself. Fixes: #4887 Change-Id: Ie073da127d292d43cb132a2b31bc90af88bfe2fd
* Drop right-nested join rewritingMike Bayer2019-10-071-158/+0
| | | | | | | | | | Dropped support for right-nested join rewriting to support old SQLite versions prior to 3.7.16, released in 2013. It is expected that all modern Python versions among those now supported should all include much newer versions of SQLite. Fixes: #4895 Change-Id: I7f0cfc2b7d988ff147b9a4c6d5e2adec87e27029
* Deprecate textual column matching in RowMike Bayer2019-10-041-3/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Deprecate query.instances() without a context Deprecate string alias with contains_eager() Deprecated the behavior by which a :class:`.Column` can be used as the key in a result set row lookup, when that :class:`.Column` is not part of the SQL selectable that is being selected; that is, it is only matched on name. A deprecation warning is now emitted for this case. Various ORM use cases, such as those involving :func:`.text` constructs, have been improved so that this fallback logic is avoided in most cases. Calling the :meth:`.Query.instances` method without passing a :class:`.QueryContext` is deprecated. The original use case for this was that a :class:`.Query` could yield ORM objects when given only the entities to be selected as well as a DBAPI cursor object. However, for this to work correctly there is essential metadata that is passed from a SQLAlchemy :class:`.ResultProxy` that is derived from the mapped column expressions, which comes originally from the :class:`.QueryContext`. To retrieve ORM results from arbitrary SELECT statements, the :meth:`.Query.from_statement` method should be used. Note there is a small bump in test_zoomark because the column._label is being calculated for each of those columns within baseline_3_properties, as it is now part of the result map. This label can't be calculated when the column is attached to the table because it needs to have all the columns present to do this correctly. Another approach here would be to pre-load the _label before the test runs however the zoomark tests don't have an easy place for this to happen and it's not really worth it. Fixes: #4877 Fixes: #4719 Change-Id: I9bd29e72e6dce7c855651d69ba68d7383469acbc
* Run row value processors up frontMike Bayer2019-10-011-1/+9
| | | | | | | | | | | | as part of a larger series of changes to generalize row-tuples, RowProxy becomes plain Row and is no longer a "proxy"; the DBAPI row is now copied directly into the Row when constructed, result handling occurs at once. Subsequent changes will break out Row into a new version that behaves fully a tuple. Change-Id: I2ffa156afce5d21c38f28e54c3a531f361345dd5
* Render LIMIT/OFFSET conditions after compile on select dialectsMike Bayer2019-08-301-24/+134
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added new "post compile parameters" feature. This feature allows a :func:`.bindparam` construct to have its value rendered into the SQL string before being passed to the DBAPI driver, but after the compilation step, using the "literal render" feature of the compiler. The immediate rationale for this feature is to support LIMIT/OFFSET schemes that don't work or perform well as bound parameters handled by the database driver, while still allowing for SQLAlchemy SQL constructs to be cacheable in their compiled form. The immediate targets for the new feature are the "TOP N" clause used by SQL Server (and Sybase) which does not support a bound parameter, as well as the "ROWNUM" and optional "FIRST_ROWS()" schemes used by the Oracle dialect, the former of which has been known to perform better without bound parameters and the latter of which does not support a bound parameter. The feature builds upon the mechanisms first developed to support "expanding" parameters for IN expressions. As part of this feature, the Oracle ``use_binds_for_limits`` feature is turned on unconditionally and this flag is now deprecated. - adds limited support for "unique" bound parameters within a text() construct. - adds an additional int() check within the literal render function of the Integer datatype and tests that non-int values raise ValueError. Fixes: #4808 Change-Id: Iace97d544d1a7351ee07db970c6bc06a19c712c6
* Revise psycopg2 execute_values approachMike Bayer2019-08-181-1/+13
| | | | | | | | | | | | Revised the approach for the just added support for the psycopg2 "execute_values()" feature added in 1.3.7 for :ticket:`4623`. The approach relied upon a regular expression that would fail to match for a more complex INSERT statement such as one which had subqueries involved. The new approach matches exactly the string that was rendered as the VALUES clause. Fixes: #4623 Change-Id: Icaae0f7b6bcf87a2cf5c6290a839c8429dd5fac3
* Invoke column_expression() for subsequent SELECTs in CompoundSelectMike Bayer2019-07-291-10/+19
| | | | | | | | | | | | | Fixed bug where :meth:`.TypeEngine.column_expression` method would not be applied to subsequent SELECT statements inside of a UNION or other :class:`.CompoundSelect`, even though the SELECT statements are rendered at the topmost level of the statement. New logic now differentiates between rendering the column expression, which is needed for all SELECTs in the list, vs. gathering the returned data type for the result row, which is needed only for the first SELECT. Fixes: #4787 Change-Id: Iceb63e430e76d2365649aa25ead09c4e2a062e10
* Fix typosMin ho Kim2019-07-211-1/+1
|
* Support tuple IN VALUES for SQLite, othersMike Bayer2019-07-191-1/+5
| | | | | | | | | | | | Added support for composite (tuple) IN operators with SQLite, by rendering the VALUES keyword for this backend. As other backends such as DB2 are known to use the same syntax, the syntax is enabled in the base compiler using a dialect-level flag ``tuple_in_values``. The change also includes support for "empty IN tuple" expressions for SQLite when using "in_()" between a tuple value and an empty set. Fixes: #4766 Change-Id: I416e1af29b31d78f9ae06ec3c3a48ef6d6e813f5
* Allow duplicate columns in from clauses and selectablesMike Bayer2019-07-111-11/+33
| | | | | | | | | | | | | | | | | | The :func:`.select` construct and related constructs now allow for duplication of column labels and columns themselves in the columns clause, mirroring exactly how column expressions were passed in. This allows the tuples returned by an executed result to match what was SELECTed for in the first place, which is how the ORM :class:`.Query` works, so this establishes better cross-compatibility between the two constructs. Additionally, it allows column-positioning-sensitive structures such as UNIONs (i.e. :class:`.CompoundSelect`) to be more intuitively constructed in those cases where a particular column might appear in more than one place. To support this change, the :class:`.ColumnCollection` has been revised to support duplicate columns as well as to allow integer index access. Fixes: #4753 Change-Id: Ie09a8116f05c367995c1e43623c51e07971d3bf0
* SelectBase no longer a FromClauseMike Bayer2019-07-061-45/+55
| | | | | | | | | | | | | | | | | | | | As part of the SQLAlchemy 2.0 migration project, a conceptual change has been made to the role of the :class:`.SelectBase` class hierarchy, which is the root of all "SELECT" statement constructs, in that they no longer serve directly as FROM clauses, that is, they no longer subclass :class:`.FromClause`. For end users, the change mostly means that any placement of a :func:`.select` construct in the FROM clause of another :func:`.select` requires first that it be wrapped in a subquery first, which historically is through the use of the :meth:`.SelectBase.alias` method, and is now also available through the use of :meth:`.SelectBase.subquery`. This was usually a requirement in any case since several databases don't accept unnamed SELECT subqueries in their FROM clause in any case. See the documentation in this change for lots more detail. Fixes: #4617 Change-Id: I0f6174ee24b9a1a4529168e52e855e12abd60667
* Turn off the is_literal flag when proxying literal_column() to LabelMike Bayer2019-06-151-0/+2
| | | | | | | | | | | | | | | | | | | Fixed a series of quoting issues which all stemmed from the concept of the :func:`.literal_column` construct, which when being "proxied" through a subquery to be referred towards by a label that matches its text, the label would not have quoting rules applied to it, even if the string in the :class:`.Label` were set up as a :class:`.quoted_name` construct. Not applying quoting to the text of the :class:`.Label` is a bug because this text is strictly a SQL identifier name and not a SQL expression, and the string should not have quotes embedded into it already unlike the :func:`.literal_column` which it may be applied towards. The existing behavior of a non-labeled :func:`.literal_column` being propagated as is on the outside of a subquery is maintained in order to help with manual quoting schemes, although it's not clear if valid SQL can be generated for such a construct in any case. Fixes: #4730 Change-Id: I300941f27872fc4298c74a1d1ed65aef1a5cdd82
* Reverse Alias nesting conceptMike Bayer2019-06-131-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | The Alias object no longer has "element" and "original", it now has "wrapped" and "element" (the name .original is also left as a descriptor for legacy access by third party dialects). These two data members refer to the dual roles Alias needs to play, where in the Python sense it needs to refer to the thing it was applied against directly, whereas in the SQL sense it needs to refer to the ultimate "non-alias" thing it refers towards. Both are necessary to maintain. However, the change here has each Alias object access the non-Alias object immediately so that the "unwrapping" is simpler and does not need any special logic. In the SQL sense, Alias objects don't nest, the only potential was that of the CTE, however there is no such thing as a nested CTE, see link below. This change is an interim change along the way to breaking Alias into more classes and breaking away Select objects from being FromClause objects. Change-Id: Ie7a0d064226cb074ca745505129b5ec7d879e389 References: https://stackoverflow.com/questions/1413516/can-you-create-nested-with-clauses-for-common-table-expressions
* Implement new ClauseElement role and coercion systemMike Bayer2019-05-181-14/+17
| | | | | | | | | | | | | | | | | | | | A major refactoring of all the functions handle all detection of Core argument types as well as perform coercions into a new class hierarchy based on "roles", each of which identify a syntactical location within a SQL statement. In contrast to the ClauseElement hierarchy that identifies "what" each object is syntactically, the SQLRole hierarchy identifies the "where does it go" of each object syntactically. From this we define a consistent type checking and coercion system that establishes well defined behviors. This is a breakout of the patch that is reorganizing select() constructs to no longer be in the FromClause hierarchy. Also includes a rename of as_scalar() into scalar_subquery(); deprecates automatic coercion to scalar_subquery(). Partially-fixes: #4617 Change-Id: I26f1e78898693c6b99ef7ea2f4e7dfd0e8e1a1bd
* Enhance documentation for string compilation use casesMike Bayer2019-04-041-3/+13
| | | | | | | | | | - Add a web link for UnsupportedCompilationError - Add new section to errors.rst - add more detail and cross-linking to the FAQ - include security caveats for parameter rendering Fixes: #4595 Change-Id: I31ea57c18d65770cd2a51276bbe2847a9eb72bba
* Remove all remaining text() coercions and ensure identifiers are safeMike Bayer2019-02-061-12/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fully removed the behavior of strings passed directly as components of a :func:`.select` or :class:`.Query` object being coerced to :func:`.text` constructs automatically; the warning that has been emitted is now an ArgumentError or in the case of order_by() / group_by() a CompileError. This has emitted a warning since version 1.0 however its presence continues to create concerns for the potential of mis-use of this behavior. Note that public CVEs have been posted for order_by() / group_by() which are resolved by this commit: CVE-2019-7164 CVE-2019-7548 Added "SQL phrase validation" to key DDL phrases that are accepted as plain strings, including :paramref:`.ForeignKeyConstraint.on_delete`, :paramref:`.ForeignKeyConstraint.on_update`, :paramref:`.ExcludeConstraint.using`, :paramref:`.ForeignKeyConstraint.initially`, for areas where a series of SQL keywords only are expected.Any non-space characters that suggest the phrase would need to be quoted will raise a :class:`.CompileError`. This change is related to the series of changes committed as part of :ticket:`4481`. Fixed issue where using an uppercase name for an index type (e.g. GIST, BTREE, etc. ) or an EXCLUDE constraint would treat it as an identifier to be quoted, rather than rendering it as is. The new behavior converts these types to lowercase and ensures they contain only valid SQL characters. Quoting is applied to :class:`.Function` names, those which are usually but not necessarily generated from the :attr:`.sql.func` construct, at compile time if they contain illegal characters, such as spaces or punctuation. The names are as before treated as case insensitive however, meaning if the names contain uppercase or mixed case characters, that alone does not trigger quoting. The case insensitivity is currently maintained for backwards compatibility. Fixes: #4481 Fixes: #4473 Fixes: #4467 Change-Id: Ib22a27d62930e24702e2f0f7c74a0473385a08eb
* Add deprecation warnings to all deprecated APIsMike Bayer2019-01-231-16/+42
| | | | | | | | | | | | | | | A large change throughout the library has ensured that all objects, parameters, and behaviors which have been noted as deprecated or legacy now emit ``DeprecationWarning`` warnings when invoked. As the Python 3 interpreter now defaults to displaying deprecation warnings, as well as that modern test suites based on tools like tox and pytest tend to display deprecation warnings, this change should make it easier to note what API features are obsolete. See the notes added to the changelog and migration notes for further details. Fixes: #4393 Change-Id: If0ea11a1fc24f9a8029352eeadfc49a7a54c0a1b
* Merge "use ..deprecated directive w/ version in all cases"mike bayer2019-01-121-5/+36
|\
| * use ..deprecated directive w/ version in all casesMike Bayer2019-01-111-5/+36
| | | | | | | | | | | | | | | | | | These changes should be ported from 1.3 back to 1.0 or possibly 0.9 to the extent they are relevant in each version. In 1.3 we hope to turn all deprecation documentation into warnings. Change-Id: I205186cde161af9389af513a425c62ce90dd54d8
* | happy new yearMike Bayer2019-01-111-1/+1
|/ | | | Change-Id: I6a71f4924d046cf306961c58dffccf21e9c03911
View Lorry Controller Status