summaryrefslogtreecommitdiff
path: root/test/sql
Commit message (Collapse)AuthorAgeFilesLines
* Update select usage to use the new 1.4 formatFederico Caselli2020-09-0831-1220/+1068
| | | | | | | | | | | | | | | | This change includes mainly that the bracketed use within select() is moved to positional, and keyword arguments are removed from calls to the select() function. it does not yet fully address other issues such as keyword arguments passed to the table.select(). Additionally, allows False / None to both be considered as "disable" for all of select.correlate(), select.correlate_except(), query.correlate(), which establishes consistency with passing of ``False`` for the legact select(correlate=False) argument. Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
* Don't rely on string col name in adapt_to_contextMike Bayer2020-09-064-27/+93
| | | | | | | | | | | | | | | fixed an issue where even though the method claims to be matching up columns positionally, it was failing on that by looking in "keymap" based on string name. Adds a new member to the _keymap recs MD_RESULT_MAP_INDEX so that we can efficiently link from the generated keymap back to the compiled._result_columns structure without any ambiguity. Fixes: #5559 Change-Id: Ie2fa9165c16625ef860ffac1190e00575e96761f
* Merge "internal test framework files for standardization of is_not/not_in;"mike bayer2020-08-308-59/+59
|\
| * internal test framework files for standardization of is_not/not_in;jonathan vanasco2020-08-298-59/+59
| | | | | | | | | | | | this is safe for 1.3.x Change-Id: Icba38fdc20f5d8ac407383a4278ccb346e09af38
* | Add support for regular expression on supported backend.Federico Caselli2020-08-271-0/+175
| | | | | | | | | | | | | | | | | | | | | | | | Two operations have been defined: * :meth:`~.ColumnOperators.regexp_match` implementing a regular expression match like function. * :meth:`~.ColumnOperators.regexp_replace` implementing a regular expression string replace function. Fixes: #1390 Change-Id: I44556846e4668ccf329023613bd26861d5c674e6
* | Updates for MariaDB sequencesFederico Caselli2020-08-223-31/+96
|/ | | | | | | | | | | | MariaDB should not run a Sequence if it has optional=True. Additionally, rework the rules in crud.py to accommodate the new combination MariaDB brings us, which is a dialect that supports both cursor.lastrowid, explicit sequences, *and* no support for returning. Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #5528 Change-Id: I9a8ea69a34983affa95dfd22186e2908fdf0d58c
* Add support for identity columnsFederico Caselli2020-08-192-0/+253
| | | | | | | | | | | | | | Added the :class:`_schema.Identity` construct that can be used to configure identity columns rendered with GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY. Currently the supported backends are PostgreSQL >= 10, Oracle >= 12 and MSSQL (with different syntax and a subset of functionalities). Fixes: #5362 Fixes: #5324 Fixes: #5360 Change-Id: Iecea6f3ceb36821e8b96f0b61049b580507a1875
* Deliver straight BinaryExpr w/ no negate for any() / all()Mike Bayer2020-08-181-0/+18
| | | | | | | | | Adjusted the :meth:`_types.ARRAY.Comparator.any` and :meth:`_types.ARRAY.Comparator.all` methods to implement a straight "NOT" operation for negation, rather than negating the comparison operator. Fixes: #5518 Change-Id: I87ee9278c321aafe51a679fcfcbb5fbb11307fda
* Merge "Create a real type for Tuple() and handle appropriately in compiler"mike bayer2020-08-173-2/+40
|\
| * Create a real type for Tuple() and handle appropriately in compilerMike Bayer2020-08-173-2/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Improved the :func:`_sql.tuple_` construct such that it behaves predictably when used in a columns-clause context. The SQL tuple is not supported as a "SELECT" columns clause element on most backends; on those that do (PostgreSQL, not surprisingly), the Python DBAPI does not have a "nested type" concept so there are still challenges in fetching rows for such an object. Use of :func:`_sql.tuple_` in a :func:`_sql.select` or :class:`_orm.Query` will now raise a :class:`_exc.CompileError` at the point at which the :func:`_sql.tuple_` object is seen as presenting itself for fetching rows (i.e., if the tuple is in the columns clause of a subquery, no error is raised). For ORM use,the :class:`_orm.Bundle` object is an explicit directive that a series of columns should be returned as a sub-tuple per row and is suggested by the error message. Additionally ,the tuple will now render with parenthesis in all contexts. Previously, the parenthesization would not render in a columns context leading to non-defined behavior. As part of this change, Tuple receives a dedicated datatype which appears to allow us the very desirable change of removing the bindparam._expanding_in_types attribute as well as ClauseList._tuple_values (which might already have not been needed due to #4645). Fixes: #5127 Change-Id: Iecafa0e0aac2f1f37ec8d0e1631d562611c90200
* | Merge "Bump minimum MySQL version to 5.0.2; use all-numeric server version"mike bayer2020-08-172-1/+3
|\ \
| * | Bump minimum MySQL version to 5.0.2; use all-numeric server versionMike Bayer2020-08-162-1/+3
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | MySQL dialect's server_version_info tuple is now all numeric. String tokens like "MariaDB" are no longer present so that numeric comparison works in all cases. The .is_mariadb flag on the dialect should be consulted for whether or not mariadb was detected. Additionally removed structures meant to support extremely old MySQL versions 3.x and 4.x; the minimum MySQL version supported is now version 5.0.2. In addition, as the "MariaDB" name goes away from server version, expand upon the change in I330815ebe572b6a9818377da56621397335fa702 to support the name "mariadb" throughout the dialect and test suite when mariadb-only mode is used. This changes the "name" field on the MariaDB dialect to "mariadb", which then implies a change throughout the testing requirements system as well as all the dialect-specific DDL argument names such as "mysql_engine" is now specified as "mariadb_engine", etc. Make use of the recent additions to test suite URL provisioning so that we can force MariaDB databases to have a "mariadb-only" dialect which allows us to test this name change fully. Update documentation to refer to MySQL / MariaDB explicitly as well as indicating the "mariadb_" prefix used for options. It seems likely that MySQL and MariaDB version numbers are going to start colliding at some point so having the "mariadb" name be available as a totally separate dialect name should give us some options in this regard. Currently also includes a date related fix to a test for the postgresql dialect that was implicitly assuming a non-UTC timezone Fixes: #4189 Change-Id: I00e76d00f62971e1f067bd61915fa6cc1cf64e5e
* | error message for LookupRamonWill2020-08-171-11/+47
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Thr proposed change will provide the user with the target Enum Class name as well as up to four possible enum values when a LookupError is raised in the Enum Class. A user requested that the enum name and possible values are included to the LookupError message to make debugging easier. The criteria included using ellipses for Enums containing more than four values and using ellipses for enum values that were greater than a certain number of characters (for this resolution the limit is 11 characters). This pull request is: - [ ] A documentation / typographical error fix - Good to go, no issue or tests are needed - [X ] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. - [ ] A new feature implementation - please include the issue number, and create an issue if none exists, which must include a complete example of how the feature would look. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. **Have a nice day!** Fixes: #4733 Closes: #5490 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5490 Pull-request-sha: 55e76f2ae796b59b7de157cfaae5235dffa359cb Change-Id: I4541f9efed1c05401587a413e9e748d46938bcd1
* Implement rudimentary asyncio support w/ asyncpgMike Bayer2020-08-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Using the approach introduced at https://gist.github.com/zzzeek/6287e28054d3baddc07fa21a7227904e We can now create asyncio endpoints that are then handled in "implicit IO" form within the majority of the Core internals. Then coroutines are re-exposed at the point at which we call into asyncpg methods. Patch includes: * asyncpg dialect * asyncio package * engine, result, ORM session classes * new test fixtures, tests * some work with pep-484 and a short plugin for the pyannotate package, which seems to have so-so results Change-Id: Idbcc0eff72c4cad572914acdd6f40ddb1aef1a7d Fixes: #3414
* Sweep through UPDATE ordered_values a second timeMike Bayer2020-08-121-9/+104
| | | | | | | | | | | | The fix in 180ae7c1a53385f72b0047496ac001ec5099cc3e didn't do much as the code was not preserving parameter order at all, in fact. Reworked stmt_parameters to be delivered in the correct order up front and preserve throughout crud.py which was not being done at all before. Fixes: #5510 Change-Id: I0795c71df73005a25d1bbf216732d41b41e11a5f
* Break scalars() and mappings() into separate objectsMike Bayer2020-08-121-0/+18
| | | | | | | | | | | | The issue of Result.fetchXXX() methods returning Row objects unless filtering is applied will not provide a clear enough API story when type annotations are applied, so break out scalars/mappings into separate wrapper objects. this makes some things more intuitive and other things a little more bumpy. however the return type story is now clearer. Fixes: #5503 Change-Id: I629a061823179680dc0723559183859a67ea4db1
* Correct for update.ordered_values() and use in new testMike Bayer2020-08-111-1/+3
| | | | | | | | | | The test for the parameter fix in c0685e5f41 was not working deterministically on Python 2, so use ordered_values(), however a second issue in ordered_values() was assuming each element was a column, so also test for array-assignment expressions with ordered_values. Change-Id: I944c72a52700ffb4ab5ae1a83ae21f1efc84b505
* render INSERT/UPDATE column expressions up front; pass stateMike Bayer2020-08-081-1/+67
| | | | | | | | | | | | | | | | | | | | | | Fixes related to rendering of complex UPDATE DML which was not correctly preserving positional parameter order in conjunction with DML features that are only known to work on the PostgreSQL database. Both pg8000 and asyncpg use positional parameters which is why these issues are suddenly apparent. crud.py now takes on the task of rendering the column expressions for SET or VALUES so that for the very unusual case that the column expression is a compound expression that includes a bound parameter (namely an array index), the bound parameter order is preserved. Additionally, crud.py passes through the positional_names keyword argument into bindparam_string() which is necessary when CTEs are being rendered, as PG supports complex CTE / INSERT / UPDATE scenarios. Change-Id: I7f03920500e19b721636b84594de78a5bfdcbc82
* Pass schema_translate_map from DDLCompiler to SQLCompilerMike Bayer2020-08-071-0/+77
| | | | | | | | | | | | Fixed issue where the :paramref:`_engine.Connection.execution_options.schema_translate_map` feature would not take effect when the :meth:`_schema.Sequence.next_value` function function for a :class:`_schema.Sequence` were used in the :paramref:`_schema.Column.server_default` parameter and the create table DDL were emitted. Fixes: #5500 Change-Id: I74a9fa13d22749d06c8202669f9ea220d9d984d9
* Implement relationship AND criteria; global loader criteriaMike Bayer2020-08-051-0/+20
| | | | | | | | | | | | | | | | | | | Added the ability to add arbitrary criteria to the ON clause generated by a relationship attribute in a query, which applies to methods such as :meth:`_query.Query.join` as well as loader options like :func:`_orm.joinedload`. Additionally, a "global" version of the option allows limiting criteria to be applied to particular entities in a query globally. Documentation is minimal at this point, new examples will be coming in a subsequent commit. Some adjustments to execution options in how they are represented in the ORMExecuteState as well as well as a few ORM tests that forgot to get merged in a preceding commit. Fixes: #4472 Change-Id: I2b8fc57092dedf35ebd16f6343ad0f0d7d332beb
* Robustness for lambdas, lambda statementsMike Bayer2020-08-054-10/+585
| | | | | | | | | | | | | | | | | | | | in order to accommodate relationship loaders with lambda caching, a lot more is needed. This is a full refactor of the lambda system such that it now has two levels of caching; the first level caches what can be known from the __code__ element, then the next level of caching is against the lambda itself and the contents of __closure__. This allows for the elements inside the lambdas, like columns and entities, to change and then be part of the cache key. Lazy/selectinloads' use of baked queries had to add distinct cache key elements, which was attempted here but overall things needed to be more robust than that. This commit is broken out from the very long and sprawling commit at Id6b5c03b1ce9ddb7b280f66792212a0ef0a1c541 . Change-Id: I29a513c98917b1d503abfdd61e6b6e8800851aa8
* Genericize str() for typesMike Bayer2020-08-013-12/+41
| | | | | | | | | | | | | | Remove lookup logic that attempts to locate a dialect for a type, just use StrSQLTypeCompiler. Cleaned up the internal ``str()`` for datatypes so that all types produce a string representation without any dialect present, including that it works for third-party dialect types without that dialect being present. The string representation defaults to being the UPPERCASE name of that type with nothing else. Fixes: #4262 Change-Id: I02149e8a1ba1e7336149e962939b07ae0df83c6b
* Allow Grouping to pass along proxy_set of elementMike Bayer2020-07-211-0/+20
| | | | | | | | | | Repaired an issue where the "ORDER BY" clause rendering a label name rather than a complete expression, which is particularly important for SQL Server, would fail to occur if the expression were enclosed in a parenthesized grouping in some cases. This case has been added to test support. Fixes: #5470 Change-Id: Ie0e27c39e5d53be78b32f7810f93d2d0536375e7
* Convert remaining ORM APIs to support 2.0 styleMike Bayer2020-07-115-86/+173
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is kind of a mixed bag of all kinds to help get us to 1.4 betas. The documentation stuff is a work in progress. Lots of other relatively small changes to APIs and things. More commits will follow to continue improving the documentation and transitioning to the 1.4/2.0 hybrid documentation. In particular some refinements to Session usage models so that it can match Engine's scoping / transactional patterns, and a decision to start moving away from "subtransactions" completely. * add select().from_statement() to produce FromStatement in an ORM context * begin referring to select() that has "plugins" for the few edge cases where select() will have ORM-only behaviors * convert dynamic.AppenderQuery to its own object that can use select(), though at the moment it uses Query to support legacy join calling forms. * custom query classes for AppenderQuery are replaced by do_orm_execute() hooks for custom actions, a separate gerrit will document this * add Session.get() to replace query.get() * Deprecate session.begin->subtransaction. propose within the test suite a hypothetical recipe for apps that rely on this pattern * introduce Session construction level context manager, sessionmaker context manager, rewrite the whole top of the session_transaction.rst documentation. Establish context manager patterns for Session that are identical to engine * ensure same begin_nested() / commit() behavior as engine * devise all new "join into an external transaction" recipe, add test support for it, add rules into Session so it just works, write new docs. need to ensure this doesn't break anything * vastly reduce the verbosity of lots of session docs as I dont think people read this stuff and it's difficult to keep current in any case * constructs like case(), with_only_columns() really need to move to *columns, add a coercion rule to just change these. * docs need changes everywhere I look. in_() is not in the Core tutorial? how do people even know about it? Remove tons of cruft from Select docs, etc. * build a system for common ORM options like populate_existing and autoflush to populate from execution options. * others? Change-Id: Ia4bea0f804250e54d90b3884cf8aab8b66b82ecf
* Merge remote-tracking branch 'origin/pr/5410'Mike Bayer2020-07-082-4/+8
|\ | | | | | | Change-Id: Ib052efc33ddd58d38e16c0cc721abee668b861d5
| * resolves #5333 typo correction 'explcit' should be 'explicit'.RamonWill2020-07-042-4/+8
| |
* | Merge "Add future=True to create_engine/Session; unify select()"mike bayer2020-07-089-1140/+1109
|\ \
| * | Add future=True to create_engine/Session; unify select()Mike Bayer2020-07-089-1140/+1109
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several weeks of using the future_select() construct has led to the proposal there be just one select() construct again which features the new join() method, and otherwise accepts both the 1.x and 2.x argument styles. This would make migration simpler and reduce confusion. However, confusion may be increased by the fact that select().join() is different Current thinking is we may be better off with a few hard behavioral changes to old and relatively unknown APIs rather than trying to play both sides within two extremely similar but subtly different APIs. At the moment, the .join() thing seems to be the only behavioral change that occurs without the user taking any explicit steps. Session.execute() will still behave the old way as we are adding a future flag. This change also adds the "future" flag to Session() and session.execute(), so that interpretation of the incoming statement, as well as that the new style result is returned, does not occur for existing applications unless they add the use of this flag. The change in general is moving the "removed in 2.0" system further along where we want the test suite to fully pass even if the SQLALCHEMY_WARN_20 flag is set. Get many tests to pass when SQLALCHEMY_WARN_20 is set; this should be ongoing after this patch merges. Improve the RemovedIn20 warning; these are all deprecated "since" 1.4, so ensure that's what the messages read. Make sure the inforamtion link is on all warnings. Add deprecation warnings for parameters present and add warnings to all FromClause.select() types of methods. Fixes: #5379 Fixes: #5284 Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51 References: #5159
* | | ensure we unwrap desc() /label() all the way w/ order byMike Bayer2020-07-071-1/+31
|/ / | | | | | | | | | | | | | | | | | | | | | | | | The deprecated logic to move order_by expressions up into the columns clause needed adjustment to accommodate for a more deeply-wrapped structure when desc() + label() are combined in an order by column. This structure now comes from coercions in 1.4. it's not clear to me at the moment why it's different from 1.3 but this shouldn't really matter. Fixes: #5443 Change-Id: If909a86f715992318d7aa283603197f7711f1d3b
* | introduce deferred lambdasMike Bayer2020-07-032-1/+746
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The coercions system allows us to add in lambdas as arguments to Core and ORM elements without changing them at all. By allowing the lambda to produce a deterministic cache key where we can also cheat and yank out literal parameters means we can move towards having 90% of "baked" functionality in a clearer way right in Core / ORM. As a second step, we can have whole statements inside the lambda, and can then add generation with __add__(), so then we have 100% of "baked" functionality with full support of ad-hoc literal values. Adds some more short_selects tests for the moment for comparison. Other tweaks inside cache key generation as we're trying to approach a certain level of performance such that we can remove the use of "baked" from the loader strategies. As we have not yet closed #4639, however the caching feature has been fully integrated as of b0cfa7379cf8513a821a3dbe3028c4965d9f85bd, we will also add complete caching documentation here and close that issue as well. Closes: #4639 Fixes: #5380 Change-Id: If91f61527236fd4d7ae3cad1f24c38be921c90ba
* | Rename Table.tometadata to to_metadataGord Thompson2020-06-294-47/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Renamed the :meth:`_schema.Table.tometadata` method to :meth:`_schema.Table.to_metadata`. The previous name remains with a deprecation warning. Updated the "decorate" utility function to support decoration of functions that include non-builtins as default values. Moves test for deprecated "databases" package into test/dialect/test_deprecations.py Fixes: #5413 Fixes: #5426 Change-Id: I6ed899871c935f9e46360127c17ccb7cf97cea6e
* | Merge "ORM executemany returning"mike bayer2020-06-281-1/+15
|\ \
| * | ORM executemany returningMike Bayer2020-06-271-1/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Build on #5401 to allow the ORM to take advanage of executemany INSERT + RETURNING. Implemented the feature updated tests to support INSERT DEFAULT VALUES, needed to come up with a new syntax for compiler INSERT INTO table (anycol) VALUES (DEFAULT) which can then be iterated out for executemany. Added graceful degrade to plain executemany for PostgreSQL <= 8.2 Renamed EXECUTEMANY_DEFAULT to EXECUTEMANY_PLAIN Fix issue where unicode identifiers or parameter names wouldn't work with execute_values() under Py2K, because we have to encode the statement and therefore have to encode the insert_single_values_expr too. Correct issue from #5401 to support executemany + return_defaults for a PK that is explicitly pre-generated, meaning we aren't actually getting RETURNING but need to return it from compiled_parameters. Fixes: #5263 Change-Id: Id68e5c158c4f9ebc33b61c06a448907921c2a657
* | | Merge "Fix a wide variety of typos and broken links"mike bayer2020-06-263-48/+48
|\ \ \ | |/ / |/| |
| * | Fix a wide variety of typos and broken linksaplatkouski2020-06-253-48/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Note the PR has a few remaining doc linking issues listed in the comment that must be addressed separately. Signed-off-by: aplatkouski <5857672+aplatkouski@users.noreply.github.com> Closes: #5371 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5371 Pull-request-sha: 7e7d233cf3a0c66980c27db0fcdb3c7d93bc2510 Change-Id: I9c36e8d8804483950db4b42c38ee456e384c59e3
* | | Default psycopg2 executemany mode to "values_only"Mike Bayer2020-06-258-38/+146
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The psycopg2 dialect now defaults to using the very performant ``execute_values()`` psycopg2 extension for compiled INSERT statements, and also impements RETURNING support when this extension is used. This allows INSERT statements that even include an autoincremented SERIAL or IDENTITY value to run very fast while still being able to return the newly generated primary key values. The ORM will then integrate this new feature in a separate change. Implements RETURNING for insert with executemany Adds support to return_defaults() mode and inserted_primary_key to support mutiple INSERTed rows, via return_defauls_rows and inserted_primary_key_rows accessors. within default execution context, new cached compiler getters are used to fetch primary keys from rows inserted_primary_key now returns a plain tuple. this is not yet a row-like object however this can be added. Adds distinct "values_only" and "batch" modes, as "values" has a lot of benefits but "batch" breaks cursor.rowcount psycopg2 minimum version 2.7 so we can remove the large number of checks for very old versions of psycopg2 simplify tests to no longer distinguish between native and non-native json Fixes: #5401 Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
* | Propose using RETURNING for bulk updates, deletesMike Bayer2020-06-231-7/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes several improvements in the area of bulk updates and deletes as well as the new session mechanics. RETURNING is now used for an UPDATE or DELETE statement emitted for a diaelct that supports "full returning" in order to satisfy the "fetch" strategy; this currently includes PostgreSQL and SQL Server. The Oracle dialect does not support RETURNING for more than one row, so a new dialect capability "full_returning" is added in addition to the existing "implicit_returning", indicating this dialect supports RETURNING for zero or more rows, not just a single identity row. The "fetch" strategy will gracefully degrade to the previous SELECT mechanics for dialects that do not support RETURNING. Additionally, the "fetch" strategy will attempt to use evaluation for the VALUES that were UPDATEd, rather than just expiring the updated attributes. Values should be evalutable in all cases where the value is not a SQL expression. The new approach also incurs some changes in the session.execute mechanics, where do_orm_execute() event handlers can now be chained to each return results; this is in turn used by the handler to detect on a per-bind basis if the fetch strategy needs to do a SELECT or if it can do RETURNING. A test suite is added to test_horizontal_shard that breaks up a single UPDATE or DELETE operation among multiple backends where some are SQLite and don't support RETURNING and others are PostgreSQL and do. The session event mechanics are corrected in terms of the "orm pre execute" hook, which now receives a flag "is_reentrant" so that the two ORM implementations for this can skip on their work if they are being called inside of ORMExecuteState.invoke(), where previously bulk update/delete were calling its SELECT a second time. In order for "fetch" to get the correct identity when called as pre-execute, it also requests the identity_token for each mapped instance which is now added as an optional capability of a SELECT for ORM columns. the identity_token that's placed by horizontal_sharding is now made available within each result row, so that even when fetching a merged result of plain rows we can tell which row belongs to which identity token. The evaluator that takes place within the ORM bulk update and delete for synchronize_session="evaluate" now supports the IN and NOT IN operators. Tuple IN is also supported. Fixes: #1653 Change-Id: I2292b56ae004b997cef0ba4d3fc350ae1dd5efc1
* | Apply dialect_options copy fixGord Thompson2020-06-181-0/+79
|/ | | | | Fixes: #5276 Change-Id: Ic608310d4a85934fc9fa4d72daef66323c6e2525
* Turn on caching everywhere, add loggingMike Bayer2020-06-104-8/+300
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* Convert bulk update/delete to new execution modelMike Bayer2020-06-061-0/+58
| | | | | | | | | | | | | | | This reorganizes the BulkUD model in sqlalchemy.orm.persistence to be based on the CompileState concept and to allow plain update() / delete() to be passed to session.execute() where the ORM synchronize session logic will take place. Also gets "synchronize_session='fetch'" working with horizontal sharding. Adding a few more result.scalar_one() types of methods as scalar_one() seems like what is normally desired. Fixes: #5160 Change-Id: I8001ebdad089da34119eb459709731ba6c0ba975
* Default create_constraint to FalseMike Bayer2020-06-032-21/+93
| | | | | | | | | | | | The :paramref:`.Enum.create_constraint` and :paramref:`.Boolean.create_constraint` parameters now default to False, indicating when a so-called "non-native" version of these two datatypes is created, a CHECK constraint will not be generated by default. These CHECK constraints present schema-management maintenance complexities that should be opted in to, rather than being turned on by default. Fixes: #5367 Change-Id: I0a3fb608ce32143fa757546cc17ba2013e93272a
* Refine IN and scalar subquery coercionsMike Bayer2020-06-013-183/+309
| | | | | | | | | | | | | | | | | Ensure IN emits a warning when it coerces a FromClause into a select(), however that it continues to allow the scalar_subquery() coercion to be automatic, particularly since it's not clear that "col IN (select)" is necessarily "scalar" in the case of tuples. Convert the "scalar_subquery()" warning emitted in other cases to be a warning, rather than a deprecation warning. I can't imagine taking this coercion out as it is intuitive and is always going to happen; we just would like to note that an implicit coercion is occurring. Fixes: #5369 Change-Id: I748f01f40bc85c64e2776f9b88ef35641fa8fb5c
* Merge "Improve rendering of core statements w/ ORM elements"mike bayer2020-06-014-0/+175
|\
| * Improve rendering of core statements w/ ORM elementsMike Bayer2020-05-314-0/+175
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch contains a variety of ORM and expression layer tweaks to support ORM constructs in select() statements, without the 1.3.x requiremnt in Query that a full _compile_context() + new select() is needed in order to get a working statement object. Includes such tweaks as the ability to implement aliased class of an aliased class, as we are looking to fully support ACs against subqueries, as well as the ability to access anonymously-labeled ColumnProperty expressions within subqueries by naming the ".key" of the label after the property key. Some tuning to query.join() as well as ORMJoin internals to allow things to work more smoothly. Change-Id: Id810f485c5f7ed971529489b84694e02a3356d6d
* | Add support for "real" sequences in mssqlGord Thompson2020-05-296-59/+98
|/ | | | | | | | | | | | | | | | | Added support for "CREATE SEQUENCE" and full :class:`.Sequence` support for Microsoft SQL Server. This removes the deprecated feature of using :class:`.Sequence` objects to manipulate IDENTITY characteristics which should now be performed using ``mssql_identity_start`` and ``mssql_identity_increment`` as documented at :ref:`mssql_identity`. The change includes a new parameter :paramref:`.Sequence.data_type` to accommodate SQL Server's choice of datatype, which for that backend includes INTEGER and BIGINT. The default starting value for SQL Server's version of :class:`.Sequence` has been set at 1; this default is now emitted within the CREATE SEQUENCE DDL for all backends. Fixes: #4235 Fixes: #4633 Change-Id: I6aa55c441e8146c2f002e2e201a7f645e667b916
* Merge "callcount reductions and refinement for cached queries"mike bayer2020-05-281-3/+16
|\
| * callcount reductions and refinement for cached queriesMike Bayer2020-05-281-3/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit includes that we've removed the "_orm_query" attribute from compile state as well as query context. The attribute created reference cycles and also added method call overhead. As part of this change, the interface for ORMExecuteState changes a bit, as well as the interface for the horizontal sharding extension which now deprecates the "query_chooser" callable in favor of "execute_chooser", which receives the contextual object. This will also work more nicely when we implement the new execution path for bulk updates and deletes. Pre-merge execution options for statement, connection, arguments all up front in Connection. that way they can be passed to the before_execute / after_execute events, and the ExecutionContext doesn't have to merge as second time. Core execute is pretty close to 1.3 now. baked wasn't using the new one()/first()/one_or_none() methods, fixed that. Convert non-buffered cursor strategy to be a stateless singleton. inline all the paths by which the strategy gets chosen, oracle and SQL Server dialects make use of the already-invoked post_exec() hook to establish the alternate strategies, and this is actually much nicer than it was before. Add caching to mapper instance processor for getters. Identified a reference cycle per query that was showing up as a lot of gc cleanup, fixed that. After all that, performance not budging much. Even test_baked_query now runs with significantly fewer function calls than 1.3, still 40% slower. Basically something about the new patterns just makes this slower and while I've walked a whole bunch of them back, it hardly makes a dent. that said, the performance issues are relatively small, in the 20-40% time increase range, and the new caching feature does provide for regular ORM and Core queries that are cached, and they are faster than non-cached. Change-Id: I7b0b0d8ca550c05f79e82f75cd8eff0bbfade053
* | Render table hints in generic SQLMike Bayer2020-05-271-0/+35
|/ | | | | | | | | | | | Added :meth:`.Select.with_hint` output to the generic SQL string that is produced when calling ``str()`` on a statement. Previously, this clause would be omitted under the assumption that it was dialect specific. The hint text is presented within brackets to indicate the rendering of such hints varies among backends. Fixes: #5353 References: #4667 Change-Id: I01d97d6baa993e495519036ec7ecd5ae62856c16
* Convert execution to move through SessionMike Bayer2020-05-254-16/+59
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-244-15/+372
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
View Lorry Controller Status