summaryrefslogtreecommitdiff
path: root/test/engine
Commit message (Collapse)AuthorAgeFilesLines
* more docs for autocommit isolation levelMike Bayer2020-07-121-0/+11
| | | | | | | | this concept is not clear that we offer real DBAPI autocommit everywhere. backport 1.3 with edits as well Change-Id: I2e8328b7fb6e1cdc5453ab29c94276f60c7ca149
* Add future=True to create_engine/Session; unify select()Mike Bayer2020-07-084-200/+184
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several weeks of using the future_select() construct has led to the proposal there be just one select() construct again which features the new join() method, and otherwise accepts both the 1.x and 2.x argument styles. This would make migration simpler and reduce confusion. However, confusion may be increased by the fact that select().join() is different Current thinking is we may be better off with a few hard behavioral changes to old and relatively unknown APIs rather than trying to play both sides within two extremely similar but subtly different APIs. At the moment, the .join() thing seems to be the only behavioral change that occurs without the user taking any explicit steps. Session.execute() will still behave the old way as we are adding a future flag. This change also adds the "future" flag to Session() and session.execute(), so that interpretation of the incoming statement, as well as that the new style result is returned, does not occur for existing applications unless they add the use of this flag. The change in general is moving the "removed in 2.0" system further along where we want the test suite to fully pass even if the SQLALCHEMY_WARN_20 flag is set. Get many tests to pass when SQLALCHEMY_WARN_20 is set; this should be ongoing after this patch merges. Improve the RemovedIn20 warning; these are all deprecated "since" 1.4, so ensure that's what the messages read. Make sure the inforamtion link is on all warnings. Add deprecation warnings for parameters present and add warnings to all FromClause.select() types of methods. Fixes: #5379 Fixes: #5284 Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51 References: #5159
* Add an extra step to pool test_syncMike Bayer2020-06-271-0/+16
| | | | | | | | | Have observed CI failure with windows where not all three connections got pulled out at the same time here as the threads got serialized. make sure all three connections get used. Change-Id: Ic2f7c7de1069358d95035f90c725c7dddd4f34d4
* Propose using RETURNING for bulk updates, deletesMike Bayer2020-06-231-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes several improvements in the area of bulk updates and deletes as well as the new session mechanics. RETURNING is now used for an UPDATE or DELETE statement emitted for a diaelct that supports "full returning" in order to satisfy the "fetch" strategy; this currently includes PostgreSQL and SQL Server. The Oracle dialect does not support RETURNING for more than one row, so a new dialect capability "full_returning" is added in addition to the existing "implicit_returning", indicating this dialect supports RETURNING for zero or more rows, not just a single identity row. The "fetch" strategy will gracefully degrade to the previous SELECT mechanics for dialects that do not support RETURNING. Additionally, the "fetch" strategy will attempt to use evaluation for the VALUES that were UPDATEd, rather than just expiring the updated attributes. Values should be evalutable in all cases where the value is not a SQL expression. The new approach also incurs some changes in the session.execute mechanics, where do_orm_execute() event handlers can now be chained to each return results; this is in turn used by the handler to detect on a per-bind basis if the fetch strategy needs to do a SELECT or if it can do RETURNING. A test suite is added to test_horizontal_shard that breaks up a single UPDATE or DELETE operation among multiple backends where some are SQLite and don't support RETURNING and others are PostgreSQL and do. The session event mechanics are corrected in terms of the "orm pre execute" hook, which now receives a flag "is_reentrant" so that the two ORM implementations for this can skip on their work if they are being called inside of ORMExecuteState.invoke(), where previously bulk update/delete were calling its SELECT a second time. In order for "fetch" to get the correct identity when called as pre-execute, it also requests the identity_token for each mapped instance which is now added as an optional capability of a SELECT for ORM columns. the identity_token that's placed by horizontal_sharding is now made available within each result row, so that even when fetching a merged result of plain rows we can tell which row belongs to which identity token. The evaluator that takes place within the ORM bulk update and delete for synchronize_session="evaluate" now supports the IN and NOT IN operators. Tuple IN is also supported. Fixes: #1653 Change-Id: I2292b56ae004b997cef0ba4d3fc350ae1dd5efc1
* Warn when transaction context manager ends on inactive transactionMike Bayer2020-06-121-1/+20
| | | | | | | | | | if .rollback() or .commit() is called inside the transaction context manager, the transaction object is deactivated. the context manager continues but will not be able to correctly fulfill it's closing state. Ensure a warning is emitted when this happens. Change-Id: I8fc3a73f7c21575dda5bcbd6fb74ddb679771630
* Turn on caching everywhere, add loggingMike Bayer2020-06-102-12/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* Add support for "real" sequences in mssqlGord Thompson2020-05-291-2/+7
| | | | | | | | | | | | | | | | | Added support for "CREATE SEQUENCE" and full :class:`.Sequence` support for Microsoft SQL Server. This removes the deprecated feature of using :class:`.Sequence` objects to manipulate IDENTITY characteristics which should now be performed using ``mssql_identity_start`` and ``mssql_identity_increment`` as documented at :ref:`mssql_identity`. The change includes a new parameter :paramref:`.Sequence.data_type` to accommodate SQL Server's choice of datatype, which for that backend includes INTEGER and BIGINT. The default starting value for SQL Server's version of :class:`.Sequence` has been set at 1; this default is now emitted within the CREATE SEQUENCE DDL for all backends. Fixes: #4235 Fixes: #4633 Change-Id: I6aa55c441e8146c2f002e2e201a7f645e667b916
* callcount reductions and refinement for cached queriesMike Bayer2020-05-281-1/+50
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit includes that we've removed the "_orm_query" attribute from compile state as well as query context. The attribute created reference cycles and also added method call overhead. As part of this change, the interface for ORMExecuteState changes a bit, as well as the interface for the horizontal sharding extension which now deprecates the "query_chooser" callable in favor of "execute_chooser", which receives the contextual object. This will also work more nicely when we implement the new execution path for bulk updates and deletes. Pre-merge execution options for statement, connection, arguments all up front in Connection. that way they can be passed to the before_execute / after_execute events, and the ExecutionContext doesn't have to merge as second time. Core execute is pretty close to 1.3 now. baked wasn't using the new one()/first()/one_or_none() methods, fixed that. Convert non-buffered cursor strategy to be a stateless singleton. inline all the paths by which the strategy gets chosen, oracle and SQL Server dialects make use of the already-invoked post_exec() hook to establish the alternate strategies, and this is actually much nicer than it was before. Add caching to mapper instance processor for getters. Identified a reference cycle per query that was showing up as a lot of gc cleanup, fixed that. After all that, performance not budging much. Even test_baked_query now runs with significantly fewer function calls than 1.3, still 40% slower. Basically something about the new patterns just makes this slower and while I've walked a whole bunch of them back, it hardly makes a dent. that said, the performance issues are relatively small, in the 20-40% time increase range, and the new caching feature does provide for regular ORM and Core queries that are cached, and they are faster than non-cached. Change-Id: I7b0b0d8ca550c05f79e82f75cd8eff0bbfade053
* Merge "Add immutabledict C code"mike bayer2020-05-231-2/+3
|\
| * Add immutabledict C codeMike Bayer2020-05-231-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Start trying to convert fundamental objects to C as we now rely on a fairly small core of things, and 1.4 is having problems with complexity added being slower than the performance gains we are trying to build in. immutabledict here does seem to bench as twice as fast as the Python one, see below. However, it does not appear to be used prominently enough to make any dent in the performance tests. at the very least it may provide us some more lift-and-copy code for more C extensions. import timeit from sqlalchemy.util._collections import not_immutabledict, immutabledict def run(dict_cls): for i in range(1000000): d1 = dict_cls({"x": 5, "y": 4}) d2 = d1.union({"x": 17, "new key": "some other value"}, None) assert list(d2) == ["x", "y", "new key"] print( timeit.timeit( "run(d)", "from __main__ import run, not_immutabledict as d", number=1 ) ) print( timeit.timeit( "run(d)", "from __main__ import run, immutabledict as d", number=1 ) ) output: python: 1.8799766399897635 C code: 0.8880784640205093 Change-Id: I29e7104dc21dcc7cdf895bf274003af2e219bf6d
* | Merge "Fix query string escaping in engine URLs"mike bayer2020-05-221-0/+5
|\ \
| * | Fix query string escaping in engine URLsMiguel Grinberg2020-05-211-0/+5
| |/ | | | | | | | | | | | | | | | | | | | | | | | | Fixed issue in :class:`.URL` object where stringifying the object would not URL encode special characters, preventing the URL from being re-consumable as a real URL. Pull request courtesy Miguel Grinberg. Fixes: #5341 Closes: #5342 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5342 Pull-request-sha: 362ca3398336a3a892e8020530f0c68d4f2d1d01 Change-Id: Ief6218122d1ec0c70479eb1a90e1c16433801924
* | Structural / performance refinementsMike Bayer2020-05-221-19/+95
|/ | | | | | | | | | | | * state connection schema_translate_map entirely in terms of execution options, support for per-execution options as well * use slots for role impls, remove superclass of the roles themselves as this is not needed * tighten loop in resolve, might become a C function Change-Id: Ib98ac9b65022fbf976e49c6060e4c37573528c5f
* Disable "check unicode returns" under Python 3Mike Bayer2020-05-192-13/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | Disabled the "unicode returns" check that runs on dialect startup when running under Python 3, which for many years has occurred in order to test the current DBAPI's behavior for whether or not it returns Python Unicode or Py2K strings for the VARCHAR and NVARCHAR datatypes. The check still occurs by default under Python 2, however the mechanism to test the behavior will be removed in SQLAlchemy 2.0 when Python 2 support is also removed. This logic was very effective when it was needed, however now that Python 3 is standard, all DBAPIs are expected to return Python 3 strings for character datatypes. In the unlikely case that a third party DBAPI does not support this, the conversion logic within :class:`.String` is still available and the third party dialect may specify this in its upfront dialect flags by setting the dialect level flag ``returns_unicode_strings`` to one of :attr:`.String.RETURNS_CONDITIONAL` or :attr:`.String.RETURNS_BYTES`, both of which will enable Unicode conversion even under Python 3. As part of this change, disabling testing of the doctest tutorials under Python 2. Fixes: #5315 Change-Id: I1260e894611409d3b7fe1a92bd90c52043bbcf19
* Update transaction / connection handlingMike Bayer2020-05-172-237/+619
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | step one, do away with __connection attribute and using awkward AttributeError logic step two, move all management of "connection._transaction" into the transaction objects themselves where it's easier to follow. build MarkerTransaction that takes the role of "do-nothing block" new connection datamodel is: connection._transaction, always a root, connection._nested_transaction, always a nested. nested transactions still chain to each other as this is still sort of necessary but they consider the root transaction separately, and the marker transactions not at all. introduce new InvalidRequestError subclass PendingRollbackError. Apply to connection and session for all cases where a transaction needs to be rolled back before continuing. Within Connection, both PendingRollbackError as well as ResourceClosedError are now raised directly without being handled by handle_dbapi_error(); this removes these two exception cases from the handle_error event handler as well as from StatementError wrapping, as these two exceptions are not statement oriented and are instead programmatic issues, that the application is failing to handle database errors properly. Revise savepoints so that when a release fails, they set themselves as inactive so that their rollback() method does not throw another exception. Give savepoints another go on MySQL, can't get release working however get support for basic round trip going Fixes: #5327 Change-Id: Ia3cbbf56d4882fcc7980f90519412f1711fae74d
* Actively unset reset agent in discard transactionMike Bayer2020-05-141-1/+15
| | | | | | | | | | | | | | | The assumptions in _discard_transaction from 916e1fea25afcd07fa1d1d2f72043b372cd02223 were too narrow, assuming that if the given transaction were not our "current" one, that this would not be the reset agent. however as the legacy behvaior is that even a "nested" transaction gets set as "self._transaction", this did not accommodate for the nested transaction being thrown away. We will attempt to refine all of this logic in #5327 for 1.4 /master assuming this is feasible for the full suite of current use cases. Fixes: #5326 Change-Id: I6787e82c9e50c23317f87d0d094122c6a6f066da
* Assert reset agent always set correctly and is activeMike Bayer2020-05-132-0/+61
| | | | | | | | | | | | | | | | | | | | Fixed fairly critical issue where the DBAPI connection could be returned to the connection pool while still in an un-rolled-back state. The reset agent responsible for rolling back the connection could be corrupted in the case that the transaction was "closed" without being rolled back or committed, which can occur in some scenarios when using ORM sessions and emitting .close() in a certain pattern involving savepoints. The fix ensures that the reset agent is always active. note that the reset agent will go away in 2.0 and the only real purpose of it is for logging of ROLLBACK. Apparently with the SQLite singleton engine in the test suite, there are some strucutral mismatches in the test fixtures where the reset agent is getting set differently than the transaction likely due to the same connection being shared in multiple context, though it's unclear. Fixes: #5326 Change-Id: If056870ea70a2d9a1749768988d5e023f3061b31
* Propose Result as immediate replacement for ResultProxyMike Bayer2020-05-011-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As progress is made on the _future.Result, including breaking it out such that DBAPI behaviors are local to specific implementations, it becomes apparent that the Result object is a functional superset of ResultProxy and that basic operations like fetchone(), fetchall(), and fetchmany() behave pretty much exactly the same way on the new object. Reorganize things so that ResultProxy is now referred to as LegacyCursorResult, which subclasses CursorResult that represents the DBAPI-cursor version of Result, making use of a multiple inheritance pattern so that the functionality of Result is also available in non-DBAPI contexts, as will be necessary for some ORM patterns. Additionally propose the composition system for Result that will form the basis for ORM-alternative result systems such as horizontal sharding and dogpile cache. As ORM results will soon be coming directly from instances of Result, these extensions will instead build their own ResultFetchStrategies that perform the special steps to create composed or cached result sets. Also considering at the moment not emitting deprecation warnings for fetchXYZ() methods; the immediate issue is Keystone tests are calling upon it, but as the implementations here are proving to be not in any kind of conflict with how Result works, there's not too much issue leaving them around and deprecating at some later point. References: #5087 References: #4395 Fixes: #4959 Change-Id: I8091919d45421e3f53029b8660427f844fee0228
* Create initial 2.0 engine implementationMike Bayer2020-04-164-51/+870
| | | | | | | | | | | | | | | | | | | Implemented the SQLAlchemy 2 :func:`.future.create_engine` function which is used for forwards compatibility with SQLAlchemy 2. This engine features always-transactional behavior with autobegin. Allow execution options per statement execution. This includes that the before_execute() and after_execute() events now accept an additional dictionary with these options, empty if not passed; a legacy event decorator is added for backwards compatibility which now also emits a deprecation warning. Add some basic tests for execution, transactions, and the new result object. Build out on a new testing fixture that swaps in the future engine completely to start with. Change-Id: I70e7338bb3f0ce22d2f702537d94bb249bd9fb0a Fixes: #4644
* The :meth:`.Inspector.reflecttable` was renamed to ↵Federico Caselli2020-04-102-4/+17
| | | | | | | :meth:`.Inspector.reflect_table`. Fixes: #5244 Change-Id: I2b12fd69ed24ce1ede8f6ed5cb14cc7761308ee3
* Remove code deprecated before version 1.1Federico Caselli2020-04-092-83/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Remove deprecated method ``get_primary_keys` in the :class:`.Dialect` and :class:`.Inspector` classes. - Remove deprecated event ``dbapi_error`` and the method ``ConnectionEvents.dbapi_error`. - Remove support for deprecated engine URLs of the form ``postgres://``. - Remove deprecated dialect ``mysql+gaerdbms``. - Remove deprecated parameter ``quoting`` from :class:`.mysql.ENUM` and :class:`.mysql.SET` in the ``mysql`` dialect. - Remove deprecated function ``comparable_property``. and function ``comparable_using`` in the declarative extension. - Remove deprecated function ``compile_mappers``. - Remove deprecated method ``collection.linker``. - Remove deprecated method ``Session.prune`` and parameter ``Session.weak_identity_map``. This change also removes the class ``StrongInstanceDict``. - Remove deprecated parameter ``mapper.order_by``. - Remove deprecated parameter ``Session._enable_transaction_accounting`. - Remove deprecated parameter ``Session.is_modified.passive``. - Remove deprecated class ``Binary``. Please use :class:`.LargeBinary`. - Remove deprecated methods ``Compiled.compile``, ``ClauseElement.__and__`` and ``ClauseElement.__or__`` and attribute ``Over.func``. - Remove deprecated ``FromClause.count`` method. - Remove deprecated parameter ``Table.useexisting``. - Remove deprecated parameters ``text.bindparams`` and ``text.typemap``. - Remove boolean support for the ``passive`` parameter in ``get_history``. - Remove deprecated ``adapt_operator`` in ``UserDefinedType.Comparator``. Fixes: #4643 Change-Id: Idcd390c77bf7b0e9957907716993bdaa3f1a1763
* Merge "Convert schema_translate to a post compile"mike bayer2020-03-241-32/+29
|\
| * Convert schema_translate to a post compileMike Bayer2020-03-241-32/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Revised the :paramref:`.Connection.execution_options.schema_translate_map` feature such that the processing of the SQL statement to receive a specific schema name occurs within the execution phase of the statement, rather than at the compile phase. This is to support the statement being efficiently cached. Previously, the current schema being rendered into the statement for a particular run would be considered as part of the cache key itself, meaning that for a run against hundreds of schemas, there would be hundreds of cache keys, rendering the cache much less performant. The new behavior is that the rendering is done in a similar manner as the "post compile" rendering added in 1.4 as part of :ticket:`4645`, :ticket:`4808`. Fixes: #5004 Change-Id: Ia5c89eb27cc8dc2c5b8e76d6c07c46290a7901b6
* | Implement autocommit isolation level for cx_oracleGord Thompson2020-03-231-19/+10
|/ | | | | | | | | | | | | | | | As part of this change Oracle also gets the concept of a default isolation level, however since Oracle does not provide a fixed method of knowing what the isolation level would be without a server side transaction actually in progress, for now we hardcode just to "READ COMMITTED". Enhanced the test suite for isolation level testing in the dialect test suite and added features to requirements so that the supported isolation levels can be reported generically for dialects. Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #5200 Change-Id: I2c4d49da9ff80ccc228c21e196ec9a961de53478
* Deprecate plain string in execute and introduce `exec_driver_sql`Federico Caselli2020-03-217-318/+624
| | | | | | | | | | | | | | | Execution of literal sql string is deprecated in the :meth:`.Connection.execute` and a warning is raised when used stating that it will be coerced to :func:`.text` in a future release. To execute a raw sql string the new connection method :meth:`.Connection.exec_driver_sql` was added, that will retain the previous behavior, passing the string to the DBAPI driver unchanged. Usage of scalar or tuple positional parameters in :meth:`.Connection.execute` is also deprecated. Fixes: #4848 Fixes: #5178 Change-Id: I2830181054327996d594f7f0d59c157d477c3aa9
* Support inspection of computed columnFederico Caselli2020-03-151-0/+36
| | | | | | | | | | | | | | | | | | | Added support for reflection of "computed" columns, which are now returned as part of the structure returned by :meth:`.Inspector.get_columns`. When reflecting full :class:`.Table` objects, computed columns will be represented using the :class:`.Computed` construct. Also improve the documentation in :meth:`Inspector.get_columns`, correctly listing all the returned keys. Fixes: #5063 Fixes: #4051 Closes: #5064 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5064 Pull-request-sha: ba00fc321ce468f8885aad23b3dd33c789e50fbe Change-Id: I789986554fc8ac7f084270474d0b2c12046b1cc2
* Fix tests failing for SQLite file databases; repair provisioningGord Thompson2020-03-132-8/+13
| | | | | | | | | | | | | | | | | | | | | | 1. ensure provision.py loads dialect implementations when running reap_dbs.py. Reapers haven't been working since 598f2f7e557073f29563d4d567f43931fc03013f . 2. add some exclusion rules to allow the sqlite_file target to work; add to tox. 3. add reap dbs target for SQLite, repair SQLite drop_db routine which also wasn't doing the right thing for memory databases etc. 4. Fix logging in provision files, as the main provision logger is the one that's enabled by reap_dbs and maybe others, have all the provision files use the provision logger. Fixes: #5180 Fixes: #5168 Change-Id: Ibc1b0106394d20f5bcf847f37b09d185f26ac9b5
* Rework select(), CompoundSelect() in terms of CompileStateMike Bayer2020-03-101-73/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Continuation of I408e0b8be91fddd77cf279da97f55020871f75a9 - add an options() method to the base Generative construct. this will be where ORM options can go - Change Null, False_, True_ to be singletons, so that we aren't instantiating them and having to use isinstance. The previous issue with this was that they would produce dupe labels in SELECT statements. Apply the duplicate column logic, newly added in 1.4, to these objects as well as to non-apply-labels SELECT statements in general as a means of improving this. - create a revised system for generating ClauseList compilation constructs that simplfies up front creation to not actually use ClauseList; a simple tuple is rendered by the compiler using the same constrcution rules as what are used for ClauseList but without creating the actual object. Apply to Select, CompoundSelect, revise Update, Delete - Select, CompoundSelect get an initial CompileState implementation. All methods used only within compilation are moved here - refine update/insert/delete compile state to not require an outside boolean - refine and simplify Select._copy_internals - rework bind(), which is going away, to not use some of the internal traversal stuff - remove "autocommit", "for_update" parameters from Select, references #4643 - remove "autocommit" parameter from TextClause , references #4643 - add deprecation warnings for statement.execute(), engine.execute(), statement.scalar(), engine.scalar(). Fixes: #5193 Change-Id: I04ca0152b046fd42c5054ba10f37e43fc6e5a57b
* Run handle_error for any exceptions raised in execute_context()Mike Bayer2020-03-031-0/+69
| | | | | | | | | | | | | | | | | | | | | Observing a SQLite connection/cursor being hung on test_resultset -> PositionalTextTest -> test_dupe_col_obj. this uses connectionless execution and the result object fails to be constructed. When that happens, there is no path for the cursor or connection to be closed / released. Recent changes with the exception assertions in #4849 seem to be causing a cycle to last a little longer than usual which is exposing this issue for one particular test on SQLite. As we want to get rid of weakref cleanup, evaluate why we dont have handle_dbapi_exception for this whole block, as after_cursor_execute can raise, result construction can raise, autocommit can raise, close can raise, there does not seem to be a reason these things should be outside of the block that gets cleaned up. Fixes: #5182 Change-Id: I640ac55e8c5f39d287f779fbb5dc0ab727218ca3
* Merge "Ensure all nested exception throws have a cause"mike bayer2020-03-024-15/+16
|\
| * Ensure all nested exception throws have a causeMike Bayer2020-03-024-15/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Applied an explicit "cause" to most if not all internally raised exceptions that are raised from within an internal exception catch, to avoid misleading stacktraces that suggest an error within the handling of an exception. While it would be preferable to suppress the internally caught exception in the way that the ``__suppress_context__`` attribute would, there does not as yet seem to be a way to do this without suppressing an enclosing user constructed context, so for now it exposes the internally caught exception as the cause so that full information about the context of the error is maintained. Fixes: #4849 Change-Id: I55a86b29023675d9e5e49bc7edc5a2dc0bcd4751
* | Remove obsolete util.compat.nestedGord Thompson2020-03-021-11/+8
|/ | | | | Fixes: #5177 Change-Id: Ie02b0c065e3833f43e056ad9c31f414871d9e8ee
* Don't call pre_ping for fresh connectionMike Bayer2020-02-261-4/+82
| | | | | | | | | | The pool "pre-ping" feature has been refined to not invoke for a DBAPI connection that was just opened in the same checkout operation. pre ping only applies to a DBAPI connection that's been checked into the pool and is being checked out again. Fixes: #4524 Change-Id: Ibe3dfb709dbdc24aa94e96513cfbea456c33b895
* Deprecate connection branchingMike Bayer2020-02-214-90/+136
| | | | | | | | | | | | | | | The :meth:`.Connection.connect` method is deprecated as is the concept of "connection branching", which copies a :class:`.Connection` into a new one that has a no-op ".close()" method. This pattern is oriented around the "connectionless execution" concept which is also being removed in 2.0. As part of this change we begin to move the internals away from "connectionless execution" overall. Remove the "connectionless execution" concept from the reflection internals and replace with explicit patterns at the Inspector level. Fixes: #5131 Change-Id: Id23d28a9889212ac5ae7329b85136157815d3e6f
* Do away with pool._refsMike Bayer2020-02-011-5/+7
| | | | | | | | | | | | | This collection was added only for the benefit of unit tests and is unnecessary for the pool to function. As SQLAlchemy 2.0 will be removing the automatic handling of connections that are garbage collection, remove this collection so that we ultimately don't need a weakref handler to do anything within the pool. The handler will do nothing other than emit a warning that a connection was dereferenced without being explicitly returned to the pool, invalidated, or detached. Change-Id: I4ca196270d5714efbac44dbf6f034e8c7f0af58a
* Deprecate empty or_() and and_()Federico Caselli2020-01-251-2/+2
| | | | | | | | | | | | | | | | | | | | | | | Creating an :func:`.and_` or :func:`.or_` construct with no arguments or empty ``*args`` will now emit a deprecation warning, as the SQL produced is a no-op (i.e. it renders as a blank string). This behavior is considered to be non-intuitive, so for empty or possibly empty :func:`.and_` or :func:`.or_` constructs, an appropriate default boolean should be included, such as ``and_(True, *args)`` or ``or_(False, *args)``. As has been the case for many major versions of SQLAlchemy, these particular boolean values will not render if the ``*args`` portion is non-empty. As there are some internal cases where an empty and_() construct is used in order to build an optional WHERE expression, a private utility function is added to suit this use case. Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #5054 Closes: #5062 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5062 Pull-request-sha: 5ca2f27281977d74e390148c0fb8deaa0e0e4ad9 Change-Id: I599b9c8befa64d9a59a35ad7dd84ff400e3aa647
* Remove jython code, remove all jython / pypy symbolsMike Bayer2020-01-172-6/+1
| | | | | | | | | | | | | | | | | | | | | | Removed all dialect code related to support for Jython and zxJDBC. Jython has not been supported by SQLAlchemy for many years and it is not expected that the current zxJDBC code is at all functional; for the moment it just takes up space and adds confusion by showing up in documentation. At the moment, it appears that Jython has achieved Python 2.7 support in its releases but not Python 3. If Jython were to be supported again, the form it should take is against the Python 3 version of Jython, and the various zxJDBC stubs for various backends should be implemented as a third party dialect. Additionally modernized logic that distinguishes between "cpython" and "pypy" to instead look at platform.python_distribution() which reliably tells us if we are cPython or not; all booleans which previously checked for pypy and sometimes jython are now converted to be "not cpython", this impacts the test suite for tests that are cPython centric. Fixes: #5094 Change-Id: I226cb55827f997daf6b4f4a755c18e7f4eb8d9ad
* Use context managers for threading.Lock()Heckad2020-01-031-4/+1
| | | | | | | | | | | | | (zzzeek:) For some reason I thought that threading.Lock() still did not support context managers, at least in Python 2, however this does not seem to be the case. Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Closes: #5069 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5069 Pull-request-sha: efeac06dda5afdbe33abcf9b27c8b5b5725c8444 Change-Id: Ic64fcd99cd587bc70b4ecc5b45d8205b5c76eff2
* Note time passage requirement for pool.invalidate()Mike Bayer2019-12-271-1/+16
| | | | | | | | | | | | | | | | | | For Windows, time.time() may only have 16 millisecond accuracy, so invalidation routines which compare the time.time() of invalidate() to the time.time() when the ConnectionRecord last connected may fail in a unit test environment that does not pause at least this much time since the ConnectionRecord startup. Using >= for comparison instead of > was considered but this only leads to more confusing results as the ConnecitonRecord goes into a re-connect loop as time continues to not pass. Overall, while using routines such as Python 3.7's time_ns() might be helpful, for now make sure tests which rely on this are marked under timing intensive and add small sleeps. Change-Id: I1a7162e67912d22c135fa517b687a073f8fd9151
* Use expanding IN for all literal value IN expressionsMike Bayer2019-12-221-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | The "expanding IN" feature, which generates IN expressions at query execution time which are based on the particular parameters associated with the statement execution, is now used for all IN expressions made against lists of literal values. This allows IN expressions to be fully cacheable independently of the list of values being passed, and also includes support for empty lists. For any scenario where the IN expression contains non-literal SQL expressions, the old behavior of pre-rendering for each position in the IN is maintained. The change also completes support for expanding IN with tuples, where previously type-specific bind processors weren't taking effect. As part of this change, a more explicit separation between "literal execute" and "post compile" bound parameters is being made; as the "ansi bind rules" feature is rendering bound parameters inline, as we now support "postcompile" generically, these should be used here, however we have to render literal values at execution time even for "expanding" parameters. new test fixtures etc. are added to assert everything goes to the right place. Fixes: #4645 Change-Id: Iaa2b7bfbfaaf5b80799ee17c9b8507293cba6ed1
* Add sequence support for MariaDB 10.3+.Gord Thompson2019-11-261-5/+12
| | | | | | | | | | | | | | | | | | | | Added support for use of the :class:`.Sequence` construct with MariaDB 10.3 and greater, as this is now supported by this database. The construct integrates with the :class:`.Table` object in the same way that it does for other databases like PostrgreSQL and Oracle; if is present on the integer primary key "autoincrement" column, it is used to generate defaults. For backwards compatibility, to support a :class:`.Table` that has a :class:`.Sequence` on it to support sequence only databases like Oracle, while still not having the sequence fire off for MariaDB, the optional=True flag should be set, which indicates the sequence should only be used to generate the primary key if the target database offers no other option. Fixes: #4976 Closes: #4996 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4996 Pull-request-sha: cb2e1426ea0b6bc6c93dbe8f033a11df9d8c4915 Change-Id: I507bc405eee6cae2c5991345d0eac53a37fe7512
* Fix exclusions for multiple fails_onMike Bayer2019-11-091-2/+0
| | | | | | | | | | | | | | The fails_on decorator was not being interpreted correctly when multiple were present. Remove obsolete fails_on from test_types that no longer take place for MySQL, Oracle. Ensure test_types tests are using __backend__ mark currently failing Oracle interval tests Change-Id: If8db0c02b31a8008fd1673c2380f1f974c3806a6
* Merge "Remove deprecated elements"mike bayer2019-10-112-89/+1
|\
| * Remove deprecated elementsAsif Saif Uddin (Auvi)2019-10-112-89/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Includes: PassiveDefault SchemaItem.quote Table.useexisting Table.quote_schema Table.append_ddl_listener MetaData.append_ddl_listener Metadata.reflect kw parameter (use reflect() method) DDL.execute_at DDL.on Partially-fixes: #4643 Closes: #4893 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4893 Pull-request-sha: 860eb6a253fe4a95685b4f5f3349b19823a304f3 Change-Id: I0f5b8a873e7581365ff8dba48eab358d9e8e7b13
* | pass executemany context to _repr_paramsMike Bayer2019-10-111-2/+132
|/ | | | | | | | | | | | | | | | | | Fixed bug where parameter repr as used in logging and error reporting needs additional context in order to distinguish between a list of parameters for a single statement and a list of parameter lists, as the "list of lists" structure could also indicate a single parameter list where the first parameter itself is a list, such as for an array parameter. The engine/connection now passes in an additional boolean indicating how the parameters should be considered. The only SQLAlchemy backend that expects arrays as parameters is that of psycopg2 which uses pyformat parameters, so this issue has not been too apparent, however as other drivers that use positional gain more features it is important that this be supported. It also eliminates the need for the parameter repr function to guess based on the parameter structure passed. Fixes: #4902 Change-Id: I086246ee0eb51484adbefd83e07295fa56576c5f
* Remove deprecated extension and similar classesMike Bayer2019-10-061-611/+0
| | | | | | | | | | All long-deprecated "extension" classes have been removed, including MapperExtension, SessionExtension, PoolListener, ConnectionProxy, AttributExtension. These classes have been deprecated since version 0.7 long superseded by the event listener system. Fixes: #4638 Change-Id: If4156d4956b10847bd93b6408a7c52ff5168db9b
* Improve testcase condition statement for dialectsj003562872019-09-051-4/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We could get dialect.requires_name_normalize rather than use hard code as "firebird" or "oracle", since we have add `normalize` attribute for quite a long time. ### Description Use `dialect.requires_name_normalize` instead `testing.against("firebird", "oracle")` ### Checklist <!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once) --> This pull request is: - [ ] A documentation / typographical error fix - Good to go, no issue or tests are needed - [x] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. - [ ] A new feature implementation - please include the issue number, and create an issue if none exists, which must include a complete example of how the feature would look. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. **Have a nice day!** Closes: #4843 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4843 Pull-request-sha: 304fe67b06c1f010a164806a12b68cfce9bd0d2b Change-Id: I276f781482779473258f9269074847e283711b05
* Add hide_parameters flag to create_engineMike Bayer2019-08-222-2/+68
| | | | | | | | | Added new parameter :paramref:`.create_engine.hide_parameters` which when set to True will cause SQL parameters to no longer be logged, nor rendered in the string representation of a :class:`.StatementError` object. Fixes: #4815 Change-Id: Ib87f868b6936cf6b42b192644e9d732ec24266c2
* Add new "exec_once_unless_exception" system; apply to dialect.initializeMike Bayer2019-08-181-0/+70
| | | | | | | | | | | | | | | | | | Fixed an issue whereby if the dialect "initialize" process which occurs on first connect would encounter an unexpected exception, the initialize process would fail to complete and then no longer attempt on subsequent connection attempts, leaving the dialect in an un-initialized, or partially initialized state, within the scope of parameters that need to be established based on inspection of a live connection. The "invoke once" logic in the event system has been reworked to accommodate for this occurrence using new, private API features that establish an "exec once" hook that will continue to allow the initializer to fire off on subsequent connections, until it completes without raising an exception. This does not impact the behavior of the existing ``once=True`` flag within the event system. Fixes: #4807 Change-Id: Iec32999b61b6af4b38b6719e0c2651454619078c
* Modernize internal reflectionMike Bayer2019-08-065-129/+282
| | | | | | | | | | | | | | | | | | | | | | | - Deprecated remaining engine-level introspection and utility methods including :meth:`.Engine.run_callable`, :meth:`.Engine.transaction`, :meth:`.Engine.table_names`, :meth:`.Engine.has_table`. The utility methods are superseded by modern context-manager patterns, and the table introspection tasks are suited by the :class:`.Inspector` object. - The internal dialect method ``Dialect.reflecttable`` has been removed. A review of third party dialects has not found any making use of this method, as it was already documented as one that should not be used by external dialects. Additionally, the private ``Engine._run_visitor`` method is also removed. - The long-deprecated ``Inspector.get_table_names.order_by`` parameter has been removed. - The :paramref:`.Table.autoload_with` parameter now accepts an :class:`.Inspector` object directly, as well as any :class:`.Engine` or :class:`.Connection` as was the case before. Fixes: #4755 Change-Id: Iec3a8b0f3e298ba87d532b16fac1e1132f464e21
View Lorry Controller Status