summaryrefslogtreecommitdiff
path: root/test/aaa_profiling
Commit message (Collapse)AuthorAgeFilesLines
* Update select usage to use the new 1.4 formatFederico Caselli2020-09-082-5/+5
| | | | | | | | | | | | | | | | This change includes mainly that the bracketed use within select() is moved to positional, and keyword arguments are removed from calls to the select() function. it does not yet fully address other issues such as keyword arguments passed to the table.select(). Additionally, allows False / None to both be considered as "disable" for all of select.correlate(), select.correlate_except(), query.correlate(), which establishes consistency with passing of ``False`` for the legact select(correlate=False) argument. Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
* normalize execute style for events, 2.0Mike Bayer2020-08-201-2/+2
| | | | | | | | | | | | | | | The _execute_20 and exec_driver_sql methods should wrap up the parameters so that they represent the single list / single dictionary style of invocation into the legacy methods. then the before_ after_ execute event handlers should be receiving the parameter dictionary as a single dictionary. this requires that we break out distill_params to work differently if event handlers are present. additionally, add deprecation warnings for old argument passing styles. Change-Id: I97cb4d06adfcc6b889f10d01cc7775925cffb116
* Convert lazy loader, selectinload, load_on_ident to lambda statementsMike Bayer2020-08-051-4/+7
| | | | | | | | | Building on newly robust lambdas in I29a513c98917b1d503abfdd61e6b6e8800851aa8, convert key loading off of the "baked" system so that baked is no longer used by the ORM. Change-Id: I3abfb45dd6e50f84f29d39434caa0b550ce27864
* Convert remaining ORM APIs to support 2.0 styleMike Bayer2020-07-111-67/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is kind of a mixed bag of all kinds to help get us to 1.4 betas. The documentation stuff is a work in progress. Lots of other relatively small changes to APIs and things. More commits will follow to continue improving the documentation and transitioning to the 1.4/2.0 hybrid documentation. In particular some refinements to Session usage models so that it can match Engine's scoping / transactional patterns, and a decision to start moving away from "subtransactions" completely. * add select().from_statement() to produce FromStatement in an ORM context * begin referring to select() that has "plugins" for the few edge cases where select() will have ORM-only behaviors * convert dynamic.AppenderQuery to its own object that can use select(), though at the moment it uses Query to support legacy join calling forms. * custom query classes for AppenderQuery are replaced by do_orm_execute() hooks for custom actions, a separate gerrit will document this * add Session.get() to replace query.get() * Deprecate session.begin->subtransaction. propose within the test suite a hypothetical recipe for apps that rely on this pattern * introduce Session construction level context manager, sessionmaker context manager, rewrite the whole top of the session_transaction.rst documentation. Establish context manager patterns for Session that are identical to engine * ensure same begin_nested() / commit() behavior as engine * devise all new "join into an external transaction" recipe, add test support for it, add rules into Session so it just works, write new docs. need to ensure this doesn't break anything * vastly reduce the verbosity of lots of session docs as I dont think people read this stuff and it's difficult to keep current in any case * constructs like case(), with_only_columns() really need to move to *columns, add a coercion rule to just change these. * docs need changes everywhere I look. in_() is not in the Core tutorial? how do people even know about it? Remove tons of cruft from Select docs, etc. * build a system for common ORM options like populate_existing and autoflush to populate from execution options. * others? Change-Id: Ia4bea0f804250e54d90b3884cf8aab8b66b82ecf
* Add future=True to create_engine/Session; unify select()Mike Bayer2020-07-082-12/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several weeks of using the future_select() construct has led to the proposal there be just one select() construct again which features the new join() method, and otherwise accepts both the 1.x and 2.x argument styles. This would make migration simpler and reduce confusion. However, confusion may be increased by the fact that select().join() is different Current thinking is we may be better off with a few hard behavioral changes to old and relatively unknown APIs rather than trying to play both sides within two extremely similar but subtly different APIs. At the moment, the .join() thing seems to be the only behavioral change that occurs without the user taking any explicit steps. Session.execute() will still behave the old way as we are adding a future flag. This change also adds the "future" flag to Session() and session.execute(), so that interpretation of the incoming statement, as well as that the new style result is returned, does not occur for existing applications unless they add the use of this flag. The change in general is moving the "removed in 2.0" system further along where we want the test suite to fully pass even if the SQLALCHEMY_WARN_20 flag is set. Get many tests to pass when SQLALCHEMY_WARN_20 is set; this should be ongoing after this patch merges. Improve the RemovedIn20 warning; these are all deprecated "since" 1.4, so ensure that's what the messages read. Make sure the inforamtion link is on all warnings. Add deprecation warnings for parameters present and add warnings to all FromClause.select() types of methods. Fixes: #5379 Fixes: #5284 Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51 References: #5159
* Remove _generate_path_cache_key()Mike Bayer2020-06-291-41/+0
| | | | | | | | | | loader options can now make a deterministic cache key based on the structure they are given, and this accommodates for aliased classes as well so that these cache keys are now "safe". Have baked query call upon the regular cache key method. Change-Id: Iaa2ef4064cfb16146f415ca73080f32003dd830d
* Propose using RETURNING for bulk updates, deletesMike Bayer2020-06-231-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes several improvements in the area of bulk updates and deletes as well as the new session mechanics. RETURNING is now used for an UPDATE or DELETE statement emitted for a diaelct that supports "full returning" in order to satisfy the "fetch" strategy; this currently includes PostgreSQL and SQL Server. The Oracle dialect does not support RETURNING for more than one row, so a new dialect capability "full_returning" is added in addition to the existing "implicit_returning", indicating this dialect supports RETURNING for zero or more rows, not just a single identity row. The "fetch" strategy will gracefully degrade to the previous SELECT mechanics for dialects that do not support RETURNING. Additionally, the "fetch" strategy will attempt to use evaluation for the VALUES that were UPDATEd, rather than just expiring the updated attributes. Values should be evalutable in all cases where the value is not a SQL expression. The new approach also incurs some changes in the session.execute mechanics, where do_orm_execute() event handlers can now be chained to each return results; this is in turn used by the handler to detect on a per-bind basis if the fetch strategy needs to do a SELECT or if it can do RETURNING. A test suite is added to test_horizontal_shard that breaks up a single UPDATE or DELETE operation among multiple backends where some are SQLite and don't support RETURNING and others are PostgreSQL and do. The session event mechanics are corrected in terms of the "orm pre execute" hook, which now receives a flag "is_reentrant" so that the two ORM implementations for this can skip on their work if they are being called inside of ORMExecuteState.invoke(), where previously bulk update/delete were calling its SELECT a second time. In order for "fetch" to get the correct identity when called as pre-execute, it also requests the identity_token for each mapped instance which is now added as an optional capability of a SELECT for ORM columns. the identity_token that's placed by horizontal_sharding is now made available within each result row, so that even when fetching a merged result of plain rows we can tell which row belongs to which identity token. The evaluator that takes place within the ORM bulk update and delete for synchronize_session="evaluate" now supports the IN and NOT IN operators. Tuple IN is also supported. Fixes: #1653 Change-Id: I2292b56ae004b997cef0ba4d3fc350ae1dd5efc1
* Turn on caching everywhere, add loggingMike Bayer2020-06-104-20/+106
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* Convert bulk update/delete to new execution modelMike Bayer2020-06-061-0/+1
| | | | | | | | | | | | | | | This reorganizes the BulkUD model in sqlalchemy.orm.persistence to be based on the CompileState concept and to allow plain update() / delete() to be passed to session.execute() where the ORM synchronize session logic will take place. Also gets "synchronize_session='fetch'" working with horizontal sharding. Adding a few more result.scalar_one() types of methods as scalar_one() seems like what is normally desired. Fixes: #5160 Change-Id: I8001ebdad089da34119eb459709731ba6c0ba975
* Inline a few ORM arguments, othersMike Bayer2020-06-031-1/+3
| | | | | | small changes Change-Id: Id89a0651196c431d0aaf6935f5a4e7b12dd70c6c
* Improve rendering of core statements w/ ORM elementsMike Bayer2020-05-311-1/+4
| | | | | | | | | | | | | | | | | | | | This patch contains a variety of ORM and expression layer tweaks to support ORM constructs in select() statements, without the 1.3.x requiremnt in Query that a full _compile_context() + new select() is needed in order to get a working statement object. Includes such tweaks as the ability to implement aliased class of an aliased class, as we are looking to fully support ACs against subqueries, as well as the ability to access anonymously-labeled ColumnProperty expressions within subqueries by naming the ".key" of the label after the property key. Some tuning to query.join() as well as ORMJoin internals to allow things to work more smoothly. Change-Id: Id810f485c5f7ed971529489b84694e02a3356d6d
* Remove loader option cycleMike Bayer2020-05-291-0/+6
| | | | | | | | removed a reference cycle set up by loader options due to the attribute dictionary containing Load objects that reference that dictionary. Change-Id: Ie3159a084f819ae44ca4992b0dbe094fb69b2fa7
* callcount reductions and refinement for cached queriesMike Bayer2020-05-281-4/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit includes that we've removed the "_orm_query" attribute from compile state as well as query context. The attribute created reference cycles and also added method call overhead. As part of this change, the interface for ORMExecuteState changes a bit, as well as the interface for the horizontal sharding extension which now deprecates the "query_chooser" callable in favor of "execute_chooser", which receives the contextual object. This will also work more nicely when we implement the new execution path for bulk updates and deletes. Pre-merge execution options for statement, connection, arguments all up front in Connection. that way they can be passed to the before_execute / after_execute events, and the ExecutionContext doesn't have to merge as second time. Core execute is pretty close to 1.3 now. baked wasn't using the new one()/first()/one_or_none() methods, fixed that. Convert non-buffered cursor strategy to be a stateless singleton. inline all the paths by which the strategy gets chosen, oracle and SQL Server dialects make use of the already-invoked post_exec() hook to establish the alternate strategies, and this is actually much nicer than it was before. Add caching to mapper instance processor for getters. Identified a reference cycle per query that was showing up as a lot of gc cleanup, fixed that. After all that, performance not budging much. Even test_baked_query now runs with significantly fewer function calls than 1.3, still 40% slower. Basically something about the new patterns just makes this slower and while I've walked a whole bunch of them back, it hardly makes a dent. that said, the performance issues are relatively small, in the 20-40% time increase range, and the new caching feature does provide for regular ORM and Core queries that are cached, and they are faster than non-cached. Change-Id: I7b0b0d8ca550c05f79e82f75cd8eff0bbfade053
* Convert execution to move through SessionMike Bayer2020-05-252-9/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-243-70/+100
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* inline one_or_noneMike Bayer2020-05-241-0/+29
| | | | | | | Remove a bunch of unnecessary functions for this case. add test coverage to ensure uniqueness logic works. Change-Id: I2e6232c5667a3277b0ec8d7e47085a267f23d75f
* Add python 3.8 profiles; remove zoomark testsFederico Caselli2020-05-232-1048/+0
| | | | | | | | | | | | | | | The zoomark tests have served us well for many years. At this point, they have been using a very antiquated calling style for many years and are no longer where we catch performance issues. The performance suite now has a large number of individual tests that catch issues very specifically and additionally record just one performance count per test. This also allows us to remove the "replay" fixtures that were not used for anything else. Fixes: #5347 Change-Id: I0a8d078e7b7240602f4f3f7068f231e98a40f17e
* Performance fixes for new result setMike Bayer2020-05-211-1/+15
| | | | | | | | | | | A few small mistakes led to huge callcounts. Additionally, the warn-on-get behavior which is attempting to warn for deprecated access in SQLAlchemy 2.0 is very expensive; it's not clear if its feasible to have this warning or to somehow alter how it works. Fixes: #5340 Change-Id: I73bdd2d7b6f1b25cc0222accabd585cf761a5af4
* Merge "Propose Result as immediate replacement for ResultProxy"mike bayer2020-05-012-2/+67
|\
| * Propose Result as immediate replacement for ResultProxyMike Bayer2020-05-012-2/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As progress is made on the _future.Result, including breaking it out such that DBAPI behaviors are local to specific implementations, it becomes apparent that the Result object is a functional superset of ResultProxy and that basic operations like fetchone(), fetchall(), and fetchmany() behave pretty much exactly the same way on the new object. Reorganize things so that ResultProxy is now referred to as LegacyCursorResult, which subclasses CursorResult that represents the DBAPI-cursor version of Result, making use of a multiple inheritance pattern so that the functionality of Result is also available in non-DBAPI contexts, as will be necessary for some ORM patterns. Additionally propose the composition system for Result that will form the basis for ORM-alternative result systems such as horizontal sharding and dogpile cache. As ORM results will soon be coming directly from instances of Result, these extensions will instead build their own ResultFetchStrategies that perform the special steps to create composed or cached result sets. Also considering at the moment not emitting deprecation warnings for fetchXYZ() methods; the immediate issue is Keystone tests are calling upon it, but as the implementations here are proving to be not in any kind of conflict with how Result works, there's not too much issue leaving them around and deprecating at some later point. References: #5087 References: #4395 Fixes: #4959 Change-Id: I8091919d45421e3f53029b8660427f844fee0228
* | repair test_fetch_resultsMike Bayer2020-04-301-1/+2
|/ | | | | | | this profiling test was not actually loading the related objects. Change-Id: I9d18a44f50f72f6653f736708829365eb561160e
* Pass connection to TablesTest.insert_data()Mike Bayer2020-04-151-18/+22
| | | | | | | | | | towards the goal of reducing verbosity and repetition in test fixtures as well as that we are moving to connection only for execution, move the insert_data() classmethod to accept a connection and adjust all fixtures to use it. Change-Id: I3bf534acca0d5f4cda1d4da8ae91f1155b829b09
* Ensure context.attrbutes dictionary is reset each timeMike Bayer2020-04-041-0/+2
| | | | | | | A memory growth issue was identified in this test which caused the profiling results to be inaccurate. Change-Id: I248dcce5833feada947bc91bdf09a8f925d31d65
* Try to measure new style caching in the ORM, take twoMike Bayer2020-04-012-7/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf Trying to see if by making the cache key memoized, we still can have the older "identity" form of caching which is the cheapest of all, at the same time as the newer "cache key each time" version that is not nearly as cheap; but still much cheaper than no caching at all. Also needed is a per-execution update of _keymap when we invoke from a cached select, so that Column objects that are anonymous or otherwise adapted will match up. this is analogous to the adaption of bound parameters from the cache key. Adds test coverage for the keymap / construct_params() changes related to caching. Also hones performance to a large extent for statement construction and cache key generation. Also includes a new memoized attribute approach that vastly simplifies the previous approach of "group_expirable_memoized_property" and finally integrates cleanly with _clone(), _generate(), etc. no more hardcoding of attributes is needed, as well as that most _reset_memoization() calls are no longer needed as the reset is inherent in a _generate() call; this also has dramatic performance improvements. Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
* Deprecate plain string in execute and introduce `exec_driver_sql`Federico Caselli2020-03-211-16/+22
| | | | | | | | | | | | | | | Execution of literal sql string is deprecated in the :meth:`.Connection.execute` and a warning is raised when used stating that it will be coerced to :func:`.text` in a future release. To execute a raw sql string the new connection method :meth:`.Connection.exec_driver_sql` was added, that will retain the previous behavior, passing the string to the DBAPI driver unchanged. Usage of scalar or tuple positional parameters in :meth:`.Connection.execute` is also deprecated. Fixes: #4848 Fixes: #5178 Change-Id: I2830181054327996d594f7f0d59c157d477c3aa9
* Fix tests failing for SQLite file databases; repair provisioningGord Thompson2020-03-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | 1. ensure provision.py loads dialect implementations when running reap_dbs.py. Reapers haven't been working since 598f2f7e557073f29563d4d567f43931fc03013f . 2. add some exclusion rules to allow the sqlite_file target to work; add to tox. 3. add reap dbs target for SQLite, repair SQLite drop_db routine which also wasn't doing the right thing for memory databases etc. 4. Fix logging in provision files, as the main provision logger is the one that's enabled by reap_dbs and maybe others, have all the provision files use the provision logger. Fixes: #5180 Fixes: #5168 Change-Id: Ibc1b0106394d20f5bcf847f37b09d185f26ac9b5
* Rework select(), CompoundSelect() in terms of CompileStateMike Bayer2020-03-102-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Continuation of I408e0b8be91fddd77cf279da97f55020871f75a9 - add an options() method to the base Generative construct. this will be where ORM options can go - Change Null, False_, True_ to be singletons, so that we aren't instantiating them and having to use isinstance. The previous issue with this was that they would produce dupe labels in SELECT statements. Apply the duplicate column logic, newly added in 1.4, to these objects as well as to non-apply-labels SELECT statements in general as a means of improving this. - create a revised system for generating ClauseList compilation constructs that simplfies up front creation to not actually use ClauseList; a simple tuple is rendered by the compiler using the same constrcution rules as what are used for ClauseList but without creating the actual object. Apply to Select, CompoundSelect, revise Update, Delete - Select, CompoundSelect get an initial CompileState implementation. All methods used only within compilation are moved here - refine update/insert/delete compile state to not require an outside boolean - refine and simplify Select._copy_internals - rework bind(), which is going away, to not use some of the internal traversal stuff - remove "autocommit", "for_update" parameters from Select, references #4643 - remove "autocommit" parameter from TextClause , references #4643 - add deprecation warnings for statement.execute(), engine.execute(), statement.scalar(), engine.scalar(). Fixes: #5193 Change-Id: I04ca0152b046fd42c5054ba10f37e43fc6e5a57b
* Ensure all nested exception throws have a causeMike Bayer2020-03-022-1/+15
| | | | | | | | | | | | | | | Applied an explicit "cause" to most if not all internally raised exceptions that are raised from within an internal exception catch, to avoid misleading stacktraces that suggest an error within the handling of an exception. While it would be preferable to suppress the internally caught exception in the way that the ``__suppress_context__`` attribute would, there does not as yet seem to be a way to do this without suppressing an enclosing user constructed context, so for now it exposes the internally caught exception as the cause so that full information about the context of the error is maintained. Fixes: #4849 Change-Id: I55a86b29023675d9e5e49bc7edc5a2dc0bcd4751
* Result initial introductionMike Bayer2020-02-212-13/+55
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This builds on cc718cccc0bf8a01abdf4068c7ea4f3 which moved RowProxy to Row, allowing Row to be more like a named tuple. - KeyedTuple in ORM is replaced with Row - ResultSetMetaData broken out into "simple" and "cursor" versions for ORM and Core, as well as LegacyCursor version. - Row now has _mapping attribute that supplies full mapping behavior. Row and SimpleRow both have named tuple behavior otherwise. LegacyRow has some mapping features on the tuple which emit deprecation warnings (e.g. keys(), values(), etc). the biggest change for mapping->tuple is the behavior of __contains__ which moves from testing of "key in row" to "value in row". - ResultProxy breaks into ResultProxy and FutureResult (interim), the latter has the newer APIs. Made available to dialects using execution options. - internal reflection methods and most tests move off of implicit Row mapping behavior and move to row._mapping, result.mappings() method using future result - a new strategy system for cursor handling replaces the various subclasses of RowProxy - some execution context adjustments. We will leave EC in but refined things like get_result_proxy() and out parameter handling. Dialects for 1.4 will need to adjust from get_result_proxy() to get_result_cursor_strategy(), if they are using this method - out parameter handling now accommodated by get_out_parameter_values() EC method. Oracle changes for this. external dialect for DB2 for example will also need to adjust for this. - deprecate case_insensitive flag for engine / result, this feature is not used mapping-methods on Row are deprecated, and replaced with Row._mapping.<meth>, including: row.keys() -> use row._mapping.keys() row.items() -> use row._mapping.items() row.values() -> use row._mapping.values() key in row -> use key in row._mapping int in row -> use int < len(row) Fixes: #4710 Fixes: #4878 Change-Id: Ieb9085e9bcff564359095b754da9ae0af55679f0
* Remove unnecessary tuple; prepare for "iterator" verbiageMike Bayer2020-02-201-1/+1
| | | | | | | | | | | | | | | Remove a tuple surrounding a generator expression that is immediately iterated in any case. Additionally note that the bulk methods can likely accept non-list objects such as arbitrary iterables, however without test coverage this is not yet guaranteed; use the term "sequence" for now. Also added a warmup to a cache key profiling test to get consistent results. Fixes: #5163 Change-Id: If838fe214da574763115855c1a65171533c96e64
* Limit non-backend critical profiling tests to SQLiteMike Bayer2020-02-173-21/+45
| | | | | | | | | | | | | | | | | | | | issues with backend-specific profiling should be limited to tests that are explcitly against resultset, compiler, etc. MySQL in particular has an often varying callcount that isn't worth running these tests against nor is it worth profiling them for other backends like Oracle and SQL Server. Also add the REQUIRE_SQLALCHEMY_CEXT flag to the regen_callcounts.tox.ini script, which is part of some review somewhere but is needed here to generate callcounts correctly. Add a "warmup" phase for some of the ORM tests for join conditions that have varying profile counts based on whether mappings have been used already or not; profiling should always be against the "warmed up" version of a function. Change-Id: If483820235fa4cc4360cbd067a9b68d83512d587
* Do away with pool._refsMike Bayer2020-02-011-8/+0
| | | | | | | | | | | | | This collection was added only for the benefit of unit tests and is unnecessary for the pool to function. As SQLAlchemy 2.0 will be removing the automatic handling of connections that are garbage collection, remove this collection so that we ultimately don't need a weakref handler to do anything within the pool. The handler will do nothing other than emit a warning that a connection was dereferenced without being explicitly returned to the pool, invalidated, or detached. Change-Id: I4ca196270d5714efbac44dbf6f034e8c7f0af58a
* Remove jython code, remove all jython / pypy symbolsMike Bayer2020-01-171-2/+2
| | | | | | | | | | | | | | | | | | | | | | Removed all dialect code related to support for Jython and zxJDBC. Jython has not been supported by SQLAlchemy for many years and it is not expected that the current zxJDBC code is at all functional; for the moment it just takes up space and adds confusion by showing up in documentation. At the moment, it appears that Jython has achieved Python 2.7 support in its releases but not Python 3. If Jython were to be supported again, the form it should take is against the Python 3 version of Jython, and the various zxJDBC stubs for various backends should be implemented as a third party dialect. Additionally modernized logic that distinguishes between "cpython" and "pypy" to instead look at platform.python_distribution() which reliably tells us if we are cPython or not; all booleans which previously checked for pypy and sometimes jython are now converted to be "not cpython", this impacts the test suite for tests that are cPython centric. Fixes: #5094 Change-Id: I226cb55827f997daf6b4f4a755c18e7f4eb8d9ad
* Merge "Fix QueryContext ref cycle on joinedload"mike bayer2020-01-071-0/+15
|\
| * Fix QueryContext ref cycle on joinedloadCarson Ip2020-01-041-0/+15
| | | | | | | | | | | | | | | | | | | | | | Avoid storing a reference to itself when dealing with create_eager_joins. Also fix a cheating test. Fixes: #5071 Closes: #5072 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5072 Pull-request-sha: 75ebaf7c91e96d7567eb5760be713dc134c58763 Change-Id: I511ddc0979b46f7928217347199eca4b1d0b4a49
* | Implement explicit autobegin step for SessionMike Bayer2020-01-032-0/+51
|/ | | | | | | | | | | | | | | | | | | | The :class:`.Session` object no longer initates a :class:`.SessionTransaction` object immediately upon construction or after the previous transaction is closed; instead, "autobegin" logic now initiates the new :class:`.SessionTransaction` on demand when it is next needed. Rationale includes to remove reference cycles from a :class:`.Session` that has been closed out, as well as to remove the overhead incurred by the creation of :class:`.SessionTransaction` objects that are often discarded immediately. This change affects the behavior of the :meth:`.SessionEvents.after_transaction_create` hook in that the event will be emitted when the :class:`.Session` first requires a :class:`.SessionTransaction` be present, rather than whenever the :class:`.Session` were created or the previous :class:`.SessionTransaction` were closed. Interactions with the :class:`.Engine` and the database itself remain unaffected. Fixes: #5074 Change-Id: I00b656eb5ee03d87104257a214214617aacae16c
* Test for short term reference cycles and resolve as many as possibleMike Bayer2019-12-301-0/+391
| | | | | | | | Added test support and repaired a wide variety of unnecessary reference cycles created for short-lived objects, mostly in the area of ORM queries. Fixes: #5056 Change-Id: Ifd93856eba550483f95f9ae63d49f36ab068b85a
* Don't apply aliasing + adaption for simple relationship joinsMike Bayer2019-12-191-0/+91
| | | | | | | | | | | Identified a performance issue in the system by which a join is constructed based on a mapped relationship. The clause adaption system would be used for the majority of join expressions including in the common case where no adaptation is needed. The conditions under which this adaptation occur have been refined so that average non-aliased joins along a simple relationship without a "secondary" table use about 70% less function calls. Change-Id: Ifbe04214576e5a9fac86ca80c1dc7145c27cd50a
* Traversal and clause generation performance improvementsMike Bayer2019-12-141-0/+79
| | | | | | | Added one traversal test, callcounts have been brought from 29754 to 5173 so far. Change-Id: I164e9831600709ee214c1379bb215fdad73b39aa
* Add anonymizing context to cache keys, comparison; convert traversalMike Bayer2019-11-041-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Created new visitor system called "internal traversal" that applies a data driven approach to the concept of a class that defines its own traversal steps, in contrast to the existing style of traversal now known as "external traversal" where the visitor class defines the traversal, i.e. the SQLCompiler. The internal traversal system now implements get_children(), _copy_internals(), compare() and _cache_key() for most Core elements. Core elements with special needs like Select still implement some of these methods directly however most of these methods are no longer explicitly implemented. The data-driven system is also applied to ORM elements that take part in SQL expressions so that these objects, like mappers, aliasedclass, query options, etc. can all participate in the cache key process. Still not considered is that this approach to defining traversibility will be used to create some kind of generic introspection system that works across Core / ORM. It's also not clear if real statement caching using the _cache_key() method is feasible, if it is shown that running _cache_key() is nearly as expensive as compiling in any case. Because it is data driven, it is more straightforward to optimize using inlined code, as is the case now, as well as potentially using C code to speed it up. In addition, the caching sytem now accommodates for anonymous name labels, which is essential so that constructs which have anonymous labels can be cacheable, that is, their position within a statement in relation to other anonymous names causes them to generate an integer counter relative to that construct which will be the same every time. Gathering of bound parameters from any cache key generation is also now required as there is no use case for a cache key that does not extract bound parameter values. Applies-to: #4639 Change-Id: I0660584def8627cad566719ee98d3be045db4b8d
* Disallow memory tests from running on WindowsMike Bayer2019-10-241-1/+1
| | | | | | | | | | These tests fail with multiprocess errors involving pickling of the profile file. The memory tests are not critical for windows development nor are the profiling tests overall as they are against platform independent measurements. Fixes: #4946 Change-Id: Iaeb3958e59ce7709ba6af3cf9d7baf2a4922bb9b
* Implement facade for pytest parametrize, fixtures, classlevelMike Bayer2019-10-201-1/+1
| | | | | | | | | | | | | | | | | | | Add factilities to implement pytest.mark.parametrize and pytest.fixtures patterns, which largely resemble things we are already doing. Ensure a facade is used, so that the test suite remains independent of py.test, but also tailors the functions to the more limited scope in which we are using them. Additionally, create a class-based version that works from the same facade. Several old polymorphic tests as well as two of the sql test are refactored to use the new features. Change-Id: I6ef8af1dafff92534313016944d447f9439856cf References: #4896
* Warn for object replaced in identity map during flushMike Bayer2019-10-041-0/+4
| | | | | | | | | | | | | A warning is emitted for a condition in which the :class:`.Session` may implicitly swap an object out of the identity map for another one with the same primary key, detaching the old one, which can be an observed result of load operations which occur within the :meth:`.SessionEvents.after_flush` hook. The warning is intended to notify the user that some special condition has caused this to happen and that the previous object may not be in the expected state. Fixes: #4890 Change-Id: Ide11c6b9f21ca67ff5a96266c521d0c56fd6af8d
* Run row value processors up frontMike Bayer2019-10-012-10/+21
| | | | | | | | | | | | as part of a larger series of changes to generalize row-tuples, RowProxy becomes plain Row and is no longer a "proxy"; the DBAPI row is now copied directly into the Row when constructed, result handling occurs at once. Subsequent changes will break out Row into a new version that behaves fully a tuple. Change-Id: I2ffa156afce5d21c38f28e54c3a531f361345dd5
* Simplify _ColumnEntity, relatedMike Bayer2019-09-301-0/+4
| | | | | | | | | | | | | | | In the interests of making Query much more lightweight up front, rework the calculations done at the top when the entities are constructed to be much less inolved. Use the new coercion system for _ColumnEntity and stop accepting plain strings, this will need to emit a deprecation warning in 1.3.x. Use annotations and other techniques to reduce the decisionmaking and complexity of Query. For the use case of subquery(), .statement, etc. we would like to do minimal work in order to get the columns clause. Change-Id: I7e459bbd3bb10ec71235f75ef4f3b0a969bec590
* bump variance on test_merge_loadMike Bayer2019-08-231-1/+1
| | | | | | | This test is very sensitive and fluctuates a lot, failing builds, bump the variance to try and resolve. Change-Id: Ia19bb8871b432059cb3917ca0050a68f75c0a0f2
* Fix typosMin ho Kim2019-07-211-1/+1
|
* Add performance improvement for Enum w/ Python 2 enum libraryMike Bayer2019-07-111-0/+37
| | | | | | | | | | Adjusted the initialization for :class:`.Enum` to minimize how often it invokes the ``.__members__`` attribute of a given PEP-435 enumeration object, to suit the case where this attribute is expensive to invoke, as is the case for some popular third party enumeration libraries. Fixes: #4758 Change-Id: Iffeb854c67393bdcb288944fc357a074e20e1325
* SelectBase no longer a FromClauseMike Bayer2019-07-061-2/+50
| | | | | | | | | | | | | | | | | | | | As part of the SQLAlchemy 2.0 migration project, a conceptual change has been made to the role of the :class:`.SelectBase` class hierarchy, which is the root of all "SELECT" statement constructs, in that they no longer serve directly as FROM clauses, that is, they no longer subclass :class:`.FromClause`. For end users, the change mostly means that any placement of a :func:`.select` construct in the FROM clause of another :func:`.select` requires first that it be wrapped in a subquery first, which historically is through the use of the :meth:`.SelectBase.alias` method, and is now also available through the use of :meth:`.SelectBase.subquery`. This was usually a requirement in any case since several databases don't accept unnamed SELECT subqueries in their FROM clause in any case. See the documentation in this change for lots more detail. Fixes: #4617 Change-Id: I0f6174ee24b9a1a4529168e52e855e12abd60667
* Add option for "sparse" backend tests and apply to memusageMike Bayer2019-06-201-1/+1
| | | | | | | | | | | The memusage tests are extremely time and memory intensive, and when CI runs against MySQL or Postgresql there are many database/driver combinations for which the "backend" tests repeatedly run; as these tests are more oriented towards basic dialect interaction, add a new "sparse" backend option that will run the tests only once per base dialect. Change-Id: I312aa0332d7ec1ff4e2faa15f6b189d6f0f68393
View Lorry Controller Status