summaryrefslogtreecommitdiff
path: root/lib
Commit message (Collapse)AuthorAgeFilesLines
...
* GitCmd implementation of gitdb base moved to git-python where it belongs. ↵Sebastian Thiel2010-06-184-4/+51
| | | | Previously it was located in gitdb, which doesn't have any facilities to use the git command
* Move LazyMixin type to gitdb, index reading now uses file_contents_ro from ↵Sebastian Thiel2010-06-153-39/+7
| | | | gitdb as well
* Moved LockedFD and its test into the gitdb projectSebastian Thiel2010-06-152-141/+3
|
* Reimplemented Lock handling to be conforming to the git lock protocol, which ↵Sebastian Thiel2010-06-153-78/+133
| | | | | | is actually more efficient than the previous implementation Index now locks its file for reading, and properly uses LockedFD when writing
* Moved small types that had their own module into the utils moduleSebastian Thiel2010-06-148-488/+482
|
* Implemented the serializable interface - by refactoring codeSebastian Thiel2010-06-141-32/+47
|
* index: split index file into multiple files of a single package. This didn't ↵Sebastian Thiel2010-06-146-1438/+1439
| | | | reduce the file size as much as I would have liked, but certainly is a start for further 'outsourcing'
* tree: added TreeModifier, allowing to adjust existing trees safely and or ↵Sebastian Thiel2010-06-144-11/+94
| | | | fast, while staying compatible with serialization which requires it to be sorted
* tree now uses less memory for its cache as it stores the bare deserialized ↵Sebastian Thiel2010-06-141-55/+63
| | | | information - this also speeds up later serialization after changes. its clear though that retrieving actual objects is slower currently as these are not cached anymore. Its worth thinking about moving these encoding, decoding routines to gitdb
* Implemented initial version of tree serialization which appears to work ↵Sebastian Thiel2010-06-143-235/+281
| | | | | | | according to a simple test ( presort still needs implementation ) submodule: added stub to allow the tree to return something, its not implemented though
* Removed odb from project, it is now used as a submodule named gitdb, which ↵Sebastian Thiel2010-06-1211-993/+24
| | | | | | was added instead Adjusted all imports to deal with the changed package names
* Removed async from treeSebastian Thiel2010-06-127-1688/+0
|
* task: improved naming of task types, improved pool test to be less dependent ↵Sebastian Thiel2010-06-121-10/+16
| | | | on starting with just the main thread
* channel: cleaned up inheritance hierarchy, adding mixing for callback ↵Sebastian Thiel2010-06-121-10/+23
| | | | functionality - previously the callback functionality was bound to channel based readers/writers
* Cleaned up channel design, Reader and Writer bases don't require a channel ↵Sebastian Thiel2010-06-123-75/+130
| | | | | | anymore, but are abstract. Added IteratorReader, implementing the reader interface from an iterator. The implementation moved from the TaskIterator to the channel
* Added performance test, improved iterator task which will now be usable by ↵Sebastian Thiel2010-06-111-0/+3
| | | | default. It shows that there must be the notion of a producer, which can work if there are no items read
* test_task: fixed import error, made all modules from x import * safeSebastian Thiel2010-06-115-1/+15
|
* Removed commented-out debug code and additional debug printings. Verified it ↵Sebastian Thiel2010-06-113-13/+0
| | | | works on py2.4, 2.5 and 2.6
* Improved shutdown handling - although its impossible to prevent some stderr ↵Sebastian Thiel2010-06-112-8/+46
| | | | printing thanks to the underlying threading implementation, we can at least make sure that the interpreter doesn't block during shutdown. Now it appears to be running smoothly
* IMPORTANT: sometimes, when notifying waiters by releasing their lock, the ↵Sebastian Thiel2010-06-113-6/+5
| | | | lock is not actually released or they are not actually notifyied, staying in a beautysleep. This glitch is probably caused by some detail not treated correctly in the thread python module, which is something we cannot fix. It works most of the time as expected though - maybe some cleanup is not done correctly which causes this
* Added dependency-task tests, and fixed plenty of ref-count related bugs, as ↵Sebastian Thiel2010-06-104-25/+44
| | | | well as concurrency issues. Now it works okay, but the thread-shutdown is still an issue, as it causes incorrect behaviour making the tests fail. Its good, as it hints at additional issues that need to be solved. There is just a little more left on the feature side, but its nearly there
* tasks can now terminate faster when no items were read, without neglecting ↵Sebastian Thiel2010-06-101-19/+24
| | | | their duty to close the channel if required. Code is a little less maintainable now, but faster, it appears
* Now tracking the amount of concurrent writers to assure the channel is ↵Sebastian Thiel2010-06-102-29/+59
| | | | closed only when there is no one else writing to it. This assures that all tasks can continue working, and put their results accordingly. Shutdown is still not working correctly, but that should be solvable as well. Its still not perfect though ...
* channel: Changed design to be more logical - a channel now has any amount of ↵Sebastian Thiel2010-06-103-125/+107
| | | | | | readers and writers, a ready is not connected to its writer anymore. This changes the refcounting of course, which is why the auto-cleanup for the pool is currently broken. The benefit of this are faster writes to the channel, reading didn't improve, refcounts should be clearer now
* Added more dependency task tests, especially the single-reads are not yet ↵Sebastian Thiel2010-06-101-4/+3
| | | | fully deterministic as tasks still run into the problem that they try to write into a closed channel, it was closed by one of their task-mates who didn't know someone else was still computing
* InputChannelTask now has interface for properly handling the reading from ↵Sebastian Thiel2010-06-102-5/+64
| | | | the same and different pools
* messy first version of a properly working depth-first graph method, which ↵Sebastian Thiel2010-06-104-16/+26
| | | | allows the pool to work as expected. Many more tests need to be added, and there still is a problem with shutdown as sometimes it won't kill all threads, mainly because the process came up with worker threads started, which cannot be
* test: prepared task dependency test, which already helped to find bug in the ↵Sebastian Thiel2010-06-092-18/+44
| | | | reference counting mechanism, causing references to the pool to be kepts via cycles
* task: redesigned write channel access to allow the task creator to set own ↵Sebastian Thiel2010-06-092-20/+30
| | | | write channels, possibly some with callbacks installed etc.. Pool.add_task will respect the users choice now, but provide defaults which are optimized for performance
* Channel: Callbacks reviewed - they are now part of Subclasses of the default ↵Sebastian Thiel2010-06-092-45/+57
| | | | channel implementation, one of which is used as base by the Pool Read channel, releasing it of the duty to call these itself. The write channel with callback subclass allows the transformation of the item to be written
* task: removed scheduled task support, which at some point was introduced to ↵Sebastian Thiel2010-06-092-65/+11
| | | | improve performance, but which now hinders performance, besides being unnecessary ;)
* Channel: removed pseudoconstructor, which clearly improves the design and ↵Sebastian Thiel2010-06-093-27/+54
| | | | | | makes it easier to constomize pool: in serial mode, created channels will be serial-only, which brings 15% of performance
* Channel: Read method revised - now it really really doesn't block anymore, ↵Sebastian Thiel2010-06-092-49/+38
| | | | and it runs faster as well, about 2/3 of the performance we have when being in serial mode
* HSCondition: Fixed terrible bug which it inherited from its default python ↵Sebastian Thiel2010-06-093-35/+21
| | | | | | Condition implementation, related to the notify method not being treadsafe. Although I was aware of it, I missed the first check which tests for the size - the result could be incorrect if the whole method wasn't locked. Testing runs stable now, allowing to move on \!
* HSCondition: now gets a lock even in the single-notify case, as it was ↵Sebastian Thiel2010-06-091-7/+18
| | | | required due to the non-atomiciy of the invovled operation. Removed one level of indirection for the lock, by refraining from calling my own 'wrapper' methods, which brought it back to the performance it had before the locking was introduced for the n==1 case
* HSCondition: now deriving from deque, as the AsyncQeue does, to elimitate ↵Sebastian Thiel2010-06-091-10/+9
| | | | one more level of indirection. Clearly this not good from a design standpoint, as a Condition is no Deque, but it helps speeding things up which is what this is about. Could make it a hidden class to indicate how 'special' it is
* HSCondition: using a deck to store waiters, for further speedupSebastian Thiel2010-06-091-3/+3
|
* thread: fixed initialization problem if an empty iterable was handed inSebastian Thiel2010-06-094-17/+38
| | | | | queue: Queue now derives from deque directly, which safes one dict lookup as the queue does not need to be accessed through self anymore pool test improved to better verify threads are started correctly
* queue: fixed critical bug in the notify method, as it was not at all ↵Sebastian Thiel2010-06-095-41/+52
| | | | | | thread-safe, causing locks to be released multiple times. Now it runs very fast, and very stable apparently. Now its about putting previous features back in, and studying their results, before more complex task graphs can be examined
* workerthread: adjusted to use a blocking queue, it will receive termination ↵Sebastian Thiel2010-06-085-34/+52
| | | | events only with its queue, with boosts performance into brigt green levels
* Revised task deletion works well, adjusted test to be creating new tasks all ↵Sebastian Thiel2010-06-082-58/+31
| | | | the time instead of reusing its own one, it was somewhat hard to manage its state over time and could cause bugs. It works okay, but it occasionally hangs, it appears to be an empty queue, have to gradually put certain things back in, although in the current mode of operation, it should never have empty queues from the pool to the user
* task: now deletes itself once its done - for the test this doesn't change a ↵Sebastian Thiel2010-06-082-48/+31
| | | | thing as the task deletes itself too late - its time for a paradigm change, the task should be deleted with its RPoolChannel or explicitly by the user. The test needs to adapt, and shouldn't assume anything unless the RPoolChannel is gone
* Its getting better already - intermediate commit before further chaning the ↵Sebastian Thiel2010-06-085-34/+38
| | | | task class
* queue: adjusted queue to be closable ( without own testing yet, except for ↵Sebastian Thiel2010-06-082-50/+62
| | | | the pool which runs it ) - its not yet stable, but should be solvable.
* The new channeldesign actually works, but it also shows that its located at ↵Sebastian Thiel2010-06-083-67/+79
| | | | the wrong spot. The channel is nothing more than an adapter allowing to read multiple items from a thread-safe queue, the queue itself though must be 'closable' for writing, or needs something like a writable flag.
* both versions of the async queue still have trouble in certain situations, ↵Sebastian Thiel2010-06-081-21/+56
| | | | at least with my totally overwritten version of the condition - the previous one was somewhat more stable it seems. Nonetheless, this is the fastest version so far
* test implementation of async-queue with everything stripped from it that ↵Sebastian Thiel2010-06-081-5/+48
| | | | didn't seem necessary - its a failure, something is wrong - performance not much better than the original one, its depending on the condition performance actually, which I don't get faster
* Task scheduled items lock now uses a dummy lock in serial mode, improving ↵Sebastian Thiel2010-06-072-1/+19
| | | | | | its performance considerably. Channels now use the AsyncQueue, boosting their throughput to about 5k items / s - this is something one can work with, considering the runtime of each item should be large enough to keep the threads busy. This could be a basis, further testing needed
* Channel now uses the AsyncQueue, boosting performance by factor 4, its a startSebastian Thiel2010-06-071-2/+2
|
* introduced a new counter keeping track of the scheduled tasks - this prevent ↵Sebastian Thiel2010-06-073-5/+59
| | | | unnecessary tasks to be scheduled as we keep track of how many items will be produced for the task at hand. This introduces additional locking, but performns well in multithreaded mode. Performance of the master queue is still a huge issue, its currently the limiting factor, as bypassing the master queue in serial moode gives 15x performance, wich is what I would need
View Lorry Controller Status