summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* bitten.Matthew Sackman2009-04-211-2/+2
|
* altered api so that deliver returns the seq_id (actually, it's a tuple of ↵Matthew Sackman2009-04-212-22/+19
| | | | | | {msgid, seqid}, but that's irrelevant), and that ack requires this seq_id (tuple) back in. This avoids extra mnesia work and makes ack much faster. Given that the amqqueue already tracks unacked messages, this seems reasonable. However, if not, back off to the parent of this revision.
* Introduced sequence IDs internally. Sadly, because of the need to lookup ↵Matthew Sackman2009-04-214-72/+153
| | | | | | seqids for each msgid in acks, acks are now very slow. Thus I'm going to alter the API so that deliver returns the seqid and then ack takes [seqid]. This should make things faster.
* so I figured it was still going too fast. Fortunately, switching mnesia to ↵Matthew Sackman2009-04-202-2/+3
| | | | disc_only has solved that problem...
* moved the message location table to dets. performance is slightly worse than ↵Matthew Sackman2009-04-202-26/+36
| | | | | | | | 10 times slower. This means that the test suite which used to take about 12 mins to run now takes about 2 hours. Looks like we could now be talking up to 40ms to publish a message. Interestingly, delivery is only twice as slow as with ets, it's publish that's taken the 10+times hit. Worryingly, the numbers show that performance per message is not constant, and wasn't in ets either. This must be the effect of buckets in both ets and dets filling up and chaining. The dets man page does say that it organises data as a linear hash list, which is a structure I know well, and I am surprised performance is dropping off in this way - maybe suggests poor distribution of their hashing algorithm or rebalancing.
* removed file_detail ets table and converted all use into use of msg_location ↵Matthew Sackman2009-04-201-37/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ets table. Even though this is slightly less optimal because of the loss of doing index lookups in file_detail, this is actually slightly faster due to not having to maintain two tables. Performance: Msg Count | Msg Size | Queue Count | Startup mu s | Publish mu s | Pub mu s/msg | Pub mu s/byte | Deliver mu s | Del mu s/msg | Del mu s/byte 1024| 512| 1| 2644.0| 41061.0| 40.098633| 0.0783176422| 156031.0| 152.374023| 0.2976055145 4096| 512| 1| 74843.0| 328683.0| 80.244873| 0.1567282677| 629441.0| 153.672119| 0.3001408577 16384| 512| 1| 373729.0| 3614155.0| 220.590515| 0.4308408499| 2969499.0| 181.243835| 0.3539918661 1024| 512| 10| 1605989.0| 281004.0| 27.441797| 0.0535972595| 1936168.0| 189.078906| 0.3692947388 4096| 512| 10| 85912.0| 2940291.0| 71.784448| 0.1402040005| 7662259.0| 187.066870| 0.3653649807 16384| 512| 10| 418213.0| 37962842.0| 231.706799| 0.4525523424| 32293492.0| 197.103833| 0.3849684238 1024| 8192| 1| 1347269.0| 144988.0| 141.589844| 0.0172839165| 173906.0| 169.830078| 0.0207312107 4096| 8192| 1| 93070.0| 606369.0| 148.039307| 0.0180712044| 829812.0| 202.590820| 0.0247303247 16384| 8192| 1| 20014.0| 4976009.0| 303.711487| 0.0370741561| 3211632.0| 196.022461| 0.0239285231 1024| 8192| 10| 77291.0| 348677.0| 34.050488| 0.0041565537| 1877374.0| 183.337305| 0.0223800421 4096| 8192| 10| 104842.0| 2722730.0| 66.472900| 0.0081143677| 7787817.0| 190.132251| 0.0232095033 16384| 8192| 10| 21746.0| 44301448.0| 270.394580| 0.0330071509| 32018244.0| 195.423853| 0.0238554507 1024| 32768| 1| 120732.0| 426700.0| 416.699219| 0.0127166510| 210704.0| 205.765625| 0.0062794685 4096| 32768| 1| 9355.0| 1925633.0| 470.125244| 0.0143470839| 824304.0| 201.246094| 0.0061415434 16384| 32768| 1| 14734.0| 10371560.0| 633.029785| 0.0193185359| 3594753.0| 219.406311| 0.0066957492 1024| 32768| 10| 6052.0| 629362.0| 61.461133| 0.0018756449| 2100901.0| 205.166113| 0.0062611729 4096| 32768| 10| 5546.0| 4203683.0| 102.628979| 0.0031319879| 8899536.0| 217.273828| 0.0066306710 16384| 32768| 10| 22657.0| 50306069.0| 307.043878| 0.0093702355| 36433817.0| 222.374371| 0.0067863273 1024| 131072| 1| 7155.0| 1913696.0| 1868.843750| 0.0142581463| 444638.0| 434.216797| 0.0033128113 4096| 131072| 1| 6671.0| 8232640.0| 2009.921875| 0.0153344870| 1907439.0| 465.683350| 0.0035528820 16384| 131072| 1| 1699.0| 33886514.0| 2068.268677| 0.0157796377| 7291762.0| 445.053833| 0.0033954913 1024| 131072| 10| 7506.0| 1991032.0| 194.436719| 0.0014834344| 4564850.0| 445.786133| 0.0034010783 4096| 131072| 10| 7486.0| 9551800.0| 233.198242| 0.0017791614| 18048697.0| 440.642017| 0.0033618318 16384| 131072| 10| 2771.0| 71072559.0| 433.792474| 0.0033095739| 81144745.0| 495.268219| 0.0037785966
* no sense running the same test twiceMatthew Sackman2009-04-201-2/+1
|
* /byte figures so small could use a couple extra dpMatthew Sackman2009-04-201-1/+1
|
* just improving display of timing stats to make it easier to graph.Matthew Sackman2009-04-201-16/+20
| | | | This then means that it's much easier to test how performance changes with modifications to the disk queue.
* just merging in default to ensure I don't get too far away from itMatthew Sackman2009-04-164-11/+265
|\
| * merge in bug20354Matthew Sackman2009-04-164-11/+265
| |\
| | * handle negative prioritiesMatthias Radestock2009-04-152-0/+6
| | |
| | * merge default into bug20354Matthias Radestock2009-04-1520-109/+166
| | |\ | | |/ | |/|
| * | Backed out changeset 4ee61cc2d6f0Matthias Radestock2009-04-142-2/+2
| | | | | | | | | | | | This was committed directly on 'default' in error.
| * | Fix double-exchange-deletion bugTony Garnock-Jones2009-04-132-2/+2
| | |
| | * add some signaturesMatthias Radestock2009-03-311-0/+21
| | |
| | * add some priority_queue testsMatthias Radestock2009-03-311-0/+53
| | |
| | * fix bugMatthias Radestock2009-03-311-1/+1
| | |
| | * small simplificationMatthias Radestock2009-03-311-2/+2
| | |
| | * don't call into queue module and don't break its encapsulationMatthias Radestock2009-03-311-12/+15
| | |
| | * do not use priorities for notify_sent and unblockMatthias Radestock2009-03-131-2/+2
| | | | | | | | | | | | since the benefit is unproven
| | * fix bugMatthias Radestock2009-03-131-1/+1
| | |
| | * change representation from gb_tree to kv listMatthias Radestock2009-03-131-42/+43
| | | | | | | | | | | | which is much more efficient for small numbers of priorities; the common case.
| | * document priority functionalityMatthias Radestock2009-03-121-0/+5
| | |
| | * make in/2 insert items that are low priorityMatthias Radestock2009-03-121-6/+6
| | | | | | | | | | | | and make to_list do something sensible.
| | * another bug fixMatthias Radestock2009-03-121-1/+1
| | |
| | * fix bugMatthias Radestock2009-03-122-6/+46
| | |
| | * first cut at extending gen_server2 with prioritiesMatthias Radestock2009-03-122-6/+131
| | | | | | | | | | | | | | | This just changes the representation of the message queue; the API changes are still to come.
* | | funny what you realise in the middle of the night...Matthew Sackman2009-04-161-11/+14
| | |
* | | Bugs fixed, and tests written. GC works. rabbit_disk_queue is functionally ↵Matthew Sackman2009-04-142-15/+78
| | | | | | | | | | | | complete. However, fortunately, Matthias this morning changed its requirements so there are now several changes to make.
* | | fixed bug. GC does seem to work, though no explicit tests for it yet.Matthew Sackman2009-04-141-1/+1
| | |
* | | well, the GC is _written_ now. Not tested yet...Matthew Sackman2009-04-141-13/+29
| | |
* | | GC file movement is done, but updating the accounting information is not ↵Matthew Sackman2009-04-141-29/+116
| | | | | | | | | | | | done. So don't use it!
* | | initial logic prior to compactionMatthew Sackman2009-04-131-1/+35
| | |
* | | Fix for that bug. Basically, mnesia wasn't being deleted correctly, which ↵Matthew Sackman2009-04-132-23/+25
| | | | | | | | | | | | meant on the subsequent startup, it would report messages which we couldn't find anywhere, as they'd been deleted from disk.
* | | tiny change to fix where the badmatch happens.Matthew Sackman2009-04-133-6/+6
| | | | | | | | | | | | Must be a bug in compaction as it would appear messages are being deleted early.
* | | initial work on compacter.Matthew Sackman2009-04-121-32/+38
| | | | | | | | | | | | If you ack messages in exactly the same order as they arrived in, then files will be deleted correctly.
* | | gah, committed the damn makefile again. Sorry.Matthew Sackman2009-04-121-2/+2
| | |
* | | removed some parenthesisMatthew Sackman2009-04-122-33/+33
| | |
* | | and now we have some substantial tests.Matthew Sackman2009-04-122-30/+36
| | | | | | | | | | | | This has already led to a good optimisation on reading, and found and fixed a bug in messages going to multiple queues.
* | | initial stab at some tests (though still only 1 queue!).Matthew Sackman2009-04-122-3/+37
| | | | | | | | | | | | Also accidentally seem to have committed a change to the Makefile a while back so am undoing that.
* | | added a stop and clean_stop api to help with tests/benchmarksMatthew Sackman2009-04-121-40/+82
| | |
* | | switched mnesia to an ordered_set.Matthew Sackman2009-04-123-15/+15
| | | | | | | | | | | | Seems to match performance with bag for startup and without the issues on fanout.
* | | refactored out common functionality between ack and cancelMatthew Sackman2009-04-122-54/+37
| | |
* | | changed the mnesia table to be a bag of MsgId -> {Q, Delivered}Matthew Sackman2009-04-113-11/+17
| | | | | | | | | | | | This makes startup MUCH faster, but delivery could be fractionally slower.
* | | combine two writes into one and alter some file:open options.Matthew Sackman2009-04-111-8/+8
| | | | | | | | | | | | | | | Profiling shows that file:write is by far the most expensive operation now. (roughly 140,000 to mnesia coming in at 27,000). So it looks like the next idea is to follow the advice in man 3 file and implement our own write buffer. Ensure we hit 4KB or more before calling file:write - though obviously chuck out early for suitable commits. This will also complicate deliver, tx_cancel and ack.
* | | corrected some bugs.Matthew Sackman2009-04-111-5/+5
| | | | | | | | | | | | Performance still isn't quite where I would like it to be. Need to do more profiling. Seem to be unnecessarily CPU bound despite use of ets throughout.
* | | typeoMatthew Sackman2009-04-111-1/+1
| | |
* | | switched summary to an ets table tooMatthew Sackman2009-04-111-64/+54
| | |
* | | switched back to ets for file detail. Also remember to call compact from ↵Matthew Sackman2009-04-111-64/+60
| | | | | | | | | | | | tx_cancel.
View Lorry Controller Status