#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

Linear Store issues:

Current/pending:
================
 Q-JIRA RHBZ     Description / Comments
 ------ -------  ----------------------
   5359 -        Linearstore: Implement new management schema and wire into store
   5360 -        Linearstore: Evaluate and rework logging to produce a consistent log output
   5361 -        Linearstore: No tests for linearstore functionality currently exist
                   svn r.1564893 2014-02-05: Added tx-test-soak.sh
                   svn r.1564935 2014-02-05: Added license text to tx-test-soak.sh
                   * No existing tests for linearstore:
                   ** Basic broker-level tests for txn and non-txn recovery
                   ** Store-level tests which check write boundary conditions
                   ** EFP tests, including file recovery, error management
                   ** Unit tests
                   ** Basic performance tests
   5362 -        Linearstore: No store tools exist for examining the journals
                   svn r.1556888 2014-01-09: WIP checkin for linearstore version of qpid_qls_analyze. Needs testing and tidy-up.
                   svn r.1560530 2014-01-22: Bugfixes for qpid_qls_analyze
                   svn r.1561848 2014-01-27: Bugfixes and enhancements for qpid_qls_analyze
                   svn r.1564808 2014-02-05: Bugfixes and enhancements for qpid_qls_analyze
                   svn r.1578899 2014-03-18: Bugfixes and enhancements for qpid_qls_analyze
                   svn r.1583778 2014-04-01: Bugfix for qpid_qls_analyze
                   * Store analysis and status
                   * Recovery/reading of message content
                   * Empty file pool status and management
   5464 -        [linearstore] Incompletely created journal files accumulate in EFP
   -    1088944  [Linearstore] store does not return all files to EFP after purging big queue
   -    1078937  [linearstore] Installation and tests for new store analysis tool qpid-qls-analyze
                   svn r.1596633 2014-05-21: Modified to run from installed location

Fixed/closed (in commit order):
===============================
 Q-JIRA RHBZ     Description / Comments
 ------ -------  ----------------------
   5357 1052518  Linearstore: Empty file recycling not functional
                   svn r.1545563 2013-11-26: Propsed fix. VERIFIED
   5358 1052727  Linearstore: Checksums not implemented in record tail
                   svn r.1547601 2013-12-03: Propsed fix. NEEDINFO on algorithm
   5387 1036071  Linearstore: Segmentation fault when deleting queue
                   svn r.1547641 2013-12-03: Propsed fix. VERIFIED
   5388 1035802  Linearstore: Segmentation fault when recovering empty queue
                   svn r.1547921 2013-12-04: Propsed fix. VERIFIED
NO-JIRA -        Added missing Apache copyright/license text
                   svn r.1551304 2013-12-16: Propsed fix
   5425 1052445  Linearstore: Transaction Prepared List (TPL) fails with jexception 0x0402 AtomicCounter::addLimit() threw JERR_JNLF_FILEOFFSOVFL
                   svn r.1551361 2013-12-16: Proposed fix VERIFIED
   5442 1039949  Linearstore: Dtx recover test fails
                   svn r.1552772 2013-12-20: Proposed fix VERIFIED
   5444 1052775  Linearstore: Recovering from qpid-txtest fails with "Inconsistent TPL 2PC count" error message
                   svn r.1553148 2013-12-23: Proposed fix NEEDIFNO on reproduction and testing
   -    1038599  [LinearStore] Abort when deleting used queue after restart
                   CLOSED-NOTABUG 2014-01-06
   5460 1051097  [linearstore] Recovery of store which contains prepared but incomplete transactions results in message loss
                   svn r.1556892 2014-01-09: Proposed fix VERIFIED
   5473 1051924  [linearstore] Recovery of journal in which last logical file contains truncated record causes crash
                   svn r.1557620 2014-01-12: Proposed fix MODIFIED
   5483 -        [linearstore] Recovery of journal with partly written record fails with "JERR_JREC_BADRECTAIL: Invalid data record tail" error message
                   svn r.1558589 2014-01-15: Proposed fix
                   * May be linked to RHBZ 1039522 - VERIFIED
                   * May be linked to RHBZ 1039525 - VERIFIED
   -    1039522  Qpid crashes while recovering from linear store around apid::linearstore::journal::JournalFile::getFqFileName() including enq_rec::decode() threw JERR_JREC_BAD_RECTAIL
                   * Possible dup of 1039525
                   * May be fixed by QPID-5483 - waiting for needinfo, recommend rebuilding with QPID-5483 fix and re-testing. VERIFIED.
   -    1039525  Qpid crashes while recovering from linear store around apid::linearstore::journal::jexception::format including enq_rec::decode() threw JERR_JREC_BAD_REC_TAIL
                   * Possible dup of 1039522
                   * May be fixed by QPID-5483 - waiting for needinfo, recommend rebuilding with QPID-5483 fix and re-testing. VERIFIED.
   5487 1054448  [linearstore] Replace use of /dev/urandom with c random generator calls
                   svn r.1558913 2014-01-16: Proposed fix VEFIFIED
   5484 1035843  Slow performance for producers
                   svn r.1558592 2014-01-15 fixes an issue with using /dev/random as a source of random numbers for Journal serial numbers.
                   svn r.1558913 2014-01-16 replaces use of /dev/urandom with several calls to rand() to construct a 64-bit random number.
                   * Recommend rebuilding and testing for performance again with these two fixes. VERIFIED.
   5479 1053701  [linearstore] Using recovered store results in "JERR_JNLF_FILEOFFSOVFL: Attempted to increase submitted offset past file size. (JournalFile::submittedDblkCount)" error message
                   * Probability: 2 of 600 (0.3%) using tx-test-soak.sh
                   * Fixed by checkin for QPID-5480, no longer able to reproduce. VERIFIED
   5480 1053749  [linearstore] Recovery of store failure with "JERR_MAP_NOTFOUND: Key not found in map." error message
                   svn r.1564877 2014-02-05: Proposed fix
                   * Probability: 6 of 600 (1.0%) using tx-test-soak.sh
                   * If broker is started a second time after failure, it starts correctly and test completes ok.
                   * Problem: File is being recycled to EFP with still-locked enqueues in it (ie dequeued transactionally).
                   * Problem: Record alignment check writes filler records to wrong file when decoding bad record moves across a file boundary
   5603 1063700  [linearstore] broker restart fails under stress test
                   svn r.1574513 2014-03-05: Proposed fix. POST
                   * jexception 0x0701 RecoveryManager::readNextRemainingRecord() threw JERR_JREC_BADRECTAIL
   5607 1064181  [linearstore] Qpidd closes transactional client session&connection with async_dequeue() failed
                   svn r.1575009 2014-03-06 Proposed fix. POST
                   * jexception 0x010b LinearFileController::getCurrentSerial() threw JERR_NULL
   -    1064230  [linearstore] Qpidd linearstore recovery sometimes fail to recover messages with recoverMessages() failed
                   * jexception 0x0701 RecoveryManager::readNextRemainingRecord() threw JERR_JREC_BADRECTAIL
                   * possible dup of 1063700
   -    1036026  [LinearStore] Qpid linear store unable to create durable queue - framing-error: Queue <q-name>: create() failed: jexception 0x0000
                   * UNABLE TO REPRODUCE - but Frantizek has additional info
                   * Retested after checkin 1575009, problem solved. VERIFIED
   5651       -  [C++ broker] segfault in qpid::linearstore::journal::jdir::clear_dir when declaring durable queue
                   svn r.1582730 2014-03-28 Proposed fix by Pavel Moravec
                   * Bug introduced by r.1578899.
   5661       -  [linearstore] Set default cmake build to exclude linearstore
                   svn r.1584379 2014-04-03 Proposed solution.
                   * Run ccmake, select BUILD_LINEARSTORE to change its value to ON to build.
   5750 1078142  [linearstore] qpidd closes connection with (distributed) transactional client while checking previous transaction, broker signals error (closed by error: Queue Ve0-2: async_dequeue() failed: exception 0x0103 wmgr::get_events() threw JERR__AIO: AIO error)
                   svn r.1594215 2014-05-13 Proposed solution.
                   * jexception 0x0103 wmgr::get_events() threw JERR__AIO: AIO error. (AIO write operation failed: Invalid argument (-22) [pg=0 size=8192 offset=4096 fh=22])
   5767 1098118  [linearstore] broker segfaults when recovering journal file with damaged header
                   svn r.1596509 2014-05-21 Proposed solution (committed by pmoravec)

Ordered checkin list:
=====================
In order to port the linearstore changes from trunk to a branch, the following svn checkins need to be ported in order:

no.   svn r  Q-JIRA     RHBZ       Date Alt Committer
--- ------- ------- -------- ---------- -------------
 1. 1545563    5357  1052518 2013-11-26
 2. 1547601    5358  1052727 2013-12-03
 3. 1547641    5387  1036071 2013-12-03
 4. 1547921    5388  1035802 2013-12-04
 5. 1551304 NO-JIRA        - 2013-12-16 (aconway)
 6. 1551361    5425  1052445 2013-12-16
 7. 1552772    5442  1039949 2013-12-20
 8. 1553148    5444  1052775 2013-12-23
 9. 1556888    5362        - 2014-01-09
10. 1556892    5460  1051097 2014-01-09
11. 1557620    5473  1051924 2014-01-12
12. 1558589    5483        - 2014-01-15
13. 1558592    5484  1035843 2014-01-15
14. 1558913    5487  1054448 2014-01-16
15. 1560530    5362        - 2014-01-22
16. 1561848    5362        - 2014-01-27
17. 1564808    5362        - 2014-02-05
18. 1564877    5480  1053749 2014-02-05
19. 1564893    5361        - 2014-02-05
20. 1564935    5361        - 2014-02-05
21. 1574513    5603  1063700 2014-03-05
22. 1575009    5607  1064181 2014-03-06
23. 1578899    5362        - 2014-03-18
24. 1582730    5651        - 2014-03-28 (pmoravec)
25. 1583778    5362        - 2014-04-01
26. 1584379    5661        - 2014-04-03
27. 1594215    5750  1078142 2014-05-13
28. 1596509    5767  1098118 2014-05-21 (pmoravec)
29. 1596633  NO-JIRA 1078937 2014-05-21

See above sections for details on these checkins.

Future work:
============
* One journal file lost when queue deleted. All files except for one are recycled back to the EFP.
* Complete exceptions - several exceptions thrown using jexception have no exception numbers
* Investigate ability of store to detect missing journal files, especially from logical end of a journal
* Investigate ability of store to handle file muddle-ups (ie journal files from EFP which are not zeroed or other journals)
* Look at improving the efficiency of recovery - right now the entire store is read once, and then each recovered record xid and data is read again

Code tidy-up
------------
* Remove old comments
* Use c++ cast templates instead of (xxx)y
* Member names: xxx_
* Rename classes, functions and variables to camel-case
* Add Doxygen docs to classes
* Make fid's consistent in name (fid, file_id, pfid) and format (hex vs decimal)
