Package com.mckoi.database

Class MasterTableDataSource

  • Direct Known Subclasses:
    V1MasterTableDataSource, V2MasterTableDataSource
    abstract class MasterTableDataSource
extends java.lang.Object
    A master table data source provides facilities for read/writing and maintaining low level data in a table. It provides primitive table operations such as retrieving a cell from a table, accessing the table's DataTableDef, accessing indexes, and providing views of transactional versions of the data.

    Logically, a master table data source contains a dynamic number of rows and a fixed number of columns. Each row has an associated state - either DELETED, UNCOMMITTED, COMMITTED_ADDED or COMMITTED_REMOVED. A DELETED row is a row that can be reused by a new row added to the table.

    When a new row is added to the table, it is marked as UNCOMMITTED. It is later tagged as COMMITTED_ADDED when the transaction that caused the row addition is committed. If a row commits a row removal, the row is tagged as COMMITTED_REMOVED and later the row garbage collector marks the row as DELETED when there are no remaining references to the row.

    A master table also maintains a list of indexes for the table.

    How the master table logical structure is translated to a form that is stored persistantly is implementation specific. This allows us flexibility with different types of storage schemes.

    • Method Summary

      All Methods Static Methods Instance Methods Abstract Methods Concrete Methods 
      Modifier and Type Method Description
      (package private) void addRootLock()
      Grabs a root lock on this table.
      (package private) int addRow​(RowData data)
      Adds a new row to this table and returns an index that is used to reference this row by the 'getCellContents' method.
      (package private) void buildIndex​(int index_number)
      Builds the given index number (from the DataIndexSetDef).
      (package private) void buildIndexes()
      Builds a complete index set on the data in this table.
      (package private) TableName cachedTableName()
      Returns a cached TableName for this data source.
      (package private) abstract void checkForCleanup()
      Checks to determine if it is safe to clean up any resources in the table, and if it is safe to do so, the space is reclaimed.
      protected void clearAllRootLocks()
      Clears all root locks on the table.
      (package private) abstract void commitIndexSet​(IndexSet index_set)
      Commits changes made to an IndexSet returned by the 'createIndexSet' method.
      (package private) void commitTransactionChange​(long commit_id, MasterTableJournal change, IndexSet index_set)
      Adds a new transaction modification to this master table source.
      (package private) abstract IndexSet createIndexSet()
      Creates and returns an IndexSet object that is used to create indices for this table source.
      (package private) SelectableScheme createSelectableSchemeForColumn​(IndexSet index_set, TableDataSource table, int column)
      Creates a SelectableScheme object for the given column in this table.
      (package private) SelectableScheme createSelectableSchemeForIndex​(IndexSet index_set, TableDataSource table, int index_i)
      Creates a SelectableScheme object for the given index in the index set def in this table.
      (package private) MutableTableDataSource createTableDataSourceAtCommit​(SimpleTransaction transaction)
      Returns a MutableTableDataSource object that represents this data source at the time the given transaction started.
      (package private) MutableTableDataSource createTableDataSourceAtCommit​(SimpleTransaction transaction, MasterTableJournal journal)
      Returns a MutableTableDataSource object that represents this data source at the time the given transaction started, and also also makes any modifications that are described by the journal in the table.
      (package private) abstract long currentUniqueID()
      Atomically returns the current 'unique_id' value for this table.
      DebugLogger Debug()
      Returns the DebugLogger object that can be used to log debug messages.
      (package private) abstract void dispose​(boolean pending_drop)
      Disposes of all in-memory resources associated with this table and invalidates this object.
      private void doHardRowRemove​(int row_index)
      Actually deletes the row from the table.
      protected void doOpeningScan()
      This is called by the 'open' method.
      (package private) abstract boolean drop()
      Disposes and drops this table.
      (package private) MasterTableJournal[] findAllJournalsSince​(long commit_id)
      Returns a list of all MasterTableJournal objects that have been successfully committed against this table that have an 'commit_id' that is greater or equal to the given.
      (package private) TObject getCellContents​(int column, int row)
      Returns the cell contents of the given cell in the table.
      (package private) DataIndexSetDef getDataIndexSetDef()
      Returns the DataIndexSetDef object that represents the indexes on this table.
      (package private) DataTableDef getDataTableDef()
      Returns the DataTableDef object that represents the topology of this table data source (name, columns, etc).
      java.lang.String getName()
      Returns the name of this table source.
      (package private) RawDiagnosticTable getRawDiagnosticTable()
      Returns an implementation of RawDiagnosticTable that we can use to diagnose problems with the data in this source.
      java.lang.String getSchema()
      Returns the schema name of this table source.
      (package private) abstract java.lang.String getSourceIdent()
      Returns a string that uniquely identifies this table within the conglomerate context.
      TransactionSystem getSystem()
      Returns the TransactionSystem for this table.
      (package private) int getTableID()
      Returns table_id - the unique identifier for this data source.
      TableName getTableName()
      Returns the TableName of this table source.
      (package private) boolean hardCheckAndReclaimRow​(int record_index)
      Checks the given record index, and if it's possible to reclaim it then it does so here.
      (package private) void hardRemoveRow​(int record_index)
      Permanently removes a row from this table.
      (package private) boolean hasTransactionChangesPending()
      Returns true if this table has any journal modifications that have not yet been incorporated into master index.
      (package private) abstract int internalAddRow​(RowData data)
      Adds a new row to this table and returns an index that is used to reference this row by the 'getCellContents' method.
      (package private) abstract void internalDeleteRow​(int row_index)
      Removes the row at the given index so that any resources associated with the row may be immediately available to be recycled.
      (package private) abstract TObject internalGetCellContents​(int column, int row)
      Returns the cell contents of the given cell in the table.
      (package private) boolean isClosed()
      Returns true if this table source is closed.
      (package private) boolean isReadOnly()
      Returns true if the source is read only.
      (package private) boolean isRootLocked()
      Returns true if the table is currently under a root lock (has 1 or more root locks on it).
      (package private) boolean isWorthCompacting()
      Returns true if a compact table is necessary.
      protected void loadInternal()
      Loads the internal variables.
      protected static java.lang.String makeTableFileName​(TransactionSystem system, int table_id, TableName table_name)
      Creates a unique table name to give a file.
      (package private) void mergeJournalChanges​(long commit_id)
      Updates the master records from the journal logs up to the given 'commit_id'.
      protected TableDataSource minimalTableDataSource​(IntegerListInterface master_index)
      Creates a minimal TableDataSource object that represents this MasterTableDataSource.
      (package private) abstract long nextUniqueID()
      Atomically returns the next 'unique_id' value from this table.
      (package private) abstract int rawRowCount()
      Returns the raw count or rows in the table, including uncommited, committed and deleted rows.
      (package private) abstract int readRecordType​(int row_index)
      Reads the record state for the given record in the table.
      (package private) abstract boolean recordDeleted​(int row_index)
      Returns true if the record with the given index is deleted from the table.
      (package private) int recordTypeInfo​(int record_index)
      Returns the record type of the given record index.
      (package private) void removeRootLock()
      Removes a root lock from this table.
      (package private) void rollbackTransactionChange​(MasterTableJournal change)
      Rolls back a transaction change in this table source.
      (package private) abstract void setUniqueID​(long value)
      Sets the unique id for this store.
      protected void setupDataIndexSetDef()
      Sets up the DataIndexSetDef object from the information set in this object.
      protected void setupDataTableDef​(DataTableDef table_def)
      Sets up the DataTableDef.
      (package private) abstract void shutdownHookCleanup()
      Called by the 'shutdown hook' on the conglomerate.
      protected StoreSystem storeSystem()
      Returns the StoreSystem object used to manage stores in the persistence system.
      (package private) java.lang.String transactionChangeString()  
      (package private) abstract int writeRecordType​(int row_index, int row_state)
      Sets the record type for the given record in the table and returns the previous state of the record.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • system

        private TransactionSystem system
        The global TransactionSystem object that points to the global system that this table source belongs to.

      • store_system

        private StoreSystem store_system
        The StoreSystem implementation that represents the data persistence layer.

      • table_id

        protected int table_id
        An integer that uniquely identifies this data source within the conglomerate.

      • is_closed

        protected boolean is_closed
        True if this table source is closed.

      • root_lock

        private int root_lock
        The number of root locks this table data source has on it.

        While a MasterTableDataSource has at least 1 root lock, it may not reclaim deleted space in the data store. A root lock means that data is still being pointed to in this file (even possibly committed deleted data).

      • table_def

        protected DataTableDef table_def
        A DataTableDef object that describes the table topology. This includes the name and columns of the table.

      • index_def

        protected DataIndexSetDef index_def
        A DataIndexSetDef object that describes the indexes on the table.

      • cached_table_name

        private TableName cached_table_name
        A cached TableName for this data source.

      • table_indices

        protected MultiVersionTableIndices table_indices
        A multi-version representation of the table indices kept for this table including the row list and the scheme indices. This contains the transaction journals.

      • column_rid_list

        protected RIDList[] column_rid_list
        The list of RIDList objects for each column in this table. This is a sorting optimization.

      • DATA_CELL_CACHING

        protected boolean DATA_CELL_CACHING
        Set to false to disable cell caching.

      • cache

        protected final DataCellCache cache
        A reference to the DataCellCache object.

      • column_count

        protected int column_count
        The number of columns in this table. This is a cached optimization.

      • open_transactions

        private OpenTransactionList open_transactions
        The list of all open transactions managed by the parent conglomerate. This is a thread safe object, and is updated whenever new transactions are created, or transactions are closed.

      • garbage_collector

        protected MasterTableGarbageCollector garbage_collector
        Manages scanning and deleting of rows marked as deleted within this data source.

      • blob_store_interface

        protected BlobStoreInterface blob_store_interface
        An abstracted reference to a BlobStore for managing blob references and blob data.

      • root_lock_key

        protected java.lang.String root_lock_key
        The keys we use for Database.stats() for information for this table.

      • total_hits_key

        protected java.lang.String total_hits_key

      • file_hits_key

        protected java.lang.String file_hits_key

      • delete_hits_key

        protected java.lang.String delete_hits_key

      • insert_hits_key

        protected java.lang.String insert_hits_key

    • Constructor Detail

      • MasterTableDataSource

        MasterTableDataSource​(TransactionSystem system,
                      StoreSystem store_system,
                      OpenTransactionList open_transactions,
                      BlobStoreInterface blob_store_interface)
        Constructs the MasterTableDataSource. The argument is a reference to an object that manages the list of open transactions in the conglomerate. This object uses this information to determine how journal entries are to be merged with the master indices.

    • Method Detail

      • getSystem

        public final TransactionSystem getSystem()
        Returns the TransactionSystem for this table.

      • Debug

        public final DebugLogger Debug()
        Returns the DebugLogger object that can be used to log debug messages.

      • getTableName

        public TableName getTableName()
        Returns the TableName of this table source.

      • getName

        public java.lang.String getName()
        Returns the name of this table source.

      • getSchema

        public java.lang.String getSchema()
        Returns the schema name of this table source.

      • cachedTableName

        TableName cachedTableName()
        Returns a cached TableName for this data source.

      • mergeJournalChanges

        void mergeJournalChanges​(long commit_id)
        Updates the master records from the journal logs up to the given 'commit_id'. This could be a fairly expensive operation if there are a lot of modifications because each change could require a lookup of records in the data source.

        NOTE: It's extremely important that when this is called, there are no transaction open that are using the merged journal. If there is, then a transaction may be able to see changes in a table that were made after the transaction started.

        After this method is called, it's best to update the index file with a call to 'synchronizeIndexFiles'

      • findAllJournalsSince

        MasterTableJournal[] findAllJournalsSince​(long commit_id)
        Returns a list of all MasterTableJournal objects that have been successfully committed against this table that have an 'commit_id' that is greater or equal to the given.

        This is part of the conglomerate commit check phase and will be on a commit_lock.

      • getTableID

        int getTableID()
        Returns table_id - the unique identifier for this data source.

      • getDataTableDef

        DataTableDef getDataTableDef()
        Returns the DataTableDef object that represents the topology of this table data source (name, columns, etc). Note that this information can't be changed during the lifetime of a data source.

      • getDataIndexSetDef

        DataIndexSetDef getDataIndexSetDef()
        Returns the DataIndexSetDef object that represents the indexes on this table.

      • makeTableFileName

        protected static java.lang.String makeTableFileName​(TransactionSystem system,
                                                    int table_id,
                                                    TableName table_name)
        Creates a unique table name to give a file. This could be changed to suit a particular OS's style of filesystem namespace. Or it could return some arbitarily unique number. However, for debugging purposes it's often a good idea to return a name that a user can recognise.

        The 'table_id' is a guarenteed unique number between all tables.

      • getSourceIdent

        abstract java.lang.String getSourceIdent()
        Returns a string that uniquely identifies this table within the conglomerate context. For example, the filename of the table. This string can be used to open and initialize the table also.

      • writeRecordType

        abstract int writeRecordType​(int row_index,
                             int row_state)
                      throws java.io.IOException
        Sets the record type for the given record in the table and returns the previous state of the record. This is used to change the state of a row in the table.
        Throws:
        java.io.IOException

      • readRecordType

        abstract int readRecordType​(int row_index)
                     throws java.io.IOException
        Reads the record state for the given record in the table.
        Throws:
        java.io.IOException

      • recordDeleted

        abstract boolean recordDeleted​(int row_index)
                        throws java.io.IOException
        Returns true if the record with the given index is deleted from the table. A deleted row can not be read.
        Throws:
        java.io.IOException

      • rawRowCount

        abstract int rawRowCount()
                  throws java.io.IOException
        Returns the raw count or rows in the table, including uncommited, committed and deleted rows. This is basically the maximum number of rows we can iterate through.
        Throws:
        java.io.IOException

      • internalDeleteRow

        abstract void internalDeleteRow​(int row_index)
                         throws java.io.IOException
        Removes the row at the given index so that any resources associated with the row may be immediately available to be recycled.
        Throws:
        java.io.IOException

      • createIndexSet

        abstract IndexSet createIndexSet()
        Creates and returns an IndexSet object that is used to create indices for this table source. The IndexSet represents a snapshot of the table and the given point in time.

        NOTE: Not synchronized because we synchronize in the IndexStore object.

      • commitIndexSet

        abstract void commitIndexSet​(IndexSet index_set)
        Commits changes made to an IndexSet returned by the 'createIndexSet' method. This method also disposes the IndexSet so it is no longer valid.

      • internalAddRow

        abstract int internalAddRow​(RowData data)
                     throws java.io.IOException
        Adds a new row to this table and returns an index that is used to reference this row by the 'getCellContents' method.

        Note that this method will not effect the master index or column schemes. This is a low level mechanism for adding unreferenced data into a conglomerate. The data is referenced by committing the change where it eventually migrates into the master index and schemes.

        Throws:
        java.io.IOException

      • internalGetCellContents

        abstract TObject internalGetCellContents​(int column,
                                         int row)
        Returns the cell contents of the given cell in the table. It is the responsibility of the implemented method to perform caching as it deems fit. Some representations may not require such extensive caching as others.

      • currentUniqueID

        abstract long currentUniqueID()
        Atomically returns the current 'unique_id' value for this table.

      • nextUniqueID

        abstract long nextUniqueID()
        Atomically returns the next 'unique_id' value from this table.

      • setUniqueID

        abstract void setUniqueID​(long value)
        Sets the unique id for this store. This must only be used under extraordinary circumstances, such as restoring from a backup, or converting from one file to another.

      • dispose

        abstract void dispose​(boolean pending_drop)
               throws java.io.IOException
        Disposes of all in-memory resources associated with this table and invalidates this object. If 'pending_drop' is true then the table is to be disposed pending a call to 'drop'. If 'pending_drop' is true then any persistant resources that are allocated may be freed.
        Throws:
        java.io.IOException

      • drop

        abstract boolean drop()
               throws java.io.IOException
        Disposes and drops this table. If the dispose failed for any reason, it returns false, otherwise true. If the drop failed, it should be retried at a later time.
        Throws:
        java.io.IOException

      • shutdownHookCleanup

        abstract void shutdownHookCleanup()
        Called by the 'shutdown hook' on the conglomerate. This method should block until the table can by put into a safe mode and then prevent any further access to the object after it returns. It must operate very quickly.

      • isWorthCompacting

        boolean isWorthCompacting()
        Returns true if a compact table is necessary. By default, we return true however it is recommended this method is overwritten and the table tested.

      • createSelectableSchemeForColumn

        SelectableScheme createSelectableSchemeForColumn​(IndexSet index_set,
                                                 TableDataSource table,
                                                 int column)
        Creates a SelectableScheme object for the given column in this table. This reads the index from the index set (if there is one) then wraps it around the selectable schema as appropriate.

        NOTE: This needs to be deprecated in support of composite indexes.

      • createSelectableSchemeForIndex

        SelectableScheme createSelectableSchemeForIndex​(IndexSet index_set,
                                                TableDataSource table,
                                                int index_i)
        Creates a SelectableScheme object for the given index in the index set def in this table. This reads the index from the index set (if there is one) then wraps it around the selectable schema as appropriate.

      • minimalTableDataSource

        protected TableDataSource minimalTableDataSource​(IntegerListInterface master_index)
        Creates a minimal TableDataSource object that represents this MasterTableDataSource. It does not implement the 'getColumnScheme' method.

      • buildIndexes

        void buildIndexes()
           throws java.io.IOException
        Builds a complete index set on the data in this table. This must only be called when either, a) we are under a commit lock, or b) there is a guarentee that no concurrect access to the indexing information can happen (such as when we are creating the table).

        NOTE: We assume that the index information for this table is blank before this method is called.

        Throws:
        java.io.IOException

      • buildIndex

        void buildIndex​(int index_number)
         throws java.io.IOException
        Builds the given index number (from the DataIndexSetDef). This must only be called when either, a) we are under a commit lock, or b) there is a guarentee that no concurrect access to the indexing information can happen (such as when we are creating the table).

        NOTE: We assume that the index number in this table is blank before this method is called.

        Throws:
        java.io.IOException

      • commitTransactionChange

        void commitTransactionChange​(long commit_id,
                             MasterTableJournal change,
                             IndexSet index_set)
        Adds a new transaction modification to this master table source. This information represents the information that was added/removed in the table in this transaction. The IndexSet object represents the changed index information to commit to this table.

        It's guarenteed that 'commit_id' additions will be sequential.

      • rollbackTransactionChange

        void rollbackTransactionChange​(MasterTableJournal change)
        Rolls back a transaction change in this table source. Any rows added to the table will be uncommited rows (type_key = 0). Those rows must be marked as committed deleted.

      • createTableDataSourceAtCommit

        MutableTableDataSource createTableDataSourceAtCommit​(SimpleTransaction transaction)
        Returns a MutableTableDataSource object that represents this data source at the time the given transaction started. Any modifications to the returned table are logged in the table journal.

        This is a key method in this object because it allows us to get a data source that represents the data in the table before any modifications may have been committed.

      • createTableDataSourceAtCommit

        MutableTableDataSource createTableDataSourceAtCommit​(SimpleTransaction transaction,
                                                     MasterTableJournal journal)
        Returns a MutableTableDataSource object that represents this data source at the time the given transaction started, and also also makes any modifications that are described by the journal in the table.

        This method is useful for merging the changes made by a transaction into a view of the table.

      • setupDataIndexSetDef

        protected void setupDataIndexSetDef()
        Sets up the DataIndexSetDef object from the information set in this object. This will only setup a default IndexSetDef on the information in the DataTableDef.

      • setupDataTableDef

        protected void setupDataTableDef​(DataTableDef table_def)
        Sets up the DataTableDef. This would typically only ever be called from the 'create' method.

      • loadInternal

        protected void loadInternal()
        Loads the internal variables.

      • isClosed

        boolean isClosed()
        Returns true if this table source is closed.

      • isReadOnly

        boolean isReadOnly()
        Returns true if the source is read only.

      • storeSystem

        protected StoreSystem storeSystem()
        Returns the StoreSystem object used to manage stores in the persistence system.

      • addRow

        int addRow​(RowData data)
    throws java.io.IOException
        Adds a new row to this table and returns an index that is used to reference this row by the 'getCellContents' method.

        Note that this method will not effect the master index or column schemes. This is a low level mechanism for adding unreferenced data into a conglomerate. The data is referenced by committing the change where it eventually migrates into the master index and schemes.

        Throws:
        java.io.IOException

      • doHardRowRemove

        private void doHardRowRemove​(int row_index)
                      throws java.io.IOException
        Actually deletes the row from the table. This is a permanent removal of the row from the table. After this method is called, the row can not be retrieved again. This is generally only used by the row garbage collector.

        There is no checking in this method.

        Throws:
        java.io.IOException

      • hardRemoveRow

        void hardRemoveRow​(int record_index)
            throws java.io.IOException
        Permanently removes a row from this table. This must only be used when it is determined that a transaction does not reference this row, and that an open result set does not reference this row. This will remove the row permanently from the underlying file representation. Calls to 'getCellContents(col, row)' where row is deleted will be undefined after this method is called.

        Note that the removed row must not be contained within the master index, or be referenced by the index schemes, or be referenced in the transaction modification list.

        Throws:
        java.io.IOException

      • hardCheckAndReclaimRow

        boolean hardCheckAndReclaimRow​(int record_index)
                        throws java.io.IOException
        Checks the given record index, and if it's possible to reclaim it then it does so here. Rows are only removed if they are marked as committed removed.
        Throws:
        java.io.IOException

      • recordTypeInfo

        int recordTypeInfo​(int record_index)
            throws java.io.IOException
        Returns the record type of the given record index. Returns a type that is compatible with RawDiagnosticTable record type.
        Throws:
        java.io.IOException

      • doOpeningScan

        protected void doOpeningScan()
                      throws java.io.IOException
        This is called by the 'open' method. It performs a scan of the records and marks any rows that are uncommitted as deleted. It also checks that the row is not within the master index.
        Throws:
        java.io.IOException

      • getRawDiagnosticTable

        RawDiagnosticTable getRawDiagnosticTable()
        Returns an implementation of RawDiagnosticTable that we can use to diagnose problems with the data in this source.

      • getCellContents

        TObject getCellContents​(int column,
                        int row)
        Returns the cell contents of the given cell in the table. This will look up the cell in the file if it can't be found in the cell cache. This method is undefined if row has been removed or was not returned by the 'addRow' method.

      • addRootLock

        void addRootLock()
        Grabs a root lock on this table.

        While a MasterTableDataSource has at least 1 root lock, it may not reclaim deleted space in the data store. A root lock means that data is still being pointed to in this file (even possibly committed deleted data).

      • removeRootLock

        void removeRootLock()
        Removes a root lock from this table.

        While a MasterTableDataSource has at least 1 root lock, it may not reclaim deleted space in the data store. A root lock means that data is still being pointed to in this file (even possibly committed deleted data).

      • isRootLocked

        boolean isRootLocked()
        Returns true if the table is currently under a root lock (has 1 or more root locks on it).

      • clearAllRootLocks

        protected void clearAllRootLocks()
        Clears all root locks on the table. Should only be used during cleanup of the table and will by definition invalidate the table.

      • checkForCleanup

        abstract void checkForCleanup()
        Checks to determine if it is safe to clean up any resources in the table, and if it is safe to do so, the space is reclaimed.

      • transactionChangeString

        java.lang.String transactionChangeString()

      • hasTransactionChangesPending

        boolean hasTransactionChangesPending()
        Returns true if this table has any journal modifications that have not yet been incorporated into master index.