Struct re_data_store::DataStore

pub struct DataStore { /* private fields */ }

A complete data store: covers all timelines, all entities, everything.

Debugging

DataStore provides a very thorough Display implementation that makes it manageable to know what’s going on internally. For even more information, you can set RERUN_DATA_STORE_DISPLAY_SCHEMAS=1 in your environment, which will result in additional schema information being printed out.

Implementations

impl DataStore

pub fn new(id: StoreId, config: DataStoreConfig) -> Self

pub fn id(&self) -> &StoreId

pub fn insert_id_component_name() -> ComponentName

The column name used for storing insert requests’ IDs alongside the data when manipulating dataframes.

See DataStoreConfig::store_insert_ids.

pub fn generation(&self) -> StoreGeneration

Return the current StoreGeneration. This can be used to determine whether the database has been modified since the last time it was queried.

pub fn config(&self) -> &DataStoreConfig

See DataStoreConfig for more information about configuration.

pub fn lookup_datatype(&self, component: &ComponentName) -> Option<&DataType>

Lookup the arrow DataType of a re_types_core::Component in the internal DataTypeRegistry.

pub fn oldest_time_per_timeline(&self) -> BTreeMap<Timeline, TimeInt>

The oldest time for which we have any data.

Ignores static data.

Useful to call after a gc.

pub fn iter_indices( &self ) -> impl ExactSizeIterator<Item = ((EntityPath, Timeline), &IndexedTable)>

Returns a read-only iterator over the raw indexed tables.

Do not use this to try and assert the internal state of the datastore.

impl DataStore

pub fn to_rows(&self) -> DataReadResult<Vec<DataRow>>

Serializes the entire datastore into one big sorted list of DataRow.

Individual re_log_types::DataRows that were split apart due to bucketing are merged back together.

Beware: this is extremely costly, don’t use this in hot paths.

pub fn to_data_table(&self) -> DataReadResult<DataTable>

Serializes the entire datastore into one big sorted DataTable.

Individual re_log_types::DataRows that were split apart due to bucketing are merged back together.

Beware: this is extremely costly, don’t use this in hot paths.

pub fn to_data_tables( &self, time_filter: Option<(Timeline, ResolvedTimeRange)> ) -> impl Iterator<Item = DataTable> + '_

Serializes the entire datastore into an iterator of DataTables, where each table corresponds 1-to-1 to an internal bucket.

impl DataStore

pub fn gc( &mut self, options: &GarbageCollectionOptions ) -> (Vec<StoreEvent>, DataStoreStats)

Triggers a garbage collection according to the desired target.

Garbage collection’s performance is bounded by the number of buckets in each table (for each RowId, we have to find the corresponding bucket, which is roughly O(log(n))) as well as the number of rows in each of those buckets (for each RowId, we have to sort the corresponding bucket (roughly O(n*log(n))) and then find the corresponding row (roughly O(log(n)). The size of the data itself has no impact on performance.

Returns the list of RowIds that were purged from the store.

Semantics

Garbage collection works on a row-level basis and is driven by RowId order, i.e. the order defined by the clients’ wall-clocks, allowing it to drop data across the different timelines in a fair, deterministic manner. Similarly, out-of-order data is supported out of the box.

The garbage collector doesn’t deallocate data in and of itself: all it does is drop the store’s internal references to that data (the DataCells), which will be deallocated once their reference count reaches 0.

Limitations

The garbage collector has limited support for latest-at semantics. The configuration option: GarbageCollectionOptions::protect_latest will protect the N latest values of each component on each timeline. The only practical guarantee this gives is that a latest-at query with a value of max-int will be unchanged. However, latest-at queries from other arbitrary points in time may provide different results pre- and post- GC.

impl DataStore

pub fn insert_component<'a, C>( &mut self, entity_path: &EntityPath, timepoint: &TimePoint, component: C )

where C: Component + Clone + 'a, Cow<'a, C>: From<C>,

Stores a single value for a given re_types_core::Component.

This is a best-effort helper, it will merely log errors on failure.

pub fn insert_empty_component( &mut self, entity_path: &EntityPath, timepoint: &TimePoint, component: ComponentName )

Stores a single empty value for a given re_types_core::ComponentName.

This is a best-effort helper, it will merely log errors on failure.

impl DataStore

pub fn all_components( &self, timeline: &Timeline, entity_path: &EntityPath ) -> Option<ComponentNameSet>

Retrieve all the ComponentNames that have been written to for a given EntityPath on the specified Timeline.

Static components are always included in the results.

Returns None if the entity doesn’t exist at all on this timeline.

pub fn entity_has_component( &self, timeline: &Timeline, entity_path: &EntityPath, component_name: &ComponentName ) -> bool

Check whether a given entity has a specific ComponentName either on the specified timeline, or in its static data.

pub fn entity_min_time( &self, timeline: &Timeline, entity_path: &EntityPath ) -> Option<TimeInt>

Find the earliest time at which something was logged for a given entity on the specified timeline.

Ignores static data.

pub fn latest_at<const N: usize>( &self, query: &LatestAtQuery, entity_path: &EntityPath, primary: ComponentName, component_names: &[ComponentName; N] ) -> Option<(TimeInt, RowId, [Option<DataCell>; N])>

Queries the datastore for the cells of the specified component_names, as seen from the point of view of the so-called primary component.

Returns an array of DataCells (as well as the associated data time and RowId, if the data is temporal) on success.

Success is defined by one thing and one thing only: whether a cell could be found for the primary component. The presence or absence of secondary components has no effect on the success criteria.

If the entity has static component data associated with it, it will unconditionally override any temporal component data.

pub fn range<'a, const N: usize>( &'a self, query: &RangeQuery, entity_path: &EntityPath, component_names: [ComponentName; N] ) -> impl Iterator<Item = (TimeInt, RowId, [Option<DataCell>; N])> + 'a

Iterates the datastore in order to return the cells of the specified component_names for the given time range.

For each and every relevant row that is found, the returned iterator will yield an array that is filled with the cells of each and every component in component_names, or None if said component is not available in that row.

This method cannot fail! If there’s no data to return, an empty iterator is returned.

⚠ Contrary to latest-at queries, range queries can and will yield multiple rows for a single timestamp if it happens to hold multiple entries.

If the entity has static component data associated with it, it will unconditionally override any temporal component data.

pub fn row_metadata( &self, row_id: &RowId ) -> Option<&(TimePoint, EntityPathHash)>

pub fn sort_indices_if_needed(&self)

Sort all unsorted indices in the store.

impl DataStore

pub fn sanity_check(&self) -> Result<(), SanityError>

Runs the sanity check suite for the entire datastore.

Returns an error if anything looks wrong.

impl DataStore

pub fn num_static_rows(&self) -> u64

Returns the number of static rows stored across this entire store.

pub fn static_size_bytes(&self) -> u64

Returns the size of the static data stored across this entire store.

pub fn num_temporal_rows(&self) -> u64

Returns the number of temporal index rows stored across this entire store, i.e. the sum of the number of rows across all of its temporal indexed tables.

pub fn temporal_size_bytes(&self) -> u64

Returns the size of the temporal index data stored across this entire store, i.e. the sum of the size of the data stored across all of its temporal indexed tables, in bytes.

pub fn num_temporal_buckets(&self) -> u64

Returns the number of temporal indexed buckets stored across this entire store.

pub fn entity_stats( &self, timeline: Timeline, entity_path_hash: EntityPathHash ) -> EntityStats

Stats for a specific entity path on a specific timeline

impl DataStore

pub fn register_subscriber( subscriber: Box<dyn StoreSubscriber> ) -> StoreSubscriberHandle

Registers a StoreSubscriber so it gets automatically notified when data gets added and/or removed to/from a DataStore.

Refer to StoreEvent’s documentation for more information about these events.

Scope

Registered StoreSubscribers are global scope: they get notified of all events from all existing DataStores, including DataStores created after the subscriber was registered.

Use StoreEvent::store_id to identify the source of an event.

Late registration

Subscribers must be registered before a store gets created to guarantee that no events were missed.

StoreEvent::event_id can be used to identify missing events.

Ordering

The order in which registered subscribers are notified is undefined and will likely become concurrent in the future.

If you need a specific order across multiple subscribers, embed them into an orchestrating subscriber.

pub fn with_subscriber<V: StoreSubscriber, T, F: FnMut(&V) -> T>( StoreSubscriberHandle: StoreSubscriberHandle, f: F ) -> Option<T>

Passes a reference to the downcasted subscriber to the given FnMut callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_subscriber_once<V: StoreSubscriber, T, F: FnOnce(&V) -> T>( StoreSubscriberHandle: StoreSubscriberHandle, f: F ) -> Option<T>

Passes a reference to the downcasted subscriber to the given FnOnce callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_subscriber_mut<V: StoreSubscriber, T, F: FnMut(&mut V) -> T>( StoreSubscriberHandle: StoreSubscriberHandle, f: F ) -> Option<T>

Passes a mutable reference to the downcasted subscriber to the given callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

impl DataStore

pub fn insert_row(&mut self, row: &DataRow) -> WriteResult<StoreEvent>

Inserts a DataRow’s worth of components into the datastore.

Trait Implementations

impl Clone for DataStore

fn clone(&self) -> Self

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Display for DataStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl SizeBytes for DataStore

fn heap_size_bytes(&self) -> u64

Returns the total size of self on the heap, in bytes.

fn total_size_bytes(&self) -> u64

Returns the total size of self in bytes, accounting for both stack and heap space.

fn stack_size_bytes(&self) -> u64

Returns the total size of self on the stack, in bytes.

fn is_pod() -> bool

Is Self just plain old data?