Storage engines

Wiki / Operations

Paged, resident and hybrid databases

Persistence and memory usage are separate choices.

EngineKept in memoryPersistent storageTypical use
Paged (Orion), from 2.0The pages in use, within one memory budgetChecksummed pages: topology, properties and every indexEvery database; the default
VolatileThe whole graph and indexesNoneDisposable tests
Durable residentThe whole graph and indexesCommit log and full checkpointsSmall, latency-sensitive catalogues
Durable hybridTopology, indexes and a bounded property cacheCommit log with properties read on demandDatabases made by 1.x that have not moved yet

The database image writes to /data/gdb by default. Mount a volume at /data to retain that data when the container is recreated. The system database is resident; new user databases are Paged. GDB_STORAGE_ENGINE=hybrid (or resident) creates them on a 1.x engine instead.

Every engine provides the same Cypher behaviour. A database remembers its engine, and one made by 1.x keeps its engine until you move it, or until GDB_AUTO_UPGRADE moves it as it opens (see upgrading to 2.0).

The Paged engine (Orion)

A Paged database keeps the graph on disk in checksummed 8 KiB pages: node and relationship records, adjacency, properties, label lookups and every index (range, hash, composite, unique constraints, full-text, vector, point and spatial). The pages a database uses stay in a page cache shared by every database on the server, and its most used stores (topology, label lookups, the upper levels of each index) are pinned there whole. From version 2.4.3 a database's property stores are pinned as well when they fit in what the others leave of GDB_PIN_BYTES, so a scan reads property values straight from memory. A property store that does not fit is read through the cache.

  • Opening is immediate. A database opens by reading its latest manifest; no index is rebuilt and nothing is replayed beyond the last few writes. A 2-million-node database opens in about 0.2 seconds.
  • One memory budget. GDB_MEMORY_LIMIT sets what the server may use (default: half the container's or machine's memory); the page cache takes half of it (GDB_PAGE_CACHE_BYTES sets it directly), and pinned pages are part of the cache, never on top of it.
  • Larger than memory. When the graph is bigger than the cache, the pages in use are kept and the rest are read as queries need them.
  • Writes flush in the background. Changes collect in memory and are written to new pages on a separate thread while readers and writers carry on. The commit log keeps only what the pages do not yet hold.
  • Crash safe. A database is published one complete generation at a time, and every page is checked against its checksum when read.

CALL gdb.storage() shows a database's engine, its size on disk and how the page cache is serving it.

Changing a live database's engine

Run one administration statement at a time in Explorer or a driver's auto-commit session:

ALTER DATABASE analytics SET ENGINE PAGED;
ALTER DATABASE analytics SET ENGINE HYBRID;
ALTER DATABASE analytics SET ENGINE RESIDENT;
SHOW DATABASES;

A switch waits for existing work on that database and blocks new work there until it finishes. Other databases can continue running. The statement cannot run inside an explicit transaction.

Switching to resident mode loads all properties into memory, so allow enough RAM for the complete property set. Conversion work grows with the number of properties. Volatile databases always remain resident.

Hybrid: RAM and disk behaviour

Hybrid storage keeps graph topology, adjacency, property locations and indexes in memory. Only base property values move behind a bounded cache; it is not a fully disk-paged database.

From version 1.8.0 the topology is bit-packed. Each node's relationship ids take only the bits the relationship table needs, and each relationship row only the bits its type and endpoints need. A relationship's structure takes about 11 bytes, and a node with two relationships about 85.

Size memory for the graph structure, indexes, query results and GDS projections as well as the property cache. Indexed values may remain in memory even when their base property value is not cached.

Hybrid checkpoints seal the active log and incrementally reclaim stale property values. Resident checkpoints write a full snapshot. See checkpointing when planning maintenance on a large graph.

Configuration

Set these values in your Compose service's environment:

environment:
  GDB_DATA: /data/gdb
  GDB_DURABILITY: group
  GDB_PROPERTY_CACHE_BYTES: 33554432
  GDB_CHECKPOINT_THRESHOLD: 67108864

This selects group commit, a 32 MiB property cache per hybrid database and a 64 MiB active-log checkpoint trigger. The cache budget counts encoded record bytes. Each cached value also takes about 110 to 160 bytes of heap beyond the decoded value (from version 1.8.0; about 240 before). Values read or written recently stay in the cache when it evicts.

GDB_PROPERTY_SHAPES (on by default) stores each distinct set of property keys once and shares it between the elements that have it, which saves memory on regular data. Set it to false for the flat layout. It affects memory only, so you can change it at any restart. Data with more than 65,536 distinct key sets falls back to a key list per element.

GDB_PROPERTY_CACHE_BYTES=0 disables the property cache. GDB_CHECKPOINT_THRESHOLD=0 disables automatic checkpointing. Configure the remaining service settings using the Docker example.

Configuration · Durability · Backups

Planning a deployment? Review compatibility and licence setup for your instance.