Batching imports over Bolt

Bulk import over Bolt: batch the statements

A Cypher export replayed one statement per RUN/PULL is bound by round trips and commits, not by the engine: every statement is its own autocommit, so it costs one network round trip plus one durable commit. Latency depends on the network and durability mode; a large export can spend most of its time waiting between statements. Group commit (topic guide) does not help here: it shares fsyncs between concurrent committers, and a single sequential client never has anything to share.

The fix is on the client: regroup the file's statements into UNWIND batches before sending them. An export's two data-carrying shapes

MERGE (n:Engram {id: 'e1'}) SET n = {id: 'e1', content: '…'};
MATCH (s:Session {id: 's1'}), (t:Engram {id: 'e1'}) MERGE (s)-[r:HAS_ENGRAM]->(t) SET r = {};

become one statement per few hundred rows, with the key values and property maps copied verbatim (they are already valid literals):

UNWIND [{k0: 'e1', p: {id: 'e1', content: 'example'}}] AS row
MERGE (n:Engram {id: row.k0}) SET n = row.p
RETURN count(n);

UNWIND [{s0: 's1', t0: 'e1', p: {}}] AS row
MATCH (s:Session {id: row.s0}), (t:Engram {id: row.t0})
MERGE (s)-[r:HAS_ENGRAM]->(t) SET r = row.p
RETURN count(r)

Three properties of the engine make this safe and fast:

A key taken from the driving row still uses the index. The anchor planner seeks the single-property and composite indexes with the pattern's value evaluated against the row, so {id: row.k0} and {type: row.k0, value: row.k1} cost the same lookups as literals.
A whole batch is one transaction. Either every row of a statement lands or none does, so a batch that errors (a uniqueness constraint on one row, say) can be replayed row by row to let the good rows through and name the bad one.
RETURN count(r) exposes silently skipped relationships. MATCH on a missing endpoint yields no row, so the one-at-a-time replay "succeeds" having created nothing. When count(r) is short of the batch size, replay the rows singly and report each one that returns 0.

Keep the file's order: flush every open node batch before the first relationship batch that follows it, so every node a relationship needs is committed first, the same guarantee the sequential replay gives.

Choose byte- and row-capped batches for the input limit. Use parameters for row data in applications. Validate endpoint counts and constraints on a disposable database before importing into a live graph. For local CSV, native batched subqueries provide independent commit boundaries.

Backup, restore and transfer · Transactions and concurrency · Server configuration

Batching imports over Bolt

Bulk import over Bolt: batch the statements

Related articles