Storage Operations

Looking for uploads?

This page covers data set management, retrieval, and lifecycle operations. For uploading data (from simple one-liner to manual store/pull/commit), see the Upload Pipeline.

Key Concepts

Data Set: A logical container of pieces stored with one provider. When a data set is created, a payment rail is established with that provider. All pieces in the data set share this single payment rail and are verified together via PDP proofs.

PieceCID: Content-addressed identifier for your data (format: bafkzcib...). Automatically calculated during upload and used to retrieve data from any provider.

Metadata: Optional key-value pairs for organization:

• Data Set Metadata: Max 10 keys (e.g., project, environment)
• Piece Metadata: Max 5 keys per piece (e.g., filename, contentType)

Copies and Durability: By default, upload() stores your data with 2 independent providers. Each provider maintains its own data set with separate PDP proofs and payment rails. If one provider goes down, your data is still available from the other.

Storage Manager: The main entry point for storage operations (synapse.storage). Handles provider selection, multi-copy orchestration, data set management, and provider-agnostic downloads.

Context Creation

Storage contexts represent a connection to a specific provider and data set. The SDK creates and manages contexts internally during upload(), but you can create them explicitly for data set management, retrieval, or split operations.

Single Context

import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })

// All options are optional
await synapse.storage.createContext({
  providerId: 1n,           // specific provider (optional)
  dataSetId: 42n,           // specific data set (optional)
  metadata: { source: "my-app" },        // data set metadata for matching/creation
  withCDN: true,            // enable fast-retrieval (paid, optional)
  excludeProviderIds: [3n], // skip specific providers (optional)
})

Multiple Contexts

For multi-copy replication, use createContexts():

const contexts = await synapse.storage.createContexts({
  copies: 3,                   // number of contexts (default: 2)
  providerIds: [1n, 2n, 3n],   // specific providers (mutually exclusive with dataSetIds)
  dataSetIds: [10n, 20n, 30n], // specific data sets (mutually exclusive with providerIds)
})
const [primary, secondary] = contexts

View creation options for createContext()

View creation options for createContexts()

Data Set Matching

Metadata Matching for Cost Efficiency

The SDK reuses existing data sets when metadata matches exactly, avoiding duplicate payment rails. To maximize reuse:

• Use consistent metadata keys and values across uploads
• Avoid changing metadata unnecessarily
• Group related content with the same metadata

Example: If you create a data set with {Application: "MyApp", Version: "1.0"}, all subsequent uploads with the same metadata will reuse that data set and its payment rail.

The SDK intelligently manages data sets to minimize on-chain transactions. The selection behavior depends on the parameters you provide:

Selection Scenarios:

1. Explicit data set ID: If you specify dataSetId, that exact data set is used (must exist and be accessible)
2. Specific provider: If you specify providerId, the SDK searches for matching data sets only within that provider’s existing data sets
3. Automatic selection: Without specific parameters, the SDK searches across all your data sets with any approved provider

Exact Metadata Matching: In scenarios 2 and 3, the SDK will reuse an existing data set only if it has exactly the same metadata keys and values as requested. This ensures data sets remain organized according to your specific requirements.

Selection Priority: When multiple data sets match your criteria:

• Data sets with existing pieces are preferred over empty ones
• Within each group (with pieces vs. empty), the oldest data set (lowest ID) is selected

Provider Selection (when no matching data sets exist):

• If you specify a provider (via providerId), that provider is used
• Otherwise, the SDK selects from endorsed providers for the primary copy and any approved provider for secondaries
• Before finalizing selection, the SDK verifies the provider is reachable via a ping test
• If a provider fails the ping test, the SDK tries the next candidate
• A new data set will be created automatically during the first commit

Retrieval

Download from any provider that has the piece. The SDK resolves the provider automatically:

import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: 'my-app' })

// Download using PieceCID from a previous upload
const pieceCid = "bafkzcib..." // from upload result
const bytes = await synapse.storage.download({ pieceCid })
const text = new TextDecoder().decode(bytes)
console.log("Downloaded:", text)

CDN-Accelerated Downloads

import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

// Enable CDN globally
const synapse = Synapse.create({
  account: privateKeyToAccount("0x..."),
  source: 'my-app',
  withCDN: true,
})

const bytes = await synapse.storage.download({ pieceCid: "bafkzcib..." })

// Or per-download:
const bytes2 = await synapse.storage.download({
  pieceCid: "bafkzcib...",
  withCDN: true,
})

Context-Specific Downloads

When using a StorageContext, downloads are restricted to that specific provider:

import { Synapse, type PieceCID } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
const pieceCid = null as unknown as PieceCID;
// Downloads from the provider associated with this context
const data = await ctx.download({ pieceCid })

CDN Option Inheritance

The withCDN option follows a clear inheritance hierarchy:

1. Synapse level: Default setting for all operations
2. StorageContext level: Can override Synapse’s default
3. Method level: Can override instance settings

await synapse.storage.download({ pieceCid })                // Uses Synapse's withCDN: true
await ctx.download({ pieceCid })                            // Uses context's withCDN: false
await synapse.storage.download({ pieceCid, withCDN: false }) // Method override: CDN disabled

When withCDN: true is set, it adds { withCDN: '' } to the data set’s metadata, ensuring CDN-enabled and non-CDN data sets remain separate.

Data Set Management

When You Need This

These APIs are useful when you want to inspect existing data sets, query stored pieces, or retrieve metadata. For basic upload/download, you don’t need these.

Getting All Data Sets

Retrieve all data sets owned by your account to inspect piece counts, CDN status, and metadata:

import { Synapse } from "@filoz/synapse-sdk";
import { privateKeyToAccount } from 'viem/accounts'

const synapse = Synapse.create({ account: privateKeyToAccount('0x...'), source: 'my-app' });
const dataSets = await synapse.storage.findDataSets();

for (const ds of dataSets) {
  console.log(`Dataset ${ds.pdpVerifierDataSetId}:`, {
    live: ds.isLive,
    cdn: ds.withCDN,
    pieces: ds.activePieceCount,
    metadata: ds.metadata
  });
}

Getting Data Set Pieces

List all pieces stored in a specific data set by iterating through a context:

const context = await synapse.storage.createContext({ dataSetId });

const pieces = [];
for await (const piece of context.getPieces()) {
  pieces.push(piece);
}
console.log(`Found ${pieces.length} pieces`);

Getting Piece Metadata

Access custom metadata attached to individual pieces:

import { WarmStorageService } from "@filoz/synapse-sdk/warm-storage";
import { privateKeyToAccount } from 'viem/accounts'

const warmStorage =  WarmStorageService.create({ account: privateKeyToAccount('0x...') });

const metadata = await warmStorage.getPieceMetadata({ dataSetId, pieceId: piece.pieceId });
console.log("Piece metadata:", metadata);

Getting Piece Size

Extract the size directly from a PieceCID using Synapse Core:

import { getSizeFromPieceCID } from "@filoz/synapse-core/piece";

const pieceCid = "bafkzcib...";
const size = getSizeFromPieceCID(pieceCid);
console.log(`Piece size: ${size} bytes`);

Storage Information

Query service-wide pricing, available providers, and network parameters:

const info = await synapse.storage.getStorageInfo();
console.log("Price/TiB/month:", info.pricing.noCDN.perTiBPerMonth);
console.log("Providers:", info.providers.length);

const providerInfo = await synapse.getProviderInfo("0x...");
console.log("PDP URL:", providerInfo.pdp.serviceURL);

Lifecycle Management

Terminating a Data Set

Irreversible Operation

Data set termination cannot be undone. Once initiated:

• The termination transaction is irreversible
• After the termination period, the provider may delete all data
• Payment rails associated with the data set will be terminated
• You cannot cancel the termination

Only terminate data sets when you’re certain you no longer need the data.

To delete an entire data set and discontinue payments for the service, call context.terminate(). This method submits an on-chain transaction to initiate the termination process. Following a defined termination period, payments will cease, and the service provider will be able to delete the data set.

You can also terminate a data set using synapse.storage.terminateDataSet({ dataSetId }), when the data set ID is known and creating a context is not necessary.

import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
// Via context
const hash = await ctx.terminate()
await synapse.client.waitForTransactionReceipt({ hash })
console.log("Dataset terminated successfully")

// Or directly by data set ID
const hash2 = await synapse.storage.terminateDataSet({ dataSetId: 42n })
await synapse.client.waitForTransactionReceipt({ hash: hash2 })

Deleting a Piece

To delete an individual piece from the data set, call context.deletePiece(). This method submits an on-chain transaction to initiate the deletion process.

Important: Piece deletion is irreversible and cannot be canceled once initiated.

import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
// List all pieces in the data set
const pieces = []
for await (const piece of ctx.getPieces()) {
  pieces.push(piece)
}

if (pieces.length > 0) {
  await ctx.deletePiece({ piece: pieces[0].pieceId })
  console.log(
    `Piece ${pieces[0].pieceCid} (ID: ${pieces[0].pieceId}) deleted successfully`
  )
}

// Delete by PieceCID
await ctx.deletePiece({ piece: "bafkzcib..." })

Next Steps

• Upload Pipeline - Upload data with multi-copy durability, from simple one-liner to manual store/pull/commit.

• Storage Costs - Calculate your monthly costs and understand funding requirements.

• Payment Management - Manage deposits, approvals, and payment rails.