# Storage Operations

:::tip[Looking for uploads?]
This page covers data set management, retrieval, and lifecycle operations. For uploading data (from simple one-liner to manual store/pull/commit), see the **[Upload Pipeline](/developer-guides/storage/upload-pipeline/)**.
:::

## Key Concepts

**Data Set**: A logical container of pieces stored with one provider. When a data set is created, a payment rail is established with that provider. All pieces in the data set share this single payment rail and are verified together via PDP proofs.

**PieceCID**: Content-addressed identifier for your data (format: `bafkzcib...`). Automatically calculated during upload and used to retrieve data from any provider.

**Metadata**: Optional key-value pairs for organization:

- **Data Set Metadata**: Max 10 keys (e.g., `project`, `environment`)
- **Piece Metadata**: Max 5 keys per piece (e.g., `filename`, `contentType`)

**Copies and Durability**: By default, `upload()` stores your data with 2 independent providers. Each provider maintains its own data set with separate PDP proofs and payment rails. If one provider goes down, your data is still available from the other.

**Storage Manager**: The main entry point for storage operations (`synapse.storage`). Handles provider selection, multi-copy orchestration, data set management, and provider-agnostic downloads.

## Context Creation

Storage contexts represent a connection to a specific provider and data set. The SDK creates and manages contexts internally during `upload()`, but you can create them explicitly for data set management, retrieval, or [split operations](/developer-guides/storage/upload-pipeline/#split-operations).

### Single Context

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })

// All options are optional
await synapse.storage.createContext({
  providerId: 1n,           // specific provider (optional)
  dataSetId: 42n,           // specific data set (optional)
  metadata: { source: "my-app" },        // data set metadata for matching/creation
  withCDN: true,            // enable fast-retrieval (paid, optional)
  excludeProviderIds: [3n], // skip specific providers (optional)
})
```

### Multiple Contexts

For multi-copy replication, use `createContexts()`:

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
// ---cut---
const contexts = await synapse.storage.createContexts({
  copies: 3,                   // number of contexts (default: 2)
  providerIds: [1n, 2n, 3n],   // specific providers (mutually exclusive with dataSetIds)
  dataSetIds: [10n, 20n, 30n], // specific data sets (mutually exclusive with providerIds)
})
const [primary, secondary] = contexts
```

[**View creation options for `createContext()`**](/reference/filoz/synapse-sdk/synapse/interfaces/storageserviceoptions/)

[**View creation options for `createContexts()`**](/reference/filoz/synapse-sdk/synapse/interfaces/contextcreatecontextsoptions/)

### Data Set Matching

:::tip[Metadata Matching for Cost Efficiency]
**The SDK reuses existing data sets when metadata matches exactly**, avoiding duplicate payment rails. To maximize reuse:

- Use consistent metadata keys and values across uploads
- Avoid changing metadata unnecessarily
- Group related content with the same metadata

**Example**: If you create a data set with `{Application: "MyApp", Version: "1.0"}`, all subsequent uploads with the same metadata will reuse that data set and its payment rail.
:::

The SDK intelligently manages data sets to minimize on-chain transactions. The selection behavior depends on the parameters you provide:

**Selection Scenarios**:

1. **Explicit data set ID**: If you specify `dataSetId`, that exact data set is used (must exist and be accessible)
2. **Specific provider**: If you specify `providerId`, the SDK searches for matching data sets only within that provider's existing data sets
3. **Automatic selection**: Without specific parameters, the SDK searches across all your data sets with any approved provider

**Exact Metadata Matching**: In scenarios 2 and 3, the SDK will reuse an existing data set only if it has **exactly** the same metadata keys and values as requested. This ensures data sets remain organized according to your specific requirements.

**Selection Priority**: When multiple data sets match your criteria:

- Data sets with existing pieces are preferred over empty ones
- Within each group (with pieces vs. empty), the oldest data set (lowest ID) is selected

**Provider Selection** (when no matching data sets exist):

- If you specify a provider (via `providerId`), that provider is used
- Otherwise, the SDK selects from endorsed providers for the primary copy and any approved provider for secondaries
- Before finalizing selection, the SDK verifies the provider is reachable via a ping test
- If a provider fails the ping test, the SDK tries the next candidate
- A new data set will be created automatically during the first commit

## Retrieval

Download from any provider that has the piece. The SDK resolves the provider automatically:

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: 'my-app' })

// Download using PieceCID from a previous upload
const pieceCid = "bafkzcib..." // from upload result
const bytes = await synapse.storage.download({ pieceCid })
const text = new TextDecoder().decode(bytes)
console.log("Downloaded:", text)
```

### CDN-Accelerated Downloads

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

// Enable CDN globally
const synapse = Synapse.create({
  account: privateKeyToAccount("0x..."),
  source: 'my-app',
  withCDN: true,
})

const bytes = await synapse.storage.download({ pieceCid: "bafkzcib..." })

// Or per-download:
const bytes2 = await synapse.storage.download({
  pieceCid: "bafkzcib...",
  withCDN: true,
})
```

### Context-Specific Downloads

When using a StorageContext, downloads are restricted to that specific provider:

```ts twoslash
// @lib: esnext,dom
import { Synapse, type PieceCID } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
const pieceCid = null as unknown as PieceCID;
// Downloads from the provider associated with this context
const data = await ctx.download({ pieceCid })
```

### CDN Option Inheritance

The `withCDN` option follows a clear inheritance hierarchy:

1. **Synapse level**: Default setting for all operations
2. **StorageContext level**: Can override Synapse's default
3. **Method level**: Can override instance settings

```ts twoslash
// @lib: esnext,dom
import { Synapse, type PieceCID } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({ withCDN: false })
const pieceCid = null as unknown as PieceCID;
// ---cut---
await synapse.storage.download({ pieceCid })                // Uses Synapse's withCDN: true
await ctx.download({ pieceCid })                            // Uses context's withCDN: false
await synapse.storage.download({ pieceCid, withCDN: false }) // Method override: CDN disabled
```

When `withCDN: true` is set, it adds `{ withCDN: '' }` to the data set's metadata, ensuring CDN-enabled and non-CDN data sets remain separate.

## Data Set Management

:::note[When You Need This]
These APIs are useful when you want to inspect existing data sets, query stored pieces, or retrieve metadata. For basic upload/download, you don't need these.
:::

### Getting All Data Sets

Retrieve all data sets owned by your account to inspect piece counts, CDN status, and metadata:

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk";
import { privateKeyToAccount } from 'viem/accounts'

const synapse = Synapse.create({ account: privateKeyToAccount('0x...'), source: 'my-app' });
const dataSets = await synapse.storage.findDataSets();

for (const ds of dataSets) {
  console.log(`Dataset ${ds.pdpVerifierDataSetId}:`, {
    live: ds.isLive,
    cdn: ds.withCDN,
    pieces: ds.activePieceCount,
    metadata: ds.metadata
  });
}
```

### Getting Data Set Pieces

List all pieces stored in a specific data set by iterating through a context:

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk";
import { privateKeyToAccount } from 'viem/accounts'

const synapse = Synapse.create({ account: privateKeyToAccount('0x...'), source: 'my-app' });
const dataSetId = 1n;
// ---cut---
const context = await synapse.storage.createContext({ dataSetId });

const pieces = [];
for await (const piece of context.getPieces()) {
  pieces.push(piece);
}
console.log(`Found ${pieces.length} pieces`);
```

### Getting Piece Metadata

Access custom metadata attached to individual pieces:

```ts twoslash
// @lib: esnext,dom
const dataSetId = 1n;
const piece = null as unknown as { pieceCid: string; pieceId: bigint };
// ---cut---
import { WarmStorageService } from "@filoz/synapse-sdk/warm-storage";
import { privateKeyToAccount } from 'viem/accounts'

const warmStorage =  WarmStorageService.create({ account: privateKeyToAccount('0x...') });

const metadata = await warmStorage.getPieceMetadata({ dataSetId, pieceId: piece.pieceId });
console.log("Piece metadata:", metadata);
```

### Getting Piece Size

Extract the size directly from a PieceCID using Synapse Core:

```ts twoslash
// @lib: esnext,dom
import { getSizeFromPieceCID } from "@filoz/synapse-core/piece";

const pieceCid = "bafkzcib...";
const size = getSizeFromPieceCID(pieceCid);
console.log(`Piece size: ${size} bytes`);
```

## Storage Information

Query service-wide pricing, available providers, and network parameters:

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk";
import { privateKeyToAccount } from 'viem/accounts'

const synapse = Synapse.create({ account: privateKeyToAccount('0x...'), source: 'my-app' });
// ---cut---
const info = await synapse.storage.getStorageInfo();
console.log("Price/TiB/month:", info.pricing.noCDN.perTiBPerMonth);
console.log("Providers:", info.providers.length);

const providerInfo = await synapse.getProviderInfo("0x...");
console.log("PDP URL:", providerInfo.pdp.serviceURL);
```

## Lifecycle Management

### Terminating a Data Set

:::warning[Irreversible Operation]
**Data set termination cannot be undone.** Once initiated:

- The termination transaction is irreversible
- After the termination period, the provider may delete all data
- Payment rails associated with the data set will be terminated
- You cannot cancel the termination

Only terminate data sets when you're certain you no longer need the data.
:::

To delete an entire data set and discontinue payments for the service, call `context.terminate()`.
This method submits an on-chain transaction to initiate the termination process. Following a defined termination period, payments will cease, and the service provider will be able to delete the data set.

You can also terminate a data set using `synapse.storage.terminateDataSet({ dataSetId })`, when the data set ID is known and creating a context is not necessary.

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
// Via context
const hash = await ctx.terminate()
await synapse.client.waitForTransactionReceipt({ hash })
console.log("Dataset terminated successfully")

// Or directly by data set ID
const hash2 = await synapse.storage.terminateDataSet({ dataSetId: 42n })
await synapse.client.waitForTransactionReceipt({ hash: hash2 })
```

### Deleting a Piece

To delete an individual piece from the data set, call `context.deletePiece()`.
This method submits an on-chain transaction to initiate the deletion process.

**Important:** Piece deletion is irreversible and cannot be canceled once initiated.

```ts twoslash
// @lib: esnext,dom
import { Synapse } from "@filoz/synapse-sdk"
import { privateKeyToAccount } from "viem/accounts"

const synapse = Synapse.create({ account: privateKeyToAccount("0x..."), source: "my-app" })
const ctx = await synapse.storage.createContext({
  metadata: { source: "my-app" },
})
// List all pieces in the data set
const pieces = []
for await (const piece of ctx.getPieces()) {
  pieces.push(piece)
}

if (pieces.length > 0) {
  await ctx.deletePiece({ piece: pieces[0].pieceId })
  console.log(
    `Piece ${pieces[0].pieceCid} (ID: ${pieces[0].pieceId}) deleted successfully`
  )
}

// Delete by PieceCID
await ctx.deletePiece({ piece: "bafkzcib..." })
```

## Next Steps

- **[Upload Pipeline](/developer-guides/storage/upload-pipeline/)** - Upload data with multi-copy durability, from simple one-liner to manual store/pull/commit.

- **[Storage Costs](/developer-guides/storage/storage-costs/)** - Calculate your monthly costs and understand funding requirements.

- **[Payment Management](/developer-guides/payments/payment-operations/)** - Manage deposits, approvals, and payment rails.