Drivers

Name: Vercube
Rating: 5 (1 reviews)
Author: Vercube

Built-in storage providers and guide to creating custom drivers

Storage providers (also called drivers) are responsible for storing and retrieving data from various backends. Vercube includes built-in providers and makes it easy to create custom ones for any storage backend.

Built-in Providers

MemoryStorage

The MemoryStorage provider stores data in memory. It's fast and ideal for caching and temporary data storage.

Usage:

import { StorageManager } from '@vercube/storage';
import { MemoryStorage } from '@vercube/storage/drivers/MemoryStorage';

container.bind(StorageManager);

const storageManager = container.get(StorageManager);
storageManager.mount({
  name: 'cache',
  storage: MemoryStorage
});

// Store data
await storageManager.setItem({
  storage: 'cache',
  key: 'mykey',
  value: { foo: 'bar' }
});

// Retrieve data
const value = await storageManager.getItem({
  storage: 'cache',
  key: 'mykey'
});
console.log(value); // { foo: 'bar' }

Characteristics:

Feature	Description
Persistence	None - data is lost on restart
Speed	Extremely fast
Memory Usage	Grows with stored data
Use Cases	Caching, temporary data, development

Best Practices:

Use for caching frequently accessed data
Ideal for development and testing
Clear periodically to prevent memory leaks
Consider size limits for production use

S3Storage

The S3Storage provider stores data in AWS S3 (or S3-compatible services like MinIO, DigitalOcean Spaces, etc.). It's ideal for distributed applications, serverless environments, and persistent cloud storage.

Installation:

You need to install the AWS SDK for S3:

$ pnpm add @aws-sdk/client-s3

$ npm install @aws-sdk/client-s3

$ bun install @aws-sdk/client-s3

Basic Usage:

import { StorageManager } from '@vercube/storage';
import { S3Storage } from '@vercube/storage/drivers/S3Storage';

container.bind(StorageManager);

const storageManager = container.get(StorageManager);

// Recommended: Use IAM roles (no credentials needed)
storageManager.mount({
  name: 's3',
  storage: S3Storage,
  initOptions: {
    bucket: 'my-app-bucket',
    region: 'us-east-1'
    // Credentials are optional - IAM role is used if available
  }
});

// Alternative: Use explicit credentials for local development
// storageManager.mount({
//   name: 's3',
//   storage: S3Storage,
//   initOptions: {
//     bucket: 'my-app-bucket',
//     region: 'us-east-1',
//     credentials: {
//       accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
//       secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
//     }
//   }
// });

// Store data
await storageManager.setItem({
  storage: 's3',
  key: 'user:123',
  value: { id: '123', name: 'John' }
});

// Retrieve data
const user = await storageManager.getItem({
  storage: 's3',
  key: 'user:123'
});

Configuration Options:

The initOptions extends AWS S3ClientConfig with an additional bucket property:

Option	Type	Required	Description
`bucket`	`string`	Yes	S3 bucket name
`region`	`string`	Yes	AWS region (e.g., 'us-east-1')
`credentials`	`object`	No	AWS credentials (accessKeyId, secretAccessKey, sessionToken). If omitted, AWS SDK uses the default credential provider chain (IAM roles, environment variables, config files)
`endpoint`	`string`	No	Custom endpoint for S3-compatible services
`forcePathStyle`	`boolean`	No	Use path-style URLs (required for some S3-compatible services like MinIO)

Note: When credentials is omitted, the AWS SDK automatically attempts to load credentials from:

IAM roles (Lambda execution role, EC2 instance profile, ECS task role)
Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
Shared credentials file (~/.aws/credentials)
ECS container credentials
EC2 instance metadata service

Using with S3-Compatible Services:

// MinIO
storageManager.mount({
  name: 'minio',
  storage: S3Storage,
  initOptions: {
    bucket: 'my-bucket',
    region: 'us-east-1',
    endpoint: 'http://localhost:9000',
    credentials: {
      accessKeyId: 'minioadmin',
      secretAccessKey: 'minioadmin'
    },
    forcePathStyle: true // Required for MinIO
  }
});

// DigitalOcean Spaces
storageManager.mount({
  name: 'spaces',
  storage: S3Storage,
  initOptions: {
    bucket: 'my-space',
    region: 'nyc3',
    endpoint: 'https://nyc3.digitaloceanspaces.com',
    credentials: {
      accessKeyId: process.env.DO_SPACES_KEY,
      secretAccessKey: process.env.DO_SPACES_SECRET
    }
  }
});

// Cloudflare R2
storageManager.mount({
  name: 'r2',
  storage: S3Storage,
  initOptions: {
    bucket: 'my-bucket',
    region: 'auto',
    endpoint: `https://${process.env.CF_ACCOUNT_ID}.r2.cloudflarestorage.com`,
    credentials: {
      accessKeyId: process.env.R2_ACCESS_KEY_ID,
      secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
    }
  }
});

Characteristics:

Feature	Description
Persistence	Yes - data persists across restarts
Speed	Network-dependent (slower than memory)
Scalability	Highly scalable, distributed
Use Cases	Production storage, file storage, distributed apps

Authentication Methods:

The S3Storage driver supports multiple authentication methods, with different security characteristics suitable for various environments.

1. IAM Roles (Recommended for AWS Environments)

When running in AWS environments (Lambda, EC2, ECS, etc.), IAM roles provide the most secure authentication method. Credentials are automatically managed by AWS without manual handling.

// AWS Lambda - No credentials needed!
storageManager.mount({
  name: 's3',
  storage: S3Storage,
  initOptions: {
    bucket: 'my-app-bucket',
    region: 'us-east-1'
    // No credentials field - IAM role is used automatically
  }
});

How it works:

AWS SDK automatically uses the IAM role attached to your Lambda function, EC2 instance, or ECS task
Credentials are temporary and automatically rotated by AWS
No secrets to manage or risk exposing in code

Setup:

Attach an IAM role to your AWS resource (Lambda function, EC2 instance, etc.)
Grant the role S3 permissions (e.g., s3:GetObject, s3:PutObject)
Omit the credentials field in initOptions

2. AWS Secrets Manager (Recommended for Secure Credential Storage)

For applications that require explicit credentials, use AWS Secrets Manager to store and retrieve them securely.

import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';

async function getS3Credentials() {
  const client = new SecretsManagerClient({ region: 'us-east-1' });
  const response = await client.send(
    new GetSecretValueCommand({ SecretId: 'my-app/s3-credentials' })
  );
  
  if (!response.SecretString) {
    throw new Error('Secret not found or uses SecretBinary');
  }
  
  return JSON.parse(response.SecretString);
}

// In your container setup
const credentials = await getS3Credentials();

storageManager.mount({
  name: 's3',
  storage: S3Storage,
  initOptions: {
    bucket: 'my-app-bucket',
    region: 'us-east-1',
    credentials: {
      accessKeyId: credentials.accessKeyId,
      secretAccessKey: credentials.secretAccessKey
    }
  }
});

Benefits:

Credentials are encrypted at rest and in transit
Centralized secret management
Automatic rotation capabilities
Audit logging of secret access

3. STS Temporary Credentials (For Cross-Account or Federated Access)

Use AWS Security Token Service (STS) to assume roles and obtain temporary credentials.

import { STSClient, AssumeRoleCommand } from '@aws-sdk/client-sts';

async function assumeRole() {
  const stsClient = new STSClient({ region: 'us-east-1' });
  const response = await stsClient.send(
    new AssumeRoleCommand({
      RoleArn: 'arn:aws:iam::123456789012:role/S3AccessRole',
      RoleSessionName: 'vercube-s3-session',
      DurationSeconds: 3600 // 1 hour
    })
  );
  
  const { Credentials } = response;
  if (!Credentials?.AccessKeyId || !Credentials?.SecretAccessKey || !Credentials?.SessionToken) {
    throw new Error('Failed to assume role - incomplete credentials returned');
  }
  
  return {
    accessKeyId: Credentials.AccessKeyId,
    secretAccessKey: Credentials.SecretAccessKey,
    sessionToken: Credentials.SessionToken
  };
}

// Get temporary credentials
const credentials = await assumeRole();

storageManager.mount({
  name: 's3',
  storage: S3Storage,
  initOptions: {
    bucket: 'my-app-bucket',
    region: 'us-east-1',
    credentials
  }
});

Use cases:

Cross-account S3 access
Federated user access
Time-limited access requirements
Enhanced security through temporary credentials

4. Environment Variables (For Development)

For local development, you can use environment variables. However, this method should be avoided in production.

// .env file
// AWS_ACCESS_KEY_ID=your-access-key
// AWS_SECRET_ACCESS_KEY=your-secret-key

storageManager.mount({
  name: 's3',
  storage: S3Storage,
  initOptions: {
    bucket: 'my-app-bucket',
    region: 'us-east-1',
    credentials: {
      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
    }
  }
});

Important: Never commit credentials to source control. Use .gitignore to exclude .env files.

Environment-Specific Configuration:

Configure different authentication methods based on your environment:

import { StorageManager } from '@vercube/storage';
import { S3Storage } from '@vercube/storage/drivers/S3Storage';

const isProduction = process.env.NODE_ENV === 'production';
const isLambda = !!process.env.AWS_LAMBDA_FUNCTION_NAME;

const storageManager = container.get(StorageManager);

if (isLambda || isProduction) {
  // Production: Use IAM roles (no credentials needed)
  storageManager.mount({
    name: 's3',
    storage: S3Storage,
    initOptions: {
      bucket: process.env.S3_BUCKET!,
      region: process.env.AWS_REGION || 'us-east-1'
      // IAM role provides credentials automatically
    }
  });
} else {
  // Development: Use local credentials
  storageManager.mount({
    name: 's3',
    storage: S3Storage,
    initOptions: {
      bucket: process.env.S3_BUCKET || 'dev-bucket',
      region: process.env.AWS_REGION || 'us-east-1',
      credentials: {
        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
      }
    }
  });
}

Best Practices:

Use IAM roles in production - Most secure method for AWS environments
Never hardcode credentials - Always use environment variables, Secrets Manager, or IAM roles
Use Secrets Manager - For storing credentials that can't use IAM roles
Rotate credentials regularly - If using static credentials, rotate them periodically
Apply least privilege - Grant only the S3 permissions your application needs
Enable S3 bucket encryption - Encrypt data at rest using S3-managed or KMS keys
Use VPC endpoints - For enhanced security when accessing S3 from within AWS VPC
Implement caching layer - Use MemoryStorage for frequently accessed data to reduce S3 calls
Use appropriate S3 bucket policies - Restrict access to authorized principals only
Consider S3 storage classes - Optimize costs based on access patterns
Monitor access logs - Enable S3 access logging for security auditing

Creating Custom Providers

Creating a custom provider allows you to store data in any backend: databases, cloud storage, external services, etc.

Basic Provider Template

import { Storage } from '@vercube/storage';
import type { StorageTypes } from '@vercube/storage';

export class CustomStorage extends Storage {
  private client: any; // Your storage client
  
  /**
   * Initialize the storage with options
   */
  public async initialize(options?: StorageTypes.Options): Promise<void> {
    // Setup code: connect to service, initialize client, etc.
    this.client = await connectToService(options);
  }
  
  /**
   * Retrieve a value by key
   */
  public async getItem<T>(key: string): Promise<T | null> {
    const data = await this.client.get(key);
    return data ? JSON.parse(data) : null;
  }
  
  /**
   * Store a value with a key
   */
  public async setItem<T>(key: string, value: T): Promise<void> {
    await this.client.set(key, JSON.stringify(value));
  }
  
  /**
   * Check if a key exists
   */
  public async hasItem(key: string): Promise<boolean> {
    return this.client.has(key);
  }
  
  /**
   * Remove a value by key
   */
  public async deleteItem(key: string): Promise<void> {
    await this.client.delete(key);
  }
  
  /**
   * Get all keys
   */
  public async getKeys(): Promise<string[]> {
    return this.client.keys();
  }
  
  /**
   * Remove all items
   */
  public async clear(): Promise<void> {
    await this.client.clear();
  }
  
  /**
   * Get the number of stored items
   */
  public async size(): Promise<number> {
    return this.client.size();
  }
}

Required Methods

Every custom storage provider must implement these methods:

Method	Description
`initialize(options)`	Initialize the storage with configuration options
`getItem<T>(key)`	Retrieve a value by key
`setItem<T, U>(key, value, options?)`	Store a value with a key and optional options
`hasItem(key)`	Check if a key exists
`deleteItem(key)`	Remove a value by key
`getKeys()`	Get all keys
`clear()`	Remove all items
`size()`	Get the number of stored items

Example: Redis Storage

A complete example of a Redis storage provider:

import { Storage } from '@vercube/storage';
import type { StorageTypes } from '@vercube/storage';
import Redis from 'ioredis';

interface RedisStorageOptions {
  host: string;
  port: number;
  password?: string;
  db?: number;
  keyPrefix?: string;
}

export class RedisStorage extends Storage {
  private redis!: Redis;
  private prefix: string = '';
  
  public async initialize(options: RedisStorageOptions): Promise<void> {
    this.redis = new Redis({
      host: options.host,
      port: options.port,
      password: options.password,
      db: options.db || 0
    });
    
    this.prefix = options.keyPrefix || '';
  }
  
  private getKey(key: string): string {
    return this.prefix + key;
  }
  
  public async getItem<T>(key: string): Promise<T | null> {
    const data = await this.redis.get(this.getKey(key));
    return data ? JSON.parse(data) : null;
  }
  
  public async setItem<T>(key: string, value: T): Promise<void> {
    await this.redis.set(this.getKey(key), JSON.stringify(value));
  }
  
  public async hasItem(key: string): Promise<boolean> {
    const exists = await this.redis.exists(this.getKey(key));
    return exists === 1;
  }
  
  public async deleteItem(key: string): Promise<void> {
    await this.redis.del(this.getKey(key));
  }
  
  public async getKeys(): Promise<string[]> {
    const keys = await this.redis.keys(this.prefix + '*');
    return keys.map(k => k.replace(this.prefix, ''));
  }
  
  public async clear(): Promise<void> {
    const keys = await this.redis.keys(this.prefix + '*');
    if (keys.length > 0) {
      await this.redis.del(...keys);
    }
  }
  
  public async size(): Promise<number> {
    const keys = await this.redis.keys(this.prefix + '*');
    return keys.length;
  }
}

Usage:

import { StorageManager } from '@vercube/storage';
import { RedisStorage } from './storages/RedisStorage';

// Register custom storage in container
container.bind(RedisStorage);

const storageManager = container.get(StorageManager);

// Mount custom storage
storageManager.mount({
  name: 'redis',
  storage: RedisStorage,
  initOptions: {
    host: 'localhost',
    port: 6379,
    keyPrefix: 'myapp:'
  }
});

// Use it like any other storage
await storageManager.setItem({
  storage: 'redis',
  key: 'user:123',
  value: { name: 'John' }
});

Example: File System Storage

A storage provider that persists data to the file system:

import { Storage } from '@vercube/storage';
import type { StorageTypes } from '@vercube/storage';
import * as fs from 'fs/promises';
import * as path from 'path';

interface FileStorageOptions {
  directory: string;
}

export class FileStorage extends Storage {
  private directory!: string;
  
  public async initialize(options: FileStorageOptions): Promise<void> {
    this.directory = options.directory;
    
    // Ensure directory exists
    await fs.mkdir(this.directory, { recursive: true });
  }
  
  private getFilePath(key: string): string {
    // Sanitize key for use as filename
    const safeKey = key.replace(/[^a-zA-Z0-9-_:]/g, '_');
    return path.join(this.directory, `${safeKey}.json`);
  }
  
  public async getItem<T>(key: string): Promise<T | null> {
    try {
      const filePath = this.getFilePath(key);
      const data = await fs.readFile(filePath, 'utf-8');
      return JSON.parse(data);
    } catch (error) {
      return null;
    }
  }
  
  public async setItem<T>(key: string, value: T): Promise<void> {
    const filePath = this.getFilePath(key);
    await fs.writeFile(filePath, JSON.stringify(value, null, 2));
  }
  
  public async hasItem(key: string): Promise<boolean> {
    try {
      await fs.access(this.getFilePath(key));
      return true;
    } catch {
      return false;
    }
  }
  
  public async deleteItem(key: string): Promise<void> {
    try {
      await fs.unlink(this.getFilePath(key));
    } catch {
      // Ignore if file doesn't exist
    }
  }
  
  public async getKeys(): Promise<string[]> {
    const files = await fs.readdir(this.directory);
    return files
      .filter(f => f.endsWith('.json'))
      .map(f => f.replace('.json', ''));
  }
  
  public async clear(): Promise<void> {
    const files = await fs.readdir(this.directory);
    for (const file of files) {
      if (file.endsWith('.json')) {
        await fs.unlink(path.join(this.directory, file));
      }
    }
  }
  
  public async size(): Promise<number> {
    const keys = await this.getKeys();
    return keys.length;
  }
}

Usage:

storageManager.mount({
  name: 'files',
  storage: FileStorage,
  initOptions: {
    directory: './data/storage'
  }
});

await storageManager.setItem({
  storage: 'files',
  key: 'config',
  value: { theme: 'dark' }
});

Provider Configuration

Register Provider in Container

If your provider has dependencies, register it in the container:

import { Container } from '@vercube/di';
import { CustomStorage } from './storages/CustomStorage';

export function setupContainer(container: Container): void {
  // Register provider
  container.bind(CustomStorage);
  
  // Configure storage manager to use it
  const storageManager = container.get(StorageManager);
  storageManager.mount({
    name: 'custom',
    storage: CustomStorage,
    initOptions: { ... }
  });
}

Provider with Dependencies

If your provider needs other services:

import { Inject } from '@vercube/di';
import { Storage } from '@vercube/storage';
import { Logger } from '@vercube/logger';

export class LoggingStorage extends Storage {
  @Inject(Logger)
  private logger!: Logger;
  
  private innerStorage!: Storage;
  
  public async initialize(options: { storage: Storage }): Promise<void> {
    this.innerStorage = options.storage;
  }
  
  public async getItem<T>(key: string): Promise<T | null> {
    this.logger.debug(`Getting item: ${key}`);
    const value = await this.innerStorage.getItem<T>(key);
    this.logger.debug(`Got item: ${key}`, { found: value !== null });
    return value;
  }
  
  public async setItem<T>(key: string, value: T): Promise<void> {
    this.logger.debug(`Setting item: ${key}`);
    await this.innerStorage.setItem(key, value);
    this.logger.info(`Item set: ${key}`);
  }
  
  // ... implement other methods with logging
}

Async Initialization

If your provider needs async initialization:

export class AsyncStorage extends Storage {
  private connection: Connection;
  
  public async initialize(options: ConnectionOptions): Promise<void> {
    // Async initialization is fully supported
    this.connection = await connectToService(options);
    
    // Wait for connection to be ready
    await this.connection.ready();
  }
  
  // ... implement other methods
}

Using Multiple Providers

Configure multiple providers for different use cases:

import { StorageManager } from '@vercube/storage';
import { MemoryStorage } from '@vercube/storage/drivers/MemoryStorage';
import { S3Storage } from '@vercube/storage/drivers/S3Storage';
import { RedisStorage } from './storages/RedisStorage';

const storageManager = container.get(StorageManager);

// Memory for fast local cache
storageManager.mount({
  name: 'cache',
  storage: MemoryStorage
});

// S3 for persistent cloud storage
storageManager.mount({
  name: 's3',
  storage: S3Storage,
  initOptions: {
    bucket: process.env.S3_BUCKET,
    region: process.env.AWS_REGION,
    credentials: {
      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
    }
  }
});

// Redis for distributed cache
storageManager.mount({
  name: 'distributed',
  storage: RedisStorage,
  initOptions: {
    host: process.env.REDIS_HOST,
    port: 6379,
    keyPrefix: 'app:'
  }
});

Troubleshooting

Provider Not Storing Data

Problem: setItem() works but getItem() returns null

Solutions:

Check serialization/deserialization:

// Make sure data can be serialized
await storage.setItem('key', {
  date: new Date().toISOString(), // ✅ Serialize dates as strings
  // date: new Date(), // ❌ May not deserialize correctly
});

Verify key format is consistent:

// Use consistent key formatting
await storage.setItem('user:123', data);
const value = await storage.getItem('user:123'); // ✅ Same key

// Not this:
await storage.setItem('user:123', data);
const value = await storage.getItem('user-123'); // ❌ Different key

Provider Initialization Error

Problem: "Failed to initialize storage provider"

Solutions:

Check initialize() method doesn't throw:

public async initialize(options: any): Promise<void> {
  try {
    // initialization code
  } catch (error) {
    console.error('Storage init failed:', error);
    // Don't throw - handle gracefully
  }
}

Verify provider is bound in container:

container.bind(CustomStorage);

Check provider initOptions:

storageManager.mount({
  name: 'custom',
  storage: CustomStorage,
  initOptions: {
    // Make sure all required options are provided
    requiredOption: 'value'
  }
});

Provider Missing Dependency

Problem: Injected dependency is undefined

Solution: Make sure dependency is registered before storage configuration:

// Register dependencies first
container.bind(Logger);
container.bind(CustomStorage);

// Then configure storage
container.bind(StorageManager);
const storageManager = container.get(StorageManager);
storageManager.mount({
  name: 'custom',
  storage: CustomStorage
});

Async Provider Not Working

Problem: Async initialize() doesn't complete before use

Solution: StorageManager handles async initialization automatically. Make sure you're using await when needed:

// Initialize is called automatically when mounting
storageManager.mount({
  name: 'async',
  storage: AsyncStorage,
  initOptions: { ... }
});

// Storage is ready to use after mount
await storageManager.setItem({
  storage: 'async',
  key: 'mykey',
  value: 'myvalue'
});

Report an issue or Edit this page on GitHub

API

Complete API reference for Storage module