S3 Storage Implementation

S3 Storage Implementation#

This package provides an S3-compatible storage backend for ncps (Nix Cache Proxy Server). It implements the same interface as the local storage but uses S3 or S3-compatible servers like Garage for backend storage.

Features#

S3 Compatibility: Works with AWS S3 and any S3-compatible server (Garage, Ceph, etc.)
Drop-in Replacement: Implements the same interface as local storage
Configurable: Supports custom endpoints, regions, and authentication
Standard S3 Client: Built using the generic minio-go S3 v4 client library
Telemetry: Includes OpenTelemetry tracing for monitoring

Configuration#

The S3 storage is configured using the Config struct:

type Config struct {
    Bucket          string // S3 bucket name (required)
    Region          string // AWS region (optional for self-hosted S3 servers)
    Endpoint        string // S3 endpoint URL with scheme (required)
    AccessKeyID     string // Access key for authentication (required)
    SecretAccessKey string // Secret key for authentication (required)
    ForcePathStyle  bool   // Force path-style addressing (required for Garage and other self-hosted S3 servers, optional for AWS S3)
}

Important Notes#

Endpoint Format: The endpoint must include the URL scheme (e.g., "http://localhost:9000" or "https://s3.us-west-2.amazonaws.com"). The scheme determines whether SSL/TLS is used.
Region: Optional for self-hosted S3 servers like Garage, but typically required for AWS S3.
ForcePathStyle: Set to true for Garage and other self-hosted S3-compatible servers that require path-style addressing. Set to false for AWS S3 (which uses virtual-hosted-style by default).

Usage#

Garage Usage (Local Development)#

import (
    "context"
    "github.com/kalbasit/ncps/pkg/storage/s3"
)

ctx := context.Background()

// Create configuration for a self-hosted S3-compatible endpoint
cfg := s3.Config{
    Bucket:          "my-nix-cache",
    Endpoint:        "http://localhost:3900", // Must include scheme; 3900 is Garage's default S3 API port
    AccessKeyID:     "your-access-key",
    SecretAccessKey: "your-secret-key",
    ForcePathStyle:  true,                    // Garage and most self-hosted S3 servers require path-style addressing
}

// Create S3 store
store, err := s3.New(ctx, cfg)
if err != nil {
    log.Fatalf("Failed to create S3 store: %v", err)
}

// Use the store - it implements the storage.Store interface
exists := store.HasNarInfo(ctx, "abc123")

AWS S3 Usage#

// Create AWS S3 configuration
cfg := s3.Config{
    Bucket:          "my-nix-cache",
    Region:          "us-west-2",
    Endpoint:        "https://s3.us-west-2.amazonaws.com", // Must include scheme
    AccessKeyID:     "your-access-key",
    SecretAccessKey: "your-secret-key",
    ForcePathStyle:  false,                                // AWS S3 uses virtual-hosted-style
}

store, err := s3.New(ctx, cfg)

Storage Structure#

The S3 storage organizes data in the following structure within the bucket:

bucket/
├── config/
│   └── cache.key          # Secret key for signing
└── store/
    ├── narinfo/
    │   └── a/ab/abc123.narinfo  # NarInfo files with sharding
    └── nar/
        └── a/ab/abc123.nar      # NAR files with sharding

Key Features#

Sharding#

Files are automatically sharded using the first 1-2 characters of their hash to prevent too many files in a single directory, which can cause performance issues with S3.

Error Handling#

The implementation properly handles S3-specific errors:

NoSuchKey errors are converted to storage.ErrNotFound
Configuration validation ensures required fields are provided
Bucket existence is verified during initialization
Returns storage.ErrAlreadyExists when attempting to overwrite existing objects

Telemetry#

All operations include OpenTelemetry tracing with relevant attributes:

Operation names (e.g., "s3.GetNarInfo", "s3.PutNar")
Object keys and paths
Hash values and NAR URLs

Streaming Uploads#

The implementation uses the S3 client's streaming upload capability for NAR files (size=-1), allowing efficient uploads of large files without buffering the entire content in memory.

Testing#

The package includes comprehensive tests:

go test ./pkg/storage/s3/...

Dependencies#

github.com/minio/minio-go/v7 - generic S3 v4 client library used for S3 operations
github.com/nix-community/go-nix - Nix-specific types and utilities
go.opentelemetry.io/otel - OpenTelemetry for tracing

Migration from Local Storage#

To migrate from local storage to S3 storage:

Create an S3 bucket or self-hosted S3-compatible instance (e.g., Garage)
Configure the S3 storage with appropriate credentials
Replace the local storage initialization with S3 storage
The rest of your application code remains unchanged

// Before (local storage)
localStore, err := local.New(ctx, "/path/to/local/storage")

// After (S3 storage)
s3Store, err := s3.New(ctx, s3.Config{
    Bucket:          "my-nix-cache",
    Endpoint:        "https://s3.us-west-2.amazonaws.com",
    AccessKeyID:     "your-key",
    SecretAccessKey: "your-secret",
    ForcePathStyle:  false,
})

Security Considerations#

Store credentials securely (environment variables, IAM roles, etc.)
Use IAM policies to restrict bucket access (for AWS S3)
Consider using temporary credentials for production
Enable bucket versioning for data protection
Use appropriate bucket policies for access control
Always use HTTPS in production (e.g., Endpoint: "https://s3.amazonaws.com")

S3 Storage Implementation