Contributing

Contributing to ncps#

Thank you for your interest in contributing to ncps! This document provides guidelines and instructions for contributing to the project.

Getting Started#

Prerequisites#

The project uses Nix flakes with direnv for reproducible development environments. You'll need:

1. Nix with flakes enabled - Installation guide
2. direnv - Installation guide

Initial Setup#

1. Clone the repository:

   git clone https://github.com/kalbasit/ncps.git
   cd ncps

2. Allow direnv:

   direnv allow

   This will automatically load the development environment with all required tools:

   • Go
   • golangci-lint
   • delve (debugger)
   • watchexec
   • sqlfluff
   • Garage (for S3 testing)
   • PostgreSQL (for database testing)
   • MySQL/MariaDB (for database testing)
   • Redis (for distributed locking testing)

Development Environment#

Available Tools#

Once in the development shell, you have access to:

Tool	Purpose
go	Go compiler and toolchain
golangci-lint	Code linting with 30+ linters
delve	Go debugger
watchexec	File watcher for hot-reloading
sqlfluff	SQL linting and formatting
garage	S3-compatible object storage
postgresql	PostgreSQL database server
mariadb	MySQL/MariaDB database server
redis	Redis server for distributed locks

Development Dependencies#

The project uses process-compose-flake for managing development services. Start dependencies with:

nix run .#deps

This starts:

• Garage - S3-compatible object store (S3 API on port 9000, admin on 3903)
  • Test bucket: test-bucket
  • Credentials: GK1234567890abcdef12345678 / 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
  • Self-validation ensures proper setup
• PostgreSQL - Database server (port 5432)
  • Test database: test-db
  • Credentials: test-user / test-password
  • Connection URL: postgresql://test-user:[email protected]:5432/test-db?sslmode=disable
• MariaDB - MySQL-compatible database server (port 3306)
  • Test database: test-db
  • Credentials: test-user / test-password
  • Connection URL: mysql://test-user:[email protected]:3306/test-db
• Redis - Distributed locking server (port 6379)
  • No authentication required (test environment)
  • Used for distributed lock testing

Development Workflow#

Running the Development Server#

The development server supports hot-reloading and multiple storage backends:

# Using local filesystem storage (default, no dependencies required)
./dev-scripts/run.sh
# or explicitly
./dev-scripts/run.sh local

# Using S3/Garage storage (requires dependencies to be running)
# In a separate terminal:
nix run .#deps

# Then run the dev server:
./dev-scripts/run.sh s3

The server automatically restarts when you modify code files.

Editing Documentation#

The documentation for this project is managed using Trilium. Follow these steps to contribute to the documentation:

1. Run the documentation editor:

   trilium-edit-docs

   This tool is available in the PATH provided by the Nix flake, see Development Environment in the Developer Guide for more information.

2. Edit the documentation through Trilium: The Trilium interface will open, allowing you to edit the notes. Trilium automatically exports the markdown files back to the repository as you make changes.

3. Wait and close: Wait 5 minutes after you have finished all your edits to ensure all changes are synced, then close Trilium.

4. Format the documentation:

   Run the project formatter to ensure the markdown files follow the project's standards:

   nix fmt

5. Submit your changes: Submit a Pull Request with your changes.

Database Migrations#

The schema is defined in Ent (ent/schema/*.go), per-dialect migrations are generated by Atlas (used as a Go library, never the CLI), and Goose applies them at runtime via the embedded migrations/<dialect>/ directories.

Creating a new migration:

1. Edit ent/schema/<entity>.go (field/edge/index/annotation change).

2. Regenerate the Ent client: go generate ./ent/... (or task ent:generate).

3. Generate per-dialect Atlas migrations:

   task migrations:gen NAME=descriptive_snake_case

   This writes one timestamp-prefixed .sql per dialect under migrations/sqlite/, migrations/postgres/, and migrations/mysql/, plus updates each dialect's atlas.sum integrity file.

Applying migrations:

ncps migrate up --cache-database-url=sqlite:/path/to/db.sqlite
ncps migrate up --cache-database-url=postgresql://user:pass@host:port/db
ncps migrate up --cache-database-url=mysql://user:pass@host:port/db
# Preview without touching the DB:
ncps migrate up --cache-database-url=... --dry-run

The same flag also accepts the env var CACHE_DATABASE_URL. Down migrations are intentionally not supported — see the expand-contract policy in CLAUDE.md for the safe column-change recipe.

Skills under .agent/skills/ codify the day-to-day workflows:

• /migrate-new — edit an Ent schema + generate the per-dialect migrations
• /migrate-up — apply migrations
• /migrate-down — read this to learn why you don't want to roll back

The cmd/ent-lint AST linter (run via task ent:lint or task ent:check) enforces the five Ent codegen invariants documented at the top of cmd/ent-lint/main.go.

Code Quality Standards#

Formatting#

IMPORTANT: Always run formatters first before making manual changes:

# Format all files (Go, Nix, SQL, YAML, Markdown)
nix fmt

The project uses:

• gofumpt - Stricter Go formatting
• goimports - Import organization
• gci - Import grouping (standard → default → alias → localmodule)
• nixfmt - Nix code formatting
• sqlfluff - SQL formatting and linting
• yamlfmt - YAML formatting
• mdformat - Markdown formatting

Linting#

IMPORTANT: Always run golangci-lint run --fix first to automatically fix issues:

# Auto-fix all fixable linting issues
golangci-lint run --fix

# Lint without auto-fix
golangci-lint run

# Lint specific package
golangci-lint run ./pkg/server/...

The project uses 30+ linters including:

• err113 - Explicit error wrapping
• exhaustive - Exhaustive switch statements
• gosec - Security checks
• paralleltest - Parallel test detection
• testpackage - Test package naming

See .golangci.yml for complete linter configuration.

SQL Linting#

# Lint SQL files
sqlfluff lint migrations/sqlite/*.sql
sqlfluff lint migrations/postgres/*.sql
sqlfluff lint migrations/mysql/*.sql

# Format SQL files
sqlfluff format migrations/sqlite/*.sql

Testing#

Running Tests#

# Run all tests with race detector (recommended)
go test -race ./...

# Run tests for specific package
go test -race ./pkg/server/...

# Run a single test
go test -race -run TestName ./pkg/server/...

Integration Tests#

The project includes integration tests for S3, PostgreSQL, MySQL, and Redis. Integration tests are disabled by default and must be explicitly enabled using shell helper functions.

Quick Start:

# In terminal 1: Start development dependencies
nix run .#deps

# In terminal 2: Enable integration tests and run tests
eval "$(enable-integration-tests)"
go test -race ./...

Available Helper Commands:

The development shell provides commands to easily enable/disable integration tests:

Command	Description
eval "$(enable-s3-tests)"	Enable S3/Garage integration tests
eval "$(enable-postgres-tests)"	Enable PostgreSQL integration tests
eval "$(enable-redis-tests)"	Enable Redis integration tests
eval "$(enable-mysql-tests)"	Enable MySQL integration tests
eval "$(enable-integration-tests)"	Enable all integration tests at once
eval "$(disable-integration-tests)"	Disable all integration tests

Running Specific Integration Tests:

# Start dependencies (in a separate terminal)
nix run .#deps

# Enable and run S3 tests only
eval "$(enable-s3-tests)"
go test -race ./pkg/storage/s3

# Enable and run database tests only
eval "$(enable-postgres-tests)"
eval "$(enable-mysql-tests)"
go test -race ./pkg/database

# Enable all tests and run everything
eval "$(enable-integration-tests)"
go test -race ./...

# Disable integration tests when done
eval "$(disable-integration-tests)"

What the Helper Commands Do:

The helper commands output shell export statements that you evaluate in your current shell:

• **enable-s3-tests** exports:
  • NCPS_TEST_S3_BUCKET=test-bucket
  • NCPS_TEST_S3_ENDPOINT=http://127.0.0.1:9000
  • NCPS_TEST_S3_REGION=us-east-1
  • NCPS_TEST_S3_ACCESS_KEY_ID=GK1234567890abcdef12345678
  • NCPS_TEST_S3_SECRET_ACCESS_KEY=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
• **enable-postgres-tests** exports:
  • NCPS_TEST_POSTGRES_URL=postgresql://test-user:[email protected]:5432/test-db?sslmode=disable
• **enable-mysql-tests** exports:
  • NCPS_TEST_MYSQL_URL=mysql://test-user:[email protected]:3306/test-db
• **enable-redis-tests** exports:
  • NCPS_ENABLE_REDIS_TESTS=1

Tests automatically skip if these environment variables aren't set, so you can run go test -race ./... without enabling integration tests and only unit tests will run.

Test Requirements#

• Use testify for assertions
• Enable race detector (-race flag)
• Use _test package suffix (enforced by testpackage linter)
• Write parallel tests where possible (checked by paralleltest linter)
• Each test should be isolated and not depend on other tests

Nix Build Tests#

# Run all checks including integration tests
nix flake check

# Build package (includes test phase)
nix build

The Nix build automatically:

1. Starts Garage, PostgreSQL, MariaDB, and Redis in preCheck phase
2. Creates test databases and buckets
3. Exports test environment variables
4. Runs all tests (including integration tests)
5. Stops services in postCheck phase

Helm Chart Testing#

The project includes comprehensive Helm chart testing using a local Kind cluster with Garage, PostgreSQL, MariaDB, and Redis. A unified CLI tool (k8s-tests) manages the complete testing workflow.

Prerequisites:

All tools are provided by the Nix development environment:

• Docker (for Kind)
• kubectl, helm, kind (via Nix)
• k8s-tests CLI (automatically available in nix develop)

Quick Start (Complete Workflow):

# Run all 5 steps in one command
k8s-tests all

This command:

1. Creates Kind cluster with all dependencies
2. Builds and pushes Docker image
3. Generates 13 test values files
4. Installs all test deployments
5. Runs comprehensive tests

Individual Steps:

# 1. Create Kind cluster with all dependencies (Garage, PostgreSQL, MariaDB, Redis)
k8s-tests cluster create

# 2. Build Docker image, push to local registry, and generate test values
k8s-tests generate --push

# 3. Install all test deployments
k8s-tests install

# 4. Run tests across all deployments
k8s-tests test

# 5. Cleanup when done
k8s-tests cleanup

The cluster can be reused across test runs. Use k8s-tests cluster destroy to clean up when done.

Alternative: Use External Image

If you've already built and pushed an image:

# Use a specific tag
k8s-tests generate sha-cf09394

# Use custom registry and repository
k8s-tests generate 0.5.1 docker.io kalbasit/ncps

# Or build externally and use the tag
DOCKER_IMAGE_TAGS="yourregistry.com/ncps:sha$(git rev-parse --short HEAD)" nix run .#push-docker-image
k8s-tests generate sha$(git rev-parse --short HEAD)-x86_64-linux yourregistry.com ncps

Test Permutations:

The tool generates 13 different test configurations:

• Single Instance (7 scenarios):
  • Local storage: SQLite, PostgreSQL, MariaDB
  • S3 storage: SQLite, PostgreSQL, MariaDB
  • S3 + PostgreSQL + CDC enabled
• External Secrets (2 scenarios):
  • S3 + PostgreSQL/MariaDB with existing Kubernetes secrets
• High Availability (4 scenarios):
  • 2 replicas with S3 storage
  • PostgreSQL or MariaDB database
  • Redis locks
  • With/without CDC enabled

Testing Individual Deployments:

# Install a single deployment
k8s-tests install single-local-sqlite

# Test specific deployment (verbose)
k8s-tests test single-local-sqlite -v

# Test specific deployment (non-verbose)
k8s-tests test single-s3-postgres

# Cleanup specific deployment
k8s-tests cleanup single-local-sqlite

Manual Installation:

# Install individual deployment manually using generated values
helm upgrade --install ncps-single-local-postgres charts/ncps \
  -f charts/ncps/test-values/single-local-postgres.yaml \
  --create-namespace \
  --namespace ncps-single-local-postgres

Cluster Management:

# Show cluster connection information
k8s-tests cluster info

# Destroy cluster
k8s-tests cluster destroy

# Help
k8s-tests --help

Adding New Test Scenarios:

To add a custom test permutation, edit nix/k8s-tests/config.nix:

{
  permutations = [
    # ... existing permutations
    {
      name = "my-new-scenario";
      description = "My custom test scenario";
      replicas = 1;
      storage = { type = "s3"; };
      database = { type = "postgresql"; };
      redis.enabled = false;
      features = [];  # Optional: ["cdc", "ha", "pod-disruption-budget"]
    }
  ];
}

Then regenerate: k8s-tests generate --push

For more details: See nix/k8s-tests/README.md

Pull Request Process#

Before Submitting#

1. Format your code:

   nix fmt

2. Fix linting issues:

   golangci-lint run --fix

3. Run tests:

   go test -race ./...

4. Build successfully:

   nix build

Commit Guidelines#

• Use clear, descriptive commit messages
• Follow Conventional Commits when possible:
  • feat: - New features
  • fix: - Bug fixes
  • docs: - Documentation changes
  • refactor: - Code refactoring
  • test: - Test additions/changes
  • chore: - Build/tooling changes

Pull Request Guidelines#

1. Create a feature branch:

   git checkout -b feature/your-feature-name

2. Make your changes following code quality standards

3. Update documentation if needed (README.md, CLAUDE.md, etc.)

4. Add tests for new functionality

5. Submit PR with:

   • Clear description of changes
   • Reference to related issues
   • Screenshots/examples if applicable

CI/CD Notes#

The project uses GitHub Actions for CI/CD:

• Workflows only run on PRs targeting main branch
• This supports Graphite-style stacked PRs efficiently
• When modifying workflows, maintain the branches: [main] restriction

Project Structure#

ncps/
├── cmd/                        # CLI commands
│   ├── serve.go                # Main serve command
│   ├── ent-lint/               # AST linter for Ent codegen invariants
│   ├── generate-migrations/    # Atlas-driven per-dialect generator
│   └── atlas-sum-check/        # CI helper verifying atlas.sum integrity
├── pkg/
│   ├── cache/                  # Core caching logic
│   ├── storage/                # Storage abstraction
│   │   ├── local/              # Local filesystem storage
│   │   └── s3/                 # S3-compatible storage
│   ├── database/               # Thin facade over the generated Ent client
│   │   └── migrate/            # State detection + adoption + apply
│   ├── lock/                   # Lock abstraction
│   │   ├── local/              # Local locks (single-instance)
│   │   └── redis/              # Redis distributed locks (HA)
│   ├── server/                 # HTTP server (Chi router)
│   └── nar/                    # NAR format handling
├── ent/                        # Ent schema definitions
│   ├── schema/                 # Hand-authored entity schemas (*.go)
│   └── ...                     # Generated client (committed)
├── migrations/                 # Per-dialect Atlas migrations
│   ├── sqlite/                 # SQLite migrations + atlas.sum
│   ├── postgres/               # PostgreSQL migrations + atlas.sum
│   └── mysql/                  # MySQL migrations + atlas.sum
├── nix/                        # Nix configuration
│   ├── packages/               # Package definitions
│   ├── devshells/              # Development shells
│   ├── formatter/              # Formatter configuration
│   └── process-compose/        # Development services
└── dev-scripts/                # Development helper scripts
    └── run.py                  # Development server orchestrator

Key Interfaces#

Storage (**pkg/storage/store.go**):

• ConfigStore - Secret key storage
• NarInfoStore - NarInfo metadata storage
• NarStore - NAR file storage

Both local and S3 backends implement these interfaces.

Locks (**pkg/lock/lock.go**):

• Locker - Exclusive locking for download deduplication
• RWLocker - Read-write locking for LRU coordination

Both local (sync.Mutex) and Redis (Redlock) backends implement these interfaces. Redis locks enable high-availability deployments with multiple instances.

Database:

• Supports SQLite, PostgreSQL, and MySQL via Ent (the type-safe ORM)
• Database selection via URL scheme in --cache-database-url
• Schema lives in ent/schema/*.go; generated client lives under ent/
• Per-dialect migrations under migrations/<dialect>/ are produced by Atlas (as a Go library) and applied at runtime by Goose

Common Development Tasks#

Adding a New Database Migration#

1. Edit ent/schema/<entity>.go — add the field/edge/index/annotation. The five codegen invariants are enforced by cmd/ent-lint; see cmd/ent-lint/main.go for the full list (table-level entsql.Check, entsql.OnDelete on edge.To, etc.).

2. Regenerate the Ent client: go generate ./ent/... (or task ent:generate).

3. Generate per-dialect Atlas migrations + update each atlas.sum:

   task migrations:gen NAME=descriptive_snake_case

4. Review the generated .sql files under migrations/sqlite/, migrations/postgres/, migrations/mysql/. Each file is a single timestamp-prefixed file with +goose Up / +goose Down markers. SQLite files that need PRAGMA foreign_keys = OFF must also carry -- +goose NO TRANSACTION.

5. Apply locally to verify:

   ncps migrate up --cache-database-url=sqlite:./test.db --dry-run
   ncps migrate up --cache-database-url=sqlite:./test.db

The expand-contract policy in CLAUDE.md documents the four-step recipe for non-additive column changes (e.g. adding NOT NULL). Down migrations are intentionally not supported.

Regenerating the Ent client#

After editing any ent/schema/*.go:

go generate ./ent/...      # or: task ent:generate
go run ./cmd/ent-lint --root .   # or: task ent:lint

task ent:check runs both and verifies the generated tree is up to date.

Adding a New Storage Backend#

1. Implement the storage interfaces in pkg/storage/
2. Add configuration flags in cmd/serve.go
3. Update documentation in README.md and CLAUDE.md
4. Add integration tests
5. Update Docker and Kubernetes examples if applicable

Debugging#

Use delve for debugging:

# Debug the application
dlv debug . -- serve --cache-hostname=localhost --cache-storage-local=/tmp/ncps

# Debug a specific test
dlv test ./pkg/server -- -test.run TestName

Note: The dev shell disables fortify hardening to allow delve to work.

Building Docker Images#

# Build Docker image
nix build .#docker

# Load into Docker
docker load < result

# Push to registry (requires DOCKER_IMAGE_TAGS environment variable)
DOCKER_IMAGE_TAGS="kalbasit/ncps:latest kalbasit/ncps:v1.0.0" nix run .#push-docker-image

Getting Help#

• Documentation Issues: Check CLAUDE.md for detailed development guidance
• Bug Reports: Open an issue
• Questions: Start a discussion
• Security Issues: Contact maintainers privately

Code of Conduct#

• Be respectful and inclusive
• Provide constructive feedback
• Focus on what's best for the project
• Show empathy towards other contributors

License#

By contributing to ncps, you agree that your contributions will be licensed under the MIT License.

---

Thank you for contributing to ncps! 🎉