Contributing

Contributing to ncps#

Thank you for your interest in contributing to ncps! This document provides guidelines and instructions for contributing to the project.

Getting Started#

Prerequisites#

The project uses Nix flakes with direnv for reproducible development environments. You'll need:

Nix with flakes enabled - Installation guide
direnv - Installation guide

Initial Setup#

Clone the repository:

git clone https://github.com/kalbasit/ncps.git
cd ncps

Allow direnv:
```
direnv allow
```
This will automatically load the development environment with all required tools:
- Go
- golangci-lint
- sqlc
- dbmate
- delve (debugger)
- watchexec
- sqlfluff
- MinIO (for S3 testing)
- PostgreSQL (for database testing)
- MySQL/MariaDB (for database testing)
- Redis (for distributed locking testing)

Development Environment#

Available Tools#

Once in the development shell, you have access to:

Tool	Purpose
`go`	Go compiler and toolchain
`golangci-lint`	Code linting with 30+ linters
`sqlc`	Type-safe SQL code generation
`dbmate`	Database migration tool
`delve`	Go debugger
`watchexec`	File watcher for hot-reloading
`sqlfluff`	SQL linting and formatting
`minio`	S3-compatible object storage
`postgresql`	PostgreSQL database server
`mariadb`	MySQL/MariaDB database server
`redis`	Redis server for distributed locks

Development Dependencies#

The project uses process-compose-flake for managing development services. Start dependencies with:

nix run .#deps

This starts:

MinIO - S3-compatible storage server (port 9000, console on 9001)
- Test bucket: test-bucket
- Credentials: test-access-key / test-secret-key
- Self-validation ensures proper setup
PostgreSQL - Database server (port 5432)
- Test database: test-db
- Credentials: test-user / test-password
- Connection URL: postgresql://test-user:[email protected]:5432/test-db?sslmode=disable
MariaDB - MySQL-compatible database server (port 3306)
- Test database: test-db
- Credentials: test-user / test-password
- Connection URL: mysql://test-user:[email protected]:3306/test-db
Redis - Distributed locking server (port 6379)
- No authentication required (test environment)
- Used for distributed lock testing

Development Workflow#

Running the Development Server#

The development server supports hot-reloading and multiple storage backends:

# Using local filesystem storage (default, no dependencies required)
./dev-scripts/run.sh
# or explicitly
./dev-scripts/run.sh local

# Using S3/MinIO storage (requires dependencies to be running)
# In a separate terminal:
nix run .#deps

# Then run the dev server:
./dev-scripts/run.sh s3

The server automatically restarts when you modify code files.

Editing Documentation#

The documentation for this project is managed using Trilium. Follow these steps to contribute to the documentation:

Run the documentation editor:
```
trilium-edit-docs
```
This tool is available in the PATH provided by the Nix flake, see Development Environment in the Developer Guide for more information.
Edit the documentation through Trilium: The Trilium interface will open, allowing you to edit the notes. Trilium automatically exports the markdown files back to the repository as you make changes.
Wait and close: Wait 5 minutes after you have finished all your edits to ensure all changes are synced, then close Trilium.
Format the documentation:
Run the project formatter to ensure the markdown files follow the project's standards:
```
nix fmt
```
Submit your changes: Submit a Pull Request with your changes.

Database Migrations#

Creating a new migration:

dbmate --migrations-dir db/migrations/sqlite new migration_name
dbmate --migrations-dir db/migrations/postgres new migration_name
dbmate --migrations-dir db/migrations/mysql new migration_name

The dbmate command is a wrapper that automatically:

Detects database type from the URL scheme
Selects the appropriate migrations directory (db/migrations/sqlite/, db/migrations/postgres/, or db/migrations/mysql/)
Creates timestamped migration files

Applying migrations:

dbmate --url "sqlite:path/to/db.sqlite" up
dbmate --url "postgresql://..." up
dbmate --url "mysql://..." up

Note: The wrapper uses the NCPS_DB_MIGRATIONS_DIR and the NCPS_DB_SCHEMA_DIR environment variables (automatically set in the dev shell) to locate migrations and schema.

Generating SQL Code#

After modifying SQL queries or migrations:

sqlc generate

This generates type-safe Go code from:

db/query.sqlite.sql → pkg/database/sqlitedb/
db/query.postgres.sql → pkg/database/postgresdb/
db/query.mysql.sql → pkg/database/mysqldb/

Code Quality Standards#

Formatting#

IMPORTANT: Always run formatters first before making manual changes:

# Format all files (Go, Nix, SQL, YAML, Markdown)
nix fmt

The project uses:

gofumpt - Stricter Go formatting
goimports - Import organization
gci - Import grouping (standard → default → alias → localmodule)
nixfmt - Nix code formatting
sqlfluff - SQL formatting and linting
yamlfmt - YAML formatting
mdformat - Markdown formatting

Linting#

IMPORTANT: Always run golangci-lint run --fix first to automatically fix issues:

# Auto-fix all fixable linting issues
golangci-lint run --fix

# Lint without auto-fix
golangci-lint run

# Lint specific package
golangci-lint run ./pkg/server/...

The project uses 30+ linters including:

err113 - Explicit error wrapping
exhaustive - Exhaustive switch statements
gosec - Security checks
paralleltest - Parallel test detection
testpackage - Test package naming

See .golangci.yml for complete linter configuration.

SQL Linting#

# Lint SQL files
sqlfluff lint db/migrations/sqlite/*.sql
sqlfluff lint db/migrations/postgres/*.sql
sqlfluff lint db/migrations/mysql/*.sql

# Format SQL files
sqlfluff format db/migrations/sqlite/*.sql

Note: sqlc query files (db/query.*.sql) are excluded from sqlfluff as they use sqlc-specific syntax.

Testing#

Running Tests#

# Run all tests with race detector (recommended)
go test -race ./...

# Run tests for specific package
go test -race ./pkg/server/...

# Run a single test
go test -race -run TestName ./pkg/server/...

Integration Tests#

The project includes integration tests for S3, PostgreSQL, MySQL, and Redis. Integration tests are disabled by default and must be explicitly enabled using shell helper functions.

Quick Start:

# In terminal 1: Start development dependencies
nix run .#deps

# In terminal 2: Enable integration tests and run tests
eval "$(enable-integration-tests)"
go test -race ./...

Available Helper Commands:

The development shell provides commands to easily enable/disable integration tests:

Command	Description
`eval "$(enable-s3-tests)"`	Enable S3/MinIO integration tests
`eval "$(enable-postgres-tests)"`	Enable PostgreSQL integration tests
`eval "$(enable-redis-tests)"`	Enable Redis integration tests
`eval "$(enable-mysql-tests)"`	Enable MySQL integration tests
`eval "$(enable-integration-tests)"`	Enable all integration tests at once
`eval "$(disable-integration-tests)"`	Disable all integration tests

Running Specific Integration Tests:

# Start dependencies (in a separate terminal)
nix run .#deps

# Enable and run S3 tests only
eval "$(enable-s3-tests)"
go test -race ./pkg/storage/s3

# Enable and run database tests only
eval "$(enable-postgres-tests)"
eval "$(enable-mysql-tests)"
go test -race ./pkg/database

# Enable all tests and run everything
eval "$(enable-integration-tests)"
go test -race ./...

# Disable integration tests when done
eval "$(disable-integration-tests)"

What the Helper Commands Do:

The helper commands output shell export statements that you evaluate in your current shell:

**enable-s3-tests** exports:
- NCPS_TEST_S3_BUCKET=test-bucket
- NCPS_TEST_S3_ENDPOINT=http://127.0.0.1:9000
- NCPS_TEST_S3_REGION=us-east-1
- NCPS_TEST_S3_ACCESS_KEY_ID=test-access-key
- NCPS_TEST_S3_SECRET_ACCESS_KEY=test-secret-key
**enable-postgres-tests** exports:
- NCPS_TEST_POSTGRES_URL=postgresql://test-user:[email protected]:5432/test-db?sslmode=disable
**enable-mysql-tests** exports:
- NCPS_TEST_MYSQL_URL=mysql://test-user:[email protected]:3306/test-db
**enable-redis-tests** exports:
- NCPS_ENABLE_REDIS_TESTS=1

Tests automatically skip if these environment variables aren't set, so you can run go test -race ./... without enabling integration tests and only unit tests will run.

Test Requirements#

Use testify for assertions
Enable race detector (-race flag)
Use _test package suffix (enforced by testpackage linter)
Write parallel tests where possible (checked by paralleltest linter)
Each test should be isolated and not depend on other tests

Nix Build Tests#

# Run all checks including integration tests
nix flake check

# Build package (includes test phase)
nix build

The Nix build automatically:

Starts MinIO, PostgreSQL, MariaDB, and Redis in preCheck phase
Creates test databases and buckets
Exports test environment variables
Runs all tests (including integration tests)
Stops services in postCheck phase

Helm Chart Testing#

The project includes comprehensive Helm chart testing using a local Kind cluster with MinIO, PostgreSQL, MariaDB, and Redis.

Prerequisites:

Docker
kubectl
helm
kind

Setup (one-time):

# Create Kind cluster with all dependencies
./dev-scripts/k8s-cluster.sh create

The cluster can be reused across test runs. Use ./dev-scripts/k8s-cluster.sh destroy to clean up when done.

Testing Workflow:

# 1. Build, push Docker image to local Kind registry, and generate test values
./dev-scripts/generate-test-values.sh --push

# 2. Quick install all test deployments (can be run from anywhere)
./charts/ncps/test-values/QUICK-INSTALL.sh

# 3. Run tests (can be run from anywhere)
./charts/ncps/test-values/TEST.sh

# 4. Cleanup when done (can be run from anywhere)
./charts/ncps/test-values/CLEANUP.sh

Helper Scripts Locations#

Alternative: Use External Registry

If you prefer to push to an external registry (e.g., Docker Hub, your own Zot instance):

# 1. Build and push Docker image to external registry
DOCKER_IMAGE_TAGS="yourregistry.com/ncps:sha$(git rev-parse --short HEAD)" nix run .#push-docker-image

# 2. Generate test values files (use the image tag from step 1)
# The tag format is: sha<commit>-<platform> (e.g., sha4954654-x86_64-linux)
./dev-scripts/generate-test-values.sh sha$(git rev-parse --short HEAD)-x86_64-linux yourregistry.com ncps

# 3. Continue with steps 2-4 from the main workflow above

Test Deployments:

The generate-test-values.sh script creates 10 different test configurations:

Single Instance with Local Storage:
- SQLite, PostgreSQL, or MariaDB database
Single Instance with S3 Storage:
- SQLite, PostgreSQL, or MariaDB database
- Tests S3 configuration and MinIO compatibility
High Availability:
- 2 replicas with S3 storage
- PostgreSQL or MariaDB database
- Redis for distributed locking

Testing Individual Deployments:

# Test with verbose output
./test-values/TEST.sh -v

# Test specific deployment
./test-values/TEST.sh -d ncps-single-local-sqlite

# Test specific deployment with verbose output
./test-values/TEST.sh -d ncps-single-s3-postgres -v

Manual Installation:

# Install individual deployment manually
helm upgrade --install ncps-single-local-postgres . \
  -f test-values/single-local-postgres.yaml \
  --create-namespace \
  --namespace ncps-single-local-postgres

Cluster Management:

# Show cluster connection information
./dev-scripts/k8s-cluster.sh info

# Destroy cluster
./dev-scripts/k8s-cluster.sh destroy

# Help
./dev-scripts/k8s-cluster.sh help

Pull Request Process#

Before Submitting#

Format your code:
```
nix fmt
```
Fix linting issues:
```
golangci-lint run --fix
```
Run tests:
```
go test -race ./...
```
Build successfully:
```
nix build
```

Commit Guidelines#

Use clear, descriptive commit messages
Follow Conventional Commits when possible:
- feat: - New features
- fix: - Bug fixes
- docs: - Documentation changes
- refactor: - Code refactoring
- test: - Test additions/changes
- chore: - Build/tooling changes

Pull Request Guidelines#

Create a feature branch:

git checkout -b feature/your-feature-name

Make your changes following code quality standards
Update documentation if needed (README.md, CLAUDE.md, etc.)
Add tests for new functionality
Submit PR with:
- Clear description of changes
- Reference to related issues
- Screenshots/examples if applicable

CI/CD Notes#

The project uses GitHub Actions for CI/CD:

Workflows only run on PRs targeting main branch
This supports Graphite-style stacked PRs efficiently
When modifying workflows, maintain the branches: [main] restriction

Project Structure#

ncps/
├── cmd/                        # CLI commands
│   └── serve.go                # Main serve command
├── pkg/
│   ├── cache/                  # Core caching logic
│   ├── storage/                # Storage abstraction
│   │   ├── local/              # Local filesystem storage
│   │   └── s3/                 # S3-compatible storage
│   ├── database/               # Database abstraction
│   │   ├── sqlitedb/           # SQLite implementation
│   │   ├── postgresdb/         # PostgreSQL implementation
│   │   └── mysqldb/            # MySQL/MariaDB implementation
│   ├── lock/                   # Lock abstraction
│   │   ├── local/              # Local locks (single-instance)
│   │   └── redis/              # Redis distributed locks (HA)
│   ├── server/                 # HTTP server (Chi router)
│   └── nar/                    # NAR format handling
├── db/
│   ├── migrations/             # Database migrations
│   │   ├── sqlite/             # SQLite migrations
│   │   ├── postgres/           # PostgreSQL migrations
│   │   └── mysql/              # MySQL migrations
│   ├── query.sqlite.sql        # SQLite queries (sqlc)
│   ├── query.postgres.sql      # PostgreSQL queries (sqlc)
│   └── query.mysql.sql         # MySQL queries (sqlc)
├── nix/                        # Nix configuration
│   ├── packages/               # Package definitions
│   ├── devshells/              # Development shells
│   ├── formatter/              # Formatter configuration
│   ├── process-compose/        # Development services
│   └── dbmate-wrapper/         # Database migration wrapper
│   └── gen-db-wrappers/        # Database wrapper generator
└── dev-scripts/                # Development helper scripts
    └── run.sh                  # Development server script

Key Interfaces#

Storage (**pkg/storage/store.go**):

ConfigStore - Secret key storage
NarInfoStore - NarInfo metadata storage
NarStore - NAR file storage

Both local and S3 backends implement these interfaces.

Locks (**pkg/lock/lock.go**):

Locker - Exclusive locking for download deduplication
RWLocker - Read-write locking for LRU coordination

Both local (sync.Mutex) and Redis (Redlock) backends implement these interfaces. Redis locks enable high-availability deployments with multiple instances.

Database:

Supports SQLite, PostgreSQL, and MySQL via sqlc
Database selection via URL scheme in --cache-database-url
Type-safe queries generated from db/query.*.sql files

Common Development Tasks#

Adding a New Database Migration#

# Create migration for all databases
dbmate --migrations-dir db/migrations/sqlite new add_new_feature
dbmate --migrations-dir db/migrations/postgres new add_new_feature
dbmate --migrations-dir db/migrations/mysql new add_new_feature

# Edit the migration files in:
# - db/migrations/sqlite/TIMESTAMP_add_new_feature.sql
# - db/migrations/postgres/TIMESTAMP_add_new_feature.sql
# - db/migrations/mysql/TIMESTAMP_add_new_feature.sql

# IMPORTANT: Do NOT wrap migrations in BEGIN/COMMIT blocks
# dbmate automatically wraps each migration in a transaction
# Adding manual transactions will cause "cannot start a transaction within a transaction" errors
#
# Example migration:
# -- migrate:up
# CREATE TABLE example (...);
# CREATE INDEX idx_example ON example (column);
#
# -- migrate:down
# DROP INDEX idx_example;
# DROP TABLE example;

# Test the migration
dbmate --url "sqlite:./test.db" up
dbmate --url "postgresql://..." up
dbmate --url "mysql://..." up

Adding New SQL Queries#

# Edit the appropriate query file:
# - db/query.sqlite.sql (SQLite-specific queries)
# - db/query.postgres.sql (PostgreSQL-specific queries)
# - db/query.mysql.sql (MySQL-specific queries)

# Generate Go code
sqlc generate

# The generated code appears in:
# - pkg/database/sqlitedb/
# - pkg/database/postgresdb/
# - pkg/database/mysqldb/

Adding a New Storage Backend#

Implement the storage interfaces in pkg/storage/
Add configuration flags in cmd/serve.go
Update documentation in README.md and CLAUDE.md
Add integration tests
Update Docker and Kubernetes examples if applicable

Debugging#

Use delve for debugging:

# Debug the application
dlv debug . -- serve --cache-hostname=localhost --cache-storage-local=/tmp/ncps

# Debug a specific test
dlv test ./pkg/server -- -test.run TestName

Note: The dev shell disables fortify hardening to allow delve to work.

Building Docker Images#

# Build Docker image
nix build .#docker

# Load into Docker
docker load < result

# Push to registry (requires DOCKER_IMAGE_TAGS environment variable)
DOCKER_IMAGE_TAGS="kalbasit/ncps:latest kalbasit/ncps:v1.0.0" nix run .#push-docker-image

Getting Help#

Documentation Issues: Check CLAUDE.md for detailed development guidance
Bug Reports: Open an issue
Questions: Start a discussion
Security Issues: Contact maintainers privately

Code of Conduct#

Be respectful and inclusive
Provide constructive feedback
Focus on what's best for the project
Show empathy towards other contributors

License#

By contributing to ncps, you agree that your contributions will be licensed under the MIT License.

Thank you for contributing to ncps! 🎉