fixes to persist packages in pypi-server

Reviewed-on: #3
Co-authored-by: Vishesh 'ironeagle' Bangotra <aetoskia@gmail.com>
Co-committed-by: Vishesh 'ironeagle' Bangotra <aetoskia@gmail.com>

README.md

@@ -0,0 +1,303 @@
+# 🏗️ Private DevOps Stack — Aetoskia Infrastructure
+
+This repository defines a private, self-hosted DevOps stack running on Raspberry Pi nodes, consisting of:
+
+* **Gitea** for Git hosting
+* **Drone** for CI/CD automation
+* **Docker Registry** with **Registry UI**
+* **Drone Runner** for build execution
+* **PyPI Server** for private Python package hosting
+
+---
+
+## 🧬 Services Overview
+
+| Service          | Role                                                 | Port(s)   | Depends On   |
+| ---------------- | ---------------------------------------------------- | --------- | ------------ |
+| **gitea**        | Git server and web UI for repositories               | 6002, 222 | —            |
+| **drone-server** | CI/CD orchestration linked with Gitea                | 6003      | gitea        |
+| **drone-runner** | Executes Drone pipelines on local Docker host        | —         | drone-server |
+| **registry**     | Private Docker registry                              | 6005      | —            |
+| **registry-ui**  | Web UI for browsing and managing the registry images | 6001      | registry     |
+| **pypi-server**  | Private Python package index and repository          | 6006      | —            |
+
+---
+
+## ⚙️ Service Details
+
+---
+
+### 🐳 **Registry**
+
+Private Docker registry storing all container images built and deployed across your network.
+
+#### Ports
+
+* **6005 → 5000** — Exposes the registry API to clients.
+
+#### Mounts
+
+* `/home/aetos/registry:/var/lib/registry` — Persistent image storage.
+* `/home/aetos/registry/auth:/auth (ro)` — Authentication credentials.
+* `./registry/config.yml:/etc/docker/registry/config.yml (ro)` — Custom registry configuration.
+
+#### Environment
+
+* **REGISTRY_AUTH:** Enables authentication using `htpasswd`.
+* **REGISTRY_AUTH_HTPASSWD_REALM:** Realm name for authentication.
+* **REGISTRY_AUTH_HTPASSWD_PATH:** Path to the `htpasswd` file.
+
+#### Extra Config Required
+
+1. Generate authentication file:
+
+   ```bash
+   mkdir -p /home/aetos/registry/auth
+   docker run --rm httpd:2.4 htpasswd -Bbn <username> <password> > /home/aetos/registry/auth/htpasswd
+   ```
+2. Set correct permissions:
+
+   ```bash
+   chmod 644 /home/aetos/registry/auth/htpasswd
+   ```
+3. Verify config file (`registry/config.yml`) includes your chosen storage and auth setup.
+
+---
+
+### 🦯 **Registry UI**
+
+Web interface for visualizing and managing images stored in the internal registry.
+
+#### Ports
+
+* **6001 → 80** — Accessible on port 6001 via browser.
+
+#### Mounts
+
+* `/home/aetos/registry/auth:/auth (ro)` — Shared authentication file with the registry.
+
+#### Environment
+
+* **REGISTRY_TITLE:** UI header title.
+* **REGISTRY_URL:** Backend registry endpoint (`https://dcr.aetoskia.com`).
+* **REGISTRY_AUTH:** Enables `htpasswd` login.
+* **REGISTRY_AUTH_HTPASSWD_PATH:** Location of `htpasswd`.
+* **SINGLE_REGISTRY:** Limits UI to a single registry source.
+
+#### Extra Config Required
+
+* Ensure DNS or local `/etc/hosts` includes:
+
+  ```
+  192.168.1.111 dcr.aetoskia.com
+  ```
+* Login using credentials from `/auth/htpasswd`.
+
+---
+
+### 🦉 **Gitea**
+
+Self-hosted Git service used as Drone’s SCM backend and code hosting platform.
+
+#### Ports
+
+* **6002 → 3000** — Web UI.
+* **222 → 22** — Git SSH access.
+
+#### Mounts
+
+* `/home/aetos/gitea:/data` — Persistent Gitea data (repos, users, configs).
+
+#### Environment
+
+* **USER_UID / USER_GID:** Runs Gitea under the specified user/group IDs.
+
+#### Extra Config Required
+
+1. Access Gitea at [https://git.aetoskia.com](https://git.aetoskia.com).
+2. Log in as the **`aetos`** admin user (repository owner).
+3. Verify Drone OAuth App is registered with:
+
+   ```
+   Application Name: Drone CI
+   Redirect URI: https://drone.aetoskia.com/login
+   ```
+4. Confirm Gitea has these extra host mappings:
+
+   ```
+   192.168.1.111 drone.aetoskia.com
+   ```
+
+---
+
+### 🚀 **Drone Server**
+
+Core CI/CD engine that connects to Gitea and manages build pipelines.
+
+#### Ports
+
+* **6003 → 80** — Drone web UI and API endpoint.
+
+#### Mounts
+
+* `/home/aetos/drone:/data` — Persistent Drone data and SQLite database.
+* `/home/aetos/certs:/usr/local/share/ca-certificates (ro)` — Trusted CA certificates.
+
+#### Environment
+
+* **DRONE_GITEA_CLIENT_ID / SECRET:** OAuth credentials from git.
+* **DRONE_GITEA_SERVER:** Gitea URL (`https://git.aetoskia.com`).
+* **DRONE_RPC_SECRET:** Shared secret for secure runner communication.
+* **DRONE_SERVER_HOST / PROTO:** External access configuration.
+
+#### Extra Config Required
+
+1. Register the OAuth app in Gitea → Admin → Applications:
+
+   * Redirect URI: `https://drone.aetoskia.com/login`
+2. Stop Drone and make admin user (if needed):
+
+   ```bash
+   docker stop drone-server
+   sqlite3 /home/aetos/drone/database.sqlite "UPDATE users SET user_admin=1 WHERE user_login='aetos';"
+   docker start drone-server
+   ```
+3. Use admin token for CLI:
+
+   ```bash
+   export DRONE_SERVER=https://drone.aetoskia.com
+   export DRONE_TOKEN=<admin_token>
+   ```
+4. Mark repositories as trusted:
+
+   ```bash
+   drone repo update apps/homepage --trusted=true
+   ```
+
+---
+
+### 🧮 **Drone Runner**
+
+Executes pipeline steps on the Docker host using the Drone RPC secret.
+
+#### Ports
+
+* *(none)* — Internal runner; connects to Drone Server API via HTTPS.
+
+#### Mounts
+
+* `/var/run/docker.sock:/var/run/docker.sock` — Allows pipeline containers to spawn Docker builds.
+
+#### Environment
+
+* **DRONE_RPC_PROTO / HOST / SECRET:** Connects to Drone Server via shared key.
+* **DRONE_RUNNER_CAPACITY:** Number of parallel jobs supported.
+* **DRONE_RUNNER_NAME / OS / ARCH:** Runner identification.
+* **DRONE_LOGS_DEBUG / TRACE:** Enables detailed logging for troubleshooting.
+
+#### Extra Config Required
+
+1. Ensure the RPC secret matches the one defined in `drone-server`.
+2. Validate connectivity:
+
+   ```bash
+   docker logs drone-runner
+   ```
+
+   Look for:
+
+   ```
+   successfully pinged the remote drone server
+   ```
+
+---
+
+### 🐍 **PyPI Server**
+
+Private Python package index for hosting internal packages and dependencies.
+
+#### Ports
+
+* **6006 → 8080** — Accessible via browser or pip client.
+
+#### Mounts
+
+* `/mnt/omnissiah-vault/data/pypiserver:/data` — Package storage directory.
+* `/mnt/omnissiah-vault/data/pypiserver/auth:/auth (ro)` — Authentication credentials.
+
+#### Environment
+
+* **PYPISERVER_AUTH=update,download,list** — Enables controlled access.
+
+#### Command
+
+```yaml
+command:
+  - -P
+  - /auth/htpasswd
+  - -a
+  - update,download,list
+  - /data/packages
+```
+
+#### Usage
+
+To upload packages:
+
+```bash
+pip install twine
+TWINE_USERNAME=<username> TWINE_PASSWORD=<password> twine upload --repository-url https://pip.aetoskia.com dist/*
+```
+
+To install packages:
+
+```bash
+pip install --index-url https://<username>:<password>@pip.aetoskia.com/simple <package-name>
+```
+
+#### Notes
+
+* Make sure `/data/packages` exists inside the container.
+* The `htpasswd` file controls user access levels.
+* Test the endpoint in browser: [https://pip.aetoskia.com](https://pip.aetoskia.com)
+
+---
+
+## 🧬 Network Integration
+
+Ensure `/etc/hosts` on all relevant devices (Pi nodes, local dev systems) includes:
+
+```
+192.168.1.111 git.aetoskia.com
+192.168.1.111 drone.aetoskia.com
+192.168.1.111 dcr.aetoskia.com
+192.168.1.111 pip.aetoskia.com
+```
+
+---
+
+## 🧠 Tips
+
+* Always start containers with profile `private-pi` to avoid unnecessary public exposure:
+
+  ```bash
+  docker compose --profile private-pi up -d
+  ```
+* Drone and Gitea must both trust the same SSL certificate chain under `/home/aetos/certs`.
+* The `registry-ui`, `drone-runner`, and `pypi-server` depend on hostname resolution between internal services.
+
+---
+
+## ✅ Quick Test Checklist
+
+| Component        | URL                                                                          | Expected Result                       |
+| ---------------- |------------------------------------------------------------------------------| ------------------------------------- |
+| **Gitea**        | [https://git.aetoskia.com](https://git.aetoskia.com)                         | Gitea login page                      |
+| **Drone**        | [https://drone.aetoskia.com](https://drone.aetoskia.com)                     | Drone CI UI linked to Gitea           |
+| **Registry UI**  | [https://registry.aetoskia.com](https://registry.aetoskia.com)               | Authenticated Docker registry browser |
+| **Registry API** | [https://dcr.aetoskia.com/v2/_catalog](https://dcr.aetoskia.com/v2/_catalog) | JSON listing of repositories          |
+| **PyPI Server**  | [https://pip.aetoskia.com/simple](https://pip.aetoskia.com/simple)           | Simple index page                     |
+
+---
+
+© Aetoskia Internal Infrastructure — All rights reserved.

docker-compose.yaml

@@ -1,22 +1,63 @@
+# Service-specific volume configurations
+x-volume-configs:
+  gitea: &gitea_volumes
+    - type: bind
+      source: /home/aetos/gitea
+      target: /data
+      bind:
+        create_host_path: true
+
+  drone: &drone_volumes
+    - /mnt/omnissiah-vault/configs/certs:/usr/local/share/ca-certificates:ro
+    - type: bind
+      source: /mnt/omnissiah-vault/configs/drone
+      target: /data
+      bind:
+        create_host_path: true
+
+  registry: &registry_volumes
+    - ./registry/config.yml:/etc/docker/registry/config.yml:ro
+
+    - /mnt/omnissiah-vault/configs/auth:/auth:ro
+    - type: bind
+      source: /mnt/omnissiah-vault/data/registry
+      target: /var/lib/registry
+      bind:
+        create_host_path: true
+
+  registry_ui: &registry_ui_volumes
+    - /mnt/omnissiah-vault/configs/auth:/auth:ro
+
+  pypi: &pypi_volumes
+    - /mnt/omnissiah-vault/configs/auth:/auth:ro
+
+    - type: bind
+      source: /mnt/omnissiah-vault/data/pypi/packages
+      target: /data/packages
+      bind:
+        create_host_path: true
+
 services:
   registry-ui:
     image: joxit/docker-registry-ui:latest
     container_name: registry-ui
     ports:
       - "6001:80"
-    volumes:
-      - /home/aetos/registry/auth:/auth:ro
+    volumes: *registry_ui_volumes
     environment:
       REGISTRY_TITLE: "Armory of the Codex"
       REGISTRY_URL: https://dcr.aetoskia.com
 
       REGISTRY_AUTH: "htpasswd"
       REGISTRY_AUTH_HTPASSWD_PATH: "/auth/htpasswd"
+      SINGLE_REGISTRY: "true"
     depends_on:
       - registry
     restart: always
     extra_hosts:
       - "dcr.aetoskia.com:192.168.1.111"
+    profiles:
+      - server-pi
 
   gitea:
     image: gitea/gitea:latest
@@ -28,69 +69,84 @@ services:
     ports:
       - "6002:3000"  # Web UI
       - "222:22"     # SSH
-    volumes:
-      - /home/aetos/gitea:/data
+    volumes: *gitea_volumes
     extra_hosts:
       - "drone.aetoskia.com:192.168.1.111"
+    profiles:
+      - private-pi
 
   drone-server:
     image: drone/drone:2
     container_name: drone-server
     ports:
       - "6003:80"   # Drone UI/API available on https://localhost:6003
-    volumes:
-      - /home/aetos/drone:/data
-      - /home/aetos/certs:/usr/local/share/ca-certificates:ro
+    volumes: *drone_volumes
     environment:
       DRONE_GITEA_CLIENT_ID: 8445594d-7c03-41a1-b66d-4296f34fa644
       DRONE_GITEA_CLIENT_SECRET: gto_f46d6utmlttauajdjxpfgikskb3j5epoohzhpp2r6j2jbed46iaq
-      DRONE_GITEA_SERVER: https://gitea.aetoskia.com
+      DRONE_GITEA_SERVER: https://git.aetoskia.com
       DRONE_RPC_SECRET: supersecretkey
       DRONE_SERVER_HOST: drone.aetoskia.com
       DRONE_SERVER_PROTO: https
     restart: always
-    depends_on:
-      - gitea
     extra_hosts:
-      - "gitea.aetoskia.com:192.168.1.111"
+      - "git.aetoskia.com:192.168.1.111"
     command: sh -c "update-ca-certificates && drone-server"
+    profiles:
+      - server-pi
 
   drone-runner:
     image: drone/drone-runner-docker:latest
     container_name: drone-runner
     restart: always
-    depends_on:
-      - drone-server
-    ports:
-      - "6004:3000"   # optional: exposes runner metrics/debug UI
+    network_mode: host
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
-      - /home/aetos/certs:/usr/local/share/ca-certificates:ro
     environment:
       DRONE_RPC_PROTO: https
       DRONE_RPC_HOST: drone.aetoskia.com
       DRONE_RPC_SECRET: supersecretkey
-      DRONE_RUNNER_CAPACITY: 2
+      DRONE_RUNNER_CAPACITY: 1
       DRONE_RUNNER_NAME: runner1
       DRONE_RUNNER_OS: linux
       DRONE_RUNNER_ARCH: arm64
       DRONE_LOGS_DEBUG: true
       DRONE_LOGS_TRACE: true
     extra_hosts:
-      - "drone.aetoskia.com:192.168.1.111"
-      - "gitea.aetoskia.com:192.168.1.111"
+      - "server-pi:192.168.1.35"
+      - "private-pi:192.168.1.111"
+    profiles:
+      - server-pi
 
   registry:
     image: registry:2
     container_name: registry
     ports:
       - "6005:5000"
-    volumes:
-      - /home/aetos/registry:/var/lib/registry
-      - /home/aetos/registry/auth:/auth:ro
-      - ./registry/config.yml:/etc/docker/registry/config.yml:ro
+    volumes: *registry_volumes
     environment:
       REGISTRY_AUTH: htpasswd
       REGISTRY_AUTH_HTPASSWD_REALM: "Registry Realm"
       REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
     restart: always
+    profiles:
+      - server-pi
+
+  pypi-server:
+    image: pypiserver/pypiserver:latest
+    container_name: pypi-server
+    restart: unless-stopped
+    ports:
+      - "6006:8080"
+    volumes: *pypi_volumes
+    environment:
+      - PYPISERVER_AUTH=update,download,list
+      - PYPISERVER_DATA_DIR=/data
+    command:
+      - -P
+      - /auth/htpasswd
+      - -a
+      - update,download,list
+      - /data/packages
+    profiles:
+      - server-pi

registry/config.yml

@@ -1,16 +1,30 @@
 version: 0.1
 log:
   level: debug
+
 http:
   addr: :5000
   headers:
-    Access-Control-Allow-Origin: ["http://registry.aetoskia.com"]
-    Access-Control-Allow-Methods: ["GET", "POST", "PUT", "DELETE", "OPTIONS"]
-    Access-Control-Allow-Headers: ["Authorization", "Content-Type"]
-    Access-Control-Expose-Headers: ["Docker-Content-Digest"]
+    Access-Control-Allow-Origin:
+      - https://registry.aetoskia.com
+    Access-Control-Allow-Methods:
+      - GET
+      - POST
+      - PUT
+      - DELETE
+      - OPTIONS
+    Access-Control-Allow-Headers:
+      - Authorization
+      - Content-Type
+    Access-Control-Expose-Headers:
+      - Docker-Content-Digest
+    Access-Control-Allow-Credentials:
+      - true
+
 storage:
   filesystem:
     rootdirectory: /var/lib/registry
+
 auth:
   htpasswd:
     realm: basic-realm