fixes to persist packages in pypi-server

Reviewed-on: #3
Co-authored-by: Vishesh 'ironeagle' Bangotra <aetoskia@gmail.com>
Co-committed-by: Vishesh 'ironeagle' Bangotra <aetoskia@gmail.com>

README.md

@@ -6,6 +6,7 @@ This repository defines a private, self-hosted DevOps stack running on Raspberry
 * **Drone** for CI/CD automation
 * **Docker Registry** with **Registry UI**
 * **Drone Runner** for build execution
+* **PyPI Server** for private Python package hosting
 
 ---
 
@@ -18,6 +19,7 @@ This repository defines a private, self-hosted DevOps stack running on Raspberry
 | **drone-runner** | Executes Drone pipelines on local Docker host        | —         | drone-server |
 | **registry**     | Private Docker registry                              | 6005      | —            |
 | **registry-ui**  | Web UI for browsing and managing the registry images | 6001      | registry     |
+| **pypi-server**  | Private Python package index and repository          | 6006      | —            |
 
 ---
 
@@ -112,7 +114,7 @@ Self-hosted Git service used as Drone’s SCM backend and code hosting platform.
 
 #### Extra Config Required
 
-1. Access Gitea at [https://gitea.aetoskia.com:6002](https://gitea.aetoskia.com:6002).
+1. Access Gitea at [https://git.aetoskia.com](https://git.aetoskia.com).
 2. Log in as the **`aetos`** admin user (repository owner).
 3. Verify Drone OAuth App is registered with:
 
@@ -143,8 +145,8 @@ Core CI/CD engine that connects to Gitea and manages build pipelines.
 
 #### Environment
 
-* **DRONE_GITEA_CLIENT_ID / SECRET:** OAuth credentials from Gitea.
+* **DRONE_GITEA_CLIENT_ID / SECRET:** OAuth credentials from git.
-* **DRONE_GITEA_SERVER:** Gitea URL (`https://gitea.aetoskia.com`).
+* **DRONE_GITEA_SERVER:** Gitea URL (`https://git.aetoskia.com`).
 * **DRONE_RPC_SECRET:** Shared secret for secure runner communication.
 * **DRONE_SERVER_HOST / PROTO:** External access configuration.
 
@@ -210,14 +212,66 @@ Executes pipeline steps on the Docker host using the Drone RPC secret.
 
 ---
 
+### 🐍 **PyPI Server**
+
+Private Python package index for hosting internal packages and dependencies.
+
+#### Ports
+
+* **6006 → 8080** — Accessible via browser or pip client.
+
+#### Mounts
+
+* `/mnt/omnissiah-vault/data/pypiserver:/data` — Package storage directory.
+* `/mnt/omnissiah-vault/data/pypiserver/auth:/auth (ro)` — Authentication credentials.
+
+#### Environment
+
+* **PYPISERVER_AUTH=update,download,list** — Enables controlled access.
+
+#### Command
+
+```yaml
+command:
+  - -P
+  - /auth/htpasswd
+  - -a
+  - update,download,list
+  - /data/packages
+```
+
+#### Usage
+
+To upload packages:
+
+```bash
+pip install twine
+TWINE_USERNAME=<username> TWINE_PASSWORD=<password> twine upload --repository-url https://pip.aetoskia.com dist/*
+```
+
+To install packages:
+
+```bash
+pip install --index-url https://<username>:<password>@pip.aetoskia.com/simple <package-name>
+```
+
+#### Notes
+
+* Make sure `/data/packages` exists inside the container.
+* The `htpasswd` file controls user access levels.
+* Test the endpoint in browser: [https://pip.aetoskia.com](https://pip.aetoskia.com)
+
+---
+
 ## 🧬 Network Integration
 
 Ensure `/etc/hosts` on all relevant devices (Pi nodes, local dev systems) includes:
 
 ```
-192.168.1.111 gitea.aetoskia.com
+192.168.1.111 git.aetoskia.com
 192.168.1.111 drone.aetoskia.com
 192.168.1.111 dcr.aetoskia.com
+192.168.1.111 pip.aetoskia.com
 ```
 
 ---
@@ -230,18 +284,19 @@ Ensure `/etc/hosts` on all relevant devices (Pi nodes, local dev systems) includ
   docker compose --profile private-pi up -d
   ```
 * Drone and Gitea must both trust the same SSL certificate chain under `/home/aetos/certs`.
-* The `registry-ui` and `drone-runner` depend on hostname resolution between internal services, so avoid using raw IPs in environment variables.
+* The `registry-ui`, `drone-runner`, and `pypi-server` depend on hostname resolution between internal services.
 
 ---
 
 ## ✅ Quick Test Checklist
 
-| Component        | URL                                                                                    | Expected Result                       |
+| Component        | URL                                                                          | Expected Result                       |
-| ---------------- | -------------------------------------------------------------------------------------- | ------------------------------------- |
+| ---------------- |------------------------------------------------------------------------------| ------------------------------------- |
-| **Gitea**        | [https://gitea.aetoskia.com:6002](https://gitea.aetoskia.com:6002)                     | Gitea login page                      |
+| **Gitea**        | [https://git.aetoskia.com](https://git.aetoskia.com)                         | Gitea login page                      |
-| **Drone**        | [https://drone.aetoskia.com:6003](https://drone.aetoskia.com:6003)                     | Drone CI UI linked to Gitea           |
+| **Drone**        | [https://drone.aetoskia.com](https://drone.aetoskia.com)                     | Drone CI UI linked to Gitea           |
-| **Registry UI**  | [http://localhost:6001](http://localhost:6001)                                         | Authenticated Docker registry browser |
+| **Registry UI**  | [https://registry.aetoskia.com](https://registry.aetoskia.com)               | Authenticated Docker registry browser |
-| **Registry API** | [https://dcr.aetoskia.com:6005/v2/_catalog](https://dcr.aetoskia.com:6005/v2/_catalog) | JSON listing of repositories          |
+| **Registry API** | [https://dcr.aetoskia.com/v2/_catalog](https://dcr.aetoskia.com/v2/_catalog) | JSON listing of repositories          |
+| **PyPI Server**  | [https://pip.aetoskia.com/simple](https://pip.aetoskia.com/simple)           | Simple index page                     |
 
 ---
 

docker-compose.yaml

@@ -1,11 +1,49 @@
+# Service-specific volume configurations
+x-volume-configs:
+  gitea: &gitea_volumes
+    - type: bind
+      source: /home/aetos/gitea
+      target: /data
+      bind:
+        create_host_path: true
+
+  drone: &drone_volumes
+    - /mnt/omnissiah-vault/configs/certs:/usr/local/share/ca-certificates:ro
+    - type: bind
+      source: /mnt/omnissiah-vault/configs/drone
+      target: /data
+      bind:
+        create_host_path: true
+
+  registry: &registry_volumes
+    - ./registry/config.yml:/etc/docker/registry/config.yml:ro
+
+    - /mnt/omnissiah-vault/configs/auth:/auth:ro
+    - type: bind
+      source: /mnt/omnissiah-vault/data/registry
+      target: /var/lib/registry
+      bind:
+        create_host_path: true
+
+  registry_ui: &registry_ui_volumes
+    - /mnt/omnissiah-vault/configs/auth:/auth:ro
+
+  pypi: &pypi_volumes
+    - /mnt/omnissiah-vault/configs/auth:/auth:ro
+
+    - type: bind
+      source: /mnt/omnissiah-vault/data/pypi/packages
+      target: /data/packages
+      bind:
+        create_host_path: true
+
 services:
   registry-ui:
     image: joxit/docker-registry-ui:latest
     container_name: registry-ui
     ports:
       - "6001:80"
-    volumes:
+    volumes: *registry_ui_volumes
-      - /home/aetos/registry/auth:/auth:ro
     environment:
       REGISTRY_TITLE: "Armory of the Codex"
       REGISTRY_URL: https://dcr.aetoskia.com
@@ -19,7 +57,7 @@ services:
     extra_hosts:
       - "dcr.aetoskia.com:192.168.1.111"
     profiles:
-      - private-pi
+      - server-pi
 
   gitea:
     image: gitea/gitea:latest
@@ -31,8 +69,7 @@ services:
     ports:
       - "6002:3000"  # Web UI
       - "222:22"     # SSH
-    volumes:
+    volumes: *gitea_volumes
-      - /home/aetos/gitea:/data
     extra_hosts:
       - "drone.aetoskia.com:192.168.1.111"
     profiles:
@@ -43,42 +80,40 @@ services:
     container_name: drone-server
     ports:
       - "6003:80"   # Drone UI/API available on https://localhost:6003
-    volumes:
+    volumes: *drone_volumes
-      - /home/aetos/drone:/data
-      - /home/aetos/certs:/usr/local/share/ca-certificates:ro
     environment:
       DRONE_GITEA_CLIENT_ID: 8445594d-7c03-41a1-b66d-4296f34fa644
       DRONE_GITEA_CLIENT_SECRET: gto_f46d6utmlttauajdjxpfgikskb3j5epoohzhpp2r6j2jbed46iaq
-      DRONE_GITEA_SERVER: https://gitea.aetoskia.com
+      DRONE_GITEA_SERVER: https://git.aetoskia.com
       DRONE_RPC_SECRET: supersecretkey
       DRONE_SERVER_HOST: drone.aetoskia.com
       DRONE_SERVER_PROTO: https
     restart: always
-    depends_on:
-      - gitea
     extra_hosts:
-      - "gitea.aetoskia.com:192.168.1.111"
+      - "git.aetoskia.com:192.168.1.111"
     command: sh -c "update-ca-certificates && drone-server"
     profiles:
-      - private-pi
+      - server-pi
 
   drone-runner:
     image: drone/drone-runner-docker:latest
     container_name: drone-runner
     restart: always
+    network_mode: host
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
     environment:
       DRONE_RPC_PROTO: https
       DRONE_RPC_HOST: drone.aetoskia.com
       DRONE_RPC_SECRET: supersecretkey
-      DRONE_RUNNER_CAPACITY: 2
+      DRONE_RUNNER_CAPACITY: 1
       DRONE_RUNNER_NAME: runner1
       DRONE_RUNNER_OS: linux
       DRONE_RUNNER_ARCH: arm64
       DRONE_LOGS_DEBUG: true
       DRONE_LOGS_TRACE: true
     extra_hosts:
+      - "server-pi:192.168.1.35"
       - "private-pi:192.168.1.111"
     profiles:
       - server-pi
@@ -88,14 +123,30 @@ services:
     container_name: registry
     ports:
       - "6005:5000"
-    volumes:
+    volumes: *registry_volumes
-      - /home/aetos/registry:/var/lib/registry
-      - /home/aetos/registry/auth:/auth:ro
-      - ./registry/config.yml:/etc/docker/registry/config.yml:ro
     environment:
       REGISTRY_AUTH: htpasswd
       REGISTRY_AUTH_HTPASSWD_REALM: "Registry Realm"
       REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
     restart: always
     profiles:
-      - private-pi
+      - server-pi
+
+  pypi-server:
+    image: pypiserver/pypiserver:latest
+    container_name: pypi-server
+    restart: unless-stopped
+    ports:
+      - "6006:8080"
+    volumes: *pypi_volumes
+    environment:
+      - PYPISERVER_AUTH=update,download,list
+      - PYPISERVER_DATA_DIR=/data
+    command:
+      - -P
+      - /auth/htpasswd
+      - -a
+      - update,download,list
+      - /data/packages
+    profiles:
+      - server-pi