fixes to persist packages in pypi-server

changed runner to 1
2025-12-17 02:32:38 +05:30 · 2025-12-16 17:50:27 +05:30 · 2025-12-16 17:46:55 +05:30 · 2025-11-01 18:21:07 +05:30 · 2025-11-01 12:44:44 +00:00 · 2025-10-21 18:46:26 +00:00
2 changed files with 135 additions and 30 deletions
--- a/README.md
+++ b/README.md
@@ -6,6 +6,7 @@ This repository defines a private, self-hosted DevOps stack running on Raspberry
 * **Drone** for CI/CD automation
 * **Docker Registry** with **Registry UI**
 * **Drone Runner** for build execution
+* **PyPI Server** for private Python package hosting

 ---

@@ -18,6 +19,7 @@ This repository defines a private, self-hosted DevOps stack running on Raspberry
 | **drone-runner** | Executes Drone pipelines on local Docker host        | —         | drone-server |
 | **registry**     | Private Docker registry                              | 6005      | —            |
 | **registry-ui**  | Web UI for browsing and managing the registry images | 6001      | registry     |
+| **pypi-server**  | Private Python package index and repository          | 6006      | —            |

 ---

@@ -112,7 +114,7 @@ Self-hosted Git service used as Drone’s SCM backend and code hosting platform.

 #### Extra Config Required

-1. Access Gitea at [https://gitea.aetoskia.com:6002](https://gitea.aetoskia.com:6002).
+1. Access Gitea at [https://git.aetoskia.com](https://git.aetoskia.com).
 2. Log in as the **`aetos`** admin user (repository owner).
 3. Verify Drone OAuth App is registered with:

@@ -143,8 +145,8 @@ Core CI/CD engine that connects to Gitea and manages build pipelines.

 #### Environment

-* **DRONE_GITEA_CLIENT_ID / SECRET:** OAuth credentials from Gitea.
-* **DRONE_GITEA_SERVER:** Gitea URL (`https://gitea.aetoskia.com`).
+* **DRONE_GITEA_CLIENT_ID / SECRET:** OAuth credentials from git.
+* **DRONE_GITEA_SERVER:** Gitea URL (`https://git.aetoskia.com`).
 * **DRONE_RPC_SECRET:** Shared secret for secure runner communication.
 * **DRONE_SERVER_HOST / PROTO:** External access configuration.

@@ -210,14 +212,66 @@ Executes pipeline steps on the Docker host using the Drone RPC secret.

 ---

+### 🐍 **PyPI Server**
+
+Private Python package index for hosting internal packages and dependencies.
+
+#### Ports
+
+* **6006 → 8080** — Accessible via browser or pip client.
+
+#### Mounts
+
+* `/mnt/omnissiah-vault/data/pypiserver:/data` — Package storage directory.
+* `/mnt/omnissiah-vault/data/pypiserver/auth:/auth (ro)` — Authentication credentials.
+
+#### Environment
+
+* **PYPISERVER_AUTH=update,download,list** — Enables controlled access.
+
+#### Command
+
+```yaml
+command:
+  - -P
+  - /auth/htpasswd
+  - -a
+  - update,download,list
+  - /data/packages
+```
+
+#### Usage
+
+To upload packages:
+
+```bash
+pip install twine
+TWINE_USERNAME=<username> TWINE_PASSWORD=<password> twine upload --repository-url https://pip.aetoskia.com dist/*
+```
+
+To install packages:
+
+```bash
+pip install --index-url https://<username>:<password>@pip.aetoskia.com/simple <package-name>
+```
+
+#### Notes
+
+* Make sure `/data/packages` exists inside the container.
+* The `htpasswd` file controls user access levels.
+* Test the endpoint in browser: [https://pip.aetoskia.com](https://pip.aetoskia.com)
+
+---
+
 ## 🧬 Network Integration

 Ensure `/etc/hosts` on all relevant devices (Pi nodes, local dev systems) includes:

 ```
-192.168.1.111 gitea.aetoskia.com
+192.168.1.111 git.aetoskia.com
 192.168.1.111 drone.aetoskia.com
 192.168.1.111 dcr.aetoskia.com
+192.168.1.111 pip.aetoskia.com
 ```

 ---
@@ -230,18 +284,19 @@ Ensure `/etc/hosts` on all relevant devices (Pi nodes, local dev systems) includ
  docker compose --profile private-pi up -d
  ```
 * Drone and Gitea must both trust the same SSL certificate chain under `/home/aetos/certs`.
-* The `registry-ui` and `drone-runner` depend on hostname resolution between internal services, so avoid using raw IPs in environment variables.
+* The `registry-ui`, `drone-runner`, and `pypi-server` depend on hostname resolution between internal services.

 ---

 ## ✅ Quick Test Checklist

 | Component        | URL                                                                          | Expected Result                       |
-| ---------------- | -------------------------------------------------------------------------------------- | ------------------------------------- |
-| **Gitea**        | [https://gitea.aetoskia.com:6002](https://gitea.aetoskia.com:6002)                     | Gitea login page                      |
-| **Drone**        | [https://drone.aetoskia.com:6003](https://drone.aetoskia.com:6003)                     | Drone CI UI linked to Gitea           |
-| **Registry UI**  | [http://localhost:6001](http://localhost:6001)                                         | Authenticated Docker registry browser |
-| **Registry API** | [https://dcr.aetoskia.com:6005/v2/_catalog](https://dcr.aetoskia.com:6005/v2/_catalog) | JSON listing of repositories          |
+| ---------------- |------------------------------------------------------------------------------| ------------------------------------- |
+| **Gitea**        | [https://git.aetoskia.com](https://git.aetoskia.com)                         | Gitea login page                      |
+| **Drone**        | [https://drone.aetoskia.com](https://drone.aetoskia.com)                     | Drone CI UI linked to Gitea           |
+| **Registry UI**  | [https://registry.aetoskia.com](https://registry.aetoskia.com)               | Authenticated Docker registry browser |
+| **Registry API** | [https://dcr.aetoskia.com/v2/_catalog](https://dcr.aetoskia.com/v2/_catalog) | JSON listing of repositories          |
+| **PyPI Server**  | [https://pip.aetoskia.com/simple](https://pip.aetoskia.com/simple)           | Simple index page                     |

 ---

--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -1,11 +1,49 @@
+# Service-specific volume configurations
+x-volume-configs:
+  gitea: &gitea_volumes
+    - type: bind
+      source: /home/aetos/gitea
+      target: /data
+      bind:
+        create_host_path: true
+
+  drone: &drone_volumes
+    - /mnt/omnissiah-vault/configs/certs:/usr/local/share/ca-certificates:ro
+    - type: bind
+      source: /mnt/omnissiah-vault/configs/drone
+      target: /data
+      bind:
+        create_host_path: true
+
+  registry: &registry_volumes
+    - ./registry/config.yml:/etc/docker/registry/config.yml:ro
+
+    - /mnt/omnissiah-vault/configs/auth:/auth:ro
+    - type: bind
+      source: /mnt/omnissiah-vault/data/registry
+      target: /var/lib/registry
+      bind:
+        create_host_path: true
+
+  registry_ui: &registry_ui_volumes
+    - /mnt/omnissiah-vault/configs/auth:/auth:ro
+
+  pypi: &pypi_volumes
+    - /mnt/omnissiah-vault/configs/auth:/auth:ro
+
+    - type: bind
+      source: /mnt/omnissiah-vault/data/pypi/packages
+      target: /data/packages
+      bind:
+        create_host_path: true
+
 services:
  registry-ui:
    image: joxit/docker-registry-ui:latest
    container_name: registry-ui
    ports:
      - "6001:80"
-    volumes:
-      - /home/aetos/registry/auth:/auth:ro
+    volumes: *registry_ui_volumes
    environment:
      REGISTRY_TITLE: "Armory of the Codex"
      REGISTRY_URL: https://dcr.aetoskia.com
@@ -19,7 +57,7 @@ services:
    extra_hosts:
      - "dcr.aetoskia.com:192.168.1.111"
    profiles:
-      - private-pi
+      - server-pi

  gitea:
    image: gitea/gitea:latest
@@ -31,8 +69,7 @@ services:
    ports:
      - "6002:3000"  # Web UI
      - "222:22"     # SSH
-    volumes:
-      - /home/aetos/gitea:/data
+    volumes: *gitea_volumes
    extra_hosts:
      - "drone.aetoskia.com:192.168.1.111"
    profiles:
@@ -43,24 +80,20 @@ services:
    container_name: drone-server
    ports:
      - "6003:80"   # Drone UI/API available on https://localhost:6003
-    volumes:
-      - /home/aetos/drone:/data
-      - /home/aetos/certs:/usr/local/share/ca-certificates:ro
+    volumes: *drone_volumes
    environment:
      DRONE_GITEA_CLIENT_ID: 8445594d-7c03-41a1-b66d-4296f34fa644
      DRONE_GITEA_CLIENT_SECRET: gto_f46d6utmlttauajdjxpfgikskb3j5epoohzhpp2r6j2jbed46iaq
-      DRONE_GITEA_SERVER: https://gitea.aetoskia.com
+      DRONE_GITEA_SERVER: https://git.aetoskia.com
      DRONE_RPC_SECRET: supersecretkey
      DRONE_SERVER_HOST: drone.aetoskia.com
      DRONE_SERVER_PROTO: https
    restart: always
-    depends_on:
-      - gitea
    extra_hosts:
-      - "gitea.aetoskia.com:192.168.1.111"
+      - "git.aetoskia.com:192.168.1.111"
    command: sh -c "update-ca-certificates && drone-server"
    profiles:
-      - private-pi
+      - server-pi

  drone-runner:
    image: drone/drone-runner-docker:latest
@@ -73,13 +106,14 @@ services:
      DRONE_RPC_PROTO: https
      DRONE_RPC_HOST: drone.aetoskia.com
      DRONE_RPC_SECRET: supersecretkey
-      DRONE_RUNNER_CAPACITY: 2
+      DRONE_RUNNER_CAPACITY: 1
      DRONE_RUNNER_NAME: runner1
      DRONE_RUNNER_OS: linux
      DRONE_RUNNER_ARCH: arm64
      DRONE_LOGS_DEBUG: true
      DRONE_LOGS_TRACE: true
    extra_hosts:
+      - "server-pi:192.168.1.35"
      - "private-pi:192.168.1.111"
    profiles:
      - server-pi
@@ -89,14 +123,30 @@ services:
    container_name: registry
    ports:
      - "6005:5000"
-    volumes:
-      - /home/aetos/registry:/var/lib/registry
-      - /home/aetos/registry/auth:/auth:ro
-      - ./registry/config.yml:/etc/docker/registry/config.yml:ro
+    volumes: *registry_volumes
    environment:
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: "Registry Realm"
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
    restart: always
    profiles:
-      - private-pi
+      - server-pi
+
+  pypi-server:
+    image: pypiserver/pypiserver:latest
+    container_name: pypi-server
+    restart: unless-stopped
+    ports:
+      - "6006:8080"
+    volumes: *pypi_volumes
+    environment:
+      - PYPISERVER_AUTH=update,download,list
+      - PYPISERVER_DATA_DIR=/data
+    command:
+      - -P
+      - /auth/htpasswd
+      - -a
+      - update,download,list
+      - /data/packages
+    profiles:
+      - server-pi
Author	SHA1	Message	Date
Vishesh 'ironeagle' Bangotra	3769f3c097	fixes to persist packages in pypi-server	2025-12-17 02:32:38 +05:30
Vishesh 'ironeagle' Bangotra	b1abed4f94	fixes to persist packages in pypi-server	2025-12-16 17:50:27 +05:30
Vishesh 'ironeagle' Bangotra	63523216e5	changed runner to 1	2025-12-16 17:46:55 +05:30
Vishesh 'ironeagle' Bangotra	3dcbd0f173	pypi server README.md changes	2025-11-01 18:21:07 +05:30
Vishesh 'ironeagle' Bangotra	4a777f7b75	devpi (#3 ) Reviewed-on: #3 Co-authored-by: Vishesh 'ironeagle' Bangotra <aetoskia@gmail.com> Co-committed-by: Vishesh 'ironeagle' Bangotra <aetoskia@gmail.com>	2025-11-01 12:44:44 +00:00
Vishesh 'ironeagle' Bangotra	2e69ade912	gitea to git (#2 ) Reviewed-on: #2 Co-authored-by: Vishesh 'ironeagle' Bangotra <aetoskia@gmail.com> Co-committed-by: Vishesh 'ironeagle' Bangotra <aetoskia@gmail.com>	2025-10-21 18:46:26 +00:00
Aetos Skia	5916e22a9f	removed dependency on gitea from drone-server	2025-10-21 17:46:10 +05:30
Aetos Skia	a8c5fe5447	cannot mount omnissia-vault as different permissions	2025-10-21 17:35:02 +05:30
Vishesh 'ironeagle' Bangotra	491f44c1f2	omnissiah-vault (#1 ) Reviewed-on: https://gitea.aetoskia.com/services/codebase/pulls/1 Co-authored-by: Vishesh 'ironeagle' Bangotra <aetoskia@gmail.com> Co-committed-by: Vishesh 'ironeagle' Bangotra <aetoskia@gmail.com>	2025-10-21 11:59:34 +00:00
Aetos Skia	af9528f6cf	Merge branch 'main' of gitea:services/codebase	2025-10-21 17:24:18 +05:30
Aetos Skia	3428d04080	only gitea on private-pi	2025-10-21 17:23:59 +05:30