rwx.work
Shell to Python Continuous Deployment
spcd.rwx.work
tilde.link
1# Shell to Python Continuous Deployment
2
3Get a common set of Python based CI / CD commands\
4from various contexts of CA, DNS, CI and OS / OCI.
5
6---
7
8## Table Of Contents {#toc}
9
10* 1 [Why](#why)
11* 2 [How](#how)
12 * 2.1 [Group environment variables](#how-variables)
13 * 2.2 [Workflow files](#how-workflows)
14 * 2.3 [Integration & Deployment](#how-actions)
15* 3 [What](#what)
16 * 3.1 [Features](#what-features)
17 * 3.2 [Environment variables](#what-variables)
18 * 3.3 [Operating Systems containers](#what-systems)
19* 4 [Who](#who)
20* 5 [Where](#where)
21 * 5.1 [Chat](#where-chat)
22 * 5.2 [Repo](#where-repo)
23 * 5.3 [Site](#where-site)
24* 6 [When](#when)
25 * 6.1 [Release tasks](#when-release)
26 * 6.2 [Further tasks](#when-further)
27
28---
29
30## 1 [Why](#toc) {#why}
31
32Project workflows can occur in:
33* an online or offline context, dealing with custom CA & DNS servers
34* various Continuous Integration platforms, with different workflow syntaxes
35* specific Operating Systems, having different commands and package names
36* specific containers for JavaScript based actions (ForgeJo, Gitea, GitHub)
37
38What if there was a unified way to do so?
39
40---
41
42## 2 [How](#toc) {#how}
43
44Picture it…
45
46### 2.1 [Group environment variables](#how) {#how-variables}
47
48* numbered
49 * Certificate Authority armored certificates
50 * Domain Name System servers IP addresses
51* ref names for deployment contexts of feature, staging & release
52* alternate locations of
53 * Operating Systems & Software packages
54 * Git repositories of
55 * this project
56 * its parent framework
57 * projects not available in some distributions
58 * shunit2
59* SSH key & hosts for deployments
60* a standalone POSIX shell script payload
61
62### 2.2 [Workflow files](#how) {#how-workflows}
63
64* calling the POSIX shell script payload as their very first step
65* using a unified YAML syntax made of actions implemented in Python
66
67### 2.3 [Integration & Deployment](#how) {#how-actions}
68
69#### Shell
70
71* [X] setting up the chosen Operating System container
72 * [X] Domain Name System resolving
73 * [X] package manager configuration & repositories
74 * [X] Certificate Authorities trusting
75 * [X] Operating System updating
76* [X] installing Git to clone
77 * [X] this project
78 * [X] its parent project
79* [X] installing Python
80 * [X] system environment
81 * [X] virtual environment
82 * [ ] managed version
83* [X] generating a Python module to switch context
84
85#### Python
86
87* [X] setting up an OpenSSH client
88 * [X] known hosts
89 * [ ] private key
90* [ ] and finally parsing commands from
91 * [X] system
92 * [ ] workflow
93* [ ] to run available actions
94 * [ ] check project
95 * [ ] build project
96 * [X] browse workspace
97 * [ ] synchronize
98
99---
100
101## 3 [What](#toc) {#what}
102
103### 3.1 [Features](#what) {#what-features}
104
105Handle project workflows in a unified way:
106
107* whether the network infrastructure is
108 * [X] online
109 * [X] offline with custom
110 * [X] Certificate Authorities
111 * [X] Domain Name System servers
112
113* whatever the Continuous Integration platform
114 * [X] ForgeJo
115 * [X] Gitea
116 * [X] GitHub
117 * [X] GitLab
118 * [ ] SourceHut
119
120* whatever the Operating System container
121| System | Latest | Previous |
122|:---------|:-----------------------|:-----------------------|
123| Alma | ☑ 9 | ☑ 8 |
124| Alpine | ☑ 3.20 | ☑ 3.19 |
125| Arch | ☑ 20240818 (.0.255804) | ☑ 20240101 (.0.204074) |
126| Debian | ☑ Trixie (13) | ☑ Bookworm (12) |
127| Fedora | ☑ 40 | ☑ 39 |
128| OpenSUSE | ☐ 15.6 | ☐ 15.5 |
129| Rocky | ☑ 9 | ☑ 8 |
130| Ubuntu | ☑ Noble (24.04) | ☐ Jammy (22.04) |
131
132### 3.2 [Environment variables](#what) {#what-variables}
133
134| Variable | Description | Default |
135|:------------------|:-------------------------|:--------------------------------|
136| SPCD_CA_n | Numbered CA certificates | |
137| SPCD_DNS_n | Numbered name servers | 9.9.9.9 |
138| SPCD_GIT_RWX | RWX Git repository | rwx |
139| SPCD_GIT_SHUNIT | ShUnit Git repository | shunit2 |
140| SPCD_GIT_SPCD | SPCD Git repository | spcd |
141| SPCD_REF_ARCHIVE | Archive deployment ref | old |
142| SPCD_REF_FEATURE | Feature deployment ref | f |
143| SPCD_REF_RELEASE | Release deployment ref | main |
144| SPCD_REF_STAGING | Staging deployment ref | dev |
145| SPCD_SSH_HOSTS | domain.tld ssh-type pub | |
146| SPCD_SSH_KEY | SSH private key | |
147| SPCD_TXT_LOCALE | Locale to use for text | en_US |
148| SPCD_URL_ALMA | Alma repository URL | https://repo.almalinux.org |
149| SPCD_URL_ALPINE | Alpine repository URL | https://dl-cdn.alpinelinux.org |
150| SPCD_URL_ARCH | Arch repository URL | https://geo.mirror.pkgbuild.com |
151| SPCD_URL_DEBIAN | Debian repository URL | https://deb.debian.org |
152| SPCD_URL_EPEL | EPEL repository URL | https://dl.fedoraproject.org |
153| SPCD_URL_FEDORA | Fedora repository URL | https://rpmfind.net |
154| SPCD_URL_OPENSUSE | OpenSUSE repository URL | https://download.opensuse.org |
155| SPCD_URL_PYTHON | Python repository URL | https://pypi.org |
156| SPCD_URL_ROCKY | Rocky repository URL | https://dl.rockylinux.org |
157| SPCD_URL_UBUNTU | Ubuntu repository URL | https://ubuntu.mirrors.ovh.net |
158
159### 3.3 [Operating Systems containers](#what) {#what-systems}
160
161#### Latest
162
163| os | https | up ca | python | ffmpeg | gource | graphviz | plantuml | shellcheck | shfmt | shunit |
164|:----------------|---|---|------------:|-------:|-----:|------:|-----------:|------:|-----:|------:|
165| Arch 20240818 | ☑ | ☑ | 3.12 | 7.0.2 | 0.54 | 12.0 | 1.2023.13 | 0.10 | 3.8 | 2.1.8 |
166| Alpine 3.20 | ☑ | ☐ | 3.12 | 6.1.1 | 0.54 | 9.0 | 1.2024.4 | 0.10 | 3.8 | 2.1.8 |
167| Fedora 40 | ☑ | ☑ | 3.12 → 3.13 | 6.1.2 | 0.55 | 9.0 | 1.2024.6 | 0.9 | 3.7 | 2.1.6 |
168| Debian Trixie | ☐ | ☐ | 3.13 | 7.1.3 | 0.54 | 2.42 | 1.2020.2 | 0.10 | 3.8 | 2.1.8 |
169| OpenSUSE 15.6 | ☐ | ☑ | 3.6 → 3.12 | 4.4.4 | 0.54 | 2.48 | 1.2020.9 | 0.8 | 3.5 | 2.1.6 |
170| Ubuntu Noble | ☐ | ☐ | 3.12 | 6.1.1 | 0.54 | u2.42 | u1.2020.2 | u0.9 | u3.8 | 2.1.8 |
171| Alma / Rocky 10 | ☑ | ☑ | 3.12 | e7.1.2 | | 9.0 | e1.2025.4 | e0.10 | | |
172
173#### Previous
174
175| os | https | up ca | python | ffmpeg | gource | graphviz | plantuml | shellcheck | shfmt | shunit |
176|:----------------|---|---|------------:|-------:|-----:|------:|-----------:|------:|-----:|------:|
177| Alpine 3.19 | ☑ | ☐ | 3.11 | 6.1.1 | 0.54 | 9.0 | 1.2023.12 | 0.9 | 3.7 | 2.1.8 |
178| Fedora 39 | ☑ | ☑ | 3.12 → 3.13 | 6.1.1 | 0.55 | 8.1 | 1.2024.6 | 0.9 | 3.5 | 2.1.6 |
179| Debian Bookworm | ☐ | ☐ | 3.11 | 5.1.6 | 0.54 | 2.42 | 1.2020.2 | 0.9 | 3.6 | 2.1.8 |
180| OpenSUSE 15.5 | ☐ | ☑ | 3.6 → 3.11 | 4.4.4 | 0.54 | 2.48 | 1.2020.9 | 0.8 | 3.5 | 2.1.6 |
181| Alma / Rocky 9 | ☑ | ☑ | 3.9 → 3.12 | e5.1.4 | | 2.44 | e1.2024.6 | e0.8 | | |
182
183#### Older
184
185| os | https | up ca | python | ffmpeg | gource | graphviz | plantuml | shellcheck | shfmt | shunit |
186|:----------------|---|---|------------:|-------:|-----:|------:|-----------:|------:|-----:|------:|
187| Alma / Rocky 8 | ☑ | ☑ | 3.6 → 3.12 | | | 2.40 | e1.2024.6 | e0.6 | | |
188
189#### Older Python
190
191| os | https | up ca | python | ffmpeg | gource | graphviz | plantuml | shellcheck | shfmt | shunit |
192|:----------------|---|---|------------:|-------:|-----:|------:|-----------:|------:|-----:|------:|
193| Ubuntu Jammy | ☐ | ☐ | 3.10 | 4.4.2 | 0.51 | u2.42 | u1.2020.2 | u0.8 | u3.4 | 2.1.6 |
194
195---
196
197## 4 [Who](#toc) {#who}
198
199* Author: [Marc Beninca](https://forge.rwx.work/marc.beninca)
200
201---
202
203## 5 [Where](#toc) {#where}
204
205### 5.1 [Chat](#where) {#where-chat}
206
207* [IRC](ircs://irc.libera.chat/#spcd)
208* [Stoat](https://stoat.chat/server/01FREKCG3P2P0YMAHQSCPSW4GD/channel/01K2S5MFK5XAAW53605NQT51Z5)
209
210### 5.2 [Repo](#where) {#where-repo}
211
212* [Code](https://forge.rwx.work/rwx.work/spcd)
213* [Feed](https://forge.rwx.work/rwx.work/spcd.rss)
214* [Actions](https://forge.rwx.work/rwx.work/spcd/actions)
215
216### 5.3 [Site](#where) {#where-site}
217
218* [Web](https://spcd.rwx.work)
219
220---
221
222## 6 [When](#toc) {#when}
223
224### 6.1 [Release tasks](#when) {#when-release}
225
226#### Documentation
227
228* build instead of readme
229* generate graphs
230* list behavior differences
231 * Continuous Integration platforms
232 * Operating Systems
233
234#### Shell
235
236* comment
237* git/pypi fallback for rwx/spcd
238* handle errors
239* packages
240 * configure
241 * apk
242 * pacman
243 * zypper
244 * repositories
245 * codecs
246 * epel
247 * fedora
248 * opensuse
249 * opensuse
250 * disable & enable https
251* locales
252 * persist
253* test
254* workspace variable
255
256#### Shell → Python
257
258* check file & variable sums
259* install system packages
260 * epel
261 * plantuml
262 * shellcheck
263 * graphviz
264 * openssh
265 * rsync
266 * shfmt
267 * shunit
268* python virtual environment
269 * install system package
270 * configure pip
271 * venv
272 * create
273 * activate
274* query package
275* relay environment module name
276
277#### Python
278
279* check
280 * .py
281 * .sh
282 * maximum line length
283* comment
284* detect ssh private key type
285* fit banner to text
286* implement substeps
287* list sys.path
288* set workflow templates
289 * actions
290 * includes
291* try git repo url variable first for shunit
292* test
293
294### 6.2 [Further tasks](#when) {#when-further}
295
296* automate versions fetching
297* gource, xvfb, xauth
298* handle openh264 repositories
299* link from workspace to actions root
300* rpm fusion
301* tex
302* translate to french
303* try to support
304 * guix
305 * nix
306* uv