*.svg

*.html

.SUFFIXES: .uml .svg .pik .md .html

.md.html:

	pandoc --toc --standalone --self-contained $< -o $@

.uml.svg:

	plantuml -tsvg --output=. $<

.pik.svg:

	pikchr-cli $< > $@.tmp

	mv $@.tmp $@

all: architecture.html

architecture.html: architecture.svg architecture-ext.svg components.svg components-ext.svg

@startuml

node -> broker : RefsUpdated event

broker -> adapter : spawn

broker -> adapter : send request

adapter-> engine : trigger run

engine -> worker : start run

note over worker : perform the run

engine -> adapter : response with run id

adapter -> broker : response: run id

broker -> node : create CI COB

worker -> engine : run finished

engine -> adapter : web hook

adapter -> broker : response: result

broker -> node : update CI COB

@enduml

---

title: Radicle CI architecture

...

# Overview

CI support in Radicle consists of several components. For native CI

they are:

* the Radicle node

* the CI broker

* the native CI executable

These all run on the same host.

For external CI, the components are:

* the Radicle node

* the CI broker

* an executable for each supported external CI instance

* the external CI instance

The first three of these run on the same host, but the external CI

instance can run anywhere.

The child process is called "the CI adapter" in this document.

CI works like this:

* a repository known to the node changes

  - a git ref is updated

  - the ref can be a branch, tag, or something else, such as a Radicle

    COB

  - the node emits an event describing the change

* the CI broker listens to node event

  - the broker subscribers to node events via the node control socket,

    which is a Unix domain socket

* the CI broker filters events, based on its configuration, and the

  configuration for the repository involved

* for an event that passes its filters, the CI broker spawns the

  adapter process

  - the adapter process is either the native CI engine, or an

    integration with an external CI engine

  - the CI broker uses the executable for native CI by default, but

    this can be overridden by the broker configuration file, and that

    can be overridden by the `.ci/config.yaml` file in the repository

    the event is for

* the broker sends a request object to the adapter as a child process,

  via the child's stdin, and reads any responses from the child's

  stdout

  - the request is JSON

  - the responses are in the form of JSON Lines: a JSON object per

    line serialized to not contain newline characters

## Native CI

![Sequence diagram for native CI](architecture.svg)

## External CI

![Sequence diagram for external CI](architecture.svg)

# The adapter

The adapter process reads the request to perform a CI run on a

specific commit in a repository, and responds with the id of the run,

then later with the status of the finished run.

Note: this is for the first version. This will be expanded later with

other requests and responses, as needed.

For native CI, the adapter actually is the CI engine and performs the

CI run itself. For external CI, the adapter process does whatever it

needs to do to get the external CI engine instance to perform the CI

run. If the CI engine calls back via web hook to notify of the run

finishing, the adapter process needs to receive the call and process

it.

# Request and response messages

Note: the JSON objects below are formatted on multiple lines to make

them easier to read. The actual wire format is one line per message.

The request that the broker sends looks like this:

~~~{.json .numberLines}

{

    "request": "trigger",

    "repo": "<RID>",

    "commit": "<COMMIT>"

}

~~~

where `<RID>` is the repository ID, and `<COMMIT>` is the commit id

(the SHA checksum). The `request` fields allows us to extend this in

the future.

The first response from the adapter looks like this:

~~~{.json .numberLines}

{

    "response": "triggered",

    "run_id": "<RUNID>"

}

~~~

where `<RUNID>` is the id of the run that has been triggered.

The second response from the adapter looks like this:

~~~{.json .numberLines}

{

    "response": "finished",

    "result": "<STATUS>"

}

~~~

where `<STATUS>` is either the string `success` or `failure`. Note

that the run id is not repeated as the context makes this clear.

@startuml

node -> broker : RefsUpdated event

broker -> adapter : spawn

broker -> adapter : send request

adapter -> broker : response: run id

note over adapter : perform the run

broker -> node : create CI COB

adapter -> broker : response: result

broker -> node : update CI COB

@enduml

down

Node: cylinder "node"

move

Broker: box "CI broker"

move

Adapter: box "external" "adapter"

right

move

Ext: [

    Engine: box "External" "CI engine"

    move

    Worker: box "CI worker"

    arrow from Engine.e to Worker.w "control" aligned above

]

Border: box thin width Ext.width+5mm height Ext.height+5mm at Ext.center

Caption: text "External host" with .n at 5mm below Ext.s

arrow from Node.s to Broker.n "event" aligned above

arrow from Broker.s to Adapter.n "spawn" aligned above

arrow from Adapter.e to Ext.w "trigger" aligned above

arc from Ext.nw to Adapter.ne ->

move from last arc.n up 0.0

text "webhook"

down

Node: cylinder "node"

move

Broker: box "CI broker"

move

Adapter: box "native" "adapter"

arrow thin from Node.s to Broker.n "event" aligned above

arc thin from Broker.sw to Adapter.nw -> "spawn" aligned above

arrow thin from Broker.s to Adapter.n "request" aligned above

arc thin from Adapter.ne to Broker.se -> "responses" aligned

Note: This is not finalized.

# Requirements for CI in Radicle

## Initial goal

Radicle itself, and Radicle users, need support for continuous

integration. Concretely, this means, at minimum, that when a change is

merged into the an integration branch, an automated process verifies

that the branch still fulfills any requirements put on it.

More concretely, for Radicle itself, and for projects using Radicle

that follow a similar development process, when a change is merged

into the `master` branch (or the `main` branch, depending on the

repository), all the code builds, and all the automated tests pass.

Simplified, the following command should pass in the Radicle

`heartwood` repository after every merge:

~~~

cargo test --locked --workspace

~~~

In addition to the `master` branch, the same checks are run on any

patch created with `rad patch` on the Radicle repositories.

Further, the any runs and their results (success vs failure) is

communicated via the Radicle network. If a contributor creates a

patch, they can query `rad` to see if the tests pass under CI. (If

there are logs from CI for a failing run, they have to be queried from

the node that runs CI, for now.)

This is an initial, minimal goal that's useful. We can build on this.

## Long-term goal

Later on, we will want to reach much more ambitious goals for

integration support in Radicle. Ideally, we should aim CI in Radicle

to surpass all other CI that has ever existed, but that's more of a

vision than a goal.

* We do more extensive checks than just building and running tests.

  For example, running clippy, checking formatting, checking for

  unmaintained or insecure dependencies (even indirect ones), etc.

* Allow projects using Radicle to freely decide what a "CI run" will

  do for their project.

* We run tests on a tentative merge, but don't publish it to the

  Radicle network unless CI is happy. This will hopefully prevent a

  broken change from even being merge, where the initial goal only

  detects it after being merged.

* We publish built artifacts, such as program binaries, installation

  packages, web site content, etc.

However, this all requires the foundation of the initial goal to work.

## Initial requirements

We want to support both external and native CI engines so that we

don't lock ourselves into supporting only one, even by mistake.

* Radicle can integrate with various external CI engines and

  instances.

  - We don't want to insist Radicle users use a specific engine,

    because whichever one we chose would be unwelcome to some Radicle

    users.

  - The first such engine we've chosen to support is Concourse.

  - Setting up and running the external engines is not part of

    Radicle.

* Radicle comes with at least a simplistic native CI engine.

  - This will not be sufficient for many other projects, but will be

    enough to at least run basic CI checks for Radicle itself, to

    start with.

* Native CI comes with Radicle and is extremely easy to set up.

  - it does not, however, need to be highly secure or performant

* A node can run CI for the repositories in its storage.

  - The node admin can opt in to run CI on their node.

  - The node admin can configure on which changes a CI run is

    triggered. A node might not have enough resources for CI to check

    arbitrary changes, but the owner might want to concentrate on

    specific ones, such as their own patches.

* A node should announce to the network what CI runs it does, on what

  commits, and what the result of each run is.

  - This allows people using other nodes to see that patches are OK in

    CI.

  - Just the metadata about each run, not build log or build

    artifacts, at least for now.

* We can set up `seed.radicle.xyz` that run CI on any patch on the

  `heartwood` repository and checks that the patch builds and its

  tests pass.

  - We can then set this up on other repositories on the seed node as

    well.

  - We are confident enough in our CI solution that this is reasonably

    safe and secure to do.

  - This can use an external CI or native CI, depending on the state

    of those at the time of setting up.

  - If we don't want to risk the central node with this, we can do it

    on another node instead.

## Requirements for a CI broker in Radicle

Based on what we've discussed and experimented with and done so far,

Radicle will have a "CI broker" component that acts as a bridge

between a Radicle node and a CI engine. The broker will support native

CI and a selection of external CI engines.

* Broker supports external CI engines.

  - first version supports at least Concourse

* Broker supports Radicle native CI.

* The internal interfaces for supporting a CI engine are the same for

  native CI and external CI.

  - this is necessary so that we don't embed unwanted assumptions in

    the interface

* Broker can be extended to support new engines.

* Broker listens to node events.

* Broker updates the Radicle network about CI runs.

* As much as is reasonably possible of the broker code is shared

  between different CI engines.