Radish alpha
r
rad:zwTxygwuz5LDGBq255RA2CbNGrz8
Radicle CI broker
Radicle
Git
feat: don't crash when failing to create trigger message
Merged liw opened 1 year ago

Signed-off-by: Lars Wirzenius liw@liw.fi

3 files changed +54 -1 f42ea5cb a7c9bd13
modified ci-broker.md
@@ -620,6 +620,48 @@ when I run ./env.sh cib --config broker.yaml --log-level error config
then stderr is exactly ""
~~~

+
## Fails run if building trigger fails, but does not crash
+

+
_Want:_ The CI broker fails a CI run if it can't create a trigger
+
message from a CI event, but it continues running and processing other
+
events.
+

+
_Why:_ If it's not possible to create a trigger message, the CI run
+
can't succeed, unless the failure is temporary. However, we have no
+
way of knowing if the failure is temporary, so the safe thing is to
+
mark the CI run as having failed and removing the CI event from the
+
queue. Further, the CI broker should not crash and should process
+
other events.
+

+
_Who:_ `cib-dev`
+

+
A failure to create a trigger message happens if the CI event refers
+
to a repository, commit, or Git ref that doesn't exist in the
+
repository on the local node. This should not ever happen, as the CI
+
event is only emitted by the node after the changes are on the node.
+
However, it has happened due to a programming error in the CI broker.
+
By handling the error and removing the event, the CI broker is a
+
little bit more robust.
+

+
We verify this by inserting two events into the queue and then running
+
`cib queued` to process them. We arrange things so that the first
+
event fails, but the second one succeeds.
+

+
~~~scenario
+
given a Radicle node, with CI configured with broker.yaml and adapter dummy.sh
+
given a Git repository xyzzy in the Radicle node
+
given a directory reports
+

+
when I run ./env.sh cibtool --db ci-broker.db event add --repo xyzzy --ref main --commit f42ea5cb9ce2dc7a9b87834ccee5b9bb3867db90 --kind branch-updated --base main
+
when I run ./env.sh cibtool --db ci-broker.db event add --repo xyzzy --ref main --commit HEAD --kind branch-updated --base main
+

+
when I run ./env.sh cib --config broker.yaml queued
+
then stderr contains "f42ea5cb9ce2dc7a9b87834ccee5b9bb3867db90"
+

+
when I run cibtool --db ci-broker.db run list
+
then stdout has one line
+
~~~
+

# Acceptance criteria for test tooling

The event synthesizer is a helper to feed the CI broker node events in
modified src/logger.rs
@@ -14,6 +14,7 @@ use crate::{
    db::{QueueId, QueuedCiEvent},
    msg::Request,
    node_event_source::NodeEventSource,
+
    queueproc::QueueError,
    run::Run,
};

@@ -202,6 +203,10 @@ pub fn queueproc_picked_event(id: &QueueId, event: &QueuedCiEvent) {
    info!(kind = %Kind::GotEvent, ?id, ?event, "picked event from queue");
}

+
pub fn queueproc_processed_event(result: &Result<bool, QueueError>) {
+
    info!(kind = %Kind::GotEvent, ?result, "result of processing event");
+
}
+

pub fn queueproc_remove_event(id: &QueueId) {
    info!(kind = %Kind::Debug, ?id, "remove event from queue");
}
modified src/queueproc.rs
@@ -78,7 +78,13 @@ impl QueueProcessor {
            while let Some(qe) = self.pick_event()? {
                self.run_tx.notify()?;
                logger::queueproc_picked_event(qe.id(), &qe);
-
                done = self.process_event(qe.event())?;
+
                let res = self.process_event(qe.event());
+
                logger::queueproc_processed_event(&res);
+
                match res {
+
                    Ok(shut_down) => done = shut_down,
+
                    Err(QueueError::BuildTrigger(_, _)) => done = false,
+
                    Err(err) => Err(err)?,
+
                }
                self.drop_event(qe.id())?;
                self.run_tx.notify()?;
            }