Radish alpha
r
radicle-ci-broker
rad:zwTxygwuz5LDGBq255RA2CbNGrz8
Radicle CI broker
Radicle
Git
Patches › 8c4ffafd
add verification scenarios for failing adapters
Merged liw opened 8 months ago
Activity Changes +159 -3
2 files changed +159 -3 57d5f7f1 → 54390185
modified Makefile
@@ -1,3 +1,5 @@
+ 
TESTS =
+ 


all: check build test doc



check:
@@ -8,10 +10,10 @@ build:
	cargo build --all-targets



fast-test: build
- 
	cargo test -- --skip upgrades
+ 
	cargo test -- --skip upgrades $(TESTS)



test: fast-test
- 
	cargo test
+ 
	cargo test  -- $(TESTS)



.PHONY: doc

doc:
modified ci-broker.md
@@ -141,6 +141,73 @@ fi
) 1>&2

~~~
+
+
+ 
## A failing adapter with a successful run
+
+ 
This adapter does nothing, just reports a run ID and a successful run,
+ 
but then fails.
+
+ 
~~~{#failing-on-success.sh .file .sh}
+ 
#!/bin/sh
+ 
set -eu
+ 
cat > /dev/null
+ 
echo '{"response":"triggered","run_id":{"id":"xyzzy"}}'
+ 
echo '{"response":"finished","result":"success"}'
+ 
(
+ 
echo "This is an adapter error: Mordor"
+ 
echo "Environment:"
+ 
env
+ 
if [ "${RADICLE_NATIVE_CI:-}" != "" ]; then
+ 
    echo "Adapter config:"
+ 
    nl "$RADICLE_NATIVE_CI"
+ 
fi
+ 
) 1>&2
+ 
exit 1
+ 
~~~
+
+ 
## A failing adapter with a failed run
+
+ 
This adapter does nothing, just reports a run ID and a failed run,
+ 
but then fails.
+
+ 
~~~{#failing-on-failure.sh .file .sh}
+ 
#!/bin/sh
+ 
set -eu
+ 
cat > /dev/null
+ 
echo '{"response":"triggered","run_id":{"id":"xyzzy"}}'
+ 
echo '{"response":"finished","result":"failure"}'
+ 
(
+ 
echo "This is an adapter error: Mordor"
+ 
echo "Environment:"
+ 
env
+ 
if [ "${RADICLE_NATIVE_CI:-}" != "" ]; then
+ 
    echo "Adapter config:"
+ 
    nl "$RADICLE_NATIVE_CI"
+ 
fi
+ 
) 1>&2
+ 
exit 1
+ 
~~~
+
+
+
+ 
## List job COBs
+
+ 
Job COBs are a way for the CI broker to record that it's run CI for a
+ 
change. This script lists the job COBs in a given repository.
+
+ 
~~~{#list-jobs.sh .file .sh}
+ 
#!/bin/bash
+
+ 
set -euo pipefail
+ 
RID="$(rad ls --all | awk -v R="$1" '$2 == R { print $3 }')"
+ 
if [ -z "$RID" ]; then
+ 
	echo "Unknown repository $1" 1>&2
+ 
	exit 1
+ 
fi
+ 
rad cob list --repo "$RID" --type xyz.radworks.job
+ 
~~~
+ 


# Custom scenario steps



In this document we use scenarios to show how to verify that the CI
@@ -312,7 +379,6 @@ This is a very rudimentary check, but if the event file is incorrect,
then Radicle code will reject it. We don't want to duplicate the logic

to do that verification in detail.
- 


# Acceptance criteria



## Shows config as JSON
@@ -408,6 +474,51 @@ given a Git repository xyzzy in the Radicle node
given the Radicle node emits a refsUpdated event for xyzzy

when I run ./env.sh synthetic-events synt.sock event.json --log log.txt

given a directory reports
+
+ 
given file list-jobs.sh
+ 
when I run bash list-jobs.sh xyzzy
+ 
then stdout is exactly ""
+
+ 
when I run ./env.sh cib --config broker-with-triggers.yaml process-events
+
+ 
then stderr contains "CibStart"
+ 
then stderr contains "CibConfig"
+ 
then stderr contains "CibEndSuccess"
+ 
then file reports/index.html exists
+ 
then file reports/status.json exists
+ 
then file reports/index.rss exists
+
+ 
when I run bash list-jobs.sh xyzzy
+ 
then stdout isn't exactly ""
+
+ 
when I run cibtool --db ci-broker.db event list
+ 
then stdout is empty
+
+ 
when I run cibtool --db ci-broker.db run list --json
+ 
then stdout contains ""id": "xyzzy""
+ 
~~~
+
+
+ 
## Handles adapter failing on a successful run
+
+ 
_Want:_ If the adapter fails, the CI broker creates a job COB and
+ 
report pages anyway.
+
+ 
_Why:_ This is necessary for the CI broker to be robust.
+
+ 
_Who:_ `cib-devs`
+
+ 
~~~scenario
+ 
given a Radicle node, with CI configured with broker-with-triggers.yaml and adapter failing-on-success.sh
+ 
given a Git repository xyzzy in the Radicle node
+ 
given the Radicle node emits a refsUpdated event for xyzzy
+ 
when I run ./env.sh synthetic-events synt.sock event.json --log log.txt
+ 
given a directory reports
+
+ 
given file list-jobs.sh
+ 
when I run bash list-jobs.sh xyzzy
+ 
then stdout is exactly ""
+ 


when I run ./env.sh cib --config broker-with-triggers.yaml process-events



then stderr contains "CibStart"
@@ -417,6 +528,49 @@ then file reports/index.html exists
then file reports/status.json exists

then file reports/index.rss exists
+ 
when I run bash list-jobs.sh xyzzy
+ 
then stdout isn't exactly ""
+
+ 
when I run cibtool --db ci-broker.db event list
+ 
then stdout is empty
+
+ 
when I run cibtool --db ci-broker.db run list --json
+ 
then stdout contains ""id": "xyzzy""
+ 
~~~
+
+
+ 
## Handles adapter failing on a failed run
+
+ 
_Want:_ If the adapter fails, the CI broker creates a job COB and
+ 
report pages anyway.
+
+ 
_Why:_ This is necessary for the CI broker to be robust.
+
+ 
_Who:_ `cib-devs`
+
+ 
~~~scenario
+ 
given a Radicle node, with CI configured with broker-with-triggers.yaml and adapter failing-on-failure.sh
+ 
given a Git repository xyzzy in the Radicle node
+ 
given the Radicle node emits a refsUpdated event for xyzzy
+ 
when I run ./env.sh synthetic-events synt.sock event.json --log log.txt
+ 
given a directory reports
+
+ 
given file list-jobs.sh
+ 
when I run bash list-jobs.sh xyzzy
+ 
then stdout is exactly ""
+
+ 
when I run ./env.sh cib --config broker-with-triggers.yaml process-events
+
+ 
then stderr contains "CibStart"
+ 
then stderr contains "CibConfig"
+ 
then stderr contains "CibEndSuccess"
+ 
then file reports/index.html exists
+ 
then file reports/status.json exists
+ 
then file reports/index.rss exists
+
+ 
when I run bash list-jobs.sh xyzzy
+ 
then stdout isn't exactly ""
+ 


when I run cibtool --db ci-broker.db event list

then stdout is empty