Radish alpha
r
rad:zwTxygwuz5LDGBq255RA2CbNGrz8
Radicle CI broker
Radicle
Git
tests: add scenario to verify behavior when adapter fails
Lars Wirzenius committed 8 months ago
commit 5439018503e998cd4d53b3654250a3349c114e60
parent a3d8d1a
1 file changed +155 -1
modified ci-broker.md
@@ -141,6 +141,73 @@ fi
) 1>&2
~~~

+

+

+
## A failing adapter with a successful run
+

+
This adapter does nothing, just reports a run ID and a successful run,
+
but then fails.
+

+
~~~{#failing-on-success.sh .file .sh}
+
#!/bin/sh
+
set -eu
+
cat > /dev/null
+
echo '{"response":"triggered","run_id":{"id":"xyzzy"}}'
+
echo '{"response":"finished","result":"success"}'
+
(
+
echo "This is an adapter error: Mordor" 
+
echo "Environment:"
+
env
+
if [ "${RADICLE_NATIVE_CI:-}" != "" ]; then
+
    echo "Adapter config:"
+
    nl "$RADICLE_NATIVE_CI"
+
fi
+
) 1>&2
+
exit 1
+
~~~
+

+
## A failing adapter with a failed run
+

+
This adapter does nothing, just reports a run ID and a failed run,
+
but then fails.
+

+
~~~{#failing-on-failure.sh .file .sh}
+
#!/bin/sh
+
set -eu
+
cat > /dev/null
+
echo '{"response":"triggered","run_id":{"id":"xyzzy"}}'
+
echo '{"response":"finished","result":"failure"}'
+
(
+
echo "This is an adapter error: Mordor" 
+
echo "Environment:"
+
env
+
if [ "${RADICLE_NATIVE_CI:-}" != "" ]; then
+
    echo "Adapter config:"
+
    nl "$RADICLE_NATIVE_CI"
+
fi
+
) 1>&2
+
exit 1
+
~~~
+

+

+

+
## List job COBs
+

+
Job COBs are a way for the CI broker to record that it's run CI for a
+
change. This script lists the job COBs in a given repository.
+

+
~~~{#list-jobs.sh .file .sh}
+
#!/bin/bash
+

+
set -euo pipefail
+
RID="$(rad ls --all | awk -v R="$1" '$2 == R { print $3 }')"
+
if [ -z "$RID" ]; then
+
	echo "Unknown repository $1" 1>&2
+
	exit 1
+
fi
+
rad cob list --repo "$RID" --type xyz.radworks.job
+
~~~
+

# Custom scenario steps

In this document we use scenarios to show how to verify that the CI
@@ -312,7 +379,6 @@ This is a very rudimentary check, but if the event file is incorrect,
then Radicle code will reject it. We don't want to duplicate the logic
to do that verification in detail.

-

# Acceptance criteria

## Shows config as JSON
@@ -408,6 +474,51 @@ given a Git repository xyzzy in the Radicle node
given the Radicle node emits a refsUpdated event for xyzzy
when I run ./env.sh synthetic-events synt.sock event.json --log log.txt
given a directory reports
+

+
given file list-jobs.sh
+
when I run bash list-jobs.sh xyzzy
+
then stdout is exactly ""
+

+
when I run ./env.sh cib --config broker-with-triggers.yaml process-events
+

+
then stderr contains "CibStart"
+
then stderr contains "CibConfig"
+
then stderr contains "CibEndSuccess"
+
then file reports/index.html exists
+
then file reports/status.json exists
+
then file reports/index.rss exists
+

+
when I run bash list-jobs.sh xyzzy
+
then stdout isn't exactly ""
+

+
when I run cibtool --db ci-broker.db event list
+
then stdout is empty
+

+
when I run cibtool --db ci-broker.db run list --json
+
then stdout contains ""id": "xyzzy""
+
~~~
+

+

+
## Handles adapter failing on a successful run
+

+
_Want:_ If the adapter fails, the CI broker creates a job COB and
+
report pages anyway.
+

+
_Why:_ This is necessary for the CI broker to be robust.
+

+
_Who:_ `cib-devs`
+

+
~~~scenario
+
given a Radicle node, with CI configured with broker-with-triggers.yaml and adapter failing-on-success.sh
+
given a Git repository xyzzy in the Radicle node
+
given the Radicle node emits a refsUpdated event for xyzzy
+
when I run ./env.sh synthetic-events synt.sock event.json --log log.txt
+
given a directory reports
+

+
given file list-jobs.sh
+
when I run bash list-jobs.sh xyzzy
+
then stdout is exactly ""
+

when I run ./env.sh cib --config broker-with-triggers.yaml process-events

then stderr contains "CibStart"
@@ -417,6 +528,49 @@ then file reports/index.html exists
then file reports/status.json exists
then file reports/index.rss exists

+
when I run bash list-jobs.sh xyzzy
+
then stdout isn't exactly ""
+

+
when I run cibtool --db ci-broker.db event list
+
then stdout is empty
+

+
when I run cibtool --db ci-broker.db run list --json
+
then stdout contains ""id": "xyzzy""
+
~~~
+

+

+
## Handles adapter failing on a failed run
+

+
_Want:_ If the adapter fails, the CI broker creates a job COB and
+
report pages anyway.
+

+
_Why:_ This is necessary for the CI broker to be robust.
+

+
_Who:_ `cib-devs`
+

+
~~~scenario
+
given a Radicle node, with CI configured with broker-with-triggers.yaml and adapter failing-on-failure.sh
+
given a Git repository xyzzy in the Radicle node
+
given the Radicle node emits a refsUpdated event for xyzzy
+
when I run ./env.sh synthetic-events synt.sock event.json --log log.txt
+
given a directory reports
+

+
given file list-jobs.sh
+
when I run bash list-jobs.sh xyzzy
+
then stdout is exactly ""
+

+
when I run ./env.sh cib --config broker-with-triggers.yaml process-events
+

+
then stderr contains "CibStart"
+
then stderr contains "CibConfig"
+
then stderr contains "CibEndSuccess"
+
then file reports/index.html exists
+
then file reports/status.json exists
+
then file reports/index.rss exists
+

+
when I run bash list-jobs.sh xyzzy
+
then stdout isn't exactly ""
+

when I run cibtool --db ci-broker.db event list
then stdout is empty