Originally on Zulip: https://radicle.zulipchat.com/#narrow/channel/380896-integrations/topic/CI.20Broker.20.26.20Adapter.28s.29.20logging.20issue/near/478122969

Copied here for posterity:

We have faced a weird issue with the CI Broker & Adapters’ logging.

The issue occurs when a rebase happens in repos with many commits. When this happens the broker execution freezes completely. At these cases the commit history of the broker message is really big (as expected). This causes the adapter to produce some really huge log messages (we log the broker’s I/O).

Investigating this issue further, I realized that as the CI broker uses pipes to handle the stderr output from the adapter, it could not handle such big payloads. The pipe’s maximum buffer size is usually up to 1MB for Ubuntu and 64KB for MacOS. Such payloads usually exceed this limit causing the broker to freeze.

So @Lars Wirzenius I think it’s good to keep that in mind for your adapter as well and maybe we might have to come up with another approach on the broker side. Please let me know if you want to open an issue to track this.

As a workaround solution for now we will log these messages in TRACE log level as we usually use DEBUG log level in our deployments.

We are extensively relied on the broker and the adapters that
are deployed in the radicle.gr for our development (CI, webhooks
etc) making such issues arise and then further investigated!