Skip to main content

Observability

Metrics come from the oncebox-metrics module, through Micrometer:

<dependency>
<groupId>io.github.dmitriy-iliyov</groupId>
<artifactId>oncebox-metrics</artifactId>
<version>2.0.0</version>
</dependency>
oncebox:
outbox:
metrics:
enabled: true
gauge:
enabled: true
inbox:
metrics:
enabled: true

The names below are as Prometheus exports them. "Tasks" are both the per-type event pollers and the background jobs: task_type is the event type for a poller, or one of outbox-processed-cleanup, outbox-stuck-recovery, outbox-dlq-transfer-to, outbox-dlq-transfer-from, outbox-dlq-cleanup, inbox-cleanup.

Outbox​

Gauges​

MetricDescriptionTags
outbox_eventsNumber of outbox events-
outbox_events_by_statusOutbox events by statusstatus={pending, in_process}
outbox_events_by_event_type_and_statusOutbox events by type and statusevent_type, status={pending, in_process}
outbox_dlq_eventsNumber of DLQ events-
outbox_dlq_events_by_statusDLQ events by statusstatus={moved, in_process, to_retry}
outbox_dlq_events_by_event_type_and_statusDLQ events by type and statusevent_type, status={moved, in_process, to_retry}
oncebox_polling_delay_millisecondsCurrent delay between task iterationstask_type

Event gauges run COUNT queries and show the exact numbers. To keep Prometheus scrapes from loading the database, their values are cached by default (gauge.cache.ttls); gauge.cache.enabled: false turns the cache off.

Counters​

MetricDescriptionTags
outbox_events_rate_totalProcessed eventsevent_type, status={processed}
outbox_events_by_action_type_rate_totalLifecycle actionsaction_type={attempt_move_to_dlq, recovered, cleaned, success_moved_to_dlq}
outbox_dlq_events_by_action_type_rate_totalDLQ actionsaction_type={attempt_move_to_outbox, success_moved_to_outbox, manual_deleted, cleaned}
oncebox_started_tasks_totalTask iterations startedtask_type
oncebox_skipped_tasks_totalTask iterations skippedtask_type
oncebox_succeeded_tasks_totalTask iterations succeededtask_type
oncebox_failed_tasks_totalTask iterations failedtask_type

Timers​

MetricDescriptionTags
oncebox_task_processing_durationDuration of a task iterationtask_type

The timer shows where time goes during bulk recovery or DLQ reprocessing.

Inbox​

MetricTypeDescriptionTags
inbox_events_totalcounterInbox events by outcometype={rejected_duplicates, consumed, cleaned, failed}
inbox_cache_action_totalcounterInbox cache hits and missesaction_type={hit, miss}
oncebox_polling_delay_millisecondsgaugeCurrent cleanup delaytask_type={inbox-cleanup}
oncebox_*_tasks_totalcounterCleanup iterations by statetask_type={inbox-cleanup}
oncebox_task_processing_durationtimerDuration of a cleanup iterationtask_type={inbox-cleanup}

What to alert on​

SignalMeaning
outbox_events_by_status{status="pending"} keeps growingPublishing does not keep up, or the broker is down
outbox_events_by_status{status="in_process"} stays above zeroBatches are stuck; check senders and emergency-timeout
outbox_dlq_events_by_status{status="moved"} above zeroEvents need review
rate(oncebox_failed_tasks_total[5m]) above zeroA poller or a background job is failing
inbox_events_total{type="failed"} growingConsumer operations are failing

Grafana​

Ready-made dashboards for the outbox and the inbox are in the repository. The example project starts Prometheus and Grafana with them provisioned.

Configuration​

See Configuration for the outbox and Configuration for the inbox.