OpenTelemetry

@julr/tenace-otel is an instrumentation that records OpenTelemetry traces and metrics for every execution.

Install

pnpm

pnpm add @julr/tenace-otel @opentelemetry/api

npm

npm install @julr/tenace-otel @opentelemetry/api

@julr/tenace and @opentelemetry/api 1.9 or later are peer dependencies. The package uses only the API, so configure the OpenTelemetry SDK as usual.

Enable it

import { Tenace } from '@julr/tenace'
import { openTelemetry } from '@julr/tenace-otel'

// every policy
Tenace.instrument(openTelemetry())

// or one policy
const policy = Tenace.policy({ name: 'users.get', instrumentation: openTelemetry() })

Option	Type	Default	Description
tracer	Tracer	trace.getTracer('@julr/tenace')	The tracer that creates the spans.
meter	Meter	metrics.getMeter('@julr/tenace')	The meter that creates the instruments.
spanEvents	boolean	true	Record one span event per Tenace event.

The default tracer and meter are looked up on each execution, so openTelemetry() can be called before the OpenTelemetry SDK starts. Instruments are created once per meter and shared by every openTelemetry() that uses it.

Name your policies, or pass name to execute(). The name becomes the span name and the tenace.operation.name attribute, so keep it low cardinality: users.get, not users.get.42.

Traces

Each execution creates one INTERNAL span, named after the operation, with these attributes:

Attribute	Value
tenace.operation.name	The operation name.
tenace.strategies	The strategy names, outermost first, joined with commas.
tenace.attempts	How many times the function was called, nested retries included.
the attributes of execute()	As given.

Each attempt creates a child span, <name> attempt, with tenace.operation.name and tenace.attempt.number, the attempt number of the innermost retry. The attempt span is active while your function runs, so the HTTP and database spans of your function nest under it.

An attempt still running when the execution ends, such as after an aggressive timeout, is ended with the execution span, with the ERROR status and tenace.attempt.abandoned: true. It is not recorded in tenace.attempt.duration.

A failed attempt gets the ERROR status, an exception event, and error.type on its own span. The execution span gets them only when the execution fails in the end, so an attempt that a retry or a fallback recovered from does not mark the execution as failed.

error.type is the _tag of a Tenace error or of any tagged error, the class name of a thrown Error, or _OTHER. The package exports the errorType(error) function that computes it.

Span events

With spanEvents: true, each Tenace event becomes an event on the execution span, named tenace.<type>, such as tenace.retry.scheduled. A custom event is named tenace.custom.<name>.

Event	Attributes
retry.scheduled	tenace.attempt.number, tenace.retry.delay_ms, error.type
retry.exhausted	tenace.attempts, error.type
retry.declined	tenace.attempt.number, tenace.retry.declined_reason, error.type
timeout	tenace.attempt.number, tenace.timeout.ms
circuit.state_changed	tenace.circuit_breaker.name, tenace.circuit_breaker.from, tenace.circuit_breaker.to
circuit.rejected	tenace.circuit_breaker.name, tenace.circuit_breaker.state, tenace.retry_after_ms when set
bulkhead.rejected	tenace.bulkhead.name, tenace.bulkhead.running, tenace.bulkhead.queued
fallback.used	error.type
fallback.stale	tenace.key, tenace.stale.age_ms
rate_limit.rejected	tenace.key, tenace.retry_after_ms
lock.rejected, dedupe.joined, cache.hit, cache.miss	tenace.key
cache.stale, cache.failed	tenace.key, error.type
custom	tenace.event.custom_name, and the attributes of the event

Metrics

Instrument	Type	Unit	Attributes
tenace.execution.duration	histogram	s	tenace.operation.name, error.type on failure
tenace.attempt.duration	histogram	s	tenace.operation.name, error.type on failure
tenace.execution.attempts	histogram	{attempt}	tenace.operation.name, error.type on failure
tenace.events	counter	{event}	tenace.operation.name, tenace.event.name
tenace.circuit_breaker.state	observable gauge		tenace.circuit_breaker.name

The execution duration includes every attempt and delay. The attempts histogram uses the bucket boundaries 1, 2, 3, 4, 5, and 10.

The circuit breaker gauge reports 0 for closed, 1 for open, 2 for half-open, and 3 for isolated. It is keyed by tenace.circuit_breaker.name, so give each breaker a unique name: unnamed breakers all report as circuitBreaker. It reports a breaker once an instrumented execution has seen one of its state changes.