ConfluentWire

dev.constructive.eo.avro.ConfluentWire
object ConfluentWire

The Confluent Schema Registry wire format for binary Avro payloads — a 5-byte header (magic byte 0x00, then the writer schema's registry id as a big-endian 4-byte int) in front of the plain Avro binary body — and the consumer surface built on it. This is the framing every Confluent-serialized Kafka topic carries; anything reading such a topic outside Confluent's own serializer stack needs exactly what lives here.

'''Registry-agnostic by design.''' No registry client, no new dependency: every member that needs a schema looked up takes the SchemaById hook (or its effectful Int => F[Schema] sibling). Plug in a registry client, a static map, a cache — the deployment owns it. All failures are values from the AvroFailure taxonomy (or an AvroFailureException raised in the caller's F), never a bare NPE or an Avro internal exception.

The surface splits along two axes.

'''Axis 1 — drift policy.''' What happens when the writer schema named by the frame's id differs from the schema the consumer reads with:

  • '''Gate''' (resolve, confluent): compare parsing-canonical-form fingerprints and REFUSE any difference (AvroFailure.SchemaMismatch). The body bytes pass through untouched — byte-exact, zero re-encode. For payloads that must stay verbatim (hashing, signatures, archiving, pass-through forwarding), and for consumers where drift is a bug to surface, not a condition to absorb.
  • '''Translate''' (resolving, resolvingRecord, resolvingBytes, reader, recordReader): run Avro schema resolution (ResolvingDecoder) writer → reader, so compatible evolution — added fields with defaults, reordered fields, promotions — is absorbed instead of refused.

'''Axis 2 — output altitude.''' Each translating member hands back a different currency:

  • typed A (via AvroCodec): resolving for a KNOWN writer schema (pure optic), reader for a per-message lookup (effectful, Stream.evalMap-ready);
  • generic IndexedRecord (no case class needed): resolvingRecord / recordReader, same split;
  • reader-layout framed '''bytes''' (no decode at all): resolvingBytes — per-message lookup, writer-id-cached, for consumers that hash / slice / forward rather than deserialize.

Underneath sit the pure frame primitives strip / attach (header handling only, no schema involvement) — the building blocks for anything the members above don't cover.

Attributes

Source
ConfluentWire.scala
Graph
Supertypes
class Object
trait Matchable
class Any
Self type

Members list

Type members

Classlikes

final case class Framed(schemaId: Int, body: Array[Byte])

A stripped Confluent frame: the schema id and the Avro binary body.

A stripped Confluent frame: the schema id and the Avro binary body.

body is a COPY of the payload bytes, not a zero-copy offset view — Array[Byte] cannot carry an offset, and every downstream consumer takes whole arrays. The copy is one arraycopy of length - 5 bytes: noise next to any decode that follows.

Attributes

Source
ConfluentWire.scala
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Types

type SchemaById = Int => Schema

Resolve a Confluent schema id to the writer Schema it names. Synchronous by design: the caller owns the registry client, the cache, and any effect wrapping, and hands this surface an already-synchronous lookup. May throw (a registry miss, a network error) — callers here catch that into AvroFailure.SchemaResolutionFailed.

Resolve a Confluent schema id to the writer Schema it names. Synchronous by design: the caller owns the registry client, the cache, and any effect wrapping, and hands this surface an already-synchronous lookup. May throw (a registry miss, a network error) — callers here catch that into AvroFailure.SchemaResolutionFailed.

Attributes

Source
ConfluentWire.scala

Value members

Concrete methods

def attach(schemaId: Int, body: Array[Byte]): Array[Byte]

Frame an Avro binary body under schemaId — the inverse of strip: strip(attach(id, body)) == Right(Framed(id, body)) for every id / body.

Frame an Avro binary body under schemaId — the inverse of strip: strip(attach(id, body)) == Right(Framed(id, body)) for every id / body.

Attributes

Source
ConfluentWire.scala
def confluent(schemaById: SchemaById, readerSchema: Schema, frameId: Int): PickMendPrism[Array[Byte], Array[Byte], Array[Byte]]

resolve as a composable eo dev.constructive.eo.optics.Prism over bytes: Array[Byte] (a full Confluent frame) ↔ Array[Byte] (the byte-exact gated body). Drop it BEFORE any byte optic and compose:

resolve as a composable eo dev.constructive.eo.optics.Prism over bytes: Array[Byte] (a full Confluent frame) ↔ Array[Byte] (the byte-exact gated body). Drop it BEFORE any byte optic and compose:

 val cf = ConfluentWire.confluent(schemaById, readerSchema, frameId)
 cf.andThen(codecPrism[A].field(_.x)).getOption(framedBytes)   // Option[X]
 cf.getOption(framedBytes)                                     // Option[Array[Byte]]
  • getOption(framed) runs strip + resolve + fingerprint-gate; a bad frame, an unresolvable id, or a fingerprint mismatch all yield None. Use resolve when you need the specific AvroFailure instead of None.
  • reverseGet(body) re-frames via attach under frameId — the schema id to publish under (typically the reader schema's registry id). Only exercised on write-back (modify / replace). A monomorphic byte Prism can't thread the original per-message id from read to write, so re-framing uses this fixed id; to preserve the incoming id, use resolve + attach by hand.

The reader fingerprint is computed once at construction, not per payload.

Attributes

Source
ConfluentWire.scala
def reader[F[_], A](schemaById: Int => F[Schema])(using F: MonadThrow[F], codec: AvroCodec[A]): Array[Byte] => F[A]

Typed per-message Confluent reader: Array[Byte] => F[A], ready for an fs2 Stream.evalMap. Per payload: strip the header, look the writer schema up by id (schemaById, effectful), resolve-decode into A (reader shape = the codec's own schema).

Typed per-message Confluent reader: Array[Byte] => F[A], ready for an fs2 Stream.evalMap. Per payload: strip the header, look the writer schema up by id (schemaById, effectful), resolve-decode into A (reader shape = the codec's own schema).

Strict on the frame: a payload that does not parse as a Confluent frame raises AvroFailure.NotConfluentFramed — no silent fallback to a direct decode, which could accidentally succeed on corrupt bytes and yield garbage. A topic with mixed framed / unframed producers opts into its own fallback by catching that failure and decoding directly (AvroCodec.decodeValue[A]).

All failures are raised in F: NotConfluentFramed / ResolveFailed / DecodeFailed / BinaryParseFailed via AvroFailureException; a schemaById failure as the effect's own error.

Attributes

Source
ConfluentWire.scala
def recordReader[F[_]](schemaById: Int => F[Schema], writeSchema: Schema)(using F: MonadThrow[F]): Array[Byte] => F[IndexedRecord]

Generic counterpart of reader — yields Array[Byte] => F[IndexedRecord] resolved into the caller-supplied writeSchema, for when no reader case class exists. Same strict frame contract as reader (opt-in fallback decode: AvroCodec.decodeRecord).

Generic counterpart of reader — yields Array[Byte] => F[IndexedRecord] resolved into the caller-supplied writeSchema, for when no reader case class exists. Same strict frame contract as reader (opt-in fallback decode: AvroCodec.decodeRecord).

Attributes

Source
ConfluentWire.scala
def resolve(framed: Array[Byte], schemaById: SchemaById, readerSchema: Schema): Either[AvroFailure, Array[Byte]]

Strip + resolve + fingerprint-gate a Confluent-framed payload down to its BODY BYTES, without decoding. Per payload:

Strip + resolve + fingerprint-gate a Confluent-framed payload down to its BODY BYTES, without decoding. Per payload:

  1. strip the 5-byte header → (schemaId, body);
  2. resolve the writer schema for schemaId via schemaById;
  3. gate by parsing-canonical-form fingerprint (org.apache.avro.SchemaNormalization.parsingFingerprint64): when the writer fingerprint equals the reader's, the body is byte-identical under both schemas — return it as-is; else refuse with AvroFailure.SchemaMismatch rather than hand back bytes that would misdecode.

The returned bytes are the caller's to decode (or hash, or forward) with whatever they own. Failures: AvroFailure.NotConfluentFramed (bad frame), AvroFailure.SchemaResolutionFailed (the hook threw), AvroFailure.SchemaMismatch (writer ≠ reader fingerprint). To ABSORB compatible drift instead of refusing it, use the translating surface below.

Attributes

Source
ConfluentWire.scala
def resolving[A](readSchema: Schema, frameId: Int)(using codec: AvroCodec[A]): Optic[Array[Byte], BridgedBytes, A, A, Affine]

Read+write Confluent optic for a KNOWN writer schema (single-schema topic, or a producer's own output). to strips the header and resolve-decodes the body from readSchema into A (the write schema = AvroCodec[A].schema); from re-encodes A and re-frames under frameId. Affine-carried with the same T = Either[AvroFailure, Array[Byte]] fallible-build shape as AvroBridge. For a mixed-schema stream use reader (typed) or resolvingBytes (bytes), which look the writer schema up per message.

Read+write Confluent optic for a KNOWN writer schema (single-schema topic, or a producer's own output). to strips the header and resolve-decodes the body from readSchema into A (the write schema = AvroCodec[A].schema); from re-encodes A and re-frames under frameId. Affine-carried with the same T = Either[AvroFailure, Array[Byte]] fallible-build shape as AvroBridge. For a mixed-schema stream use reader (typed) or resolvingBytes (bytes), which look the writer schema up per message.

Attributes

Source
ConfluentWire.scala
def resolvingBytes(schemaById: SchemaById, readerSchema: Schema, frameId: Int): Array[Byte] => Either[AvroFailure, Array[Byte]]

Framed bytes → reader-layout framed bytes for a MIXED-schema stream: per-message writer lookup (like recordReader) fused with drift translation (like resolvingRecord), handing back Array[Byte] rather than a typed A or an F[A]. Per payload: strip, resolve-decode the body writer → reader, re-encode under readerSchema, re-frame under frameId. Because the output is reader-layout, it is stable across writer-schema evolution within a reader generation — so digests, slices, and forwards computed downstream don't churn when producers upgrade. That is the property the gate (resolve) cannot give.

Framed bytes → reader-layout framed bytes for a MIXED-schema stream: per-message writer lookup (like recordReader) fused with drift translation (like resolvingRecord), handing back Array[Byte] rather than a typed A or an F[A]. Per payload: strip, resolve-decode the body writer → reader, re-encode under readerSchema, re-frame under frameId. Because the output is reader-layout, it is stable across writer-schema evolution within a reader generation — so digests, slices, and forwards computed downstream don't churn when producers upgrade. That is the property the gate (resolve) cannot give.

A factory, like confluent: the returned function closes over a per-writer-id cache (java.util.concurrent.ConcurrentHashMap), so schemaById is consulted once per DISTINCT writer id — one entry per schema version seen on the stream. Keep the returned function; re-calling resolvingBytes per message discards the cache.

Failures as Left: AvroFailure.NotConfluentFramed (bad or null frame, checked before schemaById), AvroFailure.SchemaResolutionFailed (the hook threw), AvroFailure.ResolveFailed / AvroFailure.EncodeFailed (resolve-decode / re-encode failed).

Attributes

Source
ConfluentWire.scala
def resolvingRecord(readSchema: Schema, writeSchema: Schema, frameId: Int): Optic[Array[Byte], BridgedBytes, IndexedRecord, IndexedRecord, Affine]

Generic counterpart of resolving — resolves readSchema → the caller-supplied writeSchema into an IndexedRecord, for when no reader codec / case class exists.

Generic counterpart of resolving — resolves readSchema → the caller-supplied writeSchema into an IndexedRecord, for when no reader codec / case class exists.

Attributes

Source
ConfluentWire.scala
def strip(bytes: Array[Byte] | Null): Either[AvroFailure, Framed]

Validate and strip the 5-byte Confluent header. Fails structurally (AvroFailure.NotConfluentFramed) on a null payload (a Kafka tombstone / mis-produced record — a defined failure rather than an NPE), on inputs shorter than the header, or whose magic byte isn't 0x00.

Validate and strip the 5-byte Confluent header. Fails structurally (AvroFailure.NotConfluentFramed) on a null payload (a Kafka tombstone / mis-produced record — a defined failure rather than an NPE), on inputs shorter than the header, or whose magic byte isn't 0x00.

Attributes

Source
ConfluentWire.scala

Concrete fields

val HeaderLength: Int

Header length: 1 magic byte + 4 big-endian schema-id bytes.

Header length: 1 magic byte + 4 big-endian schema-id bytes.

Attributes

Source
ConfluentWire.scala
val Magic: Byte

The Confluent magic byte — always 0x00 in the current wire format.

The Confluent magic byte — always 0x00 in the current wire format.

Attributes

Source
ConfluentWire.scala