ConfluentWire
The Confluent Schema Registry wire format for binary Avro payloads — a 5-byte header (magic byte 0x00, then the writer schema's registry id as a big-endian 4-byte int) in front of the plain Avro binary body — and the consumer surface built on it. This is the framing every Confluent-serialized Kafka topic carries; anything reading such a topic outside Confluent's own serializer stack needs exactly what lives here.
'''Registry-agnostic by design.''' No registry client, no new dependency: every member that needs a schema looked up takes the SchemaById hook (or its effectful Int => F[Schema] sibling). Plug in a registry client, a static map, a cache — the deployment owns it. All failures are values from the AvroFailure taxonomy (or an AvroFailureException raised in the caller's F), never a bare NPE or an Avro internal exception.
The surface splits along two axes.
'''Axis 1 — drift policy.''' What happens when the writer schema named by the frame's id differs from the schema the consumer reads with:
- '''Gate''' (resolve, confluent): compare parsing-canonical-form fingerprints and REFUSE any difference (AvroFailure.SchemaMismatch). The body bytes pass through untouched — byte-exact, zero re-encode. For payloads that must stay verbatim (hashing, signatures, archiving, pass-through forwarding), and for consumers where drift is a bug to surface, not a condition to absorb.
- '''Translate''' (resolving, resolvingRecord, resolvingBytes, reader, recordReader): run Avro schema resolution (
ResolvingDecoder) writer → reader, so compatible evolution — added fields with defaults, reordered fields, promotions — is absorbed instead of refused.
'''Axis 2 — output altitude.''' Each translating member hands back a different currency:
- typed
A(via AvroCodec): resolving for a KNOWN writer schema (pure optic), reader for a per-message lookup (effectful,Stream.evalMap-ready); - generic
IndexedRecord(no case class needed): resolvingRecord / recordReader, same split; - reader-layout framed '''bytes''' (no decode at all): resolvingBytes — per-message lookup, writer-id-cached, for consumers that hash / slice / forward rather than deserialize.
Underneath sit the pure frame primitives strip / attach (header handling only, no schema involvement) — the building blocks for anything the members above don't cover.
Attributes
- Source
- ConfluentWire.scala
- Graph
-
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
ConfluentWire.type
Members list
Type members
Classlikes
A stripped Confluent frame: the schema id and the Avro binary body.
A stripped Confluent frame: the schema id and the Avro binary body.
body is a COPY of the payload bytes, not a zero-copy offset view — Array[Byte] cannot carry an offset, and every downstream consumer takes whole arrays. The copy is one arraycopy of length - 5 bytes: noise next to any decode that follows.
Attributes
- Source
- ConfluentWire.scala
- Supertypes
-
trait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass AnyShow all
Types
Resolve a Confluent schema id to the writer Schema it names. Synchronous by design: the caller owns the registry client, the cache, and any effect wrapping, and hands this surface an already-synchronous lookup. May throw (a registry miss, a network error) — callers here catch that into AvroFailure.SchemaResolutionFailed.
Resolve a Confluent schema id to the writer Schema it names. Synchronous by design: the caller owns the registry client, the cache, and any effect wrapping, and hands this surface an already-synchronous lookup. May throw (a registry miss, a network error) — callers here catch that into AvroFailure.SchemaResolutionFailed.
Attributes
- Source
- ConfluentWire.scala
Value members
Concrete methods
Frame an Avro binary body under schemaId — the inverse of strip: strip(attach(id, body)) == Right(Framed(id, body)) for every id / body.
Frame an Avro binary body under schemaId — the inverse of strip: strip(attach(id, body)) == Right(Framed(id, body)) for every id / body.
Attributes
- Source
- ConfluentWire.scala
resolve as a composable eo dev.constructive.eo.optics.Prism over bytes: Array[Byte] (a full Confluent frame) ↔ Array[Byte] (the byte-exact gated body). Drop it BEFORE any byte optic and compose:
resolve as a composable eo dev.constructive.eo.optics.Prism over bytes: Array[Byte] (a full Confluent frame) ↔ Array[Byte] (the byte-exact gated body). Drop it BEFORE any byte optic and compose:
val cf = ConfluentWire.confluent(schemaById, readerSchema, frameId)
cf.andThen(codecPrism[A].field(_.x)).getOption(framedBytes) // Option[X]
cf.getOption(framedBytes) // Option[Array[Byte]]
getOption(framed)runs strip + resolve + fingerprint-gate; a bad frame, an unresolvable id, or a fingerprint mismatch all yieldNone. Use resolve when you need the specific AvroFailure instead ofNone.reverseGet(body)re-frames via attach underframeId— the schema id to publish under (typically the reader schema's registry id). Only exercised on write-back (modify/replace). A monomorphic byte Prism can't thread the original per-message id from read to write, so re-framing uses this fixed id; to preserve the incoming id, use resolve + attach by hand.
The reader fingerprint is computed once at construction, not per payload.
Attributes
- Source
- ConfluentWire.scala
Typed per-message Confluent reader: Array[Byte] => F[A], ready for an fs2 Stream.evalMap. Per payload: strip the header, look the writer schema up by id (schemaById, effectful), resolve-decode into A (reader shape = the codec's own schema).
Typed per-message Confluent reader: Array[Byte] => F[A], ready for an fs2 Stream.evalMap. Per payload: strip the header, look the writer schema up by id (schemaById, effectful), resolve-decode into A (reader shape = the codec's own schema).
Strict on the frame: a payload that does not parse as a Confluent frame raises AvroFailure.NotConfluentFramed — no silent fallback to a direct decode, which could accidentally succeed on corrupt bytes and yield garbage. A topic with mixed framed / unframed producers opts into its own fallback by catching that failure and decoding directly (AvroCodec.decodeValue[A]).
All failures are raised in F: NotConfluentFramed / ResolveFailed / DecodeFailed / BinaryParseFailed via AvroFailureException; a schemaById failure as the effect's own error.
Attributes
- Source
- ConfluentWire.scala
Generic counterpart of reader — yields Array[Byte] => F[IndexedRecord] resolved into the caller-supplied writeSchema, for when no reader case class exists. Same strict frame contract as reader (opt-in fallback decode: AvroCodec.decodeRecord).
Generic counterpart of reader — yields Array[Byte] => F[IndexedRecord] resolved into the caller-supplied writeSchema, for when no reader case class exists. Same strict frame contract as reader (opt-in fallback decode: AvroCodec.decodeRecord).
Attributes
- Source
- ConfluentWire.scala
Strip + resolve + fingerprint-gate a Confluent-framed payload down to its BODY BYTES, without decoding. Per payload:
Strip + resolve + fingerprint-gate a Confluent-framed payload down to its BODY BYTES, without decoding. Per payload:
- strip the 5-byte header →
(schemaId, body); - resolve the writer schema for
schemaIdviaschemaById; - gate by parsing-canonical-form fingerprint (
org.apache.avro.SchemaNormalization.parsingFingerprint64): when the writer fingerprint equals the reader's, the body is byte-identical under both schemas — return it as-is; else refuse with AvroFailure.SchemaMismatch rather than hand back bytes that would misdecode.
The returned bytes are the caller's to decode (or hash, or forward) with whatever they own. Failures: AvroFailure.NotConfluentFramed (bad frame), AvroFailure.SchemaResolutionFailed (the hook threw), AvroFailure.SchemaMismatch (writer ≠ reader fingerprint). To ABSORB compatible drift instead of refusing it, use the translating surface below.
Attributes
- Source
- ConfluentWire.scala
Read+write Confluent optic for a KNOWN writer schema (single-schema topic, or a producer's own output). to strips the header and resolve-decodes the body from readSchema into A (the write schema = AvroCodec[A].schema); from re-encodes A and re-frames under frameId. Affine-carried with the same T = Either[AvroFailure, Array[Byte]] fallible-build shape as AvroBridge. For a mixed-schema stream use reader (typed) or resolvingBytes (bytes), which look the writer schema up per message.
Read+write Confluent optic for a KNOWN writer schema (single-schema topic, or a producer's own output). to strips the header and resolve-decodes the body from readSchema into A (the write schema = AvroCodec[A].schema); from re-encodes A and re-frames under frameId. Affine-carried with the same T = Either[AvroFailure, Array[Byte]] fallible-build shape as AvroBridge. For a mixed-schema stream use reader (typed) or resolvingBytes (bytes), which look the writer schema up per message.
Attributes
- Source
- ConfluentWire.scala
Framed bytes → reader-layout framed bytes for a MIXED-schema stream: per-message writer lookup (like recordReader) fused with drift translation (like resolvingRecord), handing back Array[Byte] rather than a typed A or an F[A]. Per payload: strip, resolve-decode the body writer → reader, re-encode under readerSchema, re-frame under frameId. Because the output is reader-layout, it is stable across writer-schema evolution within a reader generation — so digests, slices, and forwards computed downstream don't churn when producers upgrade. That is the property the gate (resolve) cannot give.
Framed bytes → reader-layout framed bytes for a MIXED-schema stream: per-message writer lookup (like recordReader) fused with drift translation (like resolvingRecord), handing back Array[Byte] rather than a typed A or an F[A]. Per payload: strip, resolve-decode the body writer → reader, re-encode under readerSchema, re-frame under frameId. Because the output is reader-layout, it is stable across writer-schema evolution within a reader generation — so digests, slices, and forwards computed downstream don't churn when producers upgrade. That is the property the gate (resolve) cannot give.
A factory, like confluent: the returned function closes over a per-writer-id cache (java.util.concurrent.ConcurrentHashMap), so schemaById is consulted once per DISTINCT writer id — one entry per schema version seen on the stream. Keep the returned function; re-calling resolvingBytes per message discards the cache.
Failures as Left: AvroFailure.NotConfluentFramed (bad or null frame, checked before schemaById), AvroFailure.SchemaResolutionFailed (the hook threw), AvroFailure.ResolveFailed / AvroFailure.EncodeFailed (resolve-decode / re-encode failed).
Attributes
- Source
- ConfluentWire.scala
Generic counterpart of resolving — resolves readSchema → the caller-supplied writeSchema into an IndexedRecord, for when no reader codec / case class exists.
Generic counterpart of resolving — resolves readSchema → the caller-supplied writeSchema into an IndexedRecord, for when no reader codec / case class exists.
Attributes
- Source
- ConfluentWire.scala
Validate and strip the 5-byte Confluent header. Fails structurally (AvroFailure.NotConfluentFramed) on a null payload (a Kafka tombstone / mis-produced record — a defined failure rather than an NPE), on inputs shorter than the header, or whose magic byte isn't 0x00.
Validate and strip the 5-byte Confluent header. Fails structurally (AvroFailure.NotConfluentFramed) on a null payload (a Kafka tombstone / mis-produced record — a defined failure rather than an NPE), on inputs shorter than the header, or whose magic byte isn't 0x00.
Attributes
- Source
- ConfluentWire.scala
Concrete fields
Header length: 1 magic byte + 4 big-endian schema-id bytes.
The Confluent magic byte — always 0x00 in the current wire format.
The Confluent magic byte — always 0x00 in the current wire format.
Attributes
- Source
- ConfluentWire.scala