Context Propagation
It can be convenient to attach metadata to each element in the stream.
For example, when reading from an external data source it can be useful to keep track of the read offset, so it can be marked as processed when the element reaches the Sink
.
For this use case we provide the SourceWithContext
and FlowWithContext
variations on Source
and Flow
.
Essentially, a FlowWithContext
is just a Flow
that contains tuples of element and context, but the advantage is in the operators: most operators on FlowWithContext
will work on the element rather than on the tuple , allowing you to focus on your application logic rather without worrying about the context.
Restrictions¶
Not all operations that are available on Flow
are also available on FlowWithContext
. This is intentional: in the use case of keeping track of a read offset, if the FlowWithContext
was allowed to arbitrarily filter and reorder the stream, the Sink
would have no way to determine whether an element was skipped or merely reordered and still in flight.
For this reason, FlowWithContext
allows filtering operations (such as filter
, filterNot
, collect
, etc.) and grouping operations (such as grouped
, sliding
, etc.) but not reordering operations (such as mapAsyncUnordered
and statefulMapConcat
). Finally, also ‘one-to-n’ operations such as mapConcat
are allowed.
Filtering operations will drop the context along with dropped elements, while grouping operations will keep all contexts from the elements in the group. Streaming one-to-many operations such as mapConcat
associate the original context with each of the produced elements.
As an escape hatch, there is a via
operator that allows you to insert an arbitrary Flow
that can process the tuples of elements and context in any way desired. When using this operator, it is the responsibility of the implementor to make sure this Flow
does not perform any operations (such as reordering) that might break assumptions made by the Sink
consuming the context elements.
Creation¶
The simplest way to create a SourceWithContext
is to first create a regular Source
with elements from which the context can be extracted, and then use Source.asSourceWithContext.
Composition¶
When you have a SourceWithContext
source
that produces elements of type Foo
with a context of type Ctx
, and a Flow
flow
from Foo
to Bar
, you cannot simply source.via(flow)
to arrive at a SourceWithContext
that produces elements of type Bar
with contexts of type Ctx
. The reason for this is that flow
might reorder the elements flowing through it, making via
challenging to implement.
Due to this there is a unsafeDataVia
that can be used instead however no protection is offered to prevent reordering or dropping/duplicating elements from stream so use this operation with great care.
There is also a Flow.asFlowWithContext which can be used when the types used in the inner Flow
have room to hold the context. If this is not the case, a better solution is usually to build the flow from the ground up as a FlowWithContext
, instead of first building a Flow
and trying to convert it to FlowWithContext
after-the-fact.