I'm increasingly of the opinion that "basically a data warehouse, but incrementally maintained materialized views under the hood as an implementation detail" (e.g. Materialize, Feldera, anything based on Differential Dataflow / DBSP) is a good point in the design space. I get that they're more complicated than batch, and for many orgs the low latency isn't necessary for pure analytics/dashboards.
I wonder if there's a place for such architecture in the operational path, not just analytics. If you squint, "this microservice reads data from this operational DB and passes it to this other microservice / puts it in a cache / sends it in response to a request" looks sort of like an "ad hoc, informally specified, bug-ridden, slow"[0] implementation of incremental view maintenance (in that a cache, or a service's data model in memory, is a sort of "materialized view" over the source data). I've seen some success in replacing a tangle of imperative-languages-and-network-requests with a declarative, incrementally maintained model in SQL, if the read latency can be kept under control.
I'm curious about the "some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache" part. could you expand on that?
[and yes, I do find these sort of problems interesting! :)]
I’m not sure you have to squint; an mview is just a cached query. The only difference between an mview and a cache is the “incremental” part of the equation.
That is, my only disagreement is the lack of gusto
It's not a new or exotic conceptualization, it's a recovery of the semantics that Codd had in mind and well defined in fact-oriented modeling approaches like object-role modeling / NIAM and FCO-IM. It provides conceptual parsimony: attributes are just relationships between entities and values, and n-ary relationships are handled consistently rather than being reduced to binary relationships, among other things. And if more people adopted this, we might get better data abstraction layers incorporated into development frameworks than object-relational mappers (which should rather be renamed to "network data model to SQL mappers").
I wonder if there's a place for such architecture in the operational path, not just analytics. If you squint, "this microservice reads data from this operational DB and passes it to this other microservice / puts it in a cache / sends it in response to a request" looks sort of like an "ad hoc, informally specified, bug-ridden, slow"[0] implementation of incremental view maintenance (in that a cache, or a service's data model in memory, is a sort of "materialized view" over the source data). I've seen some success in replacing a tangle of imperative-languages-and-network-requests with a declarative, incrementally maintained model in SQL, if the read latency can be kept under control.
I'm curious about the "some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache" part. could you expand on that?
[and yes, I do find these sort of problems interesting! :)]
[0] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule