Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.10438 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916689103290368 |
|---|---|
| author | Sotolongo, Daniel Mills, Daniel Akidau, Tyler Santhiar, Anirudh Tóth, Attila-Péter Battiston, Ilaria Sharma, Ankur Huang, Botong Zhang, Boyuan Pauliukevich, Dzmitry Sartorello, Enrico Belianski, Igor Kalev, Ivan Benson, Lawrence Papke, Leon Geng, Ling Uhlar, Matt Shah, Nikhil Semmler, Niklas Zhou, Olivia Nowak, Saras Lionheart, Sasha Merker, Till Lifliand, Vlad Grus, Wendy Huang, Yi Zhu, Yiwen |
| author_facet | Sotolongo, Daniel Mills, Daniel Akidau, Tyler Santhiar, Anirudh Tóth, Attila-Péter Battiston, Ilaria Sharma, Ankur Huang, Botong Zhang, Boyuan Pauliukevich, Dzmitry Sartorello, Enrico Belianski, Igor Kalev, Ivan Benson, Lawrence Papke, Leon Geng, Ling Uhlar, Matt Shah, Nikhil Semmler, Niklas Zhou, Olivia Nowak, Saras Lionheart, Sasha Merker, Till Lifliand, Vlad Grus, Wendy Huang, Yi Zhu, Yiwen |
| contents | Streaming data pipelines remain challenging and expensive to build and maintain, despite significant advancements in stronger consistency, event time semantics, and SQL support over the last decade. Persistent obstacles continue to hinder usability, such as the need for manual incrementalization, semantic discrepancies across SQL implementations, and the lack of enterprise-grade operational features. While the rise of incremental view maintenance (IVM) as a way to integrate streaming with databases has been a huge step forward, transaction isolation in the presence of IVM remains underspecified, leaving the maintenance of application-level invariants as a painful exercise for the user. Meanwhile, most streaming systems optimize for latencies of 100 ms to 3 sec, whereas many practical use cases are well-served by latencies ranging from seconds to tens of minutes.
We present delayed view semantics (DVS), a conceptual foundation that bridges the semantic gap between streaming and databases, and introduce Dynamic Tables, Snowflake's declarative streaming transformation primitive designed to democratize analytical stream processing. DVS formalizes the intuition that stream processing is primarily a technique to eagerly compute derived results asynchronously, while also addressing the need to reason about the resulting system end to end. Dynamic Tables then offer two key advantages: ease of use through DVS, enterprise-grade features, and simplicity; as well as scalable cost efficiency via IVM with an architecture designed for diverse latency requirements.
We first develop extensions to transaction isolation that permit the preservation of invariants in streaming applications. We then detail the implementation challenges of Dynamic Tables and our experience operating it at scale. Finally, we share insights into user adoption and discuss our vision for the future of stream processing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_10438 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Streaming Democratized: Ease Across the Latency Spectrum with Delayed View Semantics and Snowflake Dynamic Tables Sotolongo, Daniel Mills, Daniel Akidau, Tyler Santhiar, Anirudh Tóth, Attila-Péter Battiston, Ilaria Sharma, Ankur Huang, Botong Zhang, Boyuan Pauliukevich, Dzmitry Sartorello, Enrico Belianski, Igor Kalev, Ivan Benson, Lawrence Papke, Leon Geng, Ling Uhlar, Matt Shah, Nikhil Semmler, Niklas Zhou, Olivia Nowak, Saras Lionheart, Sasha Merker, Till Lifliand, Vlad Grus, Wendy Huang, Yi Zhu, Yiwen Databases Streaming data pipelines remain challenging and expensive to build and maintain, despite significant advancements in stronger consistency, event time semantics, and SQL support over the last decade. Persistent obstacles continue to hinder usability, such as the need for manual incrementalization, semantic discrepancies across SQL implementations, and the lack of enterprise-grade operational features. While the rise of incremental view maintenance (IVM) as a way to integrate streaming with databases has been a huge step forward, transaction isolation in the presence of IVM remains underspecified, leaving the maintenance of application-level invariants as a painful exercise for the user. Meanwhile, most streaming systems optimize for latencies of 100 ms to 3 sec, whereas many practical use cases are well-served by latencies ranging from seconds to tens of minutes. We present delayed view semantics (DVS), a conceptual foundation that bridges the semantic gap between streaming and databases, and introduce Dynamic Tables, Snowflake's declarative streaming transformation primitive designed to democratize analytical stream processing. DVS formalizes the intuition that stream processing is primarily a technique to eagerly compute derived results asynchronously, while also addressing the need to reason about the resulting system end to end. Dynamic Tables then offer two key advantages: ease of use through DVS, enterprise-grade features, and simplicity; as well as scalable cost efficiency via IVM with an architecture designed for diverse latency requirements. We first develop extensions to transaction isolation that permit the preservation of invariants in streaming applications. We then detail the implementation challenges of Dynamic Tables and our experience operating it at scale. Finally, we share insights into user adoption and discuss our vision for the future of stream processing. |
| title | Streaming Democratized: Ease Across the Latency Spectrum with Delayed View Semantics and Snowflake Dynamic Tables |
| topic | Databases |
| url | https://arxiv.org/abs/2504.10438 |