variadics_macro v0.6.2

Bug Fixes

• refactor github actions workflows, make stable the default toolchain

Commit Statistics

• 2 commits contributed to the release.
• 1 commit was understood as conventional.
• 1 unique issue was worked on: #2028

Commit Details

view details

• #2028
  • Refactor github actions workflows, make stable the default toolchain (c40876e)
• Uncategorized
  • Release hydro_build_utils v0.0.1, dfir_lang v0.15.0, dfir_macro v0.15.0, variadics v0.0.10, sinktools v0.0.1, hydro_deploy_integration v0.15.0, lattices_macro v0.5.11, variadics_macro v0.6.2, lattices v0.6.2, multiplatform_test v0.6.0, dfir_rs v0.15.0, copy_span v0.1.0, hydro_deploy v0.15.0, hydro_lang v0.15.0, hydro_std v0.15.0, safety bump 5 crates (092de25)

sinktools v0.0.1

Bug Fixes

• missing description in Cargo.toml

New Features

• add LazySink and LazySource

• DemuxMap, use in hydro_deploy_integration

• sinktools crate
  this also converts variadic to use #[no_std], and adds
  feature="std", and fixes an issue causing trybuild tests to not run

  will replace pusherator in upcoming PR

Commit Statistics

• 5 commits contributed to the release.
• 4 commits were understood as conventional.
• 3 unique issues were worked on: #2157, #2163, #2182

Commit Details

view details

• #2157
  • sinktools crate (7efe1dc)
• #2163
  • DemuxMap, use in hydro_deploy_integration (335ded3)
• #2182
  • Add LazySink and LazySource (ef6f313)
• Uncategorized
  • Missing description in Cargo.toml (7925343)
  • Release hydro_build_utils v0.0.1, dfir_lang v0.15.0, dfir_macro v0.15.0, variadics v0.0.10, sinktools v0.0.1, hydro_deploy_integration v0.15.0, lattices_macro v0.5.11, variadics_macro v0.6.2, lattices v0.6.2, multiplatform_test v0.6.0, dfir_rs v0.15.0, copy_span v0.1.0, hydro_deploy v0.15.0, hydro_lang v0.15.0, hydro_std v0.15.0, safety bump 5 crates (092de25)

multiplatform_test v0.6.0

Chore

• update wasm

Commit Statistics

• 2 commits contributed to the release.
• 1 commit was understood as conventional.
• 1 unique issue was worked on: #2087

Commit Details

view details

• #2087
  • Update wasm (e5d9080)
• Uncategorized
  • Release hydro_build_utils v0.0.1, dfir_lang v0.15.0, dfir_macro v0.15.0, variadics v0.0.10, sinktools v0.0.1, hydro_deploy_integration v0.15.0, lattices_macro v0.5.11, variadics_macro v0.6.2, lattices v0.6.2, multiplatform_test v0.6.0, dfir_rs v0.15.0, copy_span v0.1.0, hydro_deploy v0.15.0, hydro_lang v0.15.0, hydro_std v0.15.0, safety bump 5 crates (092de25)

lattices_macro v0.5.11

Bug Fixes

• refactor github actions workflows, make stable the default toolchain
• hardcoded crate name issues

Other

• ensure hydro_build_utils is published in the correct order

Test

• exclude crate/module in lib snapshot file names [ci-full]
  Also removes unused
  hydro_lang/src/compile/ir/snapshots/hydro_lang__compile__ir__backtrace__tests__backtrace.snap

Commit Statistics

• 5 commits contributed to the release.
• 4 commits were understood as conventional.
• 3 unique issues were worked on: #1977, #2028, #2283

Commit Details

view details

• #1977
  • Hardcoded crate name issues (5ec8b3b)
• #2028
  • Refactor github actions workflows, make stable the default toolchain (c40876e)
• #2283
  • Exclude crate/module in lib snapshot file names [ci-full] (dc170e6)
• Uncategorized
  • Release hydro_build_utils v0.0.1, dfir_lang v0.15.0, dfir_macro v0.15.0, variadics v0.0.10, sinktools v0.0.1, hydro_deploy_integration v0.15.0, lattices_macro v0.5.11, variadics_macro v0.6.2, lattices v0.6.2, multiplatform_test v0.6.0, dfir_rs v0.15.0, copy_span v0.1.0, hydro_deploy v0.15.0, hydro_lang v0.15.0, hydro_std v0.15.0, safety bump 5 crates (092de25)
  • Ensure hydro_build_utils is published in the correct order (806a623)

lattices v0.6.2

Chore

• update pinned nightly to 2025-08-20, fix lints

New Features

• Add efficient tombstone storage for lattices (RoaringBitmap + FST)
  This PR implements space-efficient tombstone storage for
  SetUnionWithTombstones and MapUnionWithTombstones using
  RoaringBitmap (for integers) and FST (for strings).

• sinktools crate
  this also converts variadic to use #[no_std], and adds
  feature="std", and fixes an issue causing trybuild tests to not run

  will replace pusherator in upcoming PR

Bug Fixes

• refactor github actions workflows, make stable the default toolchain

Other

• ensure hydro_build_utils is published in the correct order

Commit Statistics

• 6 commits contributed to the release.
• 5 commits were understood as conventional.
• 4 unique issues were worked on: #2024, #2028, #2157, #2271

Commit Details

view details

• #2024
  • Update pinned nightly to 2025-08-20, fix lints (97426b8)
• #2028
  • Refactor github actions workflows, make stable the default toolchain (c40876e)
• #2157
  • sinktools crate (7efe1dc)
• #2271
  • Add efficient tombstone storage for lattices (RoaringBitmap + FST) (1f05026)
• Uncategorized
  • Release hydro_build_utils v0.0.1, dfir_lang v0.15.0, dfir_macro v0.15.0, variadics v0.0.10, sinktools v0.0.1, hydro_deploy_integration v0.15.0, lattices_macro v0.5.11, variadics_macro v0.6.2, lattices v0.6.2, multiplatform_test v0.6.0, dfir_rs v0.15.0, copy_span v0.1.0, hydro_deploy v0.15.0, hydro_lang v0.15.0, hydro_std v0.15.0, safety bump 5 crates (092de25)
  • Ensure hydro_build_utils is published in the correct order (806a623)

hydro_std v0.15.0

Documentation

• new template and walkthrough

New Features

• add cluster-membership ir node

• introduce sliced! syntax for processing with anonymous ticks

• replay failed simulation instances and show logs
  Previously, a failed simulation would only show the backtrace, and would
  have to be re-run with HYDRO_SIM_LOG=1 to display the simulation
  steps. This would result in lots of log pollution since logging would
  also be enabled for passing instances.

  Now, we've made some tweaks to bolero so that it re-executes the test
  after finding a failing input, passing an is_replay flag. We use this
  to enable rich logging, so that the failure is displayed with all the
  simulation steps that led to it.

• add APIs for unordered stream assertions and tests for collect_quorum
  AI Disclosure: all the tests were generated using Kiro (!)

  Currently, the core functionality test has to be split up into several
  exhaustive units because otherwise the search space becomes too large
  for CI (~400s). Eventually, keyed streams may help but splitting up the
  test is a reasonable short-term fix.

• deterministic simulator for Hydro programs
  This introduces a deterministic simulator for Hydro that can simulate
  various asynchronous scenarios to identify bugs in code that use
  non-deterministic operators, such as batch. This PR focuses on just
  the infrastructure and support for simulating batch on a
  totally-ordered, exactly-once stream. Support for additional
  non-deterministic operators will follow in separate PRs (currently, an
  exception is thrown to prevent use of the simulator on programs that use
  such operators).

  The simulator's job is to explore the space of potential asynchronous
  executions. Because "top-level" operators guarantee "eventual
  determinism" (per Flo), we do not need to simulate every possible
  interleaving of message arrivals and processing. Instead, we only need
  to simulate sources of non-determinism at the points in the program
  where a user intentionally observes them (such as batch or
  assume_ordering).

  When compiling a Hydro program for the simulator, we emit several DFIR
  programs. One of these is the async_dfir, which contains all
  asynchronously executed top-level operators in the Hydro program. Again,
  thanks to Flo semantics, we do not need to simulate the behavior of
  executing these operators on different prefixes of the input, since we
  know that none of the downstream operators change their behavior based
  on the provided prefix (this is somewhat more complicated for unbounded
  singletons, whose intermediate states are affected by the set of
  elements processed in each batch, but we will address this separately).

  Because each tick relies on a set of decisions being made to select
  their inputs (batch, snapshot), we emit each tick's code into a
  separate DFIR graph. The top-level simulator then schedules
  (LaunchedSim::scheduler) the async graph and tick graphs by always
  trying to make progress with the async graph first (so that we have the
  full set of possible inputs at each batch boundary), and when the async
  graph cannot make any further progress it selects one of the ticks,
  makes a batching decision for each of its inputs
  (autonomous_decision), and then executes the tick.

  The selection of which tick to run and which elements to release in each
  batch are driven by a source of non-determinism, which is either:
  a) libfuzzer (if using .sim().fuzz and running with cargo sim)
  b) a RNG with 8192 iterations (if using .sim().fuzz and running with
  cargo test and no reproducer is available)
  c) a static input of decisions (if using .sim().fuzz and running with
  cargo test and a reproducer is available)
  d) an exhaustive, depth-first search algorithm (if using
  .sim().exhaustive and running with cargo test)

  Whenever a fuzzer finds a failure, it stores the sequence of decisions
  that leads to the crash in a .bin file as a reproducer, so that we can
  re-execute quickly in testing environments.

  Because Hydro uses a staged compilation model, our approach to compiling
  and executing the Hydro program is also a bit unique. Because the fuzzer
  needs to track branch coverage inside the Hydro program to be effective,
  and because we need low-latency interactions between the user assertions
  and the Hydro code, we cannot run the compiled program in a separate
  process. Instead, we compile the Hydro code into a shared library and
  dynamically load it into the host process (which has the testing code).
  The shared library only provides the compiled DFIR graphs, so the
  simulator scheduler runs in the host process to enable low-latency
  switching between the DFIR and testing code.

  This PR includes a couple of toy examples testing the simulator's
  functionality. Of particular interest is
  sim_crash_with_fuzzed_batching, which takes forever with an exhaustive
  search but quickly finds a failure case with a fuzzer.

• Generalize bench_client's workload generation
  Allow custom functions for generating bench_client workloads (beyond
  u32,u32).

• add keyed singletons and optionals to eliminate unsafety in membership tracking

Bug Fixes

• Staggered client
  Client initially outputs 1 message per tick instead of all messages in 1
  giant batch, so if downstream operators do not dynamically adjust the
  batch size, they are not overwhelmed
• Client aggregator waits for all clients before outputting
  Outputs N/A as throughput and latency until all clients have responded
  with positive throughput.
• refactor github actions workflows, make stable the default toolchain
• remove strange use of batching in bench_client

Refactor

• reduce syntactic overhead of connecting test inputs / outputs
  Rather than having separate source / sink / bytes / bincode
  APIs, we use a single connect method that uses a trait to resolve the
  appropriate connection result.

• reduce atomic pollution in quorum counting
  Also fixes a sync bug in Compartmentalized Paxos. Due to batching
  semantics, we could end up in a situation where responses have missing
  metadata (for example if the batch refuses to release any elements).

  The simulator would hopefully have caught this, we can use this as an
  example.

• remove uses of legacy *_keyed APIs
  Also adds missing doctests to the aggregation APIs on KeyedStream.

Test

• add test for collecting unordered quorum

Documentation (BREAKING)

• add docs for for_each / dest_sink, restrict to strict streams
  Breaking Change: for_each / dest_sink now require a totally-ordered,
  retry-free stream since the downstream may not tolerate such
  non-determinism. This helps highlight cases where the developer needs to
  reason carefully about the safety of their side effects.

New Features (BREAKING)

• add specialized sim_input / sim_output to reduce simulation boilerplate

• cluster member ids are now clone instead of copy
  This is part one of a series of changes. The first part just changes
  MemberIds from being Copy to Clone, so that they can later support more
  use cases.

• restrict KeyedSingleton to not allow asynchronous key removal
  Previously, KeyedSingleton supported filter on unbounded collections
  and latest to yield snapshots from a tick. Both of these APIs are
  problematic because they let developers asynchronously remove elements
  from the collection, which goes against the "Singleton" naming. We have
  reasonable alternatives for all these use-sites, so remove for now to
  minimize the semantic complexity (also for the simulator).

  Eventually, we will want to bring back KeyedOptional which permits
  such asynchronous removal.

• add Ordering and Retries traits with basic axioms
  Reduces the need to use nondet! inside broadcast APIs, and also
  prepares us for graph viz metadata by requiring these traits to be
  implemented for the type parameters in Stream / KeyedStream.

• refine types when values of a KeyedSingleton are immutable
  There are many cases where once a value is released into a
  KeyedSingleton, it will never be changes. In such cases, we can permit
  APIs like .entries even if the set of entries is growing.

  We use this type to improve the API surface for looking up values for a
  request. Now, a KeyedSingleton of requests can look up values from
  another KeyedSingleton.

Bug Fixes (BREAKING)

• fix cardinality for Optional::or
  Using HydroNode::Chain is very dangerous for singletons / optionals,
  because it can lead to cardinality > 1 within a single batch, which
  breaks a fundamental invariant.

  This introduces a new HydroNode::ChainFirst operator that only emits
  the first value from the chain. This is paired with a new DFIR operator
  chain_first for this behavior.

  We also rewrite some Singleton logic to use Optional under the hood,
  which reduces the places where we deal with chaining in the IR,
  hopefully avoiding future incidents.

Refactor (BREAKING)

• don't return results from SimSender::send
  Also makes the assert_* APIs on SimReceiver more general to support
  asymmetric PartialEq
• allow sim feature without deploy feature
  Also removes leftover prototype "properties" code.
• migrate Hydro IR to have Flo semantics at all levels
  Previousl...

hydro_lang v0.15.0

Chore

• update to rust 1.91.1

• update back to Rust 1.90.0

• pin Rust to 1.89
  GitHub Actions is going to be flip/flopping between 1.89 and 1.90, and
  we have users who need support on 1.89.

  Also updates precheck.bash to use nextest for speed.

• update nightly snapshots
  [ci-full]

  This PR was automatically generated by the nightly CI run. Updated insta
  snapshots for nightly Rust builds. Please review the snapshot changes to
  ensure they are expected.

• clean up leftover snapshots

• move backtrace test to separate module for stable line numbers
  Will reduce diff noise whenever modifying Stream.

• update nightly snapshots

• update nightly snapshots
  [ci-full]

  This PR was automatically generated by the nightly CI run. Updated insta
  snapshots for nightly Rust builds. Please review the snapshot changes to
  ensure they are expected.

• update nightly snapshots
  [ci-full]

  This PR was automatically generated by the nightly CI run. Updated insta
  snapshots for nightly Rust builds. Please review the snapshot changes to
  ensure they are expected.

• update nightly snapshots
  [ci-full]

  This PR was automatically generated by the nightly CI run. Updated insta
  snapshots for nightly Rust builds. Please review the snapshot changes to
  ensure they are expected.

• update nightly snapshots

• update pinned nightly to 2025-08-20, fix lints

Documentation

• single counter tutorial and prepare code for later tutorials
• new template and walkthrough
• remaining Optional-specific APIs
• Stream::persist rustdoc
• remaining KeyedSingleton-specific APIs
  Will document all the defer_ticks in a separate PR.
• start on docs for KeyedSingleton
• Singleton/Optional::into_stream
• Singleton/Optional::all_ticks*
• Singleton/Optional::latest* and ZipResult
• Singleton::zip and Optional::zip
• add basic docs around atomics for stream / keyed streams / singleton / optional
• struct docs for Singleton / Optional, and polish others
• rustdoc for Stream::collect_vec
• rustdoc for KeyedStream::chain
• initial docs for deploy module
• avoid rustfmt thrashing
• runtime_context module
• keyed variants of networking APIs, send external
• polish for networking APIs
• start on docs for live collections
• document the nondet module
• add doc(cfg(...)) annotation for graph module
• docs for backtrace module
  Also cleans up the API interface, adds the colno field, and a unit
  test showing that we can precisely identify each element of chained
  operators.
• docs for test_util module
• set up lints so we can find missing docs
• keyed stream docs
• update crate description

New Features

• add KeyedStream APIs for compound keys and sending with keys
  Previously, the only way to send a KeyedStream including the keys
  was to get the entries (which would erase the order).

  Now, we add APIs to prefix/unprefix the key (prepend a tuple element),
  and then add a KeyedStream::send_bincode API that sends the entire set
  of entries with the sender ID prefixed in the return type. We also add
  an inverse KeyedStream::demux_bincode where the input keys are a
  compound of (MemberId, Key) and the recipient gets a
  KeyedStream<Key>.

• support KeyedStream::assume_ordering inside ticks in simulator

• support Stream::assume_ordering inside ticks in simulator

• add cluster-membership ir node

• introduce sliced! syntax for processing with anonymous ticks

• add KeyedStream::cross_singleton

• support batching unordered keyed streams in simulator

• add Location::bind_single_client for bidi single client

• support KeyedSingleton::snapshot in sim

• support batching ordered keyed streams in sim

• support atomic regions in sim

• Allow accessing HydroNodes' inputs, assign all HydroNodes (except Placeholder) an ID

• support Cluster -> Cluster network in sim

• support Process -> Cluster network in sim
  Also fixes source_iter to add type hints in generated DFIR.

• support source_cluster_members in sim

• support ticks on clusters in sim

• support Cluster -> Process network in sim

• support Process -> Process network in sim

• Migrate graph visualization to JSON format with deterministic output and semantic tagging

• support Stream::count in simulator
  This also tweaks the behavior of reduce_commutative_idempotent_trusted
  to make sure that we simulate different orders of intermediate states
  when the input is unbounded / asynchronous. Right now, we do not
  actually have any tests for this, since we do not support shuffling
  unbounded streams, but this will be supported soon.

• use dynamically linked libraries in test mode

• share __staged across trybuild targets
  Previously, each trybuild target (in test-mode) would have its own copy
  of the __staged module containing the entire sources of the original
  crate. This creates a huge compilation burden that is repeated on every
  deployment.

  With an accompanying change in Stageleft, we are now able to share the
  __staged module in the common lib.rs. This also moves the simulation
  dylibs to be compiled as an example so that they do not conflict with
  the caching of the shared lib.rs.

  Also reduces thrashing due to the lockfile having to be updated after
  each time it is written (because packages have to be removed which were
  needed in the workspace but not the trybuild).

• support singletons in simulator
  The simulator uses a different representation of singletons than
  production, streaming every intermediate state instead of just the final
  value for the current DFIR batch. This lets us simulate all possible
  snapshots when that happens.

  To support Location::singleton, which (lazily) spins in production, we
  instead introduce a proper SourceSingleton IR node which only
  (appropriately) only releases a single item in the simulator.

  To simulate snapshots, we have to decide which version of the singleton
  to advance to. This can include "stutters", where the same version is
  released multiple times. To avoid an infinite search space, stutters are
  only allowed when some other input to the tick has made a nontrivial
  decision (one that makes progress consuming its input). Therefore, if
  none of the other inputs make progress, the singleton has to make
  progress.

• avoid redundant simulations for ordering within unordered batches
  Previously, we would select elements for the next released batched in
  non-deterministic order. This is unnecessary, because the batch itself
  has NoOrder guarantees and so the permutations within the batch are
  redundant.

  Reduces instance count from 192 to 75 for
  sequence_payloads_sequences_all. To avoid regressions,
  .sim().exhaustive() now returns the number of simulation instances,
  which we can verify against a combinatorial formula.

• replay failed simulation instances and show logs
  Previously, a failed simulation would only show the backtrace, and would
  have to be re-run with HYDRO_SIM_LOG=1 to display the simulation
  steps. This would result in lots of log pollution since logging would
  also be enabled for passing instances.

  Now, we've made some tweaks to bolero so that it re-executes the test
  after finding a failing input, passing an is_replay flag. We use this
  to enable rich logging, so that the failure is displayed with all the
  simulation steps that led to it.

• ensure sim logs show up in captured stdout/stderr
  When running a sim test with cargo test, the stdout / stderr are
  captured. But this capturing mechanism only works for direct calls to
  println! and eprintln!, so the previous logic to use
  std::io::eprintln does not work (and logs are displayed out of order).

  To fix this, we change the logger to be a tri-state LogKind. If it is
  LogKind::Stderr, we directly invoke eprintln. We also pass down
  function pointers for println and eprintln to the staged code, where
  we shadow std::{println, eprintln, debug} with macros that call the
  function pointer. This ensures that any logging statements in quoted
  code show up in the right place next to simulator logs.

• add APIs for unordered stream assertions and tests for collect_quorum
  AI Disclosure: all the tests were generated using Kiro (!)

  Currently, the core functionality test has to be split up into several
  exhaustive units because otherwise the search space becomes too large
  for CI (~400s). Eventually, keyed streams may help but splitting up the
  test is a reasonable short-term fix.

• <csr-id-cea261dd8b585a3c30a8...

hydro_deploy_integration v0.15.0

New Features

• add Location::bind_single_client for bidi single client
• DemuxMap, use in hydro_deploy_integration

Bug Fixes

• compilation on non-Unix [ci-full]
• [ci-full] fix ci issues
  • run docs/website on pr
• use bash by default on windows
• build website with nightly
• increase test timeout to 45m
• run docs with nightly
• make rustfmt use native line endings instead of unix
• fix compile-fail nightly stderr outputs
• add the ability to include [ci-full] into the body/title of a PR to
  get it to run the full stable/nightly linux/windows test matrix.

New Features (BREAKING)

• start adding APIs for various forms of keyed streams "joins", add KVS counter server
  Also fixes incorrect output metadata for KeyedStream::fold, and
  removes KeyedSingleton::all_ticks for now until we a) actually need it
  and b) can think carefully about its semantics.

• expose a stream of membership events for bidi externals

• introduce support for dynamic external clients
  So far, Hydro has only supported deployments where there is a statically
  known number of single-connection external clients for sending /
  receiving data. This is not practical in production applications where
  external users will dynamically connect to the Hydro service.

  This takes the first step towards addressing this, by setting up the
  networking mechanism for handling dynamically connected clients and
  adding an API for sending/receiving data.

  This also changes the source_external_bytes APIs to return the raw
  BytesMut sent over the network, rather than adding a freezing step.

  Finally, we show an end-to-end example using this interface to write a
  simple HTTP server.

Refactor (BREAKING)

• remove unnecessary ConnectedSourceSink trait

Commit Statistics

• 11 commits contributed to the release.
• 10 commits were understood as conventional.
• 10 unique issues were worked on: #1966, #1978, #1980, #1981, #1996, #2029, #2031, #2163, #2246, #2267

Commit Details

view details

• #1966
  • Introduce support for dynamic external clients (5d40134)
• #1978
  • Remove unnecessary ConnectedSourceSink trait (97b879d)
• #1980
  • Make polling fair with round-robin (6b37888)
• #1981
  • Expose a stream of membership events for bidi externals (1bb2262)
• #1996
  • Start adding APIs for various forms of keyed streams "joins", add KVS counter server (f9f5317)
• #2029
  • [ci-full] fix ci issues (4e84a3f)
• #2031
  • Use round-robin polling for MergeSource to ensure fairness (92adacc)
• #2163
  • DemuxMap, use in hydro_deploy_integration (335ded3)
• #2246
  • Add Location::bind_single_client for bidi single client (4a43594)
• #2267
  • Compilation on non-Unix [ci-full] (e5f2004)
• Uncategorized
  • Release hydro_build_utils v0.0.1, dfir_lang v0.15.0, dfir_macro v0.15.0, variadics v0.0.10, sinktools v0.0.1, hydro_deploy_integration v0.15.0, lattices_macro v0.5.11, variadics_macro v0.6.2, lattices v0.6.2, multiplatform_test v0.6.0, dfir_rs v0.15.0, copy_span v0.1.0, hydro_deploy v0.15.0, hydro_lang v0.15.0, hydro_std v0.15.0, safety bump 5 crates (092de25)

hydro_deploy v0.15.0

Chore

• only depend on samply on supported systems, remove leftover DTrace code
  Minimizes dependencies we pull in on Unix.
• update pinned nightly to 2025-08-20, fix lints

New Features

• make glibc vs musl configurable for cloud deployments

• use dynamically linked libraries in test mode

• share __staged across trybuild targets
  Previously, each trybuild target (in test-mode) would have its own copy
  of the __staged module containing the entire sources of the original
  crate. This creates a huge compilation burden that is repeated on every
  deployment.

  With an accompanying change in Stageleft, we are now able to share the
  __staged module in the common lib.rs. This also moves the simulation
  dylibs to be compiled as an example so that they do not conflict with
  the caching of the shared lib.rs.

  Also reduces thrashing due to the lockfile having to be updated after
  each time it is written (because packages have to be removed which were
  needed in the workspace but not the trybuild).

• add support for aws via terraform

Bug Fixes

• fix full path replacement for staged code errors
• make cargo sim fuzzing command work again [ci-full]
  Accidentally broken when we moved to sharing the trybuild-lib as a
  dylib.
• simulator on Windows and test macOS in CI [ci-full]
  Now that we have a bunch of platform-specific bits for dynamic
  libraries, we should test all our supported dev platforms.
• don't emit duplicate leafs for building Rust service
  Because the .build() API creates a leaf itself, there is no need to
  wrap it in another leaf (which results in two rows in the progress
  display).

New Features (BREAKING)

• introduce support for dynamic external clients
  So far, Hydro has only supported deployments where there is a statically
  known number of single-connection external clients for sending /
  receiving data. This is not practical in production applications where
  external users will dynamically connect to the Hydro service.

  This takes the first step towards addressing this, by setting up the
  networking mechanism for handling dynamically connected clients and
  adding an API for sending/receiving data.

  This also changes the source_external_bytes APIs to return the raw
  BytesMut sent over the network, rather than adding a freezing step.

  Finally, we show an end-to-end example using this interface to write a
  simple HTTP server.

Refactor (BREAKING)

• remove deprecated add_connection API
  Has not been used for a long while...

Commit Statistics

• 14 commits contributed to the release.
• 12 commits were understood as conventional.
• 12 unique issues were worked on: #1966, #1967, #2024, #2083, #2177, #2187, #2191, #2192, #2194, #2205, #2269, #2288

Commit Details

view details

• #1966
  • Introduce support for dynamic external clients (5d40134)
• #1967
  • Remove deprecated add_connection API (9e07af1)
• #2024
  • Update pinned nightly to 2025-08-20, fix lints (97426b8)
• #2083
  • Add support for aws via terraform (1dd2606)
• #2177
  • Only depend on samply on supported systems, remove leftover DTrace code (83b77a1)
• #2187
  • Share __staged across trybuild targets (a65893a)
• #2191
  • Don't emit duplicate leafs for building Rust service (dad4abe)
• #2192
  • Use dynamically linked libraries in test mode (891a25d)
• #2194
  • Simulator on Windows and test macOS in CI [ci-full] (6d2e7d0)
• #2205
  • Make cargo sim fuzzing command work again [ci-full] (807eb0a)
• #2269
  • Make glibc vs musl configurable for cloud deployments (fc3bf5b)
• #2288
  • Fix full path replacement for staged code errors (f0acd7c)
• Uncategorized
  • Release sinktools v0.0.1, hydro_deploy_integration v0.15.0, lattices_macro v0.5.11, variadics_macro v0.6.2, lattices v0.6.2, multiplatform_test v0.6.0, dfir_rs v0.15.0, copy_span v0.1.0, hydro_deploy v0.15.0, hydro_lang v0.15.0, hydro_std v0.15.0 (ac88df1)
  • Release hydro_build_utils v0.0.1, dfir_lang v0.15.0, dfir_macro v0.15.0, variadics v0.0.10, sinktools v0.0.1, hydro_deploy_integration v0.15.0, lattices_macro v0.5.11, variadics_macro v0.6.2, lattices v0.6.2, multiplatform_test v0.6.0, dfir_rs v0.15.0, copy_span v0.1.0, hydro_deploy v0.15.0, hydro_lang v0.15.0, hydro_std v0.15.0, safety bump 5 crates (092de25)

dfir_rs v0.15.0

Chore

• update to rust 1.91.1

• update back to Rust 1.90.0

• pin Rust to 1.89
  GitHub Actions is going to be flip/flopping between 1.89 and 1.90, and
  we have users who need support on 1.89.

  Also updates precheck.bash to use nextest for speed.

• update wasm

• update pinned nightly to 2025-08-20, fix lints

Documentation

• consistently refer to Hydro as a framework
• fix "DFIR Playground" link

New Features

• add resolve_futures_blocking for resolving async calls by blocking the subgraph

• DemuxMap, use in hydro_deploy_integration

• sinktools crate
  this also converts variadic to use #[no_std], and adds
  feature="std", and fixes an issue causing trybuild tests to not run

  will replace pusherator in upcoming PR

Bug Fixes

• fix code generation for scan as a push-operator
• update backtrace snapshots for Rust 1.90 and remove unused types
  The line number of core::ops::function::FnOnce::call_once in the
  standard library changed.
• remove python udf support
• [ci-full] fix ci issues
  • run docs/website on pr
• use bash by default on windows
• build website with nightly
• increase test timeout to 45m
• run docs with nightly
• make rustfmt use native line endings instead of unix
• fix compile-fail nightly stderr outputs
• add the ability to include [ci-full] into the body/title of a PR to
  get it to run the full stable/nightly linux/windows test matrix.

Other

• ensure hydro_build_utils is published in the correct order

Refactor

• Make join_fused use new Accumulator trait
  To make the codegen less magical.

  ---

Test

• rearrange sort and anti_join tests out of surface_codegen
  does not change any code, just moves it to new files
• ensure surface_demux_enum actually assert results

Refactor (BREAKING)

• Make DFIR use sinktools for pushing to outputs [ci-bench]
  This allows DFIR to handle dest_sink directly instead of having to
  offload to a separate task, which causes latency and unwanted batching
  (on single threaded runtimes)

  pusherator crate is no longer depended upon and is on a path to removal as it has been
  replaced with Sinks

  Fixes some spanning bugs in codegen that using Sinks revealed

• remove demux operator, replace usage with demux_enum
  This is in preparation of making DFIR support async Sinks. The
  overpowered demux API is incompatible with the more constrained
  mechanics of Sinks.

  Pull Request Overview

  This PR refactors the dfir_rs library by removing the demux operator
  and replacing its usage with the demux_enum operator, which provides
  better type safety and ergonomics.

  • Removes the demux operator implementation and related code
  • Updates examples to use demux_enum with enums for message handling
  • Removes all compile-fail tests specific to the demux operator

• superficially make subgraphs/operators asynchronous
  Subgraphs now are async (create a future when running) which must be
  awaited. However this PR does not actually change any
  subgraphs/operators, so all subgraphs/operators return Poll::Ready(())
  immediately

Pull Request Overview

This PR refactors the dfir_rs library by removing the demux operator
and replacing its usage with the demux_enum operator, which provides
better type safety and ergonomics.

• Removes the demux operator implementation and related code
• Updates examples to use demux_enum with enums for message handling
• Removes all compile-fail tests specific to the demux operator
• superficially make subgraphs/operators asynchronous
  Subgraphs now are async (create a future when running) which must be
  awaited. However this PR does not actually change any
  subgraphs/operators, so all subgraphs/operators return Poll::Ready(())
  immediately

Commit Statistics

• 26 commits contributed to the release.
• 25 commits were understood as conventional.
• 24 unique issues were worked on: #1997, #2010, #2020, #2021, #2022, #2024, #2028, #2029, #2038, #2087, #2091, #2119, #2122, #2132, #2134, #2137, #2147, #2157, #2163, #2170, #2281, #2295, #2309, #2318

Commit Details

view details

• #1997
  • Fix "DFIR Playground" link (273bc1e)
• #2010
  • Check dfir warnings in tests on non-nightly (d057072)
• #2020
  • Consistently refer to Hydro as a framework (2d09687)
• #2021
  • Snapshot tests don't depend on the parent crate name (bdc54eb)
• #2022
  • Add trybuild stable tests (669aefa)
• #2024
  • Update pinned nightly to 2025-08-20, fix lints (97426b8)
• #2028
  • Refactor github actions workflows, make stable the default toolchain (c40876e)
• #2029
  • [ci-full] fix ci issues (4e84a3f)
• #2038
  • Remove python udf support (09c0770)
• #2087
  • Update wasm (e5d9080)
• #2091
  • Superficially make subgraphs/operators asynchronous (2902770)
• #2119
  • Remove demux operator, replace usage with demux_enum (8535940)
• #2122
  • Update backtrace snapshots for Rust 1.90 and remove unused types (1b818e3)
• #2132
  • Ensure surface_demux_enum actually assert results (562d108)
• #2134
  • Make DFIR use sinktools for pushing to outputs [ci-bench] (9d943ac)
• #2137
  • Pin Rust to 1.89 (bcef055)
• #2147
  • Fix code generation for scan as a push-operator (dd62edf)
• #2157
  • sinktools crate (7efe1dc)
• #2163
  • DemuxMap, use in hydro_deploy_integration (335ded3)
• #2170
  • Update back to Rust 1.90.0 (623ed9e)
• #2281
  • Add resolve_futures_blocking for resolving async calls by blocking the subgraph (f7ecb53)
• #2295
  • Update to rust 1.91.1 (d22507c)
• #2309
  • Rearrange sort and anti_join tests out of surface_codegen (356292e)
• #2318
  • Make join_fused use new Accumulator trait (f4a26b3)
• Uncategorized
  • Release hydro_build_utils v0.0.1, dfir_lang v0.15.0, dfir_macro v0.15.0, variadics v0.0.10, sinktools v0.0.1, hydro_deploy_integration v0.15.0, lattices_macro v0.5.11, variadics_macro v0.6.2, lattices v0.6.2, multiplatform_test v0.6.0, dfir_rs v0.15.0, copy_span v0.1.0, hydro_deploy v0.15.0, hydro_lang v0.15.0, hydro_std v0.15.0, safety bump 5 crates (092de25)
  • Ensure hydro_build_utils is published in the correct order (806a623)