Implement basic Aggregate Merge Engine #2255
Open
+6,537
−23
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
platinumhamburg
changed the title
platinumhamburg
force-pushed
the
base-agg
branch
21 times, most recently
from
907559e to
e7d0af7
Compare
platinumhamburg
force-pushed
the
base-agg
branch
from
e7d0af7 to
d34b27a
Compare
There was a problem hiding this comment.
Pull request overview
This PR implements the Aggregation Merge Engine for Fluss, which allows field-level aggregation of rows with the same primary key. The implementation includes 12 aggregate functions (sum, product, max, min, last_value, last_value_ignore_nulls, first_value, first_value_ignore_nulls, listagg, string_agg, bool_and, bool_or) with comprehensive schema evolution support.
Key changes:
- Core aggregation engine with field-level aggregators and caching
- Schema API enhancements to support aggregation function configuration
- Comprehensive documentation with examples for all supported functions
- Utility classes for string concatenation and value comparison
Reviewed changes
Copilot reviewed 56 out of 56 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| website/docs/table-design/table-types/pk-table/merge-engines/aggregation.md | Comprehensive documentation for the Aggregation Merge Engine with examples |
| fluss-server/src/main/java/org/apache/fluss/server/kv/rowmerger/AggregateRowMerger.java | Main row merger implementation with partial update support |
| fluss-server/src/main/java/org/apache/fluss/server/kv/rowmerger/aggregate/*.java | Aggregation context, caching, and field processing logic |
| fluss-server/src/main/java/org/apache/fluss/server/kv/rowmerger/aggregate/functions/*.java | Field aggregator implementations for all functions |
| fluss-server/src/main/java/org/apache/fluss/server/kv/rowmerger/aggregate/factory/*.java | Factory classes for creating aggregators via SPI |
| fluss-common/src/main/java/org/apache/fluss/metadata/AggFunction.java | Enum defining all supported aggregation functions |
| fluss-common/src/main/java/org/apache/fluss/metadata/Schema.java | Schema enhancements to support aggregation functions |
| fluss-common/src/main/java/org/apache/fluss/utils/*.java | Utility classes for string operations and comparisons |
| fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java | New configuration options for aggregation |
| Test files | Comprehensive test coverage for all components |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| * <p>Efficiently encodes schema ID and target column indices for cache lookup. Uses array | ||
| * content-based equality and hashCode for correct cache behavior. | ||
| */ | ||
| private static class CacheKey { |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #2254
Brief change log
Tests
API and Format
Documentation