[FLUSS-CLIENT] Return log end offset in UpsertResult and DeleteResult for exactly-once semantics #2530
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… for exactly-once semantics This change enhances the UpsertWriter API to return bucket and log end offset information in UpsertResult and DeleteResult, which is essential for implementing exactly-once semantics in Flink sink connectors. Changes: - Add bucket and logEndOffset fields to UpsertResult and DeleteResult - Extend WriteCallback interface to include bucket and logEndOffset parameters - Add log_end_offset field to PbPutKvRespForBucket protobuf message - Update Sender to propagate log end offset from server response to callbacks - Add sendWithResult() method in AbstractTableWriter for result-aware writes - Add integration tests for log end offset tracking The log end offset (LEO) represents the offset of the next record to be written, allowing Flink sink to track checkpoint offsets for exactly-once delivery.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change enhances the UpsertWriter API to return bucket and log end offset information in UpsertResult and DeleteResult, which is essential for implementing exactly-once semantics in Flink sink connectors.
Changes:
The log end offset (LEO) represents the offset of the next record to be written, allowing Flink sink to track checkpoint offsets for exactly-once delivery.
Purpose
Linked issue: close #2529 2529
Brief change log
Tests
API and Format
Documentation