branch-4.0: [fix](test) fix unstable test cases - #66502
Open
morningman wants to merge 14 commits into
Open
Conversation
…ackup_restore_colocate test 6 "restore to a new db" intermittently failed with ColocateMismatchNum != 0. After RESTORE creates a new colocate group, some tablets are transiently COLOCATE_MISMATCH until the TabletScheduler clones them onto the group's BackendsPerBucketSeq. checkColocateTabletHealth asserted the live mismatch count once, right after waiting only for the group's cached IsStable flag, so it could observe the mismatch before the scheduler converged (self-heals in ~23/25 runs). Build apache#317 shows the same failure predates the recent wait-helper rename (apache#64980), confirming a long-standing race rather than a regression. Poll ColocateMismatchNum to zero via awaitUntil(60) instead of asserting once; it still fails loud if the mismatch never heals. Applied to both suites in the file as they share the identical closure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The last query of nereids_syntax_p1/mv/aggregate/agg_sync_mv still
expected map_agg v1 behavior (null keys dropped, e.g. `\N {}`). apache#51343
switched map_agg to v2, which keeps null keys; the stale golden was
masked by earlier failures (fixed in apache#64974 and apache#65133) and has kept
every branch-4.0 P1 run red since 2026-07-03 (the test was muted in
TeamCity on 07-06, but the regression runner still exits 1, so the
pipeline stays red anyway).
Simply aligning the golden to the observed map literal is not enough:
entry order inside one map is not deterministic across runs. For the
kbint=1 group (keys 0 and null), branch-4.1 CI produced
{0:"string1", null:"string1"} in one build and
{null:"string1", 0:"string1"} in the next; branch-4.1's one-line golden
alignment (apache#65879) is failing on exactly this today. The p0 sibling
pins {null:..., 0:...} and stays green only by luck of its smaller
data set.
Replace the rendered-map comparison with order_qt over exploded entries
(lateral view explode_map), which pins the exact per-group key/value
pairs and entry counts without depending on map entry order. Map
literal rendering stays covered by the p0 sibling.
Verification: replayed the fixture (stream load of agg_mv_test.dat plus
the 13 self-inserts) on a local cluster and confirmed
- the `null` literals in the .dat parse as SQL NULL for id/kbint but as
the 4-char string "null" for kstr, reproducing the CI-observed
{null:"null"} for the null-kbint group;
- the new exploded query returns the same 14 rows across repeated runs;
- the .out block is byte-identical to the real output sorted with the
framework's DataUtils.sortByToString semantics.
The local cluster runs 4.1.3-rc02, not a branch-4.0 build; map_agg v2
and explode_map semantics match the CI-observed branch-4.0 values. The
groovy change compiles clean offline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ons apache#64822 Backport upstream 3226e0c. The s3() smoke query selected nested complex columns with 'order by k1 limit 10'; duplicate k1 values at the limit boundary make the chosen rows nondeterministic, so the complex-column contents drift between runs (1 red / 11 builds on branch31-P2). Keep the direct-s3 smoke query scalar (k1 only) and take the regenerated .out baselines. branch-4.0 had backported apache#64706 (apache#64709) but missed this follow-up. (cherry picked from commit 3226e0c) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ail fast on failed compaction Two chronic branch31-P2 reds rooted in trigger_and_wait_compaction: 1. test_base_compaction_no_value (7/11): a lagging publish makes an earlier cumulative trigger a silent no-op (E-2000, already ignored) on one replica, so its cumulative point stays behind and the later base trigger hits the by-design E-808 (BE_NO_SUITABLE_VERSION, base_compaction.cpp:143/176 'nothing to base-compact') -> plugin threw. Treat base's E-808 exactly like cumulative's E-2000/E-2010: skip that replica. Suites still verify the compaction effect via their own rowset/segment-count asserts. 2. compaction_width_array_column (11/11): the wait loop only watched 'last <type> success time', so a compaction that FAILED on the BE (here [E-232] No space left on device, 2.5min after trigger) idled the full 40min timeout. Port master's completion detection (apache#64945/apache#65209): watch failure time too, with the E-2010 cumulative-point handoff special case, so failures surface within one poll interval. Branch-local curl transient-failure re-poll hardening is kept; master's binlog compaction support is intentionally not ported. Verified: offline groovy compile clean; remaining diff vs master is exactly {binlog feature (master-only), e-808 ignore (ours), curl re-poll (ours)}. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ts to fit compaction on CI disks The fixture loads ~113GB total; with BUCKETS 2 a single tablet holds one ~56.46GB 197-segment overlapping rowset (x3 forced replicas). Compaction writes the full output on the same mount before deleting the input, so on 100GB CI data disks ENOSPC is structural ([E-232] No space left on device, observed in P2 apache#210; 11/11 recent builds red, ASAN runs hit MEM_LIMIT for the same oversize). 16 buckets => ~7GB/tablet, ~14GB transient per mount. Test intent is per-tablet and preserved: every tablet still has plenty of overlapping segments to merge and the final assertFalse(isOverLap) checks all replicas of all tablets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-writer-path index size compares All four sub-suites compare physical index sizes produced by different writer paths (inline-at-load vs ALTER+BUILD INDEX, index-compaction merge vs rebuild). Settle-sequence analysis of P2 apache#210 shows the values are genuinely stable (13+ identical polls, all replica update steps complete) yet differ by a quantized ~7.05KB per index file per replica between paths (x3 replicas = the observed ~21KB deltas, 12.2-14.5% relative). Physical size equality across writer paths is not a stable invariant; the tests' intent is 'index present, no gross bloat/corruption'. - test_show_data (6/11 red) / test_show_data_for_bkd (3/11 red) / test_show_data_multi_add (6/11 red): exact assertEquals -> 20% tolerance (a missing index still shows as a ~37% deficit, bloat as 2x+). - test_show_data_with_compaction (4/11 red): earlier 10% tolerance was exceeded (12.2%) -> widen to 20%. - add the file's idiomatic wait_timeout guards before the numeric compares. The exact no_index_size compares (pure data, no index writer variance, never flaked) are deliberately left exact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r cooldown on ALL replicas Both phases gated only on tablets[0] reaching local_data_size==0, then asserted remote_data_size>0 for EVERY row of SHOW TABLETS (9 rows with the forced 3 replicas). Replicated cooldown is leader-uploads/followers-follow- async, so a follower whose meta sync lagged failed the assert with zero grace (7/11 recent branch31-P2 builds; apache#210: tablet 1785686520128 on .26 remote size 0 while row 0 had cooled during the 10min sleep). Make the wait condition match the asserts: poll until every replica row reports local==0 && remote>0, same 100x10s budget, fail-loud on timeout. Same fix applied to the byte-identical _by_hdfs twin (runs in hdfs-enabled pipelines; self-skipped in P2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…conflict on its own label P2 apache#210: load_four_step's broker load was CANCELLED 2s after submit with 'Label [lineorder_...] has already been used, relate to txn [19118], status [PREPARE]' - the label conflicted with the job's OWN transaction (TaskNumber:0, no BE work started). BrokerLoadPendingTask.executeTask() = getAllFileStatus() then beginTxn(), and LoadTask.exec retries the whole task on failure (retryTime=3). If the first attempt registers the txn but throws before transactionId is assigned - e.g. DatabaseTransactionMgr.beginTransaction fails on the edit log write after registering the label (FE meta dir shared the mount that went ENOSPC 11s earlier in this build) - every retry's beginTxn() hits LabelAlreadyUsedException against the job's own PREPARE txn, burns all retries in seconds and cancels the job with a misleading message. The variant where transactionId WAS assigned before a later Throwable self-conflicts the same way. Make beginTxn() retry-idempotent: - transactionId already assigned -> reuse it; - on LabelAlreadyUsedException, look up the label's txn and adopt it iff it is ours (callbackId == job id, status PREPARE); foreign labels and lookup failures still rethrow the original exception. Adds 3 JMockit tests (reuse / self-adopt / foreign-label rethrow). Verified: mvn test -pl fe-core -am, BrokerLoadJobTest 13/13 green, 0 checkstyle violations, build cache disabled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… guard's "concurrently dropped" surface too TC P0 build 1016049 (pull/66459, unrelated PR): the auto-partition INSERT raced with DynamicPartitionScheduler retention cleanup again, but this time failed the suite despite the retry loop added in apache#65357. Cause: apache#65357 itself changed the error surface. Its FE null-check guard in FrontendServiceImpl.createPartition() reports the race as a clean retryable status "partition %s was concurrently dropped, please retry" (the load is cancelled with that reason), while the retry condition added in the same commit only matches the pre-guard THRIFT_RPC_ERROR text "createPartition". So when the race hits the guarded path, the retry never triggers and the exception is rethrown. Match both surfaces. History: 1 such hit in the last 250 P0 runs; the trigger (a concurrent suite lowering dynamic_partition_check_interval_seconds, 12 suites do) is permanent, so the retry must honor the "please retry" contract the FE now emits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tyuj2w8ZHVtAnB4i4MoKbN
…ceberg partition evolution After Iceberg partition evolution, data files written under an older partition spec carry only that spec's identity partition values (none, for a formerly unpartitioned table), so the ranges of one scanner can have different columns_from_path_keys. FileScanner built its partition slot index map from the FIRST range only and then required every later range to supply a value at that index, failing with "Invalid partition value index 0, value count 0 for column ..." whenever a post-evolution file's range happened to be scheduled before a pre-evolution file's range in the same scanner (range order depends on Iceberg's parallel manifest reading, hence flaky: test_iceberg_partition_evolution_ddl, External Regression build 1016050). Fix, following the per-range semantics master already has via the data-lake reader refactor line (apache#62306/apache#62821, which the focused paimon backport apache#65583 deliberately did not bring to this branch): - _generate_partition_columns: for query ranges that carry columns_from_path_keys, bind values by THIS range's own key list; a partition slot absent from the range's keys is read from the data file instead (such slots are always file slots, see IcebergScanNode.getPathPartitionKeys). Load ranges keep the original index-map binding unchanged. - _process_runtime_filters_partition_prune: skip pruning a range that bound only a subset of the partition slots, since prune conjuncts may reference unbound slots whose block columns would stay empty. For scans whose ranges all share one key set (hive/hudi/paimon/tvf and non-evolved iceberg) the by-name binding yields exactly the previous indices, so behavior is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tyuj2w8ZHVtAnB4i4MoKbN
…ction cases apache#65211 Backport upstream 8ea88fc. Suites that drive compaction explicitly via trigger_and_wait_compaction() race the background auto compaction that their own setup loads trigger (e.g. 6 inserts >= cumulative_compaction_min_deltas; the cloud scheduler scores tablets off write-path approximate counters, so it reacts within milliseconds of the load). Seen on CloudP0 build 63404 (PR apache#66459), test_variant_custom_analyzer, tablet 1785913774936, from be.INFO/WARNING of the uploaded log archive: 15:27:11.757 auto CloudCumulativeCompaction starts, input [2-7] 15:27:11.784 manual full compaction accepted (prepare picked the same [2-7]) 15:27:12.093 cumulative commits its job in MS: cumulative_compaction_cnt 0->1 15:27:12.876 full's request_global_lock rejected by MS: "meta_service_job.cpp:218 could not perform compaction on expired tablet cache. req_cumulative_compaction_cnt=0, cumulative_compaction_cnt=1" The rejected manual full is never retried, so "last full success time" never changes and the branch-4.0 plugin (which has no failure-time fail-fast) waits out the entire 1800s timeout. Upstream fixed this family-wide in apache#65211 (2026-07-06) by disabling table-level auto compaction in every suite that triggers compaction manually; branch-4.0 never picked it up. All 39 files apply with their added/removed lines byte-identical to upstream; the two *_ck suites only differ in surrounding context (branch-4.0 still uses CLUSTER BY where master has ORDER BY). (cherry picked from commit 8ea88fc) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tyuj2w8ZHVtAnB4i4MoKbN
…o the BE syncs the tablet The suite inserts into a non-MOW cloud table and then polls information_schema.backend_tablets until its sizes match SHOW TABLETS. But backend_tablets reports the BE's cached tablet view, and in cloud mode a load on a non-MOW table never refreshes that cache (only MOW loads call sync_rowsets; CloudRowsetBuilder::init caches the pre-commit view, which for a fresh table is just the empty [0-1] rowset). The suite never reads the table, so the only thing that can sync the cache is an incidental background compaction attempt on a score-1 tablet; on a busy CloudP0 host higher-score tablets can starve it for the whole 120-iteration window and the BE keeps answering local/remote = 0/0 while SHOW TABLETS already says 0/535. Auto-muted since 2026-07-30 (the day after the suite landed via apache#66058); 4/40 recent CloudP0 runs failed with exactly this signature, all on unrelated PRs. Passing runs show the inverse shape (backend 0/547 immediately, SHOW catching up), confirming the BE-cache sync is the only unreliable edge. Fix: after the insert, read the table once. The scan carries the committed query version, so CloudTablet::sync_rowsets runs synchronously and the poll starts from an aligned BE view. The 120s loop still covers the FE side (meta-service stats propagation), which was already reliable. The suite file is byte-identical on master, which has the same MOW-only sync in CloudRowsetBuilder, so master needs the same fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tyuj2w8ZHVtAnB4i4MoKbN
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
…ereids planning fails
org.apache.doris.nereids.exceptions.AnalysisException extends
RuntimeException, so a Nereids planning failure inside createLoadingTask
escaped onPendingTaskFinished's catch (UserException) and fell into the
generic pending-task retry path ('Unexpected failed to execute load
task') instead of cancelling the job with the real message. In the
observed incident this masked 'disk ... exceed limit usage' behind a
misleading 'Label has already been used' cancellation (see previous
commit).
Catch it alongside UserException so deterministic planning/analysis
errors cancel the job immediately with the actual cause.
The new test drives the real propagation path - the mocked
NereidsLoadingTaskPlanner.plan throws the Nereids exception through the
real LoadLoadingTask.init/createLoadingTask/onPendingTaskFinished chain -
and asserts the job is CANCELLED with the real message and no loading
tasks were created.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lect compaction profile BE by tablet replica apache#62178 apache#65552 Backport two upstream regression-test fixes that never reached branch-4.0. Both cause recurring failures in the daily branch-4.0 P0 pipeline. 1. ann_index_basic vs ivf_index_test table-name collision (apache#62178) ann_index_basic and ivf_index_test run in the same regression database (ann_index_p0) and both used tbl_ann_l2 / tbl_ann_ip. With suiteParallel=10 they can run concurrently: FE logs of the failing run show ivf_index_test dropping and recreating tbl_ann_ip 230ms after ann_index_basic created it, then inserting 6 rows. ann_index_basic then reads the neighbor's table by name and fails. This also explains the historical intermittent empty result of sql_ip_asc (query landing between the neighbor's create and publish); the insert itself publishes in ~80ms, so the visibility-window theory behind the waitRowsVisible gate (apache#65942) was wrong, and the 30s gate now times out against the neighbor's 6-row table instead. Rename the shared tables with basic_/ivf_ prefixes as upstream did in 9c226f5 (apache#62178) and drop the gate. ann_index_basic.groovy becomes byte-identical to the upstream post-fix file. ivf_index_test takes only the renames because the upstream file also carries the apache#60358 behavior change (insufficient train rows no longer throws) which branch-4.0 BE does not have. No .out changes needed. 2. test_compaction_profile_action queries an arbitrary BE (apache#65552) The suite built the /api/compaction/profile URL from backendId_to_backendIP.keySet()[0]. On a multi-BE pipeline (4 BEs, replication forced to 3) the chosen BE has ~1/4 chance of not hosting the tablet replica, so the tablet_id filter returns an empty list (observed 5/26 failures; the case is currently muted as DORIS-26131). Derive the endpoint from the BackendId of the selected tablet row as upstream did in fd16ebd (apache#65552); the file becomes byte-identical to the upstream post-fix version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
run buildall |
Contributor
Possible file(s) that should be tracked in LFS detected: 🚨The following file(s) exceeds the file size limit:
Consider using |
github-actions
Bot
added
the
lfs-detected!
Contributor
Possible file(s) that should be tracked in LFS detected: 🚨The following file(s) exceeds the file size limit:
Consider using |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.