branch-4.1: [feature](file-cache) Disable cache writes after remote scan threshold

[bobhan1]

What problem does this PR solve?

Issue Number: None

Related PR: #65058

Problem Summary:

This backports #65058 to branch-4.1.

Remote scans can read a large amount of data from object storage. Before this change, a query that had already exceeded its useful cache-write budget could still keep writing later remote cache misses into file cache, increasing cache churn and IO overhead.

This PR adds the session variable file_cache_query_limit_bytes to control remote-scan read-through file-cache writes per query on each BE:

• < 0: disabled. This is the default.
• = 0: do not write remote-scan cache misses into local file cache from the start of the query.
• > 0: allow remote-scan cache writes until the next cache block would exceed the byte threshold. Once the threshold is exceeded, later cache misses continue reading from remote storage but skip additional local file-cache writes.

The limiter is query-wide within a BE and is propagated through scan, segment loading, preload, segment footer, column-reader metadata, and inverted-index metadata paths. The BE dynamic config file_cache_query_limit_segment_meta, default false, optionally includes segment footer and segment metadata cache writes in the same limit. Profile counters expose the threshold, whether remote-only-on-miss was triggered, and specialized cache-write bytes and timers.

The backport keeps the branch-4.1 scanner and short-circuit layouts: it wires the shared file-scan IO context through OlapScanner::prepare with ReaderType::READER_QUERY, preserves the target branch's short-circuit schema/version lifecycle, and does not pull in unrelated master-only refactors.

Release note

Add file_cache_query_limit_bytes for query-level remote-scan file-cache write limiting, and add optional segment metadata accounting with file_cache_query_limit_segment_meta.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

  Validation:

  • git diff --check
  • build-support/check-format.sh
  • ./build.sh --be --fe --cloud -j100
  • ./run-fe-ut.sh --run org.apache.doris.qe.SessionVariablesTest,org.apache.doris.qe.ShortCircuitQueryContextTest (15 tests passed)
  • Targeted BE unit tests for file-cache limiter/profile behavior, cached remote reads, scanner propagation, external scan metadata, primary-key index, segment footer, inverted-index IO context, and column-reader cache, run with -j100 (22 tests across 9 suites passed)
  • env -u HTTP_PROXY -u HTTPS_PROXY -u http_proxy -u https_proxy -u ALL_PROXY -u all_proxy ./run-regression-test.sh --run -d cloud_p0/cache/remote_scan_no_write_file_cache -s test_remote_scan_no_write_file_cache_threshold -g docker -runMode=cloud -dockerSuiteParallel 1 (passed)
  • env -u HTTP_PROXY -u HTTPS_PROXY -u http_proxy -u https_proxy -u ALL_PROXY -u all_proxy ./run-regression-test.sh --run -d cloud_p0/cache/remote_scan_no_write_file_cache -s test_file_cache_query_limit_segment_meta_profile -g docker -runMode=cloud -dockerSuiteParallel 1 (passed)

• Behavior changed:

  • No.
  • Yes. Remote scans can stop writing local file cache after file_cache_query_limit_bytes is reached while continuing to read misses from remote storage. Optional segment metadata accounting can include segment footer and metadata cache writes in the same query-wide threshold.

• Does this need documentation?

  • No.
  • Yes. The new user-visible knobs should be documented in a follow-up docs PR.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[bobhan1]

run buildall

[bobhan1]

run feut

[hello-stephen]

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.62% (1904/2453)
Line Coverage	64.48% (34029/52774)
Region Coverage	64.49% (17215/26694)
Branch Coverage	53.97% (9210/17064)

[hello-stephen]

FE UT Coverage Report

Increment line coverage 76.19% (16/21) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 0.38% (6/1570) 🎉
Increment coverage report
Complete coverage report