Skip to content

Conversation

@mbutrovich
Copy link
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change

We're seeing issues on main with Iceberg tests due to CometIcebergNativeScan convert() failing to serialize a dynamicpruning expression, which leaves the CometBatchScan operator. When doExecuteColumnar() is called on that, we get the underlying Iceberg reader producing a type that Comet is not prepared for at the native columnar to row operator.

I think this is a new bug exposed by #3299.

What changes are included in this PR?

  • Fall back on a dynamicpruning expression.
  • Stricter exception throwing for failure scenarios at serialization time.

How are these changes tested?

New test, existing Iceberg tests should pass again.

case _
if scanExec.scan.getClass.getName ==
"org.apache.iceberg.spark.source.SparkBatchQueryScan" =>
if (scanExec.runtimeFilters.exists(isDynamicPruningFilter)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this check more widely for all v2 scans?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about the blast radius on that, it's possible/likely.

Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mbutrovich

@codecov-commenter
Copy link

codecov-commenter commented Jan 29, 2026

Codecov Report

❌ Patch coverage is 50.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.96%. Comparing base (f09f8af) to head (0a3691f).
⚠️ Report is 906 commits behind head on main.

Files with missing lines Patch % Lines
.../comet/serde/operator/CometIcebergNativeScan.scala 25.00% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #3335      +/-   ##
============================================
+ Coverage     56.12%   59.96%   +3.83%     
- Complexity      976     1474     +498     
============================================
  Files           119      175      +56     
  Lines         11743    16165    +4422     
  Branches       2251     2681     +430     
============================================
+ Hits           6591     9693    +3102     
- Misses         4012     5124    +1112     
- Partials       1140     1348     +208     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants