fix: hour/minute/second handling for TimestampNTZ

[vigneshsiva11]

Which issue does this PR close?

Closes #3180

Rationale for this change

Spark TimestampNTZ values represent local time without any timezone
information. However, the current Comet implementation applies timezone
conversion when evaluating hour, minute, and second, which leads to
incorrect results for TimestampNTZ inputs in non-UTC session timezones.

This change aligns Comet behavior with Spark semantics by ensuring that
timezone conversion is only applied to Timestamp values with an explicit
timezone, and not to TimestampNTZ.

What changes are included in this PR?

• Updated the Rust implementation of extract_date_part used by
  hour, minute, and second to:
  • Bypass timezone conversion for TimestampNTZ inputs
  • Preserve existing behavior for Timestamp inputs with timezone
• Added defensive handling for unsupported input types

How are these changes tested?

• Verified via Rust unit tests:

  cargo test -p datafusion-comet-spark-expr

[Copilot]

Pull request overview

Aligns Comet’s hour/minute/second behavior with Spark semantics by avoiding session-timezone conversion for TimestampNTZ (timestamp without timezone), while preserving the existing conversion behavior for timezone-aware timestamps.

Changes:

• Bypass timezone conversion for DataType::Timestamp(_, None) (TimestampNTZ) inputs.
• Preserve timezone conversion for DataType::Timestamp(_, Some(_)) inputs via array_with_timezone.
• Add an explicit execution error for unsupported (non-timestamp) input types.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

invoke_with_args now pattern-matches directly on array.data_type() and rejects DataType::Dictionary(_, _) with an execution error. This is a regression for dictionary-encoded timestamp columns (the function’s return_type explicitly supports dictionary input, and array_with_timezone also has dictionary handling). Add explicit handling for DataType::Dictionary(_, value_type) where value_type is Timestamp(_, None) (no conversion) or Timestamp(_, Some(_)) (apply array_with_timezone), instead of falling into the error branch.

[andygrove]

@vigneshsiva11 this is a valid concern

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.14%. Comparing base (f09f8af) to head (c5cf290).
⚠️ Report is 901 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3265      +/-   ##
============================================
+ Coverage     56.12%   60.14%   +4.02%     
- Complexity      976     1451     +475     
============================================
  Files           119      175      +56     
  Lines         11743    16067    +4324     
  Branches       2251     2663     +412     
============================================
+ Hits           6591     9664    +3073     
- Misses         4012     5059    +1047     
- Partials       1140     1344     +204     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andygrove]

Thanks for fixing this! The TimestampNTZ handling looks correct - skipping timezone conversion for TimestampNTZ while keeping it for regular timestamps makes sense.

One thing I wanted to check: the new match pattern handles Timestamp(_, None) and Timestamp(_, Some(_)) directly, but I'm wondering about dictionary-encoded timestamps. Looking at array_with_timezone in utils.rs, it has explicit handling for DataType::Dictionary(_, value_type) where the value type is a timestamp. With the new code, dictionary-encoded timestamps would fall into the error branch. Could this be a problem for Iceberg tables where timestamp columns are often dictionary-encoded?

Also, it might be worth adding a Scala test that exercises the TimestampNTZ path specifically, maybe something that verifies hour(timestamp_ntz_column) returns the correct value in a non-UTC session timezone. The bug this fixes is subtle and having a regression test would help catch it if anything changes in the future.

---

This review was generated with AI assistance.

[vigneshsiva11]

I’ve added the regression test for datediff with dictionary-encoded timestamp columns and pushed the update. Thanks for the guidance!

[andygrove]

I’ve added the regression test for datediff with dictionary-encoded timestamp columns and pushed the update. Thanks for the guidance!

@vigneshsiva11 I don't see the test

[andygrove]

@vigneshsiva11 I have moved this to draft for now. Please mark as ready for review once feedback has been addressed. Thank you.