[native_datafusion] [Spark SQL Tests] _tmp_metadata_row_index not populated

[andygrove]

Summary

2 Spark SQL tests fail because native_datafusion doesn't populate the _tmp_metadata_row_index metadata column.

Failing Tests

• FileMetadataStructRowIndexSuite: "reading _tmp_metadata_row_index - not present in a table" — returns 0 instead of expected row indices
• FileMetadataStructRowIndexSuite: "reading _tmp_metadata_row_index - present in a table" — returns 0 instead of expected row indices

Root Cause

native_datafusion scan doesn't generate row index metadata. The count returned is 0 instead of the expected 100 rows.

Possible Fix

In CometScanRule.nativeDataFusionScan(), detect when row index metadata columns are requested in the schema and fall back to native_iceberg_compat.

Related

Discovered in CI for #3307 (enable native_datafusion in auto scan mode).