GH-38558: [C++] Add support for null sort option per sort key #46926

Taepper · 2025-06-27T12:12:56Z

See #38584 for original PR. Will be quoted for this PR description.

Rationale for this change

support multi sortkey nulls first.

order by i nulls first, j, k nulls first;

The current null sorting only supports all sortkeys, not a certain sortkey, so NullPlacement is extended to the SortKey field. Since the underlying framework is very well written, when modifying this function, you only need to pass the null_placement of each SortKey in. That’s it.

What changes are included in this PR?

1.SortKey structure, NullPlacemnt transfer logic, sorting logic and Ording related, test related
2.Substriait related.
3.c_glib related.
4.SelectK related.
5.RankOptions related.

Are these changes tested?

yes, I changed the code inside vector_sort_test.cc and performed additional tests.

Are there any user-facing changes?

yes, pg database include null sorting of multiple sort keys.

This PR includes breaking changes to public APIs. (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)

I amended the original PR to be less breaking in public APIs.

Still Ordering, SortOptions, RankOptions, and RankQuantileOptions now accept a std::optional<NullPlacement> instead of NullPlacement, which did lead to some changes in downstream APIs and bindings. ~~I also need some help with fixing the c_glib bindings.~~

GitHub Issue: [C++][Compute] Fix null sorting of multiple sort keys #38558

1.Reconstruct the SortKey structure and add NullPlacement. 2.Remove NullPlacement from SortOptions 3.Fix selectk not displaying non-empty results in null AtEnd scenario. When limit k is greater than the actual table data and the table contains Null/NaN, the data cannot be obtained and only non-empty results are available. Therefore, we support returning non-null and supporting the order of setting Null for each SortKey. 4.Add relevant unit tests and change the interface implemented by multiple versions

…8558 # Conflicts: # c_glib/arrow-glib/compute.cpp # c_glib/arrow-glib/compute.h # cpp/src/arrow/compute/kernels/vector_rank.cc # cpp/src/arrow/compute/kernels/vector_select_k.cc # cpp/src/arrow/compute/kernels/vector_sort.cc # cpp/src/arrow/compute/kernels/vector_sort_internal.h # python/pyarrow/_acero.pyx # python/pyarrow/_compute.pyx # python/pyarrow/array.pxi # python/pyarrow/tests/test_compute.py # python/pyarrow/tests/test_table.py

# Conflicts: # cpp/src/arrow/compute/api_vector.cc # cpp/src/arrow/compute/api_vector.h # cpp/src/arrow/compute/kernels/vector_rank.cc # cpp/src/arrow/compute/kernels/vector_select_k.cc # cpp/src/arrow/compute/kernels/vector_sort.cc # cpp/src/arrow/compute/kernels/vector_sort_internal.h # cpp/src/arrow/compute/kernels/vector_sort_test.cc # cpp/src/arrow/compute/ordering.cc # cpp/src/arrow/compute/ordering.h

…most likely human-error while merging)

c_glib/arrow-glib/compute.cpp

c_glib/arrow-glib/compute.h

kou · 2026-01-23T09:42:51Z

c_glib/arrow-glib/compute.cpp

+    options->null_placement = garrow_optional_null_placement_to_raw(
+      static_cast<GArrowOptionalNullPlacement>(g_value_get_enum(value)));


Suggested change

options->null_placement = garrow_optional_null_placement_to_raw(

static_cast<GArrowOptionalNullPlacement>(g_value_get_enum(value)));

{

auto null_placement = static_cast<GArrowOptionalNullPlacement>(g_value_get_enum(value));

if (null_placement == GARROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED) {

options->null_placement = std::nullopt;

} else {

options->null_placement = null_placement

}

}

This would be missing a static_cast:

{ auto null_placement = static_cast<GArrowOptionalNullPlacement>(g_value_get_enum(value)); if (null_placement == GARROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED) { options->null_placement = std::nullopt; } else { options->null_placement = static_cast<arrow::compute::NullPlacement>(null_placement); } }

Are you sure this is the cleaner way to go? Should I replace all usage of the helper function garrow_optional_null_placement_to_raw like this?

Ah, we have multiple places that use these helper functions. I missed them. Sorry.

Could you export these helper functions from compute.hpp instead of defining them in an anonymous namespace? And could you simplify these helper functions?

Okay, I did this, but this lead to the following change:

The return type is a cpp object and not a pointer as for most other exported functions. Therefore, its definition must be outside the G_BEGIN_DECLS/G_END_DECLS (these define extern "C"). I moved it there, but it makes it feel a little out of place. No other definition before the line G_BEGIN_DECLS is exported.

Could you move them after the G_END_DECLS

arrow/c_glib/arrow-glib/compute.cpp

Line 10666 in 75ef031

G_END_DECLS

? We place exported C++ functions after G_END_DECLS.

Oh, this was embarrassing to miss. Thanks for the pointer!

c_glib/arrow-glib/compute.cpp

…LACEMENT_UNSPECIFIED

Taepper · 2026-01-23T10:07:56Z

Thank you for triggering the CI!

I fixed the remaining c_glib (sorry for not merging in an up-to-date origin/main before), and all your comments except your last two.

I pushed them onto a separate branch, but they are not required for passing the CI:
Taepper@8aa6733

Taepper · 2026-01-23T10:25:46Z

If you dislike them, I could remove the helper functions entirely as seen in:
Taepper@1c15e9b

Taepper · 2026-01-23T11:54:09Z

I see that there is an unintended API breakage for ruby:

/Users/runner/work/arrow/arrow/c_glib/test/test-select-k-options.rb:33:in `test_sort_keys'
     30: 
     31:   def test_sort_keys
     32:     sort_keys = [
  => 33:       Arrow::SortKey.new("column1", :ascending),
     34:       Arrow::SortKey.new("column2", :descending),
     35:     ]
     36:     @options.sort_keys = sort_keys
Error: ArgumentError: wrong arguments: Arrow::SortKey#initialize("column1", :ascending):

SortKey without a specified null-placement should be constructable with a default NullPlacement of AtEnd

ruby/red-arrow/lib/arrow/sort-key.rb

kou · 2026-01-23T12:06:00Z

ruby/red-arrow/lib/arrow/sort-key.rb

+      # for recreatable, we should remove suffix
+      if target.end_with?("_at_start")
+        suffix_length = "_at_start".length
+        target = target[0..-(suffix_length + 1)]
+      elsif target.end_with?("_at_end")
+        suffix_length = "_at_end".length
+        target = target[0..-(suffix_length + 1)]
+      end


Hmm, how about ^ for :at_start and $ for :at_end?

For example:

-column -> ["column", :descending, null_placement]

+column -> ["column", :ascending, null_placement]

-^column -> ["column", :descending, :at_start]`

-$column -> ["column", :descending, :at_end]`

+^column -> ["column", :ascending, :at_start]`

+$column -> ["column", :ascending, :at_end]`

^column -> ["column", order, :at_start]`

$column -> ["column", order, :at_end]`

Sounds good to me

Should we omit the $ in the to_s() function (as :at_end is the default value)? Otherwise, it might be confusing for users that never used the null_placement option

It's a good idea. Let's do it.

Hmm, maybe this was a bad idea, I just noticed that this makes them no longer safely round-trip. E.g.:

Arrow::SortKey.new( # Inner key will printed as "+^count" Arrow::SortKey.new( "^count", Arrow::SortOrder::ASCENDING, Arrow::NullPlacement::AtEnd ).to_s )

will have a null_placement of AtStart

Taepper · 2026-01-23T12:25:11Z

Just to confirm, I just noticed that the breakage I pasted above:

/Users/runner/work/arrow/arrow/c_glib/test/test-select-k-options.rb:33:in `test_sort_keys'
     30: 
     31:   def test_sort_keys
     32:     sort_keys = [
  => 33:       Arrow::SortKey.new("column1", :ascending),
     34:       Arrow::SortKey.new("column2", :descending),
     35:     ]
     36:     @options.sort_keys = sort_keys
Error: ArgumentError: wrong arguments: Arrow::SortKey#initialize("column1", :ascending):

Is not in the red-arrow ruby gem, but only in the GI bindings. Is this tolerable? I.e. are they only used for internal tests as the one I pasted, or are they part of arrow's published api?

Sorry for my lack of knowledge in this regard

…ement

kou · 2026-01-23T22:27:49Z

Is not in the red-arrow ruby gem, but only in the GI bindings. Is this tolerable?

Yes. You're right.

I.e. are they only used for internal tests as the one I pasted, or are they part of arrow's published api?

They are part of published API like c_glib/*/*.{h,hpp}. But these API breakage are acceptable because we'll use "major version up" for the next release. "Major version up" means that we have API breakages in semantic versioning.

Sorry for my lack of knowledge in this regard

No problem. It's (a bit?) confusing...

kou · 2026-01-23T22:31:55Z

c_glib/arrow-glib/compute.cpp

+    options->null_placement = garrow_optional_null_placement_to_raw(
+      static_cast<GArrowOptionalNullPlacement>(g_value_get_enum(value)));


Could you move them after the G_END_DECLS

arrow/c_glib/arrow-glib/compute.cpp

Line 10666 in 75ef031

G_END_DECLS

? We place exported C++ functions after G_END_DECLS.

kou · 2026-01-23T22:33:38Z

c_glib/arrow-glib/compute.h

+ *   NaNs will come before nulls.
+ * @GARROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED:
+ *   Do not specify null placement.
+ *   Null placement should instead


Is this garbage?

Suggested change

* Null placement should instead

kou · 2026-01-23T22:35:06Z

c_glib/arrow-glib/compute.h

+ *   Do not specify null placement.
+ *   Null placement should instead
+ *
+ * They are corresponding to `std::optional<arrow::compute::NullPlacement>` values.


Suggested change

* They are corresponding to `std::optional<arrow::compute::NullPlacement>` values.

* They are corresponding to `arrow::compute::NullPlacement` values except

* `GARROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED`.

* `GARROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED` is used to specify

* `std::nullopt`.

kou · 2026-01-23T22:37:03Z

c_glib/arrow-glib/compute.h

+  GARROW_OPTIONAL_NULL_PLACEMENT_AT_START,
+  GARROW_OPTIONAL_NULL_PLACEMENT_AT_END,
+  GARROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED,


How about using -1 that will be never used in arrow::compute::NullPlacement?

Suggested change

GARROW_OPTIONAL_NULL_PLACEMENT_AT_START,

GARROW_OPTIONAL_NULL_PLACEMENT_AT_END,

GARROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED,

GARROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED = -1,

GARROW_OPTIONAL_NULL_PLACEMENT_AT_START,

GARROW_OPTIONAL_NULL_PLACEMENT_AT_END,

…RROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED correspond to -1. All other (possibly future) values will have a 1:1 mapping

…unctions in c_glib/arrow-glib/compute.{hpp,cpp}

Light-City and others added 30 commits November 9, 2023 09:57

fix test

d8bca7c

fix sortkey assert

970f5bf

fix serde test

6c0bc76

better api

25984c5

merge follow-up

dafbe14

merge follow-up

48a3e0c

formatting

5c9eb50

formatting

12d1b0d

missing unwrap_null_placement in python

0b442f7

do not remove demoted null_placement from python api

4398d16

fix member name in hash_aggregate_test

bdc4069

update ToString method output

8fc630c

format remove extra empty line

dcb3b7a

fix python interface

682a37a

python formatting

e6afb8a

do not pass None to python C binding

82299bc

format python

370cdac

fix minor python api mistakes

503a7d1

python formatting

1d0883e

do not rename member of exposed API struct

7833add

do not rename member of exposed API struct, missed test file

6e2759b

format cc file

0a029ca

updating api that was not using additional argument for some reason (…

3463a59

…most likely human-error while merging)

make null_placement optional in the python acero api

e01a009

minor additional fixes

794a2cd

amend c_glib api to use std::optional<NullPlacement> for RankOptions

adaae37

amend c_glib api to use std::optional<NullPlacement> for RankOptions

14c3914

kou reviewed Jan 23, 2026

View reviewed changes

Taepper added 3 commits January 23, 2026 10:49

also update RankQuantileOptions to new NullPlacement api

e07a673

update library version documentation

290812f

rename GARROW_OPTIONAL_NULL_PLACEMENT_UNSET to GARROW_OPTIONAL_NULL_P…

5320622

…LACEMENT_UNSPECIFIED

formatting

bf31270

kou reviewed Jan 23, 2026

View reviewed changes

Taepper added 8 commits January 23, 2026 13:33

ruby default value formatting

2e58463

change red-arrow gem to use different syntax for specifying null_plac…

79995df

…ement

update ruby test functions that use direct GObject introspection

bac578a

simplify helper functions

1411115

move and export helper functions

8f041b5

fixup move and export helper functions

d36fbb6

format ruby

272ec3f

fix another ruby test

f1fda48

Taepper force-pushed the GH-38558 branch from 128478f to f1fda48 Compare January 23, 2026 14:20

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Jan 23, 2026

kou reviewed Jan 23, 2026

View reviewed changes

Taepper added 4 commits January 24, 2026 09:16

fix another ruby test

e398fe4

Improve GArrowOptionalNullPlacement documentation

4778d97

Make static_cast of GARROW_OPTIONAL_NULL_PLACEMENT safer by having GA…

15542b0

…RROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED correspond to -1. All other (possibly future) values will have a 1:1 mapping

improve placement of helper garrow_optional_null_placement_* helper f…

b7293cb

…unctions in c_glib/arrow-glib/compute.{hpp,cpp}

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jan 24, 2026

more ruby test fixes

b33465e

		options->null_placement = garrow_optional_null_placement_to_raw(
		static_cast<GArrowOptionalNullPlacement>(g_value_get_enum(value)));

-    options->null_placement = garrow_optional_null_placement_to_raw(
-      static_cast<GArrowOptionalNullPlacement>(g_value_get_enum(value)));
+    {
+      auto null_placement = static_cast<GArrowOptionalNullPlacement>(g_value_get_enum(value));
+      if (null_placement == GARROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED) {
+        options->null_placement = std::nullopt;
+      } else {
+        options->null_placement = null_placement
+      }
+    }

- * They are corresponding to `std::optional<arrow::compute::NullPlacement>` values.
+ * They are corresponding to `arrow::compute::NullPlacement` values except
+ * `GARROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED`.
+ * `GARROW_OPTIONAL_NULL_PLACEMENT_UNSPECIFIED` is used to specify
+ * `std::nullopt`.

GH-38558: [C++] Add support for null sort option per sort key #46926

Are you sure you want to change the base?

GH-38558: [C++] Add support for null sort option per sort key #46926

Conversation

Taepper commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Taepper commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Taepper commented Jan 23, 2026

Uh oh!

Taepper commented Jan 23, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Taepper commented Jan 23, 2026

Uh oh!

kou commented Jan 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Taepper commented Jun 27, 2025 •

edited

Loading

Taepper commented Jan 23, 2026 •

edited

Loading