[Ruby] Literal expressions gets wrong value

[stenlarsson]

Describe the bug, including details regarding any error messages, version, and platform.

Running this test

require 'bundler/setup'
require 'arrow'

table = Arrow::Table.new(
  'foo' => [1, 2],
)
plan = Arrow::ExecutePlan.new
source_node = plan.build_source_node(table)
expresions = [
  Arrow::FieldExpression.new('foo'),
  Arrow::LiteralExpression.new('hello'),
]
names = [
  'foo',
  'bar',
]
project_options = Arrow::ProjectNodeOptions.new(expresions, names)
project_node = plan.build_project_node(source_node, project_options)

10_000.times do |i|
  Arrow::LiteralExpression.new("test #{i}")
end

sink_node_options = Arrow::SinkNodeOptions.new
plan.build_sink_node(project_node, sink_node_options)
plan.validate
plan.start
plan.wait
reader = sink_node_options.get_reader(project_node.output_schema)
pp reader.read_all

I expect the following output:

	    foo	   bar
	(uint8)	(utf8)
0	      1	hello
1	      2	hello

However the output is something like this:

	    foo	   bar
	(uint8)	(utf8)
0	      1
1	      2

In this case the bar column became the empty string, but sometimes I get other junk values.

If I comment out the times-block I get consistently the expected result. I suspect the literals gets garbage collected before the plan is executed.

I can reproduce this with both Arrow 22 and main.

Component(s)

Ruby

[stenlarsson]

From what I can gather the problem is that there is an Arrow::Buffer that points to data owned by a Ruby string, but the string is destroyed by the garbage collector. There is a long chain of references which must be maintained for the string to survive. It looks something like this in the below example:

ExecutePlan => ExecuteNode => ProjectNodeOptions => CallExpression => LiteralExpression => ScalarDatum => StringScalar => Buffer => string

I tried adding all of these classes here to keep the references, but ExecutePlan is not so easy. Have a look at the following example:

require 'bundler/setup'
require 'arrow'

table = Arrow::Table.new(
  'foo' => [1, 2],
  'bar' => [%w[a b], %w[c d]],
)
plan = Arrow::ExecutePlan.new
node = plan.build_source_node(table)
node = plan.build_project_node(
  node,
  Arrow::ProjectNodeOptions.new(
    [
      :foo,
      Arrow::CallExpression.new('binary_join', [:bar, ',']),
    ],
    %w[foo bar],
  ),
)
puts plan.nodes.map(&:object_id).join(', ')
puts plan.nodes.map(&:object_id).join(', ')
GC.start
puts plan.nodes.map(&:object_id).join(', ')
puts plan.nodes.map(&:object_id).join(', ')

When I run this example I get the following output:

6128, 6136
6128, 6136
6128, 6144
6128, 6144

It looks like the project-node gets represented by a new Ruby object after running the garbage collector. @kou, what could be the cause of this?

[kou]

Ah, sorry. I missed this.

GH-48919 fixes this.

[kou]

Workaround is keeping references of all nodes except the last node: source_node = node = plan.build_source_node(table)

[stenlarsson]

Nice! However, to completely fix this I also need to add FilterNodeOptions and ProjectNodeOptions to gc_guard, so something is still missing. 🤔

[kou]

How to test it? Options must be referred from node. So it must not be GC-ed.