GH-343: Fix ListVector offset buffer not properly serialized for nested empty arrays

[Yicong-Huang]

What's Changed

Fix ListVector/LargeListVector IPC serialization when valueCount is 0.

Problem

When valueCount == 0, setReaderAndWriterIndex() was setting offsetBuffer.writerIndex(0), which means readableBytes() == 0. IPC serializer uses readableBytes() to determine buffer size, so 0 bytes were written to the IPC stream. This crashes IPC readers in other libraries because Arrow spec requires offset buffer to have at least one entry [0].

@viirya:

The offset buffers are allocated properly. But during IPC serialization, they are ignored.

  public long readableBytes() {
      return writerIndex - readerIndex;
  }

So when ListVector.setReaderAndWriterIndex() sets writerIndex(0) and readerIndex(0), readableBytes() returns 0 - 0 = 0.

Then when MessageSerializer.writeBatchBuffers() calls WriteChannel.write(buffer), it writes 0 bytes.

So the flow is:

valueCount=0 → ListVector.setReaderAndWriterIndex() sets offsetBuffer.writerIndex(0)
VectorUnloader.getFieldBuffers() returns the buffer with writerIndex=0
MessageSerializer.writeBatchBuffers() writes the buffer
WriteChannel.write(buffer) checks buffer.readableBytes() which is 0
0 bytes are written to the IPC stream
PyArrow read the batch with the missing buffer → crash when other libraries to read

Fix

Simplify setReaderAndWriterIndex() to always use (valueCount + 1) * OFFSET_WIDTH for offset buffer's writerIndex. When valueCount == 0, this correctly sets writerIndex to OFFSET_WIDTH, ensuring offset[0] is included in serialization.

Testing

Added tests for nested empty lists verifying offset buffer has correct readableBytes().

Closes #343.

[jbonofre]

@Yicong-Huang can you please rebase the PR ? Thanks !

[viirya]

When outer array is empty, nested writers are never invoked, so child list's offset buffer remains unallocated (capacity = 0). This violates Arrow spec which requires offset[0] = 0.

I think you are referring the writers in Spark. It is out of context here and not related to the root cause. We should update the description to explain the issue clearly.

The offset buffers are actually allocated properly. But during IPC serialization, they are ignored.

  public long readableBytes() {
      return writerIndex - readerIndex;
  }

So when ListVector.setReaderAndWriterIndex() sets writerIndex(0) and readerIndex(0), readableBytes() returns 0 - 0 = 0.

Then when MessageSerializer.writeBatchBuffers() calls WriteChannel.write(buffer), it writes 0 bytes.

So the flow is:

1. valueCount=0 → ListVector.setReaderAndWriterIndex() sets offsetBuffer.writerIndex(0)
2. VectorUnloader.getFieldBuffers() returns the buffer with writerIndex=0
3. MessageSerializer.writeBatchBuffers() writes the buffer
4. WriteChannel.write(buffer) checks buffer.readableBytes() which is 0
5. 0 bytes are written to the IPC stream
6. PyArrow read the batch with the missing buffer → crash when other libraries to read

[viirya]

Hi @lidavidm @jbonofre, do you think this can catch up the Arrow Java 19.0.0 release?

[viirya]

When valueCount == 0, I think validity buffer writer index should be validityBuffer.writerIndex(0);.

[Yicong-Huang]

I believe when valueCount==0, BitVectorHelper.getValidityBufferSizeFromCount(valueCount) also returns 0, so it is equivalent. The current version might be simpler. If you prefer an if branch to handle it separately, I can also apply it.

[Yicong-Huang]

I added the if branch back to handle valueCount==0 case. But I still think it is not necessary?

[viirya]

Oh, if BitVectorHelper.getValidityBufferSizeFromCount(valueCount) returns 0 for valueCount == 0, then it is okay.

[viirya]

I think innerList should also call allocateNew? Not allocated is different to not written.

[Yicong-Huang]

added

[viirya]

ditto

[Yicong-Huang]

added!

[viirya]

I looked these tests again. I think this nested structure is not necessary for the unit tests here. It only matter for our usage at Spark side (on the ArrowWriters). But for here, we just need to make sure that a ListVector/LargeListVector has meaningful and correct readableBytes value after they are allocated.

Maybe we can simplify these tests.

[viirya]

I only have some comments on the tests to simplify them. Otherwise, the fix looks correct to me.

@jbonofre @lidavidm Can you take a look and see if we can get this into the next release?

Thank you!