[REQUEST]: More nuanced array length handling to deal with arcane protocols

[acamilo]

Description

I would like to begin a discussion about the difficulty of adding more nuanced array length handling.
This is related to #45 and #60 and #60 would solve my problem but i suspect it's not the ideal solution.

I have a packet where the length of the array is not immediately preceding he packet but is instead another field in the packet. The feature i'm requesting is to use a syntax similar to a LookbackDecider to use a previously decoded field as the array length.

I have looked around in the code and have thought about implementing this myself but i would almost certainly need to increase my understanding about the internals of the library first.

class TestPacket(Structured):
     seq: uint32
     id: uint32
     length: uint32
     unrelated: uint32
     stuff: uint8
     data: array[Header[attrgetter('length')],uint32]

Use

No response

Anything else?

Your library is awesome.

[acamilo]

Hmm, thinking about it more, it does make serialization and deserialization a pain in the butt. Especially if there are additional fields after data.

[acamilo]

I've been thinking about this a little and reading through the library code. It could be done by making a new array serializer. Unpacking shouldn't be hard, you'd add logic to the preunpack method in the new array variant's serializer to use a provided attrgetter to retrieve the length from the partial object.

Packing is different. A decision regarding the way the length attribute would be handled needs to be made. Would you simply omit the length attribute from the constructor? and in the prepack add logic to calculate the length of the array and set it's value? Throw an exception if it's defined or does not match the array length?

But, by the time the custom array's pack method would be called, presumably the value would have been(incorrectly) packed already.

Perhaps, adding a prepack function to the serializer API,that gets run before any serialization begins so each serializer has a chance to do something to the partial object before any packing begins? This would also make union deciders capable of setting their corresponding field based on the specific class used?

[acamilo]

I just saw #42, interesting.

[lojack5]

Hey, thanks for the interest in this!

I agree, I do want some more robust capabilities in any "list-like" unpacking. Most of the capabilities are what I've needed as I've used this on various projects. I didn't have any particular plans to look back into adding stuff yet, but I'm more than happy to look at a pull request if you come up with improvements.

I'll take a look at this...eventually (no promises on how quickly though) if you don't. One of the goals to keep in mind - try to push as much computation/logic into the class creation code which boils down to usually (best place to worst, usually):

• In the constructors of various serializers
• In the serializer selection code (annotated.register_transform)
• The general type-hint handling code (transform_typehint / filter_typehints)
• Structured.__init_subclass__ and its directly called methods
• The metaclass (currently this doesn't do much, thankfully)

But yeah, I've run into a few problems (and there are plenty of binary formats I've run into) that place data sizes or array lengths in various locations before the actual data/list.

Packing as you noted is an issue currently, serializers don't have access to the object they're packing until right before they pack, so an array serializer wouldn't be able to modify the length attribute until after it's already been packed.

Most likely the best solution would be another, earlier pre-pack phase, where every serializer is given the opportunity to make modifications to the object? Then any array serializers could update length/size attributes in that phase. I'll have to think about this more before I settle on an option.

[acamilo]

If you could give a quick TL;DR; on the way the library architecture is set up it would help me try and take a stab at it using my mediocre python skills, Your comments are good but I am still somewhat unfamiliar with all the fun ways you're utilizing advanced features of the language.