Inference Batching

[nitron532]

Hello, I'm Alvin, I'm a 2nd Year CS undergrad at UCSB! I work with Professor Majedi in the CS3E Embedded Ethics lab.

I found ArchLab's pyrtlenet and thought it would be fun to help with the TODO's since I've built my own neural net in C++ and have experience working with their fundamental theory and logic.

I would like to add batching to the inference implementations of pyrtlenet!
I already created a batched version of the numpy inference on my own fork, with modified comments and batching logic implemented. I plan on adding batching for the other two versions as well.
It has passed both the tests and the ruff linter, although I had to leave the parameter "test_image" in the inference class as it is named despite it being a batch so it would pass the tests.
Check it out and let me know if anything should be changed.
Thanks!

[fdxmw]

Hi, nice to meet you, and thank you for looking into input batching!

I took a look at your fork and your changes look good so far, though I haven't studied your changes in detail yet. I agree that the parameters should be renamed to test_batch since it's no longer a single image after your change, and we should update other callers and tests accordingly.

If feasible, it would be much better to first send a separate pull request that just adds batching to NumPyInference, rather than merging the changes to all the inference implementations at once. Smaller pull requests are much easier to review :)

[nitron532]

Yeah I can send a PR for just NumPyInference!