Dynamic shapes support for Llama2 model

[mbencer]

TODO

• Add capabilities to test models with dynamic inputs #35 (blocked)
• Enable handling tensors with dynamic shapes
• Adjust remove_redundant_reshape pass
• Handle SymSize operator
• dynamic shapes support for Reshape op
• dynamic shapes support for Expand op
• dynamic shapes support for Slice op
• Enable E2E tests for Lama2 with dynamic inputs

[mhs4670go]

Hello,

It’s great to see that you’re working on dynamic shape support. I was wondering if I could rewrite the test infra for dynamic shapes. I know you’ve already prepared some, so I apologize for the overlap, but if we add the ONERT backend (#64) to the test infrastructure, we can test dynamic shapes directly with the Circle model without needing a separate resizer, which I think would be beneficial.

How about I set up the test infrastructure first, and then you could build the dynamic shape logic on top of that?

We could also consider integrating the resizer as a fallback mechanism later for cases where ONERT lacks support for specific kernels.

[mbencer]

Hi @mhs4670go,

Thank you for sharing feedback ;-)

I know you’ve already prepared some, so I apologize for the overlap

No problem. The purpose of #35 was to present using circle-resizer in TICO. I don't treat it as a final solution.

we can test dynamic shapes directly with the Circle model without needing a separate resizer, which I think would be beneficial.

Yes, it seems to be reasonable. IMO dynamic shape tests for particular ops in general should be independent of backends (both Circle interpreter and ONERT).

I see 2 possibilities here: in dynamic shapes tests we are testing only produced Circle graph (topological comparison of nodes, attributes, const values) without inference on the particular backend or create infrastructure to run such tests on different backends.
Both approaches have some cons and pros.

How about I set up the test infrastructure first, and then you could build the dynamic shape logic on top of that?

It's no problem for me. Probably finishing Samsung/ONE#14727 takes also some time.

We could also consider integrating the resizer as a fallback mechanism later for cases where ONERT lacks support for specific kernels.

Agree that fallback makes a sense for ONERT. Do you have plans to skip using circle interpreter completely?

Note that (almost) all described in this issue todo items I've presented in circle-exir in pull requests 871.

[mhs4670go]

Thanks for the reply:) Now I see what you was going to do a little more.

in dynamic shapes tests we are testing only produced Circle graph (topological comparison of nodes, attributes, const values) without inference on the particular backend

Yeah, it could be enough at some point. But, while the first option (graph-level comparison) is simpler and backend-independent, it doesn't guarantee runtime correctness. The second option, on the other hand, not only validate the graph structure indirectly but also confirm that the model runs correctly end-to-end.

In fact, if the output values match expected results, we don't necessarily need to exhaustive graph checks — the correctness of the graph is essentially verified by execution.

And, I don't know if you've seen #28 but we have plans to run models with dynamic shapes on onert runtime. So, setting up the infrastructure this way would allow us to incorporate fallback testing or kernel support validation later on.

Do you have plans to skip using circle interpreter completely?

No. ONERT is a fallback right now. I've chosen this because ONERT supports dynamic shapes in some level but circle-interpreter doesn't.

[mhs4670go]

For curiosity, why are you working with Llama2 instead of latest Llama 3 model?

[mbencer]

And, I don't know if you've seen #28 but we have plans to run models with dynamic shapes on onert runtime. So, setting up the infrastructure this way would allow us to incorporate fallback testing or kernel support validation later on.

Thank you for this information. It affects my plans from circle-exir/pull/871

For curiosity, why are you working with Llama2 instead of latest Llama 3 model?

I've just took circle-exir/issues/293 but I haven't any knowledge about priorities of such models. Here I am open to suggestions ;-)

[mbencer]

@mhs4670go Just for confirmation, do you have plans to merge dynamic shapes support from #29 into master? If yes, only for ops which are already in #29 or something more?
I'm asking because I'm considering which parts from my draft circle-exir/pull/871 should be planned to be productized ;-)

[mhs4670go]

@mbencer

do you have plans to merge dynamic shapes support from #29 into master?

No. I've just implemented that for PoC. You can push the PRs for what you've done or what you will gonna make. I will review those then.

[mbencer]

@mhs4670go Are you maybe aware about some plans to add onert python package to CI? I am asking because it is needed to enable unit tests of operators in dynamic shapes scenario (at least without circle-resizer fallback), like:

TICO/test/modules/op/linear.py

Line 30 in 601996b

# TODO enable this after introducing onert in CI.

I can even try to do it myself but not sure about package (do we have any official one?) which can be used.

The other possibility is using circle-resizer fallback but also here any official distribution is not available. Please note that Samsung/ONE#15301 is on hold, the new separate python package proposed in https://github.com/Samsung/ONE/pull/14727/files also hasn't got the green light.

[mhs4670go]

@mbencer

I can even try to do it myself but not sure about package

You can use this comment to use onert. Here is the example.

Are you maybe aware about some plans to add onert python package to CI?

I was waiting for Samsung/ONE#15342 being merged. But, I think you can draft some codes in advance and patch them according to the updates from onert.

And, how about enabling dynamic shape tests only if onert is installed? Then, it can be integrated smoothly to the main branch.

[mbencer]

@mhs4670go

You can use #2 (comment) to use onert. Here is the example.

I've meant here rather about a package which should be used in CI (locally I'm building the onert wheel from scratch)

I was waiting for Samsung/ONE#15342 being merged. But, I think you can draft some codes in advance and patch them according to the updates from onert.

Thank you! I haven't been aware about this feature

And, how about enabling dynamic shape tests only if onert is installed? Then, it can be integrated smoothly to the main branch.

If you are ok with tests run only locally to confirm the changes (as a temporary solution), it can speed up the process ;)