[circle-resizer] Implementation a new tool for model resizing

[mbencer]

In the current moment we have no a capabilities for model resizing and dynamic input specialization.
The best place in the whole tool-chain seems to be operating on circle representation (common solution for all front-ends).

What types of API should be exposed?

We need both C++ and Python API. Python API is especially needed for Python-based frontends, like Pytorch->Circle converter.
There are also some C++ tools run from Python as sub-processes but I am against such approach (very hard to debug, need to determine paths for every usage, no versioning control, each step requires saving the whole model to file).

Why a separate tool instead of extending existing API:

I am just following convention from https://github.com/Samsung/ONE/tree/master/compiler where an each step is realized as a separate app/service. What's more the tool should be easy accessible for front-ends.

Initial use-cases:

In the beginning I am focused on LLM Pytoch models. My first candidate to provide e2e support LLama2.

cmd app

The tool should be accessible also as command line app with args:

• --input_path
• --output_path
• --input_shapes
  Supported format of new inputs shapes: comma separated. An example for model with 2 inputs: [1,2],[3,4].

Way of tests:

• unit tests (c++ and python parts)
• e2e for the models, like: Pytorch->Circle conversion to dynamic version, resizing/shape specialization using circle-resizer, numerical results testing on circle-interpreter

Proposition of implementation:

TODO:

• adjust/expand current shape inference capabilities (luci/Pass/CircleShapeInferencePass) to support LLama2 dynamic shapes

• [luci/service] Enable CircleOutput shape inference #14845

• [luci] Handle dynamic dimensions in Squeeze shape inference #14857

• [luci] Add additional check of unknown dimension for Reshape #14858

• [luci] Add additional negative unit test for Reshape #14859

• [common-artifacts] Exclude Inf_Squeeze_000 recipe #14864

• [res] Add TFLiteRecipe for Squeeze with dynamic shape #14865

• [circle2circle-dredd-recipe-test] Add Inf_Squeeze_000 test #14866

• adjust luci-interpreter to enable inference model with specialized shape (like LLama2)

• [luci-interpreter] Relax StrideSlice rank limitations #14507

• [luci-interpreter] Relax Transpose rank limitations #14512

• [luci/service] Enable CircleOutput shape inference #14845

• [luci-interpreter] Support S32 for Gather #14836

• Implementation circle-resizer tool with cmd interface

• [circle-resizer] Bring up entrypoint infrastructure #15062

• [circle-resizer] Add Shape parser #15082

• [circle-resizer] Add CircleModel #15185

• [circle-resizer] Add ModelEditor #15224

• [circle-resizer] Resolve warning of operator<< for Shape #15225

• [circle-resizer] Expose cmd interface #15249

• Integration the tool with ONE toolchain

• one-cmds integration

• one-build integration

• infra/debian integration

• separate circle-resizer-dredd-recipe-test target to run res/TensorFlowLiteRecipes tests

Test models (cut from dynamic LLama2):
dyn_llama2_query.zip
dyn_llama2_norm.zip

[mbencer]

DRAFT: #14727

[seanshpark]

1/ Can you explain more about what to resize?
Providing some examples should help understanding.

2/ Do you have some specific real model that you are interested in? Can you share it?

[mbencer]

@seanshpark Sure

1/ Can you explain more about what to resize?
Providing some examples should help understanding.

1. Shape specialization. There are many models with dynamic inputs (inputs which have dynamic shapes) which have to be static before inference (I am not aware of any backend in ONE which accept inputs with dynamic shapes).
2. Input tensor resizing from static-to-static. Like changing batch or input image size in some cases.

Such tool on circle representation level seems to be a good solution for all frontends. Note that some time ago I've proposed sth similar to legacy onnx frontend - #13638. In that time it was attempt to resolve a request from our user.

Something to consider - I'm thinking now of changing name of the tool to sth more general - like circle-editor. Maybe in the future features like changing data type, layout order, inserting pre-processing layers, cutting some fragments can be also useful.

2/ Do you have some specific real model that you are interested in? Can you share it?

My first candidate in LLama2 with dynamic batch and sequence length. It's proposed in circle-exir repo under issue 293 (not sure if I can paste the full link here).

[seanshpark]

Thanks for the explanation.

... which have to be static before inference

have you checked circle2circle tool ? restricted but there is dynamic_batch_to_single_batch option

Maybe in the future features like changing data type, layout order, inserting pre-processing layers, cutting ...

have you checked circle2circle tools options? there seems similar options, but I don't know what you are thinking, so, please describe more about the future thoughts if you like, and describe what are the differences.

As I know you are to provide a new tool.
Our individual tools are integrated through one-cmds, onecc tool and also through sub tools,
if the feature is for users, to manipulate model files.
Please integrate it though this tool, and also prepare some UI tests like others,
though your draft PR, so that reviewers can provide some comments if something goes in different direction.

[seanshpark]

It's proposed in circle-exir repo under issue 293 (not sure if I can paste the full link here).

please preapre a res/TensorFlowLiteRecipes recipes with simplified version of the model or part of the model; each dim < 10 in the shape, so that luci-interpreter execution doesn't take too much time.
as I assume, there are lots of Ops in the model, you may have to prepare multiple models,
some for single Ops, and with multiple Ops.

we don't need to do tests with full model, only Ops and similar shapes consideration are enough,
but must be done in our CI.

[mbencer]

@seanshpark

have you checked circle2circle tool ? restricted but there is dynamic_batch_to_single_batch option

Yes. I've seen it but it doesn't covert cases fulfilled by circle-resizer.
Do you think that resizing feature such be an option of circle2circle? The integration is possible but I've tried to follow rule of rather small services (similar to the rest from https://github.com/Samsung/ONE/tree/master/compiler).

have you checked circle2circle tools options? there seems similar options, but I don't know what you are thinking, so, please describe more about the future thoughts if you like, and describe what are the differences.

That's right. At least layout order order can be changed by circle2circle. circle-editor instead of circle-resizer was just a proposition without any real case. Agree that's maybe it is not a good idea. Let's keep circle-resizer (at least now).

Our individual tools are integrated through one-cmds, onecc tool and also through sub tools,
if the feature is for users, to manipulate model files.
Please integrate it though this tool, and also prepare some UI tests like others,
though your draft PR, so that reviewers can provide some comments if something goes in different direction.

I see. I'll introduce such integration to draft PR but still I would like to keep Python API (built via setup.py) (in parallel to one-cmds). The first user (in my plans) of such API will be circle-exir.

please preapre a res/TensorFlowLiteRecipes recipes with simplified version of the model or part of the model; each dim < 10 in the shape, so that luci-interpreter execution doesn't take too much time.

I see. I'll add such tests.

Thank you for reviewing the proposition.

[seanshpark]

Do you think that resizing feature such be an option of circle2circle?

I'll introduce such integration to draft PR but still I would like to keep Python API (built via setup.py) (in parallel to one-cmds). The first user (in my plans) of such API will be circle-exir.

These two seems to be linked.
So you are thinking of providing this tool for circle-exir, and not for one-cmds ? right?

[seanshpark]

... it doesn't covert cases fulfilled by ...
Do you think that resizing feature such be an option of circle2circle?

As you know, current functionality may not satisfy your requirements but it is a duplicate. For users and other developers, this may give confusion. "What should I use? why are there two? where should I contribute? ..."

/CC @lemmaa @jinevening

Anyway before we discuss about this, IMHO, we may need to define about how we provide circle-exir to the users and what use-cases it will solve.