Multi-dimensional batch calculation: list of update batch dataset as a cartesian product

[TonyXiang8787]

Closes #1176

Let user to provide a list of update batch datasets, which are used as cartesian product to produce a multi-dimentional batch calculation.

The implementation is a cartesian product over a linked list.

check list

• C-API and core
• C-API test
• C-API in-code documentation
• Python wrapper
• Python test
• Python in-code documentation
• Adjustment on general documentation and examples

[TonyXiang8787]

@mgovers @figueroa1395 @nitbharambe,

The main part is almost finished. We still need to add general documentation and some notebook examples. I propose one of you to add this. This make sure we actually have the same understanding about how cartesian product of MD batch calculation works.

[figueroa1395]

@mgovers @figueroa1395 @nitbharambe,

The main part is almost finished. We still need to add general documentation and some notebook examples. I propose one of you to add this. This make sure we actually have the same understanding about how cartesian product of MD batch calculation works.

Good idea. I can probably pick this up. I'll bring it to Peter today so we allocate some time for it.

Edit: No further comments from me. It looks good to me at this stage besides the additional stuff you pointed out. I would only also silence the Sonar Cloud warnings as I deem them false positives.

[mgovers]

are we sure that we want to do this logic here and not in the C++ job dispatch where it's easier to test and also better multi-thread-able? E.g. we could only create separate threads for the outermost dimensions.

[TonyXiang8787]

We can think about this kind of refactor/optimization in later stage. The C-API will not be affected.

[mgovers]

however, this is one of those things in which it's hard to refactor at a later stage. i'd much rather refactor now to ensure that we design for change.

[TonyXiang8787]

I do not think it is difficult to refactor in later stage. The current implementation is pretty separated in the C-API files. If we decide to migrate this into the core, we can just reset this change to the old C-API file and implement the MD batch in the core.

[mgovers]

then isn't it easier to do the migration now? we of course don't need to fully support the multi threading thing i thought of, but why not put it in the right location from the start?

[TonyXiang8787]

We have a working implementation now and well-defined user-API. I think the priority should be releasing this fast to the user and hear user experience feedback, before we do another round of internal refactoring.

At a later stage, if and how we are going to refactor this will depend on user business case.

[figueroa1395]

@mgovers @figueroa1395 @nitbharambe,
The main part is almost finished. We still need to add general documentation and some notebook examples. I propose one of you to add this. This make sure we actually have the same understanding about how cartesian product of MD batch calculation works.

Good idea. I can probably pick this up. I'll bring it to Peter today so we allocate some time for it.

Edit: No further comments from me. It looks good to me at this stage besides the additional stuff you pointed out. I would only also silence the Sonar Cloud warnings as I deem them false positives.

@TonyXiang8787 I can pick this up whenever you give me the green light and finish it off.

[TonyXiang8787]

@mgovers @figueroa1395 @nitbharambe,
The main part is almost finished. We still need to add general documentation and some notebook examples. I propose one of you to add this. This make sure we actually have the same understanding about how cartesian product of MD batch calculation works.

Good idea. I can probably pick this up. I'll bring it to Peter today so we allocate some time for it.
Edit: No further comments from me. It looks good to me at this stage besides the additional stuff you pointed out. I would only also silence the Sonar Cloud warnings as I deem them false positives.

@TonyXiang8787 I can pick this up whenever you give me the green light and finish it off.

@figueroa1395 it's ready now. So go ahead.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
83.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud