No Validation Losses?

[DominikVincent]

Hello,

Why is it that your code doesn't log validation losses? How do you monitor overfitting? How do you observe convergence?

Did you remove the validation loss logging on purpose, or do you plan to release code that logs validation losses?

[BoifZ]

There is validation code in motGPT/metrics, we log every LOGGER.VAL_EVERY_STEPS (set in config files)

[DominikVincent]

Thanks for the reply. I see that there are metrics, which in my training seem to be stagnating, e.g. not really any improvement.

Hence, I am confused about how you can monitor the training progress of your model, especially if you do not monitor how the loss generalizes to the validation set.

Do you observe the same metric curves? Have you ever verified the validation loss curve?

Thanks for your help :)

[BoifZ]

Could you please check the generated motion metrics? It is reasonable for the GT metrics shown in the figure to remain stable.

[DominikVincent]

Ah sorry, sent the wrong screenshot. The evaluation curves indeed change slightly, but seem to converge quite quickly:

[anon123x]

@DominikVincent Could check the validation_modtion videos? It cannot make sense.

https://drive.google.com/file/d/1F4Zmu_a1PabVyllLzj6AVMSf9u2hpZBn/view?usp=sharing

[BoifZ]

@DominikVincent Could check the validation_modtion videos? It cannot make sense.

https://drive.google.com/file/d/1F4Zmu_a1PabVyllLzj6AVMSf9u2hpZBn/view?usp=sharing

This behavior is expected before the model is trained: an untrained diffusion head will produce unstable latents and drifting motion. However, if you still see motions like this after >50 epochs, it indicates a training issue.

Here is the generated motion with id M007732 and 012523, after 20 epoch training in stage1 (from validate_motion/epoch_19):
https://github.com/user-attachments/assets/3a3a348f-bf3b-458e-a2d5-a12092d71c1a
https://github.com/user-attachments/assets/af5bb861-4606-482d-86a2-3cedb8aa0de9

[anon123x]

@DominikVincent Could check the validation_modtion videos? It cannot make sense.
https://drive.google.com/file/d/1F4Zmu_a1PabVyllLzj6AVMSf9u2hpZBn/view?usp=sharing

This behavior is expected before the model is trained: an untrained diffusion head will produce unstable latents and drifting motion. However, if you still see motions like this after >50 epochs, it indicates a training issue.

Here is the generated motion with id M007732 and 012523, after 20 epoch training in stage1 (from validate_motion/epoch_19): https://github.com/user-attachments/assets/3a3a348f-bf3b-458e-a2d5-a12092d71c1a https://github.com/user-attachments/assets/af5bb861-4606-482d-86a2-3cedb8aa0de9

So, do we need to complete all states (1 ,2, 3) to obtain the correct validation_motion videos?

[BoifZ]

So, do we need to complete all states (1 ,2, 3) to obtain the correct validation_motion videos?

About 50 epochs on t2m training (stage1) provides reasonable visulization. See the examples above:

after 20 epoch training in stage1 (from validate_motion/epoch_19)

[anon123x]

I checked every epoch, all of the stageI epoch are not working, pls see the below link.

https://drive.google.com/drive/folders/1_9qOIGU7MjmUZalLR0sLkFB2Yj9gpwdp?usp=sharing

Please provide the correct configs, current version is really confusing.