Cache of compressed files got deleted, DCOR-Aid cannot upload data and "retry" doesn't help

[B-Hartmann]

Windows 10
DCOR-Aid 0.11.13

I loaded 55 task files into DCOR-Aid for uploading, it started compressing and uploading files as expected. But then, suddenly, some compressed files got deleted (I didn't do it myself) and that caused DCOR-Aid to present the error messages shown below for all those datasets with missing compressed files. This might be related to the fact that my free disk space run out, because the compression was faster than the upload itself and I only had like 200 GB free space left..

Clicking the retry button doesn't help, because that doesn't start the compression again. I suspect that the current state of the upload job is saved and once it's in "transfer" and the compressed file is not there, it cannot go back to "compress" again.

So I would propose that in case a compressed file is missing, it just tries to compresses the data again and uploads it (as long as the access to the original file is still there and there is enough disk space available, of course).

Error message:

Traceback (most recent call last):
  File "dcoraid\worker\daemon.py", line 45, in run
    task()
  File "dcoraid\upload\job.py", line 317, in task_compress_resources
    with IntegrityChecker(path) as ic:
  File "dclab\rtdc_dataset\check.py", line 195, in __init__
  File "dclab\rtdc_dataset\load.py", line 22, in load_file
  File "dclab\rtdc_dataset\fmt_hdf5.py", line 280, in __init__
  File "h5py\_hl\files.py", line 533, in __init__
  File "h5py\_hl\files.py", line 226, in make_fid
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = 'C:\Users\bhartma\AppData\Local\DCOR\dcoraid\cache\compress-d7beaaeb-0df4-4e60-a060-60e8b8c68e94\0\M001_data.rtdc', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

[B-Hartmann]

I should add that the Windows settings to automatically free up space by deleting temporary files is deactivated and that a restart of DCOR-Aid helped and it started compressing the files again.

[paulmueller]

Thanks!. As restarting DCOR-Aid resolves the issue (the fail-safe is working), I am treating this as a minor bug.

[paulmueller]

There is already some logic that checks the data integrity. I assume there is only a check at the end of the compression missing that makes sure the file got created.