Term 2023 issues

[mmaelicke]

@maxschmi for reference

This is a report on errors, warnings and issues logged by the server while uploading this years data.

Referencing @data-hydenv/students for your information. If you recognize some of the information below, you are welcome to comment, otherwise just ignore the mentioning.

metadata

I guess this is a student who dropped out.

Some entries are missing metadata:
----------------------------------
            name  hobo_id github_name data_available region  latitude  longitude exposition altitude influence description
14  ***, ***      NaN         NaN            NaN    NaN       NaN        NaN        NaN      NaN       NaN         NaN

Additionally, there was an exception, which means there might be missing metadata

[ERROR]: (psycopg2.errors.UndefinedFunction) operator does not exist: character varying = timestamp without time zone
LINE 3: ...RE details.key = 'altitude' AND details.str_value = '1900-01...
                                                             ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.

[SQL: SELECT details.id AS details_id, details.key AS details_key, details.str_value AS details_str_value, details.int_value AS details_int_value, details.float_value AS details_float_value, details.bool_value AS details_bool_value
FROM details
WHERE details.key = %(key_1)s AND details.str_value = %(str_value_1)s
 LIMIT %(param_1)s]
[parameters: {'key_1': 'altitude', 'str_value_1': datetime.datetime(1900, 1, 2, 12, 0), 'param_1': 1}]
(Background on this error at: https://sqlalche.me/e/14/f405)

This basically means one student had a date in the altitude column, which was parsed as the 2.1.1900 12:00. Maybe putted 2.1 into the cell and Excel formatted as a date?

Raw data

These are either extra HOBO time series and can be ignored, or the Metadata file references the wrong HOBO (typo?)

File ./data-master/hobo/2023/raw/10350089.csv references HOBO ID=10350089, which is not found.
File ./data-master/hobo/2023/raw/10347324.csv references HOBO ID=10347324, which is not found.
File ./data-master/hobo/2023/raw/10347327.csv references HOBO ID=10347327, which is not found.
File ./data-master/hobo/2023/raw/10347333.csv references HOBO ID=10347333, which is not found.
File ./data-master/hobo/2023/raw/10350107.csv references HOBO ID=10350107, which is not found.
File ./data-master/hobo/2023/raw/10347366.csv references HOBO ID=10347366, which is not found.

[MarilenaAlmeida]

The value in the altitude column was supposed to be 2. Unfortunately I can no longer access the file to correct this.

[mmaelicke]

The value in the altitude column was supposed to be 2. Unfortunately I can no longer access the file to correct this.

Thanks for reporting. I corrected this.

[maxschmi]

Hey Mirko,
Yes their are additional HObos that are not in the Meta table. Those were my spare Hobos I did install as a backup.

[mmaelicke]

@data-hydenv/students to let you all know, I have a minor issue uploading the quality-checked data. I will solve that during the day and then the quality checked data will be available in the database, as well.

[mmaelicke]

@data-hydenv/students the 2023 quality checked data is uploaded.

There were some warnings and errors, read through them and reply if you feel like you know something about it.

File ./data-master/hobo/2023/hourly/10347324_Th.csv references HOBO ID=10347324, which is not found.

Parsing file './data-master/hobo/2023/hourly/10347355_Th.csv' was not successfull.
Do not edit the files by hand!
Error: 'DataFrame' object has no attribute 'tstamp' 

File ./data-master/hobo/2023/hourly/10350089_Th.csv references HOBO ID=10350089, which is not found.

Parsing file './data-master/hobo/2023/hourly/10349996_Th.csv' was not successfull.
Do not edit the files by hand!
Error: time data '2022.12.01. 00:00:00' does not match format '%Y.%m.%d %H:%M:%S' 

File ./data-master/hobo/2023/hourly/10347333_Th.csv references HOBO ID=10347333, which is not found.

Parsing file './data-master/hobo/2023/hourly/10350049_Th.csv' was not successfull.
Do not edit the files by hand!
Error: 'DataFrame' object has no attribute 'tstamp' 

File ./data-master/hobo/2023/hourly/10350107_Th.csv references HOBO ID=10350107, which is not found.

The error Error: 'DataFrame' object has no attribute 'tstamp' indicates, that reading the ddtm column failed. If it was your file, please check it carefully.

Respond to this thread, if you changed something about your data

[arrpit19]

Oh, alright I'll make the changes. object has no attribute 'tstamp. Could you please point me in the right direction? As how to do it? Thanks.

…

On Fri, 20 Jan, 2023, 10:09 Mirko Mälicke, ***@***.***> wrote: ***@***.***/students <https://github.com/orgs/data-hydenv/teams/students> the 2023 quality checked data is uploaded.* There were some warnings and errors, read through them and reply if you feel like you know something about it. File ./data-master/hobo/2023/hourly/10347324_Th.csv references HOBO ID=10347324, which is not found. Parsing file './data-master/hobo/2023/hourly/10347355_Th.csv' was not successfull. Do not edit the files by hand! Error: 'DataFrame' object has no attribute 'tstamp' File ./data-master/hobo/2023/hourly/10350089_Th.csv references HOBO ID=10350089, which is not found. Parsing file './data-master/hobo/2023/hourly/10349996_Th.csv' was not successfull. Do not edit the files by hand! Error: time data '2022.12.01. 00:00:00' does not match format '%Y.%m.%d %H:%M:%S' File ./data-master/hobo/2023/hourly/10347333_Th.csv references HOBO ID=10347333, which is not found. Parsing file './data-master/hobo/2023/hourly/10350049_Th.csv' was not successfull. Do not edit the files by hand! Error: 'DataFrame' object has no attribute 'tstamp' File ./data-master/hobo/2023/hourly/10350107_Th.csv references HOBO ID=10350107, which is not found. The error Error: 'DataFrame' object has no attribute 'tstamp' indicates, that reading the ddtm column failed. If it was your file, please check it carefully. *Respond to this thread, if you changed something about your data* — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXKWE4UHRICAD7MHLKDCXZTWTJI53ANCNFSM6AAAAAATYFY5WI> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[kg200]

Hello Mirko,  concerning:  File ./data-master/hobo/2023/hourly/10350089_Th.csv references HOBO ID=10350089, which is not found. just changed my Hobo ID in the metafile yesterday because I had to use a spare one and forgot to change it until now. I hope it works now.  all the best Kilian  Am Freitag, 20. Januar 2023 um 10:47:48 MEZ hat ARPIT KUMAR ***@***.***> Folgendes geschrieben: Oh, alright I'll make the changes. object has no attribute 'tstamp. Could you please point me in the right direction? As how to do it? Thanks.

On Fri, 20 Jan, 2023, 10:09 Mirko Mälicke, ***@***.***> wrote: ***@***.***/students <https://github.com/orgs/data-hydenv/teams/students> the 2023 quality checked data is uploaded.* There were some warnings and errors, read through them and reply if you feel like you know something about it. File ./data-master/hobo/2023/hourly/10347324_Th.csv references HOBO ID=10347324, which is not found. Parsing file './data-master/hobo/2023/hourly/10347355_Th.csv' was not successfull. Do not edit the files by hand! Error: 'DataFrame' object has no attribute 'tstamp' File ./data-master/hobo/2023/hourly/10350089_Th.csv references HOBO ID=10350089, which is not found. Parsing file './data-master/hobo/2023/hourly/10349996_Th.csv' was not successfull. Do not edit the files by hand! Error: time data '2022.12.01. 00:00:00' does not match format '%Y.%m.%d %H:%M:%S' File ./data-master/hobo/2023/hourly/10347333_Th.csv references HOBO ID=10347333, which is not found. Parsing file './data-master/hobo/2023/hourly/10350049_Th.csv' was not successfull. Do not edit the files by hand! Error: 'DataFrame' object has no attribute 'tstamp' File ./data-master/hobo/2023/hourly/10350107_Th.csv references HOBO ID=10350107, which is not found. The error Error: 'DataFrame' object has no attribute 'tstamp' indicates, that reading the ddtm column failed. If it was your file, please check it carefully. *Respond to this thread, if you changed something about your data* — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXKWE4UHRICAD7MHLKDCXZTWTJI53ANCNFSM6AAAAAATYFY5WI> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[nico51two]

Hi, the missing timestamp was most likely due to me having the wrong column name for the dttm column. Fixed and reuploaded. Sorry :)

…

On 11.01.2023 16:39, Mirko Mälicke wrote: @maxschmi [1] for reference This is a report on errors, warnings and issues logged by the server while uploading this years data. Referencing @data-hydenv/students [2] for your information. If you recognize some of the information below, you are welcome to comment, otherwise just ignore the mentioning. METADATA I guess this is a student who dropped out. Some entries are missing metadata: ---------------------------------- name hobo_id github_name data_available region latitude longitude exposition altitude influence description 14 ***, *** NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN Additionally, there was an exception, which means there might be missing metadata [ERROR]: (psycopg2.errors.UndefinedFunction) operator does not exist: character varying = timestamp without time zone LINE 3: ...RE details.key = 'altitude' AND details.str_value = '1900-01... ^ HINT: No operator matches the given name and argument types. You might need to add explicit type casts. [SQL: SELECT details.id AS details_id, details.key AS details_key, details.str_value AS details_str_value, details.int_value AS details_int_value, details.float_value AS details_float_value, details.bool_value AS details_bool_value FROM details WHERE details.key = %(key_1)s AND details.str_value = %(str_value_1)s LIMIT %(param_1)s] [parameters: {'key_1': 'altitude', 'str_value_1': datetime.datetime(1900, 1, 2, 12, 0), 'param_1': 1}] (Background on this error at: https://sqlalche.me/e/14/f405) This basically means one student had a date in the _altitude_ column, which was parsed as the 2.1.1900 12:00. Maybe putted 2.1 into the cell and Excel formatted as a date? RAW DATA These are either extra HOBO time series and can be ignored, or the Metadata file references the wrong HOBO (typo?) File ./data-master/hobo/2023/raw/10350089.csv references HOBO ID=10350089, which is not found. File ./data-master/hobo/2023/raw/10347324.csv references HOBO ID=10347324, which is not found. File ./data-master/hobo/2023/raw/10347327.csv references HOBO ID=10347327, which is not found. File ./data-master/hobo/2023/raw/10347333.csv references HOBO ID=10347333, which is not found. File ./data-master/hobo/2023/raw/10350107.csv references HOBO ID=10350107, which is not found. File ./data-master/hobo/2023/raw/10347366.csv references HOBO ID=10347366, which is not found. -- Reply to this email directly, view it on GitHub [3], or unsubscribe [4]. You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***> Links: ------ [1] https://github.com/maxschmi [2] https://github.com/orgs/data-hydenv/teams/students [3] #18 [4] https://github.com/notifications/unsubscribe-auth/ARPD5RSPNZYVV5SBFHFEKADWR3H3TANCNFSM6AAAAAATYFY5WI

[adia-saurey]

Hello,
The ref_id and ref_r2 columns for hobo id 10350095 have been added.
Thank you!

[MarilenaAlmeida]

Hi,
I've corrected and re-uploaded the data from id 10347333.
I hope now it works!

[arrpit19]

Hi, I have added all the information for 10350107. Hope it works this time :)

…

On Fri, 20 Jan, 2023, 10:09 Mirko Mälicke, ***@***.***> wrote: ***@***.***/students <https://github.com/orgs/data-hydenv/teams/students> the 2023 quality checked data is uploaded.* There were some warnings and errors, read through them and reply if you feel like you know something about it. File ./data-master/hobo/2023/hourly/10347324_Th.csv references HOBO ID=10347324, which is not found. Parsing file './data-master/hobo/2023/hourly/10347355_Th.csv' was not successfull. Do not edit the files by hand! Error: 'DataFrame' object has no attribute 'tstamp' File ./data-master/hobo/2023/hourly/10350089_Th.csv references HOBO ID=10350089, which is not found. Parsing file './data-master/hobo/2023/hourly/10349996_Th.csv' was not successfull. Do not edit the files by hand! Error: time data '2022.12.01. 00:00:00' does not match format '%Y.%m.%d %H:%M:%S' File ./data-master/hobo/2023/hourly/10347333_Th.csv references HOBO ID=10347333, which is not found. Parsing file './data-master/hobo/2023/hourly/10350049_Th.csv' was not successfull. Do not edit the files by hand! Error: 'DataFrame' object has no attribute 'tstamp' File ./data-master/hobo/2023/hourly/10350107_Th.csv references HOBO ID=10350107, which is not found. The error Error: 'DataFrame' object has no attribute 'tstamp' indicates, that reading the ddtm column failed. If it was your file, please check it carefully. *Respond to this thread, if you changed something about your data* — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXKWE4UHRICAD7MHLKDCXZTWTJI53ANCNFSM6AAAAAATYFY5WI> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[mmaelicke]

@arrpit19, @MarilenaAlmeida, @nico51two, @kg200 thanks for your quick replies!

We produced a new error :)

[ERROR]: File: './data-master/hobo/2023/hourly/10350049_Th.csv' errored. Message: (psycopg2.errors.DatatypeMismatch) column "value" is of type real but expression is of type text
LINE 1: ...ity_flag_id) SELECT meta_id, variable_id, tstamp, value, qua...
                                                             ^
HINT:  You will need to rewrite or cast the expression.

[SQL: INSERT INTO data (meta_id, variable_id, tstamp, value, quality_flag_id) SELECT meta_id, variable_id, tstamp, value, quality_flag_id FROM hnnwjjpjmcizpgpz ON CONFLICT (meta_id, variable_id, tstamp) DO NOTHING]
(Background on this error at: https://sqlalche.me/e/14/f405)

it is the file: './data-master/hobo/2023/hourly/10350049_Th.csv', please check the temperature column. I guess there is some kind of character there, as it presents itself as a character datatype.

[nico51two]

yeah sorry I only now managed to work my backlog from being sick in the first week so I guess my file still had NAs in there... i'm finishing up as we speak and will upload in ~5 mins so check again in 15 :D ...sorry for the inconvenience

[nico51two]

Aaaaaaand it's up...thanks for your patience

[xamfru]

@mmaelicke

My uploaded file was read in correctly, when i was trying to aggregate for calculating indices of all HOBOS in 2023 i saw that there are only 38 unique timestamps in my file. So i uploaded it with corrected timestamps. Is it possible to update the db data table once again?

Sorry for the inconvience