Skip to content

Conversation

@hubgeter
Copy link
Contributor

What problem does this PR solve?

Problem Summary:
This PR fixes the semantics of reading Parquet INT96 values. Previously, INT96 was interpreted as a time zone–aware type, causing the read results to vary with the set timezone configuration.
This PR changes the behavior to always convert INT96 values to UTC, treating them as time zone–independent timestamps.

Release note

Parse Parquet INT96 physical type data into a no-timezone datetime.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 31, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31901 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 666297a2c335cd13b722c2b4d535e8f716ab6bf2, data reload: false

------ Round 1 ----------------------------------
q1	17636	4216	4042	4042
q2	2031	366	237	237
q3	10185	1270	734	734
q4	10204	898	316	316
q5	7553	2159	1915	1915
q6	194	171	137	137
q7	946	807	668	668
q8	9283	1430	1150	1150
q9	5510	4736	4721	4721
q10	6815	1810	1428	1428
q11	488	303	290	290
q12	736	742	602	602
q13	17826	3819	3065	3065
q14	296	287	274	274
q15	594	515	518	515
q16	707	689	644	644
q17	720	749	617	617
q18	6650	6505	6381	6381
q19	1112	961	618	618
q20	402	373	247	247
q21	3138	2562	2356	2356
q22	1102	1055	944	944
Total cold run time: 104128 ms
Total hot run time: 31901 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4175	4062	4042	4042
q2	338	395	333	333
q3	2128	2589	2222	2222
q4	1324	1749	1297	1297
q5	4078	4014	3992	3992
q6	208	170	129	129
q7	1890	1845	2102	1845
q8	2610	2464	2361	2361
q9	7224	7161	7124	7124
q10	2499	2784	2338	2338
q11	529	486	458	458
q12	730	769	622	622
q13	3622	4045	3391	3391
q14	406	301	291	291
q15	542	500	498	498
q16	660	657	649	649
q17	1138	1344	1347	1344
q18	7858	7934	8137	7934
q19	920	911	932	911
q20	2095	2018	1911	1911
q21	4697	4477	4237	4237
q22	1139	1086	988	988
Total cold run time: 50810 ms
Total hot run time: 48917 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174160 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 666297a2c335cd13b722c2b4d535e8f716ab6bf2, data reload: false

query5	4490	593	449	449
query6	332	227	213	213
query7	4224	478	279	279
query8	351	254	242	242
query9	8758	2600	2637	2600
query10	492	379	318	318
query11	15178	15043	15060	15043
query12	181	118	115	115
query13	1272	505	416	416
query14	6216	2957	2733	2733
query14_1	2631	2641	2678	2641
query15	198	195	179	179
query16	984	403	469	403
query17	1123	736	570	570
query18	2530	425	333	333
query19	225	221	185	185
query20	117	118	112	112
query21	224	151	115	115
query22	3951	4002	3824	3824
query23	15981	15694	15438	15438
query23_1	15435	15568	15500	15500
query24	7366	1579	1238	1238
query24_1	1216	1201	1181	1181
query25	528	469	401	401
query26	1228	262	157	157
query27	2779	457	291	291
query28	4549	2201	2194	2194
query29	776	523	429	429
query30	313	236	214	214
query31	803	627	556	556
query32	78	71	69	69
query33	528	326	290	290
query34	891	882	540	540
query35	755	801	697	697
query36	903	887	853	853
query37	127	99	78	78
query38	2725	2679	2618	2618
query39	778	743	730	730
query39_1	733	714	710	710
query40	217	138	121	121
query41	73	68	66	66
query42	104	105	107	105
query43	457	449	421	421
query44	1339	762	765	762
query45	191	186	181	181
query46	872	964	617	617
query47	1447	1530	1378	1378
query48	334	337	257	257
query49	647	429	340	340
query50	663	295	232	232
query51	3754	3819	3804	3804
query52	113	112	100	100
query53	300	333	277	277
query54	305	289	260	260
query55	81	81	76	76
query56	302	318	317	317
query57	1017	1064	978	978
query58	270	268	262	262
query59	2092	2141	2172	2141
query60	336	328	302	302
query61	192	181	182	181
query62	383	375	324	324
query63	304	268	270	268
query64	5128	1360	995	995
query65	3761	3688	3768	3688
query66	1433	438	311	311
query67	14934	15281	15815	15281
query68	8237	1010	725	725
query69	510	339	313	313
query70	1048	903	941	903
query71	355	304	279	279
query72	6130	3413	3508	3413
query73	767	723	316	316
query74	8810	8796	8645	8645
query75	2835	2845	2499	2499
query76	3889	1066	662	662
query77	546	386	277	277
query78	9865	9997	9080	9080
query79	1214	929	618	618
query80	669	585	519	519
query81	494	264	230	230
query82	212	145	107	107
query83	263	248	248	248
query84	258	120	109	109
query85	882	525	462	462
query86	319	315	316	315
query87	2872	2858	2758	2758
query88	3197	2324	2291	2291
query89	404	356	332	332
query90	1960	154	159	154
query91	179	164	155	155
query92	70	65	61	61
query93	1008	919	573	573
query94	568	335	289	289
query95	586	344	307	307
query96	588	448	205	205
query97	2310	2381	2291	2291
query98	220	196	196	196
query99	600	577	513	513
Total cold run time: 253913 ms
Total hot run time: 174160 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 27.17 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 666297a2c335cd13b722c2b4d535e8f716ab6bf2, data reload: false

query1	0.06	0.05	0.05
query2	0.11	0.06	0.05
query3	0.25	0.10	0.09
query4	1.61	0.11	0.11
query5	0.27	0.24	0.26
query6	1.15	0.64	0.66
query7	0.04	0.02	0.03
query8	0.05	0.04	0.04
query9	0.58	0.51	0.49
query10	0.57	0.55	0.56
query11	0.16	0.11	0.11
query12	0.16	0.13	0.13
query13	0.63	0.60	0.59
query14	0.97	0.99	0.98
query15	0.82	0.80	0.78
query16	0.42	0.42	0.37
query17	1.03	1.05	1.08
query18	0.22	0.21	0.21
query19	1.95	1.91	1.89
query20	0.02	0.02	0.01
query21	15.45	0.30	0.14
query22	4.86	0.05	0.05
query23	16.18	0.30	0.10
query24	0.94	0.61	0.40
query25	0.11	0.08	0.08
query26	0.13	0.14	0.12
query27	0.06	0.06	0.04
query28	4.97	1.05	0.88
query29	12.60	3.97	3.18
query30	0.28	0.13	0.12
query31	2.81	0.67	0.38
query32	3.23	0.57	0.48
query33	2.97	3.03	3.08
query34	16.64	5.09	4.48
query35	4.45	4.48	4.45
query36	0.65	0.50	0.49
query37	0.11	0.06	0.06
query38	0.08	0.04	0.04
query39	0.04	0.04	0.03
query40	0.19	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 97.98 s
Total hot run time: 27.17 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 78.05% (32/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.39% (18964/35518)
Line Coverage 39.26% (175969/448170)
Region Coverage 33.82% (136138/402479)
Branch Coverage 34.76% (58796/169164)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 78.05% (32/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.15% (24357/34723)
Line Coverage 56.51% (252588/446978)
Region Coverage 51.06% (207627/406642)
Branch Coverage 52.49% (89083/169726)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants