T5435 tests are fragile
A megaissue for the following big table. It groups failures by job name and month. There are multiple variants of the test, and all variants seen in the given bucket are listed. (I.e., among the 5 jobs that failed in the first row, there was at least one failure of type v_gcc
and one of type v_asm_a
: possibly the same job.)
To summarize, it's clear that something got a lot better after November! But it's also clear that T5435 tests still fail on all platforms. I am considering just marking all of them fragile: this has been a problem since forever and still impacts CI.
Count of failures for T5435* jobs
count(di | month | types seen | job |
---|---|---|---|
5 | 2023-02 | v_gcc,v_asm_a | x86_64-windows-validate |
1 | 2023-02 | dyn_asm | validate-aarch64-darwin |
21 | 2023-01 | v_gcc,v_asm_a,dyn_asm | x86_64-windows-validate |
4 | 2023-01 | v_asm_a,v_asm_b,v_gcc | x86_64-linux-alpine3_12-validate+fully_static |
2 | 2023-01 | v_asm_a,v_gcc | nightly-x86_64-windows-validate |
2 | 2023-01 | v_asm_a | release-x86_64-windows-int_native-release+no_split_sections |
1 | 2023-01 | dyn_asm | validate-aarch64-darwin |
1 | 2023-01 | v_asm_a | nightly-x86_64-windows-int_native-validate |
1 | 2022-12 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-validate+fully_static |
1 | 2022-12 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static |
366 | 2022-11 | v_asm_a,v_asm_b,v_gcc | x86_64-linux-alpine3_12-validate+fully_static |
28 | 2022-11 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-validate+fully_static |
27 | 2022-11 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static |
10 | 2022-11 | v_asm_a,v_gcc | x86_64-windows-validate |
2 | 2022-11 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-release+fully_static |
2 | 2022-11 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-int_native-release+fully_static |
1 | 2022-11 | v_gcc | nightly-x86_64-windows-validate |
1 | 2022-11 | v_gcc | nightly-x86_64-windows-int_native-validate |
351 | 2022-10 | v_asm_a,v_asm_b,v_gcc,dyn_asm | x86_64-linux-alpine3_12-validate+fully_static |
29 | 2022-10 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-validate+fully_static |
28 | 2022-10 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static |
15 | 2022-10 | v_asm_a,v_gcc,dyn_asm | x86_64-windows-validate |
3 | 2022-10 | dyn_asm,v_asm_a,v_asm_b,v_gcc | x86_64-linux-deb10-numa-slow-validate |
2 | 2022-10 | dyn_asm,v_asm_a,v_asm_b,v_gcc | x86_64-linux-fedora33-release |
2 | 2022-10 | dyn_asm,v_asm_a,v_asm_b,v_gcc | x86_64-linux-deb10-validate+debug_info |
2 | 2022-10 | dyn_asm,v_asm_a,v_asm_b,v_gcc | x86_64-linux-deb10-unreg-validate |
2 | 2022-10 | dyn_asm,v_asm_a,v_asm_b,v_gcc | x86_64-linux-deb10-int_native-validate |
2 | 2022-10 | dyn_asm,v_asm_a,v_asm_b,v_gcc | i386-linux-deb9-validate |
2 | 2022-10 | dyn_asm,v_asm_a,v_asm_b,v_gcc | aarch64-linux-deb10-validate |
2 | 2022-10 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-release+fully_static |
2 | 2022-10 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-int_native-release+fully_static |
1 | 2022-10 | dyn_asm,v_asm_a,v_gcc | x86_64-darwin-validate |
1 | 2022-10 | dyn_asm | release-aarch64-darwin |
1 | 2022-10 | v_asm_a | nightly-x86_64-windows-int_native-validate |
280 | 2022-09 | v_asm_a,v_asm_b,v_gcc | x86_64-linux-alpine3_12-validate+fully_static |
29 | 2022-09 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static |
28 | 2022-09 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-validate+fully_static |
7 | 2022-09 | v_gcc,v_asm_a | x86_64-windows-validate |
1 | 2022-09 | dyn_asm,v_asm_a,v_gcc | x86_64-darwin-validate |
3 | 2022-09 | v_asm_a,v_gcc | nightly-x86_64-windows-validate |
1 | 2022-09 | v_gcc | nightly-x86_64-windows-int_native-validate |
383 | 2022-08 | v_asm_a,v_asm_b,v_gcc | x86_64-linux-alpine3_12-validate+fully_static |
20 | 2022-08 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-validate+fully_static |
19 | 2022-08 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static |
12 | 2022-08 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-release+fully_static |
11 | 2022-08 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-int_native-release+fully_static |
20 | 2022-08 | v_asm_a | nightly-x86_64-freebsd13-validate |
17 | 2022-08 | v_gcc,v_asm_a | x86_64-windows-validate |
6 | 2022-08 | v_asm_a | x86_64-freebsd13-validate |
4 | 2022-08 | v_asm_a | nightly-x86_64-windows-int_native-validate |
2 | 2022-08 | v_gcc,v_asm_a | release-x86_64-windows-int_native-release |
2 | 2022-08 | v_asm_a,v_gcc | nightly-x86_64-windows-validate |
1 | 2022-08 | v_asm_a | release-x86_64-windows-release |
348 | 2022-07 | v_asm_a,v_asm_b,v_gcc | x86_64-linux-alpine3_12-validate+fully_static |
32 | 2022-07 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-validate+fully_static |
32 | 2022-07 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static |
8 | 2022-07 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-release+fully_static |
8 | 2022-07 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-int_native-release+fully_static |
5 | 2022-07 | v_asm_a,v_gcc | x86_64-windows-validate |
1 | 2022-07 | v_asm_a,v_gcc | x86_64-darwin-validate |
2 | 2022-07 | v_asm_a | nightly-x86_64-windows-validate |
2 | 2022-07 | v_gcc,v_asm_a | nightly-x86_64-windows-int_native-validate |
286 | 2022-06 | v_asm_a,v_asm_b,v_gcc | x86_64-linux-alpine3_12-validate+fully_static |
30 | 2022-06 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static |
30 | 2022-06 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-validate+fully_static |
5 | 2022-06 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-release+fully_static |
5 | 2022-06 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-int_native-release+fully_static |
11 | 2022-06 | v_gcc,v_asm_a | x86_64-windows-validate |
1 | 2022-06 | v_asm_a,v_asm_b,v_gcc | validate-x86_64-linux-alpine-hadrian |
1 | 2022-06 | v_asm_a | release-x86_64-windows-release |
1 | 2022-06 | v_asm_a | release-x86_64-windows-int_native-release |
479 | 2022-05 | v_asm_a,v_asm_b,v_gcc,dyn_asm | x86_64-linux-alpine3_12-validate+fully_static |
29 | 2022-05 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-validate+fully_static |
29 | 2022-05 | v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static |
6 | 2022-05 | dyn_asm,v_asm_a,v_asm_b,v_gcc | nightly-x86_64-linux-deb10-validate+thread_sanitizer |
19 | 2022-05 | v_gcc,v_asm_a | x86_64-windows-validate |
6 | 2022-05 | v_asm_a,v_asm_b,v_gcc | validate-x86_64-linux-alpine-hadrian |
4 | 2022-05 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-perf+fully_static |
3 | 2022-05 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-int_native-perf+fully_static |
2 | 2022-05 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-int_native-release+fully_static |
1 | 2022-05 | dyn_asm,v_asm_a,v_asm_b,v_gcc | x86_64-linux-fedora33-release |
1 | 2022-05 | dyn_asm,v_asm_a,v_asm_b,v_gcc | x86_64-linux-deb10-validate+debug_info |
1 | 2022-05 | dyn_asm,v_asm_a,v_asm_b,v_gcc | x86_64-linux-deb10-numa-slow-validate |
1 | 2022-05 | dyn_asm,v_asm_a,v_asm_b,v_gcc | x86_64-linux-deb10-int_native-validate |
2 | 2022-05 | v_asm_b,v_asm_a,v_gcc | release-x86_64-linux-alpine3_12-release+fully_static |
1 | 2022-05 | dyn_asm,v_asm_a,v_gcc | aarch64-darwin-validate |
1 | 2022-05 | v_asm_a | nightly-x86_64-windows-int_native-validate |
5 | 2022-04 | v_asm_a,v_asm_b,v_gcc | x86_64-linux-alpine3_12-validate+fully_static |
1 | 2022-04 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-perf+fully_static |
1 | 2022-04 | v_asm_a,v_asm_b,v_gcc | release-x86_64-linux-alpine3_12-int_native-perf+fully_static |
.width 8 7 30 100
.mode markdown
with
typ as (
select rowid, 'v_asm_a' 'typ' from job_trace where trace match 'NEAR(unexpected T5435_v_asm a)'
union
select rowid, 'v_asm_b' 'typ' from job_trace where trace match 'NEAR(unexpected T5435_v_asm b)'
union
select rowid, 'v_gcc' from job_trace where trace match 'NEAR(unexpected T5435_v_gcc)'
union
select rowid, 'dyn_asm' from job_trace where trace match 'NEAR(unexpected T5435_dyn_asm)'
)
select
count(distinct job_id),
strftime('%Y-%m', json ->> 'created_at') 'month',
group_concat(distinct typ.typ) 'types seen',
json->>'$.name' 'job'
from job
left join typ on typ.rowid = job_id
where job_id in (
select rowid from job_trace where trace match 'NEAR(unexpected T5435*)')
group by
strftime('%Y-%m', json ->> 'created_at'),
json->>'$.name'
order by strftime('%Y-%m', json ->> 'created_at') desc, count(*) desc
Besides these unexpected failures, T5435_v_asm_a, T5435_v_asm_b, and T5435_v_gcc are expected broken on Linux on the static way (#20706), and they are also expected broken on 'arm' (Does that mean armv7 and aarch64, or just armv7?) (#17559).
List of most recent failures:
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1365036
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1364128
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1360471
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1359386
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1358109
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1357098
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1349313
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1348955
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1346809
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1346228
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1336013
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1327990
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1325759
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1325698
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1325418
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1324166
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1323698
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1322979
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1322914
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1320955
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1320202
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1317169
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1314997
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1311431
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1308140
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1306602
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1303797
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1296938
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1291845
- https://gitlab.haskell.org/ghc/ghc/-/jobs/1288740
Related issues: #21788, #17559, #13283 (closed), #11299 (closed), #8458 (closed), #22072