Skip to content
GitLab
Open
Issue created by Bryan R@chreekat🚒Maintainer

T5435 tests are fragile

A megaissue for the following big table. It groups failures by job name and month. There are multiple variants of the test, and all variants seen in the given bucket are listed. (I.e., among the 5 jobs that failed in the first row, there was at least one failure of type v_gcc and one of type v_asm_a: possibly the same job.)

To summarize, it's clear that something got a lot better after November! But it's also clear that T5435 tests still fail on all platforms. I am considering just marking all of them fragile: this has been a problem since forever and still impacts CI.

Count of failures for T5435* jobs
count(di month types seen job
5 2023-02 v_gcc,v_asm_a x86_64-windows-validate
1 2023-02 dyn_asm validate-aarch64-darwin
21 2023-01 v_gcc,v_asm_a,dyn_asm x86_64-windows-validate
4 2023-01 v_asm_a,v_asm_b,v_gcc x86_64-linux-alpine3_12-validate+fully_static
2 2023-01 v_asm_a,v_gcc nightly-x86_64-windows-validate
2 2023-01 v_asm_a release-x86_64-windows-int_native-release+no_split_sections
1 2023-01 dyn_asm validate-aarch64-darwin
1 2023-01 v_asm_a nightly-x86_64-windows-int_native-validate
1 2022-12 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-validate+fully_static
1 2022-12 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static
366 2022-11 v_asm_a,v_asm_b,v_gcc x86_64-linux-alpine3_12-validate+fully_static
28 2022-11 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-validate+fully_static
27 2022-11 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static
10 2022-11 v_asm_a,v_gcc x86_64-windows-validate
2 2022-11 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-release+fully_static
2 2022-11 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-int_native-release+fully_static
1 2022-11 v_gcc nightly-x86_64-windows-validate
1 2022-11 v_gcc nightly-x86_64-windows-int_native-validate
351 2022-10 v_asm_a,v_asm_b,v_gcc,dyn_asm x86_64-linux-alpine3_12-validate+fully_static
29 2022-10 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-validate+fully_static
28 2022-10 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static
15 2022-10 v_asm_a,v_gcc,dyn_asm x86_64-windows-validate
3 2022-10 dyn_asm,v_asm_a,v_asm_b,v_gcc x86_64-linux-deb10-numa-slow-validate
2 2022-10 dyn_asm,v_asm_a,v_asm_b,v_gcc x86_64-linux-fedora33-release
2 2022-10 dyn_asm,v_asm_a,v_asm_b,v_gcc x86_64-linux-deb10-validate+debug_info
2 2022-10 dyn_asm,v_asm_a,v_asm_b,v_gcc x86_64-linux-deb10-unreg-validate
2 2022-10 dyn_asm,v_asm_a,v_asm_b,v_gcc x86_64-linux-deb10-int_native-validate
2 2022-10 dyn_asm,v_asm_a,v_asm_b,v_gcc i386-linux-deb9-validate
2 2022-10 dyn_asm,v_asm_a,v_asm_b,v_gcc aarch64-linux-deb10-validate
2 2022-10 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-release+fully_static
2 2022-10 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-int_native-release+fully_static
1 2022-10 dyn_asm,v_asm_a,v_gcc x86_64-darwin-validate
1 2022-10 dyn_asm release-aarch64-darwin
1 2022-10 v_asm_a nightly-x86_64-windows-int_native-validate
280 2022-09 v_asm_a,v_asm_b,v_gcc x86_64-linux-alpine3_12-validate+fully_static
29 2022-09 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static
28 2022-09 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-validate+fully_static
7 2022-09 v_gcc,v_asm_a x86_64-windows-validate
1 2022-09 dyn_asm,v_asm_a,v_gcc x86_64-darwin-validate
3 2022-09 v_asm_a,v_gcc nightly-x86_64-windows-validate
1 2022-09 v_gcc nightly-x86_64-windows-int_native-validate
383 2022-08 v_asm_a,v_asm_b,v_gcc x86_64-linux-alpine3_12-validate+fully_static
20 2022-08 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-validate+fully_static
19 2022-08 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static
12 2022-08 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-release+fully_static
11 2022-08 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-int_native-release+fully_static
20 2022-08 v_asm_a nightly-x86_64-freebsd13-validate
17 2022-08 v_gcc,v_asm_a x86_64-windows-validate
6 2022-08 v_asm_a x86_64-freebsd13-validate
4 2022-08 v_asm_a nightly-x86_64-windows-int_native-validate
2 2022-08 v_gcc,v_asm_a release-x86_64-windows-int_native-release
2 2022-08 v_asm_a,v_gcc nightly-x86_64-windows-validate
1 2022-08 v_asm_a release-x86_64-windows-release
348 2022-07 v_asm_a,v_asm_b,v_gcc x86_64-linux-alpine3_12-validate+fully_static
32 2022-07 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-validate+fully_static
32 2022-07 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static
8 2022-07 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-release+fully_static
8 2022-07 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-int_native-release+fully_static
5 2022-07 v_asm_a,v_gcc x86_64-windows-validate
1 2022-07 v_asm_a,v_gcc x86_64-darwin-validate
2 2022-07 v_asm_a nightly-x86_64-windows-validate
2 2022-07 v_gcc,v_asm_a nightly-x86_64-windows-int_native-validate
286 2022-06 v_asm_a,v_asm_b,v_gcc x86_64-linux-alpine3_12-validate+fully_static
30 2022-06 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static
30 2022-06 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-validate+fully_static
5 2022-06 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-release+fully_static
5 2022-06 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-int_native-release+fully_static
11 2022-06 v_gcc,v_asm_a x86_64-windows-validate
1 2022-06 v_asm_a,v_asm_b,v_gcc validate-x86_64-linux-alpine-hadrian
1 2022-06 v_asm_a release-x86_64-windows-release
1 2022-06 v_asm_a release-x86_64-windows-int_native-release
479 2022-05 v_asm_a,v_asm_b,v_gcc,dyn_asm x86_64-linux-alpine3_12-validate+fully_static
29 2022-05 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-validate+fully_static
29 2022-05 v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static
6 2022-05 dyn_asm,v_asm_a,v_asm_b,v_gcc nightly-x86_64-linux-deb10-validate+thread_sanitizer
19 2022-05 v_gcc,v_asm_a x86_64-windows-validate
6 2022-05 v_asm_a,v_asm_b,v_gcc validate-x86_64-linux-alpine-hadrian
4 2022-05 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-perf+fully_static
3 2022-05 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-int_native-perf+fully_static
2 2022-05 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-int_native-release+fully_static
1 2022-05 dyn_asm,v_asm_a,v_asm_b,v_gcc x86_64-linux-fedora33-release
1 2022-05 dyn_asm,v_asm_a,v_asm_b,v_gcc x86_64-linux-deb10-validate+debug_info
1 2022-05 dyn_asm,v_asm_a,v_asm_b,v_gcc x86_64-linux-deb10-numa-slow-validate
1 2022-05 dyn_asm,v_asm_a,v_asm_b,v_gcc x86_64-linux-deb10-int_native-validate
2 2022-05 v_asm_b,v_asm_a,v_gcc release-x86_64-linux-alpine3_12-release+fully_static
1 2022-05 dyn_asm,v_asm_a,v_gcc aarch64-darwin-validate
1 2022-05 v_asm_a nightly-x86_64-windows-int_native-validate
5 2022-04 v_asm_a,v_asm_b,v_gcc x86_64-linux-alpine3_12-validate+fully_static
1 2022-04 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-perf+fully_static
1 2022-04 v_asm_a,v_asm_b,v_gcc release-x86_64-linux-alpine3_12-int_native-perf+fully_static 
.width 8 7 30 100
.mode markdown
with
    typ as (
        select rowid, 'v_asm_a' 'typ' from job_trace where trace match 'NEAR(unexpected T5435_v_asm a)'
        union 
        select rowid, 'v_asm_b' 'typ' from job_trace where trace match 'NEAR(unexpected T5435_v_asm b)'
        union 
        select rowid, 'v_gcc' from job_trace where trace match 'NEAR(unexpected T5435_v_gcc)'
        union
        select rowid, 'dyn_asm' from job_trace where trace match 'NEAR(unexpected T5435_dyn_asm)'
    )
select
    count(distinct job_id),
    strftime('%Y-%m', json ->> 'created_at') 'month',
    group_concat(distinct typ.typ) 'types seen',
    json->>'$.name' 'job'
from job
left join typ on typ.rowid = job_id
where job_id in (
    select rowid from job_trace where trace match  'NEAR(unexpected T5435*)')
group by
    strftime('%Y-%m', json ->> 'created_at'),
    json->>'$.name'
order by strftime('%Y-%m', json ->> 'created_at') desc, count(*) desc

Besides these unexpected failures, T5435_v_asm_a, T5435_v_asm_b, and T5435_v_gcc are expected broken on Linux on the static way (#20706), and they are also expected broken on 'arm' (Does that mean armv7 and aarch64, or just armv7?) (#17559).

List of most recent failures:

Related issues: #21788, #17559, #13283 (closed), #11299 (closed), #8458 (closed), #22072

Edited by Bryan R
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information