Trouble with GitLab bots

Description of the problem

I will avoid trying to diagnose the problem in this description.

1. 403s are showing up in ghc-perf-gitlab-import-bot.service.

They have done so as far back as journalctl can see. However, there are far more failures recorded in the time range than there are 403s in the log. Does this indicate that 403s only happen sometimes?

Number of Recorded Failures: 231

ghc_perf=> select count(distinct job_id) from ci_failure where job_date > '2022-12-10 00:00:00';
count
-------
231
(1 row)

Number of 403 Results in the Log: 94

[bryan@gitlab:~]$ journalctl -u ghc-perf-gitlab-import-bot.service  -g '403 Forbidden' |wc
    94     846   11402

Earliest log entry: Dec 09

[bryan@gitlab:~]$ journalctl -u ghc-perf-gitlab-import-bot.service |head -n1
Dec 09 03:11:22 gitlab ghc-perf-gitlab-import-bot-start[336780]: INFO:root:event: {'object_kind': 'build', 'ref': 'wip....

Earliest 403: Dec 09

[bryan@gitlab:~]$ journalctl -u ghc-perf-gitlab-import-bot.service  -g '403 Forbidden' |head -n1
Dec 09 20:47:41 gitlab ghc-perf-gitlab-import-bot-start[2121060]: gitlab.exceptions.GitlabHttpError: 403: 403 Forbidden

Latest 403: Dec 22

[bryan@gitlab:~]$ journalctl -u ghc-perf-gitlab-import-bot.service  -g '403 Forbidden' |tail -n1
Dec 21 05:33:46 gitlab ghc-perf-gitlab-import-bot-start[618560]: gitlab.exceptions.GitlabJobRetryError: 403: 403 Forbidden

2. The latest perf data is from the 19th.

False alarm.

3. The latest commit data is from the 19th.

False alarm.