Trouble with GitLab bots
Description of the problem
I will avoid trying to diagnose the problem in this description.
1. 403s are showing up in ghc-perf-gitlab-import-bot.service.
They have done so as far back as journalctl can see. However, there are far more failures recorded in the time range than there are 403s in the log. Does this indicate that 403s only happen sometimes?
Number of Recorded Failures: 231
ghc_perf=> select count(distinct job_id) from ci_failure where job_date > '2022-12-10 00:00:00';
count
-------
231
(1 row)
Number of 403 Results in the Log: 94
[bryan@gitlab:~]$ journalctl -u ghc-perf-gitlab-import-bot.service -g '403 Forbidden' |wc
94 846 11402
Earliest log entry: Dec 09
[bryan@gitlab:~]$ journalctl -u ghc-perf-gitlab-import-bot.service |head -n1
Dec 09 03:11:22 gitlab ghc-perf-gitlab-import-bot-start[336780]: INFO:root:event: {'object_kind': 'build', 'ref': 'wip....
Earliest 403: Dec 09
[bryan@gitlab:~]$ journalctl -u ghc-perf-gitlab-import-bot.service -g '403 Forbidden' |head -n1
Dec 09 20:47:41 gitlab ghc-perf-gitlab-import-bot-start[2121060]: gitlab.exceptions.GitlabHttpError: 403: 403 Forbidden
Latest 403: Dec 22
[bryan@gitlab:~]$ journalctl -u ghc-perf-gitlab-import-bot.service -g '403 Forbidden' |tail -n1
Dec 21 05:33:46 gitlab ghc-perf-gitlab-import-bot-start[618560]: gitlab.exceptions.GitlabJobRetryError: 403: 403 Forbidden
2. The latest perf data is from the 19th.
False alarm.
3. The latest commit data is from the 19th.
False alarm.