0001-eventlog-importing.md

Deploy instructions for head.hackage eventlog importing

1. Merge the MR that introduces the eventlog importing into the ghc-perf-import repo
2. Run the SQL to set up the new tables and types in the database (below)
3. Update the submodule pin in the ghc-servers repo to include the merged code
4. nixos-rebuild the server

SQL for creating tables

create type heap_measure as enum ('megablock','block', 'live');
create table eventlog_import (
    import_id bigserial primary key,
    gitlab_job_id text not null,
    commit_id bigserial not null references commits (commit_id),
    date timestamp with time zone not null
);
create table package (
    package_id bigserial primary key,
    name text not null,
    ver text not null,
    sha text,
    unique (name, ver, sha)
);
create table heap_event (
    heap_event_id bigserial primary key,
    import_id bigserial not null references eventlog_import (import_id),
    package_id bigserial not null references package (package_id),
    timestamp bigint,
    measure heap_measure not null,
    value bigint
);

SQL privileges

Read and write privs for the script runner.

Read privs for grafana.

During deployment, privileges were actually applied with trial and error. This was written after the fact. Caveat emptor.

grant insert, select
on eventlog_import, heap_event, package
to ghc_perf;
;
grant usage
on eventlog_import_import_id_seq,
   package_package_id_seq,
   heap_event_heap_event_id_seq
to ghc_perf;
;
grant select
on eventlog_import, heap_event, package
to ghc_perf_web;

SQL indices

This index was added to speed up the Eventlog Import Overview panel on Grafana.

create index heap_event_import_id_idx on heap_event (import_id);

Followup

Here's a query that will inspect the output of the script:

select c.commit_sha, i.gitlab_job_id, i.date "import date", p.name, p.ver,  e.timestamp, e.measure, e.value
from commits c
join eventlog_import i
on c.commit_id = i.commit_id
join heap_event e
on e.import_id = i.import_id
join package p
on e.package_id = p.package_id
limit 20;