Trim Mailman archive for data deletion policy

Submitted by Jeff on Thu, 06/07/2023 - 08:40

Pseudo howto

1. Set list to emergency moderation

Visit https://lists.waspa.org.za/mailman/admin/complaints and switch on the Emergency moderation of all list traffic button.

2. Optionally send a test email to list to make sure it's getting held in mod queue

Emails that are held for moderation are located in ./mailman/data/. E.g., /var/virtual/waspa.org.za/mailman/data/heldmsg-complaints-195.pck.

3. Run https://console.waspa.org.za/toolkit/popspan.php

Select the appropriate domain (if applicable) and list, and enter the regex (regular expression).

Here we have entered Dec 31 [0-9:]+ 2019 to provide us with the (chronological) numeric sequence of all emails sent to the list on 31 December, 2019.

After some processing time, the result was provided as:

from 217045 to 217051, so we therefore want to 'trim' the archive with all emails starting the next day - 1 January 2020 - which will be from the reported "to" number, and use the next numeric increment, 217052)  in the "from" field "to" the present; for which we just need to enter a large out of range number.

Clicking on the "Proceed" button will create an mbox file named, in this case, complaints.trim which we will use to rebuild the archive.

4. Rearrange and apply ownership and permissions to the mbox files as required

cd /var/virtual/waspa.org.za/mailman/archives/private/complaints.mbox
mv complaints.mbox complaints.original
mv complaints.trim complaints.mbox
chown mailman:mailman *
chmod g+rw *

5. Rebuild the archive

In screen or tmux, first delete the archive directory (NB) containing the html, txt, gz and attachment files, and rebuild the archive with the arch command:

screen
mv /var/virtual/waspa.org.za/mailman/archives/private/complaints/ /tmp/
/var/virtual/waspa.org.za/mailman/bin/arch complaints

6. Check that the archives have been rebuilt correctly, and tidy up

Visit the list archive at https://lists.waspa.org.za/mailman/private/complaints/ and ensure they start and end correctly.

N.B.: Restart mailman service!

Delete the original mbox and /tmp/complaints/ dir.

Optionally revisit popspan.php and make a trimmed mbox of the current and previous day's emails in case anything should happen to go wrong and we need to restore the previous day's backup from Stu. This is left to the user's discretion.

cd /var/virtual/waspa.org.za/mailman/archives/private/complaints.mbox
rm complaints.original
rm -rf /tmp/complaints/

 

Category