MISP Purge Events tool v.0.1 released

In various cases it would be useful to expire/purge older events, cleanup blocklist or delete events from a specific organization.

The MISP Purge Event tool is written in python and can assist in various cleaning operations

The tool supports deletion of events based on first seen and last seen and as default covers all events, however the tool also supports only deleting events based on a specific organizations UUID or perform clean up within the “Blocklisted Events” (Previously deleted events that should not be re-synced)

To protect events from specific Organizations this can be added to an exclusions list, and last but not least Feeds with fixed event will as default be excluded.

Download the tool from: https://github.com/eCrimeLabs/MISP-PurgeEvents

Warning

  • It is always recommended to perform a backup prior to deletion of data, and as minimum do a dryrun first to understand the events to be deleted.

  • Never run this on an MISP instance you do not own.

Notice

If a large set of events has to be purged it is highly recommended to disable/flush the correlation table prior, else we have seen MISP databases getting into unstable states, due to waits for cleanups in correlations.

Why can the deletion of events take so long time, the answer is that there is much cleanup going on that can be quite database heavy, with deletes of:

  • attributes

  • eventtags

  • attribute tags

  • objects

  • proposals

  • event reports

  • discussion threads

  • correlations

And the correlations typically being the biggest, depending on relations to other events.

MISP disable correlations database

This can be achieved by logging in to MISP as a site admin, and go through

  • "Administration"

    • "Server Settings & Administration"

      • "MISP Settings"

        • Change "MISP.completely_disable_correlation" to True

Validate that the correlations table has been clean by checking "Diagnostics" under "SQL database status".

When the large task is completed remember to enable correlations again.





Benchmarks

For a MISP instance with the below data volume (Correlations were removed prior to running the tool)

  • Events: 9.460 (From 2011 and until 2022)

  • Attributes: 1.424.251

Execution time 12m11.239s

MISP Purge Event Config

The config file is divided up into 3 elements

  • MISP connection information

  • Chunk size (How many events to delete at a time)

  • Exclusions (Contains organizations UUID’s where events created by these are excluded from any deletion)