Set Up ParseDMARC on Ubuntu 20.04 to Analyze DMARC Reports

ParseDMARC is an open-source, self-hosted DMARC report analyzer. For those who don’t know, DMARC is an email security standard that can protect your domain name from email spoofing and also identify incoming spoofed emails to protect end users. In a previous article, we discussed how to create DMARC DNS record for your domain name and used a third-party tool called PostMark to analyze DMARC reports. In this tutorial, we are going to set up ParseDMARC on Ubuntu 20.04 server so we can analyze DMARC reports without having to share the data with a third-party.

Prerequisites

It is assumed that you have created a DMARC DNS record for your domain name.

To generate visualized data, ParseDMARC relies on Elasticsearch and Kibana, both of which are RAM hungry. They will use about 2G RAM after installation and require more RAM to process data. So you need a server with at least 3G RAM. It’s also recommended that you use a server with at least 4 CPU cores.

You can click this special link to get $100 free credit on DigitalOcean. (For new users only). If you are already a DigitalOcean user, then you can click this special link to get $50 free credit on Vultr (for new users only). Once you have an account at DigitalOcean or Vultr, install Ubuntu 20.04 on your server and follow the instructions below.

Installing ParseDMARC on Ubuntu 20.04 Server

ParseDMARC is a Python program and can be installed from the PyPI (Python Package Index) software repository. First, we need to install the Python package installer on Ubuntu 20.04. In the following command, we install the python3-pip package because ParseDMARC can only work with Python 3. The geoipupdate package is used to update the MaxMind GeoIP database.

sudo apt install python3-pip geoipupdate

Then install the latest stable version of ParseDMARC with the following command.

sudo -H pip3 install -U parsedmarc

To check information about the parsedmarc package, you can run

pip3 show -f parsedmarc

To see command line options, run

parsedmarc --help

Installing Elasticsearch and Kibana on Ubuntu 20.04

ParseDMARC is a command-line program, which produces hard-to-read output. If you want to see visualized DMARC reports in a web-based interface, you need to install two other open-source programs: Elasticsearch and Kibana. Elasticsearch is a search and analytics engine and Kibana allows users to visualize data with charts and graphs in Elasicsearch.

We can install Elasticsearch and Kibana from the official repository with the following commands. Elasticsearch is written in Java programming language, so we also install the default-jre-headless (Java runtime environment) package in the last command.

sudo apt-get install -y apt-transport-https

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt-get update

sudo apt-get install -y default-jre-headless elasticsearch kibana

By default, the Elasticsearch systemd service is disabled. You can start and enable it with the following commands.

sudo systemctl start elasticsearch

sudo systemctl enable elasticsearch

Then check the status.

systemctl status elasticsearch

We can see that it’s now enabled and running, and it uses 1.3G RAM. (Hint: If this command doesn’t quit immediately, press Q to make it quit.)

parsedmarc-elasticsearch-ubuntu 20.04

We also need to do the same for Kibaba.

sudo systemctl start kibana

sudo systemctl enable kibana

Check the status.

systemctl status kibana

parsedmarc kibana ubuntu 20.04

Setting Up Nginx Reverse Proxy for Elastic

Elastic web server listens on 127.0.0.1:5601. We can use Nginx to set up reverse proxy to allow for remote access and also protect Elastic web interface. Run the following command to install Nginx from Ubuntu 20.04 repository.

sudo apt install nginx

Create a Nginx virtual host file for Elastic.

sudo nano /etc/nginx/conf.d/elastic.conf

Put the following lines into the file. Replace the placeholder as necessary and you should create a DNS A record for the sub-domain.

server {
      listen 80;
      listen [::]:80; 
      server_name dmarc.yourdomain.com;

      access_log /var/log/nginx/dmarc.access;
      error_log /var/log/nginx/dmarc.error;

      add_header X-Frame-Options SAMEORIGIN; 
      add_header X-Content-Type-Options nosniff;

      location / {
         proxy_pass http://127.0.0.1:5601;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      }
}

Save and close the file. Then test Nginx configurations.

sudo nginx -t

If the test is successful, reload Nginx for the change to take effect.

sudo systemctl reload nginx

Now you can access Elastic web interface at dmarc.yourdomain.com. Click the Explore on my own button.

dmarc-report-analyzer-open-source-ubuntu-20.04

 

Enabling HTTPS

To encrypt the HTTP traffic when you visit Elastic web interface via a domain name, we can enable HTTPS by installing a free TLS certificate issued from Let’s Encrypt. Run the following command to install Let’s Encrypt client (certbot) on Ubuntu 20.04 server.

sudo apt install certbot

If you use Nginx, then you also need to install the Certbot Nginx plugin.

sudo apt install python3-certbot-nginx

Next, run the following command to obtain and install TLS certificate.

sudo certbot --nginx --agree-tos --redirect --hsts --staple-ocsp --email [email protected] -d dmarc.yourdomain.com

The certificate should now be obtained and automatically installed.

self-hosted-dmarc-analyzer-https

Restricting Access to the Elastic Web Interface

By default, Elastic doesn’t provide user authentication. To allow only trusted users to access Elastic, we can enable HTTP basic authentication in Nginx. First, we need to install the apache2-utils package which provides the htpasswd password file create tool.

sudo apt install apache2-utils

Then run the following command to create the admin user. You will need to set a password.

sudo htpasswd -c /etc/nginx/htpasswd admin

Next, edit the Elastic virtual host file.

sudo nano /etc/nginx/conf.d/elastic.conf

Add the following two lines in the listen 443 ssl server block.

auth_basic "Login required";
auth_basic_user_file /etc/nginx/htpasswd;

Save and close the file. Then test Nginx configurations.

sudo nginx -t

If the test is successful, reload Nginx for the change to take effect.

sudo systemctl reload nginx

Now if you reload the Elastic in your web browser, you will be asked to enter an username and password.

kibana HTTP basic authentication

Configuring ParseDMARC

In order to use ParseDMARC and send data to Elastisearch, we need to create a configuration file.

sudo nano /etc/parsedmarc.ini

Put the following lines in the file. Replace the palceholders as necessary.

[general]
# Save aggregate and forensic reports to Elasticsearch
save_aggregate = True
save_forensic = True

[imap]
# Log into the DMARC report email address and download data.
host = mail.yourdomain.com
port = 993
ssl = True
user = [email protected]
password = your_password_here
watch = True

[elasticsearch]
# Send data to Elastichsearch, which listens on port 9200.
hosts = 127.0.0.1:9200
ssl = False

[smtp]
# For sending email
host = mail.yourdomain.com
port = 587
ssl = True
user = [email protected]
password = your_password_here
from = [email protected]

# send results to this address
to = [email protected]

Save and close the file.

Running ParseDMARC as a Systemd Service

We can manually run ParseDMARC with parsedmarc -c /etc/parsedmarc.ini (Don’t run this command just yet), but it’s more convenient to run ParseDMARC as a systemd service in the background. Create a systemd service unit file for ParseDMARC with the following command.

sudo nano /etc/systemd/system/parsedmarc.service

Put the following lines into the file.

[Unit]
Description=parsedmarc mailbox watcher
Documentation=https://domainaware.github.io/parsedmarc/
Wants=network-online.target
After=network.target network-online.target elasticsearch.service

[Service]
ExecStart=/usr/local/bin/parsedmarc -c /etc/parsedmarc.ini
User=parsedmarc
Group=parsedmarc
Restart=always
RestartSec=5m

[Install]
WantedBy=multi-user.target

Save and close the file. This systemd service will run as the parsedmarc user, so we need to create the user with the following command.

sudo adduser --system --no-create-home --group parsedmarc

We also need to protect the /etc/parsedmarc.ini file so only root and parsedmarc group users can read it.

sudo chown parsedmarc /etc/parsedmarc.ini

sudo chmod 600 /etc/parsedmarc.ini

ParseDMARC will need to download MaxMind GeoIP database to /usr/share/elasticsearch/modules/ingest-geoip directory, so the parsedmarc user needs to have write permission.

sudo setfacl -R -m "u:parsedmarc:rwx" /usr/share/elasticsearch/modules/ingest-geoip/

Now we can start and enable the parsedmarc systemd service.

sudo systemctl start parsedmarc

sudo systemctl enable parsedmarc

Check status.

systemctl status parsedmarc

parsedmarc-systemd-service-ubuntu-20.04

ParseDMARC will start fetching DMARC reports from the report email address and send them to Elasticsearch for analysis. If you have lots of DMARC report emails, please be patient to let ParseDMARC finish its work.

If the parsedmarc service isn’t active (running), you can run the following command to see what’s wrong.

sudo journalctl -eu parsedmarc

Importing ParseDMARC Objects In Kibana

To use the Kibana dashboard with ParseDMARC, first you need to import the ParseDMARC objects. You can use the following command to download it on your local Linux or Mac computer.

wget https://raw.githubusercontent.com/domainaware/parsedmarc/master/kibana/export.ndjson

If you use Windows, just copy the https link and paste it in browser address bar to download the data.

Then in Elastic Home page, click Kibana to access the Kibana dashboard.

elastic kibana dashboard

Then in Kibana dashboard, click Add your data button.

add data in kibana dashboard

Next, select Saved Objects and click the Import button to import the ParseDMARC objects you downloaded. import parsedmarc kibana objects

Analyzing DMARC Reports in Kibana

Once the ParseDMARC systemd service finishes reading emails, the DMARC report data will show up in Kibana. The most useful report I think is the DMARC Summary. You can check it by going to Saved Objects -> DMARC Summary. The most useful thing you can do with DMARC summary is to identify legitimate sources that failed DMARC check.

  1. Filter the results to only show messages that didn’t pass DMARC.
  2. Then you can see the message dispositions (How the receiving email servers handle them: do nothing, put into quarantine, or reject them).
  3. Scroll down to see the SPF alignment and DKIM alignment details to identify the reason why some legitimate sources failed DMARC check.

If the sender doesn’t include DKIM signature or doesn’t have a reverse DNS record, then it’s clearly spam.

DKIM alignment details

What surprises me is that some mailbox providers uses their own domain name in the DKIM signature when forwarding my legitimate emails. Other mailbox providers uses my domain name in the DKIM signature but still failed DKIM alignment. To further investigate the cause, you can check the individual forensic report.

Conclusion

I hope this tutorial helped you set up ParseDMARC on Ubuntu 20.04 to analyze DMARC reports. As always, if you found this post useful, then subscribe to our free newsletter to get more tips and tricks. Take care 🙂

Rate this tutorial
[Total: 6 Average: 4.5]

31 Responses to “Set Up ParseDMARC on Ubuntu 20.04 to Analyze DMARC Reports

  • Michel Bisson
    4 years ago

    This titorial is excellent. I must make some remarks on 2 things:
    1 – in the result of the following command(sudo -H pip3 install -U parsedmarc) (in your tutorial) I got the follwoing error in the list of results:
    ——————————————-
    Successfully built expiringdict maxminddb
    ERROR: geoip2 4.1.0 has requirement requests=2.24.0, but you’ll have requests 2.22.0 which is incompatible.
    ——————————————
    I wonder if this will create problems.

    2 – after a clean installation of Ubuntu 20.0x the package ‘acl’ was not installed and therefore the following command could not work:
    sudo setfacl -R -m “u:parsedmarc:rwx” /usr/share/elasticsearch/modules/ingest-geoip/
    Maybe an indication that this package should be previously installed would be helpful.

    Otherwise this is one of the best tutorials I’ve come across.

  • Ken Wright
    4 years ago

    Hello Xiao!

    So I’m up to the step of running parseDMARC as a service, but I can’t get it to start. When I run journalctl I get the following output:


    Dec 08 04:30:00 grace parsedmarc[1177860]: [38B blob data]
    Dec 08 04:30:00 grace parsedmarc[1177860]: ERROR:cli.py:615:IMAP Error: [AUTHENTICATIONFAILED] Authentication failed
    Dec 08 04:30:00 grace systemd[1]: parsedmarc.service: Main process exited, code=exited, status=1/FAILURE
    Dec 08 04:30:00 grace systemd[1]: parsedmarc.service: Failed with result ‘exit-code’.

    Do I need to add a user dmarc to authenticate?

    • Xiao Guoan (Admin)
      4 years ago

      You need to edit the /etc/parsedmarc.ini and enter the correct password for the email address that receives DMARC reports.

      The email address is specified in your DMARC record, and you need to create this email address on your mail server if it doesn’t exist.

  • Ken Wright
    4 years ago

    That worked. Thanks!

  • Jeffery
    3 years ago

    Can I analysis loacal dmarc file? (Not connect [email protected])

  • Thanks. Overall good. You might put in an http:// URL when you first say to go look at Elastic, or put the Certbot stuff ahead of that point.

    Also might be worth adding an alternative for those with a wildcard cert they could apply, which would be useful if you wanted to put this server on a network only reachable by VPN (Certbot needs public accessibility).

  • I have finished the entire configuration everything is working as expected, however all the reports i receive are .zip or .gz extensions that have xml formats inside and i attempted to import data and all this thing reads is json…

    what good is a report viewer that cant read the format of the reports i receive from everywhere…

    • Xiao Guoan (Admin)
      3 years ago

      You should not import the DMARC report files directly. The parsedmarc.service systemd service will take care of parsing those XML reports and present them in Kibana.

      If you don’t use the parsedmarc.service systemd service, then it’s just Kibana you are using, it’s not ParseDmarc.

      Why not upload the DMARC reports to your mailbox, so ParseDMARC can read them? Be flexible.

      And it’s an open-source project. You don’t pay any money to the software author. If you are not satisfied with the current features, why not take the source code and improve it by yourself?

  • Hi. I setup all as per yr instructions. However, there are 2 things that is giving me issues as per below.

    1. the GeoLite2-Country.mmdb is missing. I followed the instructions as per the link, but don’t seems to work.
    2. cli.py:637:Connection unexpectedly closed. Not sure what is this error.

    Thanks for helping.

    Apr 24 13:33:06 ubuntu systemd[1]: Started parsedmarc mailbox watcher.
    Apr 24 13:33:13 ubuntu parsedmarc[1944]: [38B blob data]
    Apr 24 13:33:13 ubuntu parsedmarc[1944]: WARNING:utils.py:296:GeoLite2-Country.mmdb is missing. Please follow the instructions at https://dev.maxmind.com/geoip/geoipupdate/ to get the latest version.
    Apr 24 13:35:59 ubuntu parsedmarc[1944]: ERROR:cli.py:637:Connection unexpectedly closed
    Apr 24 13:36:00 ubuntu systemd[1]: parsedmarc.service: Main process exited, code=exited, status=1/FAILURE
    Apr 24 13:36:00 ubuntu systemd[1]: parsedmarc.service: Failed with result ‘exit-code’.
    Apr 24 13:41:00 ubuntu systemd[1]: parsedmarc.service: Scheduled restart job, restart counter is at 1.
    Apr 24 13:41:00 ubuntu systemd[1]: Stopped parsedmarc mailbox watcher.

    • Same issue here, can’t update GeoIP.
      Have you been able to resolve it?
      https://dev.maxmind.com/geoip/geoipupdate/ to get the latest version

  • I understand from the configuration ini file that parsedmarc processes the aggregate and forsenic reports in one email address. if my org has chosen to use two email addresses: [email protected] and [email protected], any idea to go around this problem?

  • I can’t get any of the Forensics stuff to work. I just get “ERROR” messages and things like “this index no longer exists”. Any ideas? Screenshot attached.

    • I have the same problem, did you manage to fix it?

    • Could anyone solve this?
      There seems to be a problem with the Index patterns.

    • I have the same problem, any solution for this?

      • I got everything to work again. I followed the guide here:
        https://domainaware.github.io/parsedmarc/

        Beware that some commands there are incomplete or incorrect, e.g. there is a section where it says you should execute “sudo -u parsedmarc /opt/parsedmarc/venv -U parsedmarc” <== NOT a valid command in my experience. Correct: "sudo -u parsedmarc /opt/parsedmarc/venv/bin/pip3 install parsedmarc"

        BUT: This will install the current version 8.3.1 … which seems buggy? I have had way way better luck by forcing "pip3" to install an older version:

        sudo -u parsedmarc /opt/parsedmarc/venv/bin/pip3 install parsedmarc==6.12.0

        And now all is working again.

  • Following this installation guide. Fine so far but my fresh install of Ubuntu 20.04.3 didn’t recognise the command ‘setfacl’ out of the box. Had to ‘sudo apt install acl’ before I could continue.

  • Thanks for the great setup instructions.

    All is working fine for me except that when running as a service it only checks the email on startup, not continually. I am having to restart the service everyday to get results.

    What am I missing?

  • Khazaar
    2 years ago

    It seems that those index are generated with datas

    i waited one night and everything is fine now

  • You say:

    “We also need to protect the /etc/parsedmarc.ini file so only root and parsedmarc group users can read it.”

    But the commands you give make it owned by the parsedmarc user and no rights for the root group:

    sudo chown parsedmarc /etc/parsedmarc.ini
    
    sudo chmod 600 /etc/parsedmarc.ini
    
    /etc/systemd/system# ls -l /etc/parsedmarc.ini
    -rw------- 1 parsedmarc root 663 Apr 22 20:04 /etc/parsedmarc.ini
    

    Those commands should be:

    sudo chown root:parsedmarc /etc/parsedmarc.ini
    sudo chmod 660 /etc/parsedmarc.ini
    

    If that is your intent

    ls -l /etc/parsedmarc.ini
    -rw-rw---- 1 root parsedmarc 663 Apr 22 20:04 /etc/parsedmarc.ini
    
    • Xiao Guoan (Admin)
      2 years ago

      Yes, your commands are right, but the root user can read any file even if it’s not the owner of a file and you don’t grant read/write permission to root.

  • Hey, so I followed everything in this guide but my dashboards are saying no data. what could be the issue? the output files?

  • any Idea how to solve the following error message?

  • i got this error: No matching indices found: No indices match pattern “dmarc_aggregate*”

    how do you mange to solve it ?

  • Sylvain
    1 year ago

    Hi,

    Has anyone already configured parsedmarc with gmail api? I’m a bit lost

  • First, you need to create an app-specific password in your Google account.
    Then, use the following config in parsedmarc.ini to establish the link:

    #########################
    [general]
    save_aggregate = True
    save_forensic = True

    [imap]
    host = imap.gmail.com
    user = [email protected]
    password = yourAppSpecificPassword

    [smtp]
    # For sending email
    host = smtp.gmail.com
    port = 587
    ssl = True
    user = [email protected]
    password = yourAppSpecificPassword
    from = [email protected]
    to = [email protected]

    [mailbox]
    watch = True
    delete = False

    [elasticsearch]
    hosts = 127.0.0.1:9200
    ssl = False
    #########################

  • Scott J.
    7 months ago

    I realize this is a bit of a long shot since this is an older post but if anyone has any info on using the gmail_api integration with parsedmarc, I’d appreciate the assist.

  • Hi,

    I’m up to the step of running parseDMARC as a service. I could start the service but when I went to the browser and tried to click the ‘import’ from saved object, the service failed. Then I ran journalctl and I got the following output

    parsedmarc[499698] ERROR:cli.py:615:IMAP Error: 'NoneType' object has no attribute 'decode'

Leave a Comment

  • Comments with links are moderated by admin before published.
  • Your email address will not be published.
  • Use <pre> ... </pre> HTML tag to quote the output from your terminal/console.
  • Please use the community (https://community.linuxbabe.com) for questions unrelated to this article.
  • I don't have time to answer every question. Making a donation would incentivize me to spend more time answering questions.

The maximum upload file size: 2 MB. You can upload: image. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here