Converting Auth logs to JSON with Golang

On Ubuntu Server (and other Linux distros) all authentication attempts are registered in var/log/auth.log. While it’s convenient to have this information in one dedicated file, it can quickly become very large and difficult to extract meaningful information from. We’re going to implement a small program written in Golang that parses auth.log and exports information on failed access attempts to a JSON file.

The log I’m using for this was generated by Ubuntu Server 18.04, running fail2ban to block multiple access attempts. Your log format may be a bit different.

---

The first step will be to (assuming you have a working Go installation) is just opening the auth.log file and logging any errors. It’s important to note that these logs require root access, so you can either run the script with sudo or make a copy with different permissions.

package main

import (
  "log"
  "os"
)

func main() {
  file, err := os.Open("auth.log")

  if err != nil {
    log.Fatal(err)
  }

  defer file.Close()
}

For this example, we’ll read the file line by line, and check for failed attempts to log in. If there’s a failed attempt, we use a regex to get the IP address of whoever tried to connect. We’ll use a map to store the result; a golang map with integer values, will initialize the values to 0, so we can simply increment for repeated attempts from the same IP.

m := make(map[string]int)

scanner := bufio.NewScanner(file)

for scanner.Scan() {
  text := scanner.Text()

  if strings.Contains(text, ": Failed ") {
    ip := ipAddress(text)

    if ip != "" {
      m[ip]++
    }
  }
}

In the function to parse the IP address we can use a fairly basic regex, as we don’t need to validate it.

func ipAddress(text string) string {
  var rgx = regexp.MustCompile(`\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}`)

  rs := rgx.FindStringSubmatch(text)

  if len(rs) > 0 {
    return rs[0]
  }

  return ""
}

Golang has a module, encoding/json, that makes transforming maps to json easy, and by using jsonMarshal.indent we can even a nicely formatted file.

json, err := json.MarshalIndent(m, "", "  ")

err = ioutil.WriteFile("results.json", json, 0644)

if err != nil {
  log.Fatal(err)
}

For example for these two entries in auth.log (with some made up example ips);

Dec  4 23:00:00 test-server sshd[11222]: Failed password for root from 999.99.99.999 port 21546 ssh2
Dec  4 23:00:03 test-server sshd[11222]: Failed password for root from 999.99.99.999 port 21546 ssh2
Dec  4 23:00:10 test-server sshd[11222]: Failed password for root from 999.99.99.999 port 50570 ssh2
Dec  5 11:44:43 test-server sshd[16666]: Failed password for root from 999.99.99.888 port 51770 ssh2

Would result in this JSON:

{
  "999.99.99.999": 3
  "999.99.99.888": 1,
}

And now putting it all together we’ve got:

package main

import (
  "bufio"
  "encoding/json"
  "io/ioutil"
  "log"
  "os"
  "regexp"
  "strings"
)

func main() {
  file, err := os.Open("auth.log")

  if err != nil {
    log.Fatal(err)
  }

  defer file.Close()

  m := make(map[string]int)

  scanner := bufio.NewScanner(file)
    
  for scanner.Scan() {
    text := scanner.Text()

    if strings.Contains(text, ": Failed ") {
      ip := ipAddress(text)

      if ip != "" {
        m[ip]++
      }
    }
  }
    
  json, err := json.MarshalIndent(m, "", "  ")

  err = ioutil.WriteFile("results.json", json, 0644)
    
  if err != nil {
    log.Fatal(err)
  }
}

func ipAddress(text string) string {
  var rgx = regexp.MustCompile(`\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}`)

  rs := rgx.FindStringSubmatch(text)

  if len(rs) > 0 {
    return rs[0]
  }

  return ""
}