Loggregate

Aggregates, Analyses and Generates reports from log files.

Loggregate can read your log files given a glob pattern and generates a report with stats based on the logs.

I am bad at designing web pages, so any design changes or suggestions are welcome.

Compile from source

• You might need to install some basic packages for this tool to work mentioned in requirements.txt. They are mentioned with software name (arch linux package name & debian package name)

pkg-config (base-devel / build-essential)
libfreetype (freetype2 / libfreetype6-dev)
libfontconfig (fontconfig / libfontconfig-dev)

• Make sure you have rust toolchain installed.
• Go to the releases page and download loggregate-<version>.tar.gz along with the loggregate-<version>.tar.gz.asc (recommended)
• You can optionally check the signature of the tarball to verify the authenticity.
• To verify the sinature you need my public pgp key. Run this command to get my key.

gpg --recv-key C8B2A95D8D855A9D8C6F0C78BCBCAE31ECE05007

• Then verify the signature by runnig the below command

gpg --verify loggregate-<version>.tar.gz.asc loggregate-<version>.tar.gz

Runnig this should say Good Signature from “Akash Doppalapudi”

• You can skip the signature verification part if you want to though I’d recommned it. Next you need to extract the tar archive.

tar xvf loggregate-<version>.tar.gz

• You can also simply clone this repository and cd into it but I recommned get the release tarball.
• Run cargo build. You can use --release flag to make an optimized release build.
• Compiled binary can be found inside target folder.
• You can also copy the binary to a location included in $PATH to run it like any other program.

cp target/release/loggregate /usr/local/bin

Test loggregate by running loggregate -h

Usage

You can get detailed usage by running loggregate with --help flag.

Required Args

• --datetime-format / -d Datetime string’s format. eg: “%d/%m/%y %H:%M:%S”. Refer to the below table to see how to write datetime format.

  Spec.	Example	Description
  %Y	2001	Zero padded year in 4 digits.
  %y	01	The proleptic Gregorian year modulo 100, zero-padded to 2 digits.
  %m	07	Month number (01–12), zero-padded to 2 digits.
  %b	Jul	Abbreviated month name. Always 3 letters.
  %B	July	Full month name.
  %h	Jul	Same as %b.
  %d	08	Day number (01–31), zero-padded to 2 digits.
  %e	8	Same as %d but space-padded.
  %F	2001-07-11	Year-month-day format (ISO 8601). Same as %Y-%m-%d.
  %H	00	Hour number (00–23), zero-padded to 2 digits.
  %k	0	Same as %H but space-padded.
  %I	12	Hour number in 12-hour clocks (01–12), zero-padded to 2 digits.
  %l	12	Same as %I but space-padded.
  %P	am	am or pm in 12-hour clocks.
  %p	AM	AM or PM in 12-hour clocks.
  %M	34	Minute number (00–59), zero-padded to 2 digits.
  %S	60	Second number (00–60), zero-padded to 2 digits.

• You also need to pass file glob pattern as a positional argument

Optional Args

• --user / -u User can optionally give their name and it will be shown in the report

Supported log level tyes: Emergency, Alert, Critical, Error, Warning/Warn, Notice, Info/Information, Debug and any other type will be considered of type ‘Others’

Example

Say we have log files in /var/log ending with .log and log statement looks like this

2015-07-29 17:41:44,747 - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@774] - Notification time out: 3200
2015-07-29 19:04:12,394 - INFO  [/10.10.34.11:3888:QuorumCnxManager$Listener@493] - Received connection request /10.10.34.11:45307

then the command looks like

loggregate --datetime-format "%Y-%m-%d %H:%M:%S" --user "John Doe" "/var/log/*.log"

Running this will generate bar plots and prepares a html file in current directory and you can open it in browser to see the report.

If the program has any problems reading datetime it writes the line it failed to read to stderr. You can redirect all the errors to a seperate file by adding 2> errors.txt to the end of the command.

loggregate -d "%Y-%m-%d %H:%M:%S" -u "John Doe" "/var/log/*.log" 2> errors.txt