`logs`

Read log files into records (maps) you can search and count: syslog, Linux auth.log (SSH and sudo), Apache/nginx access logs, JSON lines, CSV and Windows Security event exports. Every record has "time" (a timestamp or nil), "when" (the time as text), "format", "message" and "raw" (the original line), plus the fields of its format.

import logs

Example

import logs

let records = logs.read("apps/data/auth.log", "auth")
say(f"{len(records)} records")
let failed = records.filter(fn(r) { return r.get("action") == "failed_login" })
say(f"{len(failed)} failed logins")
for row in logs.top(failed, "ip", 3) {
  say(row)
}

Functions

parse_line(line, format?)

returns map

Parse one log line into a record. format is "auto" (default), "syslog", "auth", "apache", "nginx", "json", "windows" (one JSON event) or "csv" (gives {"fields": [...]}). A line that doesn't fit becomes a "text" record with just time, message and raw.

parse(text, format?, options?)

returns list

Parse a whole log that is already in a text value (handy in the browser playground). Same formats and options as read().

read(path, format?, options?)

returns list

Read a log file into a list of records. format "auto" (default) looks at the start of the file and picks the format for you. Files are read a line at a time, so big logs are fine. options: {"limit": n} to stop after n records, {"year": 2024} for syslog lines (which have no year; otherwise the most recent year that isn't in the future is used).

each(path, fn, format?, options?)

returns number

Call fn(record) for every record in a log file without keeping them all in memory (for huge logs). Gives back how many records there were. If fn gives back false, reading stops early.

sniff(path)

returns text

Guess a log file's format from its first lines: "auth", "syslog", "apache", "json", "csv", "windows" or "text". This is what "auto" uses.

parse_time(text, year?)

returns number or nil

A timestamp from the time formats logs use: ISO 8601 (2024-03-10T09:15:02Z), syslog (Mar 10 09:15:02), Apache (10/Mar/2024:09:15:02 +0000), Windows (3/10/2024 9:15:02 AM, /Date(1710062102000)/) and Unix seconds or milliseconds. Times without a time zone are taken as this computer's local time. nil if it isn't a time.

top(records, field, n?)

returns list

The most common values of a field, as [{"value", "count"}, ...] from most to least (default top 10). Records without the field are skipped. top(records, "ip") shows who talks the most.

Values

Name	Value	What it is
`logs.formats`	`["auto", "syslog", "auth", "apache", "nginx", "json", "csv", "windows", "text"]`	The formats logs knows: auto, syslog, auth, apache, nginx, json, csv, windows, text.