Regex for parsing nginx access logs

You

I need to parse the default nginx combined log format. Every regex I write breaks on user agents with quotes in them.

ChatGPT

Do not try to be clever about the quoting, anchor on the structure instead:

import re

LINE = re.compile(
    r'^(?P<ip>\S+) \S+ (?P<user>\S+) \\[(?P<time>[^\\]]+)\\] \'\n    r\'"(?P<method>[A-Z]+) (?P<path>[^"]*?) (?P<proto>HTTP/[\\d.]+)" \'\n    r\'(?P<status>\\d{3}) (?P<bytes>\\d+|-) \'\n    r\'"(?P<ref>[^"]*)" "(?P<agent>.*)"$\'\n)\n

\n\nThe trick is `(?P<agent>.*)$` at the end: the user agent is the last field, so anything goes until the closing quote at end of line.

You

It still misses a few lines. How do I find out which?

ChatGPT

Never silently drop them. Count and sample:

bad = []
for n, line in enumerate(stream, 1):
    if not LINE.match(line):
        bad.append((n, line[:120]))
print(f"unmatched: {len(bad)}")
for n, sample in bad[:5]:
    print(n, sample)

In practice the leftovers are almost always malformed requests from scanners, where the request line is not METHOD PATH PROTO at all but raw junk or a TLS handshake landing on a plain HTTP port.