[logback-dev] [JIRA] Updates for LOGBACK-1734: Add support for NDJSON / structured logging - popular log collector/shipper format

[logback developers list]

logback / LOGBACK-1734 [Open]
Add support for NDJSON / structured logging - popular log collector/shipper format

==============================

Here's what changed in this issue in the last few minutes.

This issue has been created
This issue is now assigned to you.


View or comment on issue using this link
https://jira.qos.ch/browse/LOGBACK-1734

==============================
 Issue created
------------------------------

Oleksandr Gavenko created this issue on 20/Apr/23 1:19

Summary:              Add support for NDJSON / structured logging - popular log collector/shipper format
Issue Type:           Bug
Assignee:             Logback dev list
Components:           logback-classic
Created:              20/Apr/23 1:19
Labels:               layouts
Priority:             Major
Reporter:             Oleksandr Gavenko
Description:
  Modern centralized logging assumes you work with structured data and even *beyond logger/message/exception* you enhance data with **MDC** (like httpUrl / httpMethod or customerEmail, etc) and with `keyValue` of SLF4J fluent API. 
  
  Some tracing solution (OpenTelemetry/OpenTracing) might add additional dimension to data, via MDC, for example: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/logger-mdc-instrumentation.md
  
  So:
  
  * logging data gets lots of independent dimensions (via MDC or keyValue)
  * plain text file ambiguity is no longer acceptable - who wants to deal with parsing multiline stack traces anymore?
  
  One of the popular structured format is NDJSON.
  
  http://ndjson.org/ tells that NDJSON is a file with valid JSON on each line.
  
  Popular file log collectors supports NDJSON parsers:
  
  * FluentBit https://docs.fluentbit.io/manual/pipeline/parsers/json
  * FileBeat (from Elastic) https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html
  
  Today to log in NDJSON format you need to relay on 3rd party layout providers, like https://github.com/logfellow/logstash-logback-encoder
  
  I with Logback has support for NDJSON out of the box.
  
  https://github.com/qos-ch/logback-contrib/blob/master/json/classic/src/main/java/ch/qos/logback/contrib/json/classic/JsonLayout.java is not a solution:
  
  * it lacks customization: possibility to rename attributes
  * it lacks customization: possibility to flatten MDC
  * it lacks support for Markers
  * it lacks support for keyValue of fluent API
  * it lacks support for nanoseconds 
  * it lacks auto-incremented sequence number - centralized logging solutions store events in random order, if a query returns several events withing millisecond (or nanosecond) - they will be presented in random order, not the original order! 
  


==============================
 This message was sent by Atlassian Jira (v9.6.0#960000-sha1:a3ee8af)