[logback-dev] RFC 5424 and Structured Data support.

[Ralph Goers]

On Dec 3, 2009, at 1:05 AM, Ceki Gülcü wrote:

> 
> RFC 5424 is a big mental leap in the sense that one needs to start
> seeing logging events as a map with named fields, with the entailing
> changes at the API level. At the same time, and very contradictorily,
> it can be viewed as just a new encoding technique, i.e. business as
> usual.
> 
> Could you, without going into the business details, give an example of
> a use case. In particular, I am curious with the kind of
> post-processing you do with the logs. Do you do any post processing?
> 

Sure. We deal with internet banking for 1800 financial institutions. The applications are clustered with a user for any institution being serviced by any of the nodes in the cluser. Here is a flow that isn't completely accurate but is representative of what would happen.

1. The user hits the web site and gets the sign on screen. Typically a session will be created and information such as the user's ip address and the bank's id will be stored. On every subsequent interaction the MDC will be populated with that data along with a unique request id.
2. The user enters their log in data. A "login attempt" audit event will be generated and the user's id will be part of the event data (the MDC data is always included in every event).
3. The login suceeds. The user's id will be added to the MDC. A "login succeeded" audit event will be generated (no extra event data is necessary).
4. The user goes to their account summary page. A "displayAccountSummary" audit event will be generated and the account number will be included in the event data.
5. The request to obtain the summary data will be routed to an internal service (which could be running in a separate data center). The MDC data is passed to the service where it populates an MDC with that data. A "getSummaryData" event will be generated and the relevant data included in the event data (this is  not an "audit" event).
6. The request succeeds and the service generates a "getSummaryData succeeded" event and returns the data.
7. The application generates a "displayAccountSummary succeeded" audit event.

This will go on for every activity the end user performs and each back end service that participates. The only difference between an "Audit" event and a non-Audit event is that Financial Institution administrators can view and generate reports on audit events while our internal administrators can view audit and non-audit events co-mingled.

The basic design calls for RFC 5424 messages to be routed through a message bus to a "logging service" - I am still working with other Architects within Intuit to determine what that will be. However, for my purposes I will have portlets running in our Administrative Portal that will allow FI administrators to view end user activity, generate daily or weekly reports (or raw data for download) that meet certain filtering criteria (for example, showing all transactions where money was transferred from one account to another over a certain dollar amount).  In addition, we might use the data to add promotional messages to the end user's screen. So having the data in a self describing format is of great benefit. I've already seen that Splunk handles the structured data pretty well without me having to do anything. Syslog-ng can also filter on the structured data even to the point of identifying it as SD-ID.key.  If, on the other hand, the data was to be stored in a database it could easily be done in the same manner that the DBAppender stores the MDC data.

In short, post-processing is one of the most important aspects of the solution.

Ralph