Introduction

For each RESTful http request, we currently only log the URL of the request. We need to log the request body or response body if useful information (dataset properties, program runtime arguments, etc.) is included. 

Goals

Improve the audit logging in the router.

User Stories 

User would like to know what more for some kind of HTTP request. For example, when user updates the dataset property, in the audit log, the request body which contains the newly updated dataset property will be very helpful. 

Design 

All RESTful endpoints that needs to be logged with more information are in this table, click to expand. This table has all POST, DELETE, PUT endpoints other than GET. All GET methods do not have a body or header params so the URL is enough for audit logging.

Approach

Currently, in SecurityAuthenticationHttpHandler, we already have each request audit logged. For each incoming request, we log the clientIP, the URL from HTTPRequest and the responseCode from HTTPResponse. We will still keep this as the default behavior for audit logging.

To log the request body, header name or response body, we annotate the RESTful endpoint methods with a new custom annotation AuditPolicy such as: 

/**
 * Annotates a REST endpoint method to indicate which content needs to be audit logged.
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface AuditPolicy {

  AuditDetail[] value();
}

And in the AuditDetail:

/**
 * Declares which content needs to be audit logged.
 */
public enum AuditDetail {
  HEADERS,
  REQUEST_BODY,
  RESPONSE_BODY
}

For example, for the deploy app REST call, we want to log several header names and the response body, so we annotate the method as:

/**
 * Annotate the method with the @AuditPolicy, with response body set to true, and set header names to true.
 */
@POST
@Path("/apps")
@AuditPolicy({AuditDetail.HEADERS, AuditDetail.RESPONSE_BODY})
public BodyConsumer deploy(HttpRequest request, HttpResponder responder, ...) throws Exception {
 ...
}

When the router starts up, we will walk through all methods in each handler class in the class path to find if the method has a @AuditPolicy annotation. If the method does not have this annotation, the default behavior is used(log the URL, response body, etc). If it is present, then we will log the additional information provided by this annotation.

How to walk through all handler classes

We will use the class path to get all the handler classes and filter out based on the handler class.

How to audit log

Currently we are doing audit log in SecurityAuthenticationHttpHandler, we will refactor the handler to separate the audit logging since they are unrelated features.

When we receive a http request(in MessageReceived() method), we will create a AuditLogEntry and have the username, URL, request body and headers if needed to log, etc, stored in the object, and attach it to the ChannelHandlerContext. 

When a http response comes(in WriteRequested() method), we get the AuditLogEntry in the context and have the response body, if necessary, stored in the object.

In WriteComplete() method, we log the AuditLogEntry:

Audit log entry

For each http REST call, we currently are using the AuditLogEntry to log the request. We will add the header, request body, response body fields to it. Here is how it looks:

/**
 * Annotate the method with the @AuditPolicy, with response body set to true, and set header names that need to be logged as an array.
 */
public final class AuditLogEntry {
  ...
  private InetAddress clientIP;
  private String userName;
  private Date date;
  private String requestLine;
  private String requestBody;
  private Integer responseCode;
  private Long responseContentLength;
  private String userIdentity;
  private Map<String, String> headers;
  private String responseBody;
  ...
  
  // log the string
  public String toString() {
  return String.format("%s %s %s [%s] \"%s\" %s %s %s %s %s",
                       clientIP != null ? clientIP.getHostAddress() : DEFAULT_VALUE,
                       fieldOrDefault(userIdentity),
                       fieldOrDefault(userName),
                       DEFAULT_DATE_FORMAT.format(date),
                       fieldOrDefault(requestLine),
                       fieldOrDefault(headers),
                       fieldOrDefault(requestBody),
                       fieldOrDefault(responseCode),
                       fieldOrDefault(responseContentLength),
                       fieldOrDefault(responseBody));
  }
}

Security Impact 

The audit logging only happens when security is enabled.

Impact on Infrastructure Outages 

Test Scenarios

Releases

release 4.1.0

Related Work