Adding Error stack trace and filter options in Log4j Kafka Appender
In the previous article , we discussed how to direct spark logs to kafka but one issue with org.apache.kafka.log4jappender.KafkaLog4jAppender class is that error stack trace is not pushed into kafka. Error stack trace is vital and forms a crucial part of log analysis. In this article, we will discuss creation of our own kafka appender class(myKafkaAppender) by extending org.apache.kafka.log4jappender.KafkaLog4jAppender and customizing this class to cater to our needs. Also, this class needs to be specified in all log4j properties files. Adding error stack trace:- To add error stack trace, we will have to create a subAppend method (we cannot override subAppend method of base class as it is private) and store the error stack trace in a string. We will append this string with the error log message. Code snippet of subAppend method is given below:- private String subAppend (LoggingEvent event) { StringBuilder str= new StringBuilder() ; if ( layout .ignoresThrowable()) { Str