Posts

Adding Error stack trace and filter options in Log4j Kafka Appender

-
In the previous article , we discussed how to direct spark logs to kafka but one issue with org.apache.kafka.log4jappender.KafkaLog4jAppender class is that error stack trace is not pushed into kafka. Error stack trace is vital and forms a crucial part of log analysis. In this article, we will discuss creation of our own kafka appender class(myKafkaAppender) by extending org.apache.kafka.log4jappender.KafkaLog4jAppender and customizing this class to cater to our needs. Also, this class needs to be specified in all log4j properties files. Adding error stack trace:- To add error stack trace, we will have to create a subAppend method (we cannot override subAppend method of base class as it is private) and store the error stack trace in a string. We will append this string with the error log message. Code snippet of subAppend method is given below:- private String subAppend (LoggingEvent event) { StringBuilder str= new StringBuilder() ; if ( layout .ignoresThrowable()) { Str
Post a Comment
Read more

Redirecting Spark logs to Kafka

-
Spark logs can be redirected to Kafka which can be further used for analysis. These logs can be fed from kafka to elastic search and further to kibana to get data and resource related insights w.r.t our job. In this article, we will be discussing about how to stream log4j application logs to Apache Kafka using maven artifact kafka-log4j-appender . We need to add kafka-log4j-appender as a dependency in pom.xml. <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-log4j-appender</artifactId> <version>1.0.0</version> </dependency> We need to add kafka properties in log4j.properties file for using KafkaLog4jAppender. Kafka broker list and topic need to specified. This topic will be used for storing logs. # Root logger option log4j.rootLogger=
Post a Comment
Read more