Redirecting Spark logs to Kafka
Spark logs can be redirected to Kafka which can be further used for analysis. These logs can be fed from kafka to elastic search and further to kibana to get data and resource related insights w.r.t our job. In this article, we will be discussing about how to stream log4j application logs to Apache Kafka using maven artifact kafka-log4j-appender . We need to add kafka-log4j-appender as a dependency in pom.xml. <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-log4j-appender</artifactId> <version>1.0.0</version> </dependency> We need to add kafka properties in log4j.properties file for using KafkaLog4jAppender. Kafka broker list and topic need to specified. This topic will be used for storing logs. # Root logger option log4j.rootLogger=