package kafka
Type Members
-
class
KafkaOutputWriter extends DatasourceWriter
A custom KafkaOutputWriter class that extends the DatasourceWriter.
A custom KafkaOutputWriter class that extends the DatasourceWriter. This class is responsible for writing data from a DataFrame to a Kafka topic based on the specified options.
-
case class
KafkaWriterOptions(brokers: String, topic: String, schemaRegistryUrl: Option[String], format: Option[String], compressionType: Option[String], keyColumn: Option[String], saveMode: SaveMode = SaveMode.Append, replicationFactor: Int, partitions: Int, options: Map[String, String]) extends Product with Serializable
A case class representing the Kafka writer options required for writing data from a DataFrame to a Kafka topic.
A case class representing the Kafka writer options required for writing data from a DataFrame to a Kafka topic.
- brokers
The Kafka brokers in the format "host1:port1,host2:port2,...".
- topic
The name of the Kafka topic to which data will be written.
- schemaRegistryUrl
Optional URL for the Confluent Schema Registry when writing data in Avro format (default: None).
- format
Optional format for the data (json or avro) when writing to Kafka (default: None).
- compressionType
Optional compression type for the Kafka producer (default: None).
- keyColumn
Optional column name to be used as the Kafka message key (default: None).
- saveMode
The SaveMode for writing data to Kafka (default: SaveMode.Append).
- replicationFactor
The replication factor for creating the Kafka topic.
- partitions
The number of partitions for creating the Kafka topic.
- options
Additional options to be passed to the Kafka producer (default: Map.empty).