Packages

c

io.dataos.spark.writers.kafka

KafkaWriterOptions

case class KafkaWriterOptions(brokers: String, topic: String, schemaRegistryUrl: Option[String], format: Option[String], compressionType: Option[String], keyColumn: Option[String], saveMode: SaveMode = SaveMode.Append, replicationFactor: Int, partitions: Int, options: Map[String, String]) extends Product with Serializable

A case class representing the Kafka writer options required for writing data from a DataFrame to a Kafka topic.

brokers

The Kafka brokers in the format "host1:port1,host2:port2,...".

topic

The name of the Kafka topic to which data will be written.

schemaRegistryUrl

Optional URL for the Confluent Schema Registry when writing data in Avro format (default: None).

format

Optional format for the data (json or avro) when writing to Kafka (default: None).

compressionType

Optional compression type for the Kafka producer (default: None).

keyColumn

Optional column name to be used as the Kafka message key (default: None).

saveMode

The SaveMode for writing data to Kafka (default: SaveMode.Append).

replicationFactor

The replication factor for creating the Kafka topic.

partitions

The number of partitions for creating the Kafka topic.

options

Additional options to be passed to the Kafka producer (default: Map.empty).

Linear Supertypes
Serializable, Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. KafkaWriterOptions
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new KafkaWriterOptions(brokers: String, topic: String, schemaRegistryUrl: Option[String], format: Option[String], compressionType: Option[String], keyColumn: Option[String], saveMode: SaveMode = SaveMode.Append, replicationFactor: Int, partitions: Int, options: Map[String, String])

    brokers

    The Kafka brokers in the format "host1:port1,host2:port2,...".

    topic

    The name of the Kafka topic to which data will be written.

    schemaRegistryUrl

    Optional URL for the Confluent Schema Registry when writing data in Avro format (default: None).

    format

    Optional format for the data (json or avro) when writing to Kafka (default: None).

    compressionType

    Optional compression type for the Kafka producer (default: None).

    keyColumn

    Optional column name to be used as the Kafka message key (default: None).

    saveMode

    The SaveMode for writing data to Kafka (default: SaveMode.Append).

    replicationFactor

    The replication factor for creating the Kafka topic.

    partitions

    The number of partitions for creating the Kafka topic.

    options

    Additional options to be passed to the Kafka producer (default: Map.empty).

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. val brokers: String
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. val compressionType: Option[String]
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. val format: Option[String]
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. val keyColumn: Option[String]
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. val options: Map[String, String]
  18. val partitions: Int
  19. val replicationFactor: Int
  20. val saveMode: SaveMode
  21. val schemaRegistryUrl: Option[String]
  22. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  23. val topic: String
  24. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped