Packages

c

io.dataos.spark.writers.pulsar

PulsarOutputWriter

class PulsarOutputWriter extends DatasourceWriter

A custom PulsarOutputWriter class that extends the DatasourceWriter. This class is responsible for writing data from a DataFrame to a Pulsar topic based on the specified options.

Linear Supertypes
DatasourceWriter, Writer, Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PulsarOutputWriter
  2. DatasourceWriter
  3. Writer
  4. Serializable
  5. Serializable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PulsarOutputWriter(outputOptions: PulsarWriterOptions)

    outputOptions

    The Pulsar writer options containing the necessary parameters for writing to the Pulsar topic.

Type Members

  1. case class MissingWriterArgumentException(message: String = "", cause: Throwable = None.orNull) extends Exception with Product with Serializable
    Definition Classes
    Writer

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def getMetastoreUrl(addressInfo: Option[AddressInfo]): String

    Get the metastore URL from the provided AddressInfo.

    Get the metastore URL from the provided AddressInfo.

    addressInfo

    An optional AddressInfo that contains the connection information.

    returns

    The metastore URL as a String.

    Definition Classes
    Writer
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. val log: Logger

    Logger for logging messages during the Pulsar writing process.

  14. def mergeStreamingConfig(streamingConfig: Option[Streaming], outputOpts: Map[String, AnyRef]): Streaming

    Merge the streaming configuration with the output options for streaming queries.

    Merge the streaming configuration with the output options for streaming queries. This method is used to combine options from both streamingConfig and outputOptions. If the streamingConfig is None, it uses options from the outputOptions and applies default values as needed.

    streamingConfig

    Optional streaming configuration for the streaming query.

    outputOpts

    Output options for the writer.

    returns

    A Streaming object containing the merged streaming configuration.

    Definition Classes
    Writer
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. def setMandatoryArguments(mandatoryArgs: String*): Unit

    Set the mandatory arguments required by the writer implementation.

    Set the mandatory arguments required by the writer implementation.

    mandatoryArgs

    The names of mandatory arguments as varargs.

    Attributes
    protected
    Definition Classes
    Writer
  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. def toString(): String
    Definition Classes
    AnyRef → Any
  21. def validateMandatoryArguments(outputOptions: Map[String, Any]): Unit

    Validate if all mandatory arguments are present in the outputOptions map.

    Validate if all mandatory arguments are present in the outputOptions map. If any argument is missing, it throws a MissingWriterArgumentException.

    outputOptions

    The output options map containing the configuration for the writer.

    Definition Classes
    Writer
    Exceptions thrown

    MissingWriterArgumentException if any mandatory argument is missing.

  22. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  25. def write(dataFrame: DataFrame): Unit

    Writes the data from the DataFrame to the specified Pulsar topic based on the provided options.

    Writes the data from the DataFrame to the specified Pulsar topic based on the provided options.

    dataFrame

    The DataFrame containing the data to be written to the Pulsar topic.

    Definition Classes
    PulsarOutputWriter → Writer
  26. def writeStream(dataFrame: DataFrame, streamingConfig: Option[Streaming]): StreamingQuery

    Writes the data from the DataFrame to the specified Pulsar topic as a streaming query based on the provided options.

    Writes the data from the DataFrame to the specified Pulsar topic as a streaming query based on the provided options.

    dataFrame

    The DataFrame containing the streaming data to be written to the Pulsar topic.

    streamingConfig

    Optional streaming configuration to be merged with the Pulsar options (default: None).

    returns

    A StreamingQuery representing the active streaming query.

    Definition Classes
    PulsarOutputWriter → Writer

Inherited from DatasourceWriter

Inherited from Writer

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped