class IcebergOutputWriter extends DatasourceWriter
A custom IcebergOutputWriter class that extends the DatasourceWriter. This class provides methods to write DataFrame data to Iceberg tables and handle Iceberg-specific properties.
- Alphabetic
- By Inheritance
- IcebergOutputWriter
- DatasourceWriter
- Writer
- Serializable
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
IcebergOutputWriter(writerOptions: FileWriterOptions)
- writerOptions
The configuration options for the Iceberg output writer.
Type Members
-
case class
MissingWriterArgumentException(message: String = "", cause: Throwable = None.orNull) extends Exception with Product with Serializable
- Definition Classes
- Writer
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
- val fow: FileOutputWriter
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getMetastoreUrl(addressInfo: Option[AddressInfo]): String
Get the metastore URL from the provided AddressInfo.
Get the metastore URL from the provided AddressInfo.
- addressInfo
An optional AddressInfo that contains the connection information.
- returns
The metastore URL as a String.
- Definition Classes
- Writer
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- val logger: Logger
-
def
mergeStreamingConfig(streamingConfig: Option[Streaming], outputOpts: Map[String, AnyRef]): Streaming
Merge the streaming configuration with the output options for streaming queries.
Merge the streaming configuration with the output options for streaming queries. This method is used to combine options from both streamingConfig and outputOptions. If the streamingConfig is None, it uses options from the outputOptions and applies default values as needed.
- streamingConfig
Optional streaming configuration for the streaming query.
- outputOpts
Output options for the writer.
- returns
A Streaming object containing the merged streaming configuration.
- Definition Classes
- Writer
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
setMandatoryArguments(mandatoryArgs: String*): Unit
Set the mandatory arguments required by the writer implementation.
Set the mandatory arguments required by the writer implementation.
- mandatoryArgs
The names of mandatory arguments as varargs.
- Attributes
- protected
- Definition Classes
- Writer
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
def
validateMandatoryArguments(outputOptions: Map[String, Any]): Unit
Validate if all mandatory arguments are present in the outputOptions map.
Validate if all mandatory arguments are present in the outputOptions map. If any argument is missing, it throws a MissingWriterArgumentException.
- outputOptions
The output options map containing the configuration for the writer.
- Definition Classes
- Writer
- Exceptions thrown
MissingWriterArgumentException
if any mandatory argument is missing.
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
write(dataFrame: DataFrame): Unit
Writes the content of the DataFrame to the Iceberg table.
Writes the content of the DataFrame to the Iceberg table.
- dataFrame
The DataFrame to be written to the Iceberg table.
- Definition Classes
- IcebergOutputWriter → Writer
-
def
writeStream(dataFrame: DataFrame, streamingConfig: Option[Streaming]): StreamingQuery
Writes the content of the DataFrame to the Iceberg table in a streaming context.
Writes the content of the DataFrame to the Iceberg table in a streaming context.
- dataFrame
The DataFrame to be written to the Iceberg table.
- streamingConfig
The streaming configuration options.
- returns
The StreamingQuery representing the writing process.
- Definition Classes
- IcebergOutputWriter → Writer