package file
Type Members
-
class
FileOutputWriter extends DatasourceWriter
A custom FileOutputWriter class that extends the DatasourceWriter.
A custom FileOutputWriter class that extends the DatasourceWriter. This class provides methods to write DataFrame data to files and handle various file format options.
-
case class
FileWriterOptions(path: Option[String], catalogName: Option[String], schemaName: Option[String], tableName: Option[String], saveMode: SaveMode = SaveMode.Append, partitionBy: Option[Seq[String]], format: String, alwaysUpdateSchemaInCatalog: Boolean, sort: Option[SortOptions], iceberg: Option[IcebergOutputOptions], metastoreUris: Option[String], warehousePath: Option[String], icebergCatalogType: Option[String], options: Map[String, String]) extends Product with Serializable
A case class that represents the configuration options for writing data to files.
A case class that represents the configuration options for writing data to files.
- path
The optional base path for the files to be written.
- catalogName
The optional name of the catalog.
- schemaName
The optional name of the schema.
- tableName
The optional name of the table.
- saveMode
The SaveMode used for writing data to files (default is Append).
- partitionBy
The optional list of column names to partition the data by.
- format
The file format to be used for writing data (e.g., parquet, csv, etc.).
- alwaysUpdateSchemaInCatalog
A flag indicating whether to always update the schema in the catalog.
- sort
The optional configuration for sorting the data before writing.
- iceberg
The optional configuration options for writing data in Iceberg format.
- metastoreUris
The optional URI(s) for the Hive metastore.
- warehousePath
The optional base path for the warehouse storage.
- icebergCatalogType
The optional type of the Iceberg catalog.
- options
Additional options as key-value pairs to configure the writer.
-
class
IcebergOutputWriter extends DatasourceWriter
A custom IcebergOutputWriter class that extends the DatasourceWriter.
A custom IcebergOutputWriter class that extends the DatasourceWriter. This class provides methods to write DataFrame data to Iceberg tables and handle Iceberg-specific properties.
- case class IcebergProps(properties: Option[Map[String, String]], partitionSpec: Option[List[PartitionSpecItem]], merge: Option[MergeProps]) extends Product with Serializable
- case class MergeProps(onClause: String, whenClause: String) extends Product with Serializable
- case class PartitionSpecItem(type: String, column: String, name: Option[String], numBuckets: Option[Int]) extends Product with Serializable
- case class SortColumnSpecItem(name: String, order: String) extends Product with Serializable
- case class SortProps(mode: String, columns: Option[List[SortColumnSpecItem]]) extends Product with Serializable