Packages

case class FileInput(name: String, path: Option[String], paths: Option[Seq[String]], warehousePath: Option[String], catalogName: Option[String], schemaName: Option[String], tableName: Option[String], metastoreUris: Option[String], options: Option[Map[String, String]], format: Option[String], isMetric: Option[Boolean], icebergCatalogType: Option[String], incremental: Option[Incremental], schema: Option[Schema]) extends IncrementalReader with DatasourceReader with FileInputBase with Product with Serializable

Case class representing a file input configuration.

name

The name of the file input.

path

Optional single file path.

paths

Optional list of file paths.

warehousePath

Optional warehouse path.

catalogName

Optional catalog name.

schemaName

Optional schema name.

tableName

Optional table name.

metastoreUris

Optional metastore URIs.

options

Optional additional options for reading files.

format

Optional file format.

isMetric

Optional flag indicating if the input is for metrics.

icebergCatalogType

Optional catalog type for Iceberg tables.

incremental

Optional incremental configuration.

schema

Optional schema info.

Linear Supertypes
Serializable, Serializable, Product, Equals, FileInputBase, DatasourceReader, Reader, IncrementalReader, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FileInput
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. FileInputBase
  7. DatasourceReader
  8. Reader
  9. IncrementalReader
  10. AnyRef
  11. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new FileInput(name: String, path: Option[String], paths: Option[Seq[String]], warehousePath: Option[String], catalogName: Option[String], schemaName: Option[String], tableName: Option[String], metastoreUris: Option[String], options: Option[Map[String, String]], format: Option[String], isMetric: Option[Boolean], icebergCatalogType: Option[String], incremental: Option[Incremental], schema: Option[Schema])

    name

    The name of the file input.

    path

    Optional single file path.

    paths

    Optional list of file paths.

    warehousePath

    Optional warehouse path.

    catalogName

    Optional catalog name.

    schemaName

    Optional schema name.

    tableName

    Optional table name.

    metastoreUris

    Optional metastore URIs.

    options

    Optional additional options for reading files.

    format

    Optional file format.

    isMetric

    Optional flag indicating if the input is for metrics.

    icebergCatalogType

    Optional catalog type for Iceberg tables.

    incremental

    Optional incremental configuration.

    schema

    Optional schema info.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. val catalogName: Option[String]
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. val filesInput: FilesInput
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. val format: Option[String]
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def getFormat(format: Option[String], path: String): String

    Determines the file format based on the specified format option or file path.

    Determines the file format based on the specified format option or file path.

    format

    The optional format specified by the user.

    path

    The file path.

    returns

    The determined file format.

    Definition Classes
    FileInputBase
  13. def getOptions(readFormat: String, options: Option[Map[String, String]]): Option[Map[String, String]]

    Retrieves the options for reading files based on the file format.

    Retrieves the options for reading files based on the file format.

    readFormat

    The file format.

    options

    The optional user-provided options.

    returns

    The options for reading files.

    Definition Classes
    FileInputBase
  14. def getSchemaStruct(schema: Option[Schema], sparkSession: SparkSession): Option[StructType]

    Retrieves the schema structure based on the provided schema path, schema string, and schema type.

    Retrieves the schema structure based on the provided schema path, schema string, and schema type.

    schema

    Schema info

    sparkSession

    The Spark session.

    returns

    The schema structure as a StructType.

    Definition Classes
    FileInputBase
  15. val icebergCatalogType: Option[String]
  16. val incremental: Option[Incremental]
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. val isMetric: Option[Boolean]
  19. val metastoreUris: Option[String]
  20. val name: String

    The name of the data reader.

    The name of the data reader.

    Definition Classes
    FileInput → Reader
  21. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  22. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  23. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  24. val options: Option[Map[String, String]]
  25. val path: Option[String]
  26. val pathList: Seq[String]
  27. val paths: Option[Seq[String]]
  28. def persistIncrementalState(): Unit

    Persists the incremental state if any.

    Persists the incremental state if any.

    Definition Classes
    IncrementalReader
  29. def read(sparkSession: SparkSession): DataFrame

    Reads data from the file input.

    Reads data from the file input.

    sparkSession

    The SparkSession instance.

    returns

    The DataFrame containing the read data.

    Definition Classes
    FileInput → Reader
  30. def readIncremental(df: DataFrame, incremental: Option[Incremental]): DataFrame

    Reads data from a DataFrame with optional incremental settings.

    Reads data from a DataFrame with optional incremental settings.

    df

    The DataFrame to read data from.

    incremental

    Optional Incremental settings to apply.

    returns

    A new DataFrame after applying incremental settings if provided, otherwise the original DataFrame.

    Definition Classes
    IncrementalReader
  31. val schema: Option[Schema]
  32. val schemaName: Option[String]
  33. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  34. val tableName: Option[String]
  35. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  36. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  37. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  38. val warehousePath: Option[String]

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from FileInputBase

Inherited from DatasourceReader

Inherited from Reader

Inherited from IncrementalReader

Inherited from AnyRef

Inherited from Any

Ungrouped