pyflare.sdk.core package

Submodules

pyflare.sdk.core.dataos_input module

class pyflare.sdk.core.dataos_input.DataOSInput(name, parsed_inputs, spark, apikey, is_stream=None, source_format=None, driver=None, query=None, options=None)

Bases: object

process_inputs() → Tuple[Any, Any]

Read dataset from a source with the supplied parameters and create a temp view with the name passed in the dataos_source decorator.

pyflare.sdk.core.dataos_output module

class pyflare.sdk.core.dataos_output.DataOSOutput(name, dataframe, parsed_outputs, spark, apikey, is_stream=None, sink_format=None, mode=None, driver=None, options=None)

Bases: object

process_outputs()

Write the transformed dataset to sink, with the supplied parameters to dataos_sink decorator.

pyflare.sdk.core.decorator module

pyflare.sdk.core.minerva_input module

class pyflare.sdk.core.minerva_input.MinervaInput(name, parsed_inputs, spark, source_format='jdbc', driver='io.trino.jdbc.TrinoDriver', query=None, options=None)

Bases: object

process_inputs() → DataFrame

Run query on minerva and the result is stored as a temp view with the name passed in the dataos_source decorator.

pyflare.sdk.core.session_builder module

class pyflare.sdk.core.session_builder.SparkSessionBuilder(log_level: str)[source]

Bases: object

add_reader_instance(depot_name, format_list: list)[source]
add_writer_instance(depot_name, format_list: list)[source]
api_token: str = ''
build_session() → SparkSession[source]
dataos_fqdn: str = ''
load_default_spark_conf() → SparkSessionBuilder[source]
log_level: str = 'INFO'
logger: Logger = None
parsed_inputs: dict = {}
parsed_outputs: dict = {}
spark: SparkSession = None
spark_conf = []
with_dataos_fqdn(dataos_fqdn: str)[source]
with_depot(depot_name: str, acl: str = 'r') → SparkSessionBuilder[source]
with_readers(reader_address_list) → SparkSessionBuilder[source]
with_spark_conf(conf) → SparkSessionBuilder[source]
with_user_apikey(apikey: str)[source]
with_writers(writer_address_list) → SparkSessionBuilder[source]
pyflare.sdk.core.session_builder.g_dataos_token: str
pyflare.sdk.core.session_builder.g_inputs: dict
pyflare.sdk.core.session_builder.g_outputs: dict
pyflare.sdk.core.session_builder.load(name, format, driver=None, query=None, options=None)[source]
pyflare.sdk.core.session_builder.minerva_input(name, query, driver='io.trino.jdbc.TrinoDriver', options=None)[source]
pyflare.sdk.core.session_builder.refresh_global_data(spark_session_builder: SparkSessionBuilder)[source]
pyflare.sdk.core.session_builder.save(name: str, dataframe, format: Optional[str] = None, mode='append', driver=None, options=None)[source]
pyflare.sdk.core.session_builder.spark: SparkSession

Module contents