dlt.common.destination.capabilities
LoaderFileFormatSelector Objects
class LoaderFileFormatSelector(Protocol)
Selects preferred and supported file formats for a given table schema
MergeStrategySelector Objects
class MergeStrategySelector(Protocol)
Selects right set of merge strategies for a given table schema
ReplaceStrategySelector Objects
class ReplaceStrategySelector(Protocol)
Selects right set of replace strategies for a given table schema
DataTypeMapper Objects
class DataTypeMapper(ABC)
__init__
def __init__(capabilities: "DestinationCapabilitiesContext") -> None
Maps dlt data types into destination data types
to_destination_type
@abstractmethod
def to_destination_type(column: TColumnSchema,
table: PreparedTableSchema) -> str
Gets destination data type for a particular column in prepared table
from_destination_type
@abstractmethod
def from_destination_type(db_type: str, precision: Optional[int],
scale: Optional[int]) -> TColumnType
Gets column type from db type
ensure_supported_type
@abstractmethod
def ensure_supported_type(column: TColumnSchema, table: PreparedTableSchema,
loader_file_format: TLoaderFileFormat) -> None
Makes sure that dlt type in column in prepared table is supported by the destination for a given file format
UnsupportedTypeMapper Objects
class UnsupportedTypeMapper(DataTypeMapper)
Type Mapper that can't map any type
DestinationCapabilitiesContext Objects
@configspec
class DestinationCapabilitiesContext(ContainerInjectableContext)
Injectable destination capabilities required for many Pipeline stages ie. normalize
loader_file_format_selector
Callable that adapts preferred_loader_file_format and supported_loader_file_formats at runtime.
recommended_file_size
Recommended file size in bytes when writing extract/load files
escape_identifier
Escapes table name, column name and other identifiers
escape_literal
Escapes string literal
casefold_identifier
Casing function applied by destination to represent case insensitive identifiers.
has_case_sensitive_identifiers
Tells if destination supports case sensitive identifiers
timestamp_precision
Default precision of the timestamp type
max_timestamp_precision
Maximum supported timestamp precision
supports_clone_table
Destination supports CREATE TABLE ... CLONE ... statements
max_table_nesting
Allows a destination to overwrite max_table_nesting from source
max_parallel_load_jobs
The destination can set the maximum amount of parallel load jobs being executed
loader_parallelism_strategy
The destination can override the parallelism strategy
max_query_parameters
The maximum number of parameters that can be supplied in a single parametrized query
supports_native_boolean
The destination supports a native boolean type, otherwise bool columns are usually stored as integers
supports_nested_types
Tells if destination can write nested types, currently only destinations storing parquet are supported
enforces_nulls_on_alter
Tells if destination enforces null constraints when adding NOT NULL columns to existing tables
sqlglot_dialect
The SQL dialect used by sqlglot to transpile a query to match the destination syntax.
parquet_format
Parquet format preferred by this destination
supports_tz_aware_datetime
The destination can store datetime with timezone
supports_naive_datetime
The destination can store datetime without timezone
generates_case_sensitive_identifiers
def generates_case_sensitive_identifiers() -> bool
Tells if capabilities as currently adjusted, will generate case sensitive identifiers
merge_caps_file_formats
def merge_caps_file_formats(
destination: str, staging: str, dest_caps: DestinationCapabilitiesContext,
stage_caps: DestinationCapabilitiesContext
) -> Tuple[TLoaderFileFormat, Sequence[TLoaderFileFormat]]
Merges preferred and supported file formats from destination and staging. Returns new preferred file format and all possible formats.
adjust_column_schema_to_capabilities
def adjust_column_schema_to_capabilities(
column: TColumnSchema,
caps: DestinationCapabilitiesContext) -> TColumnSchema
Modify column schema in place according to destination capabilities.
- timestamp and time precision are limited by min(destination max, parquet writer max)
- timezone is always removed (enables default behavior)
- default timestamp precision for destination is removed
adjust_schema_to_capabilities
def adjust_schema_to_capabilities(
columns: TTableSchemaColumns,
caps: DestinationCapabilitiesContext) -> TTableSchemaColumns
Modifies columns schema in place according to destination caps. See adjust_column_schema_to_capabilities
for a list of adjustments.
NOTE: that function is intended to be used on schemas inferred from the data,
before user explicit hints are applied as it overrides ie. timezone and precision hints.
Returns modified columns schema