dlt.destinations.impl.dremio.pydremio
Copyright (C) 2017-2021 Dremio Corporation
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
The code in this module was original from https://github.com/dremio-hub/arrow-flight-client-examples/tree/main/python. The code has been modified and extended to provide a PEP 249 compatible interface.
This implementation will eagerly gather the full result set after every query. Eventually, this module should be replaced with ADBC Flight SQL client. See: https://github.com/apache/arrow-adbc/issues/1559
DremioCursor Objects
@dataclass
class DremioCursor()
execute
def execute(query: AnyStr,
parameters: Optional[Tuple[Any, ...]] = None,
*args: Any,
**kwargs: Any) -> None
Execute a query and prepare to stream results.
this configures a Flight stream reader so that results can be consumed incrementally via fetchone/fetchmany/fetchall or iter_arrow_tables.
Arguments:
query- SQL query string (or bytes)parameters- optional positional parameters for the query
fetchmany
def fetchmany(size: Optional[int] = None) -> List[Tuple[Any, ...]]
Fetch up to size rows, streaming from the server as needed.
Arguments:
size- number of rows to return. If None, uses self.arraysize.
Returns:
list of row tuples with up to size elements. May be empty when
no more rows are available.
fetch_arrow_table
def fetch_arrow_table() -> pyarrow.Table
Read and return the full remaining result as an Arrow table.
this drains the underlying Flight stream (if active) and returns all rows currently buffered plus all remaining rows from the server. The cursor buffer is reset to empty afterwards.
iter_arrow_tables
def iter_arrow_tables(size: Optional[int] = None) -> pyarrow.Table
Fetch up to size rows and return them as an Arrow table.
this mirrors fetchmany but returns a pyarrow.Table instead of a list of row tuples. It streams from the server as needed to satisfy the size.
Arguments:
size- number of rows to return. If None, uses self.arraysize.
Returns:
pyarrow.Table containing up to size rows. May be empty when no
more rows are available.
DremioClientAuthMiddlewareFactory Objects
class DremioClientAuthMiddlewareFactory(flight.ClientMiddlewareFactory)
A factory that creates DremioClientAuthMiddleware(s).
DremioClientAuthMiddleware Objects
class DremioClientAuthMiddleware(flight.ClientMiddleware)
A ClientMiddleware that extracts the bearer token from the authorization header returned by the Dremio Flight Server Endpoint.
Parameters
factory : ClientHeaderAuthMiddlewareFactory The factory to set call credentials if an authorization header with bearer token is returned by the Dremio server.
CookieMiddlewareFactory Objects
class CookieMiddlewareFactory(flight.ClientMiddlewareFactory)
A factory that creates CookieMiddleware(s).
CookieMiddleware Objects
class CookieMiddleware(flight.ClientMiddleware)
A ClientMiddleware that receives and retransmits cookies. For simplicity, this does not auto-expire cookies.
Parameters
factory : CookieMiddlewareFactory The factory containing the currently cached cookies.