Package 'datapond'

Connect to one or more datapond databases

Description

Opens an in-memory DuckDB connection and attaches each requested database read-only, either remotely from its attach_url (the default; DuckDB's httpfs extension fetches only the byte ranges a query touches, so there is no full download) or from a local copy made with dp_download().

Usage

dp_connect(id, local = FALSE, quiet = FALSE)

Arguments

id	One or more database ids (see dp_list()).
local	Attach local copies under dp_data_dir() instead of remote files?
quiet	Suppress progress messages?

Details

With a single id the database is made the default catalog (USE), so tables can be referenced unqualified. With several ids, qualify tables with the database id, double-quoting ids that contain a hyphen: ⁠"cms-medicare".physician_summary⁠.

Value

A duckdb_connection (a DBIConnection). Close it with dp_disconnect().

Examples

## Not run: 
con <- dp_connect("eoir")
DBI::dbGetQuery(con, "SELECT * FROM proceedings LIMIT 5")
dp_disconnect(con)

con <- dp_connect(c("cms-medicare", "openpayments"))
DBI::dbGetQuery(con, 'SELECT * FROM "cms-medicare".physician_summary LIMIT 5')
dp_disconnect(con)

## End(Not run)

---

Local directories used by datapond

Description

The registry is cached under tools::R_user_dir("datapond", "cache") and downloaded databases live under tools::R_user_dir("datapond", "data"). Override either with options(datapond.cache_dir = ...) / options(datapond.data_dir = ...). Setting options(datapond.data_dir = "~/.datapond") shares downloads with the Python package, which names local files the same way (⁠<id>.duckdb⁠).

Usage

dp_data_dir()

dp_cache_dir()

Value

A path (character scalar).

Examples

dp_data_dir()

---

Explore a database's data dictionary

Description

Every datapond database ships ⁠_metadata⁠ (one row per table) and ⁠_columns⁠ (types, null rates, example values, join hints). dp_describe() reads them and falls back to information_schema when they are absent.

Usage

dp_describe(x, table = NULL, search = NULL)

Arguments

x	A database id, or an open connection from dp_connect().
table	Describe the columns of this table instead of listing tables.
search	Find columns whose name contains this text (case-insensitive).

Value

A data frame (a tibble when the tibble package is installed).

Examples

## Not run: 
dp_describe("eoir")
dp_describe("eoir", table = "proceedings")
dp_describe("eoir", search = "judge")

## End(Not run)

---

Close a datapond connection

Description

Close a datapond connection

Usage

dp_disconnect(con)

Arguments

con	A connection from dp_connect().

Value

TRUE, invisibly.

---

Download a database for local use

Description

Downloads the full .duckdb file so later queries run at disk speed. Files are saved as ⁠<id>.duckdb⁠ under dp_data_dir() (or path), which is where dp_connect(id, local = TRUE) looks for them.

Usage

dp_download(id, path = NULL, quiet = FALSE, resume = TRUE)

Arguments

id	Database id.
path	Destination file or directory. Defaults to dp_data_dir().
quiet	Suppress the progress bar?
resume	Resume a partial download if one exists?

Details

The transfer goes to ⁠<file>.part⁠ and replaces the destination only after it is complete and opens as a DuckDB database, so an existing copy is never damaged by a failed download. A partial download is resumed only when the server identifies the file's revision (ETag, else Last-Modified), the partial file was started against that revision, and the ranged request is bound to it with If-Range; otherwise it restarts. A sidecar ⁠<file>.datapond.json⁠ records the remote file identity for dp_update().

Value

The local path, invisibly.

Examples

## Not run: 
dp_download("dol-visas")
con <- dp_connect("dol-visas", local = TRUE)

## End(Not run)

---

Information about one database

Description

Information about one database

Usage

dp_info(id)

Arguments

id	Database id.

Value

The registry entry (a list of class datapond_db), printed in a readable layout.

Examples

## Not run: 
dp_info("eoir")

## End(Not run)

---

List available databases

Description

List available databases

Usage

dp_list()

dp_databases()

Value

dp_list() returns a character vector of database ids; dp_databases() returns one row per database with its registry fields.

Examples

## Not run: 
dp_list()
dp_databases()[, c("id", "rows", "size_gb")]

## End(Not run)

---

Path of a locally downloaded database

Description

Path of a locally downloaded database

Usage

dp_local_path(id, must_exist = FALSE)

Arguments

id	Database id (see dp_list()).
must_exist	Error if the file is not present?

Value

A path (character scalar).

Examples

dp_local_path("eoir")

---

The datapond registry

Description

Fetches registry.json from GitHub (cached for an hour under dp_cache_dir(); a stale cache is used if the network is unavailable).

Usage

dp_registry(refresh = FALSE)

Arguments

refresh

Ignore the cache and fetch again?

Value

The parsed registry: a list with a databases element, one list per database.

Examples

## Not run: 
reg <- dp_registry()
length(reg$databases)

## End(Not run)

---

Update a local database if the remote file has changed

Description

Compares the identity of the remote file (ETag, else Last-Modified, from a HEAD request) with the one recorded when the local copy was downloaded, and re-downloads when they differ or when the revision cannot be verified. Copies without that record fall back to the registry's updated date versus the file's modification time and are re-downloaded whenever that comparison is inconclusive. The registry is re-fetched first.

Usage

dp_update(id, quiet = FALSE)

Arguments

id	Database id.
quiet	Suppress the progress bar?

Value

The local path, invisibly.

---