---
title: "Introduction to the IPUMS API for R Users"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Introduction to the IPUMS API for R Users}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

```{r, echo=FALSE, results="hide"}
library(vcr)

vcr_dir <- "fixtures"

have_api_access <- TRUE

if (!nzchar(Sys.getenv("IPUMS_API_KEY"))) {
  if (dir.exists(vcr_dir) && length(dir(vcr_dir)) > 0) {
    # Fake API token to fool ipumsr API functions
    Sys.setenv("IPUMS_API_KEY" = "foobar")
  } else {
    # If there are no mock files nor API token, can't run API tests
    have_api_access <- FALSE
  }
}

vcr_configure(
  filter_sensitive_data = list(
    "<<<IPUMS_API_KEY>>>" = Sys.getenv("IPUMS_API_KEY")
  ),
  write_disk_path = vcr_dir,
  dir = vcr_dir
)

modify_ready_extract_cassette_file <- function(cassette_file_name,
                                               fixture_path = NULL,
                                               n_requests = 1) {
  fixture_path <- fixture_path %||% vcr::vcr_test_path("fixtures")

  ready_extract_cassette_file <- file.path(
    fixture_path, cassette_file_name
  )

  ready_lines <- readLines(ready_extract_cassette_file)
  request_lines <- which(grepl("^- request:", ready_lines))

  start_line <- request_lines[length(request_lines) - n_requests + 1]

  writeLines(
    c(
      ready_lines[[1]],
      ready_lines[start_line:length(ready_lines)]
    ),
    con = ready_extract_cassette_file
  )
}
```

The IPUMS API provides two asset types, both of which are supported by
ipumsr:

-   **IPUMS extract** endpoints can be used to submit extract requests
    for processing and download completed extract files.
-   **IPUMS metadata** endpoints can be used to discover and explore
    available IPUMS data as well as retrieve codes, names, and other
    extract parameters necessary to form extract requests.

Use of the IPUMS API enables the adoption of a programmatic workflow
that can help users to:

-   Precisely recreate the specifications of previous extract requests,
    making analysis scripts reproducible and self-contained
-   Save extract request definitions that can be shared with others
    without violating IPUMS conditions
-   Integrate the extract download process with functions to load data
    into R
-   Quickly identify and explore available IPUMS data sources

The basic workflow for interacting with the IPUMS API is as follows:

1.  [Define](#define) the parameters of an extract request
2.  [Submit](#submit) the extract request to the IPUMS API
3.  [Wait](#wait) for an extract to complete
4.  [Download](#download) a completed extract

Before getting started, we'll load the necessary packages for the
examples in this vignette:

```{r, message=FALSE}
library(ipumsr)
library(dplyr)
library(purrr)
```

## API availability

IPUMS **extract** support is currently available via API for the
following collections:

-   IPUMS microdata collections
    *   IPUMS USA
    *   IPUMS CPS
    *   IPUMS International
    *   IPUMS Time Use (ATUS, AHTUS, MTUS)
    *   IPUMS Health Surveys (NHIS, MEPS)
    *   IPUMS DHS

-   IPUMS aggregate data collections
    *   IPUMS NHGIS
    *   IPUMS IHGIS

Note that this support only includes data available via a collection's
extract engine. Many collections provide supplemental data files via direct
download. Currently, API support for downloading these files is limited to
IPUMS NHGIS. See `download_supplemental_data()` for more information.

IPUMS **metadata** support is currently available via API for the IPUMS
aggregate data collections (NHGIS and IHGIS).

API support will continue to be added for more collections in the
future. You can check general API availability for all IPUMS collections
with `ipums_data_collections()`.

```{r}
ipums_data_collections()
```

Note that the tools in ipumsr may not necessarily support all the functionality
currently supported by the IPUMS API. See the [API
documentation](https://developer.ipums.org/docs/apiprogram/) for more
information about its latest features.

## Set up your API key {#set-key}

To interact with the IPUMS API, you'll need to register for access with
the IPUMS project you'll be using. If you have not yet registered, you
can find the link to register for each project at the top of its website, which 
can be accessed from [ipums.org](https://www.ipums.org).

Once you're registered, you'll be able to [create an API
key](https://account.ipums.org/api_keys).

By default, ipumsr API functions assume that your key is stored in the
`IPUMS_API_KEY` environment variable. You can also provide your key
directly to these functions, but storing it in an environment variable
saves you some typing and helps prevent you from inadvertently sharing
your key with others (for instance, on GitHub).

You can save your API key to the `IPUMS_API_KEY` environment variable
with `set_ipums_api_key()`. To save your key for use in future sessions,
set `save = TRUE`. This will add your API key to your `.Renviron` file
in your user home directory.

```{r, eval=FALSE}
# Save key in .Renviron for use across sessions
set_ipums_api_key("paste-your-key-here", save = TRUE)
```

The rest of this vignette assumes you have obtained an API key and
stored it in the `IPUMS_API_KEY` environment variable.

## Define an extract request {#define}

ipumsr contains two extract definition functions used to specify the parameters
of a new extract request from scratch.

  -   For microdata projects, use `define_extract_micro()`.
  -   For aggregate data projects, use `define_extract_agg()`.

When you define an extract request, you can specify the data to be
included in the extract and indicate the desired format and layout.

For instance, the following defines a simple IPUMS USA extract request
for the `AGE`, `SEX`, `RACE`, `STATEFIP`, and `MARST` variables from the
2018 and 2019 American Community Survey (ACS):

```{r}
usa_extract_definition <- define_extract_micro(
  collection = "usa",
  description = "USA extract for API vignette",
  samples = c("us2018a", "us2019a"),
  variables = c("AGE", "SEX", "RACE", "STATEFIP", "MARST")
)

usa_extract_definition
```

The exact extract definition options vary across collections, but all
collections can be used with the same general workflow. For more details
on the available extract definition options, see the associated
[microdata](ipums-api-micro.html) and [aggregate data](ipums-api-agg.html)
vignettes.

For the purposes of demonstrating the overall workflow, we will continue
to work with the sample IPUMS USA extract definition created above.

### Extract request objects

`define_extract_micro()` and `define_extract_agg()` always produce an 
`ipums_extract` object, which can be handled by other API functions 
(see `?ipums_extract`). Furthermore, these objects will have a subclass for 
the particular collection with which they are associated.

```{r}
class(usa_extract_definition)
```

Many of the specifications for a given extract request object can be
accessed by indexing the object:

```{r}
names(usa_extract_definition$samples)

names(usa_extract_definition$variables)

usa_extract_definition$data_format
```

`ipums_extract` objects also contain information about the extract
request's processing status and its assigned extract number, which
serves as an identifier for the extract request. Since this extract
request is still unsubmitted, it has no request number:

```{r}
usa_extract_definition$status

usa_extract_definition$number
```

To obtain the data requested in the extract definition, we must first
submit it to the IPUMS API for processing.

## Submit an extract request {#submit}

```{r, include=FALSE}
insert_cassette("submit-placeholder-extract-usa")

# We submit these extracts so that the output of requests for things like
# `get_extract_history` below is more "natural".
# Otherwise the output is a bunch of duplicate extract requests for the
# primary extract in this vignette, as this vignette usually gets rebuilt
# several times during editing before it is complete.
submit_extract(
  define_extract_micro(
    "usa",
    description = "Data from 2017 PRCS",
    samples = "us2017b",
    variables = c("RACE", "YEAR")
  )
)

submit_extract(
  define_extract_micro(
    "usa",
    description = "Data from long ago",
    samples = "us1880a",
    variables = c("SEX", "AGE", "LABFORCE")
  )
)

eject_cassette()
```

```{r, echo=FALSE, results="hide", message=FALSE}
insert_cassette("submit-extract")
```

To submit an extract definition, use `submit_extract()`.

If no errors are detected in the extract definition, a submitted extract
request will be returned with its assigned number and status. Storing
the returned object can be useful for checking the extract request's
status later.

```{r}
usa_extract_submitted <- submit_extract(usa_extract_definition)
```

The extract number will be stored in the returned object:

```{r}
usa_extract_submitted$number
usa_extract_submitted$status
```

Note that some fields of a submitted extract may be automatically
updated by the API upon submission. For instance, for microdata
extracts, additional preselected variables may be added to the extract
even if they weren't specified explicitly in the extract definition.

```{r}
names(usa_extract_submitted$variables)
```

If you forget to store the updated extract object returned by
`submit_extract()`, you can use the `get_last_extract_info()` helper to
request the information for your most recent extract request for a given
collection:

```{r}
usa_extract_submitted <- get_last_extract_info("usa")

usa_extract_submitted$number
```

```{r, echo=FALSE, results="hide", message=FALSE}
eject_cassette()
```

## Wait for an extract request to complete {#wait}

It may take some time for the IPUMS servers to process your extract
request. You can ensure that an extract has finished processing before
you attempt to download its files by using `wait_for_extract()`. This
polls the API regularly until processing has completed (by default, each
interval increases by 10 seconds). It then returns an `ipums_extract`
object containing the completed extract definition.

```{r, echo=FALSE, results="hide", message=FALSE}
insert_cassette("wait-for-extract")

usa_extract_complete <- wait_for_extract(usa_extract_submitted)

eject_cassette()

# Leave an extract request to simulate wait_for_extract() output for USA
modify_ready_extract_cassette_file(
  "wait-for-extract.yml",
  fixture_path = "fixtures",
  n_requests = 2
)
```

```{r, echo=FALSE, results="hide", message=FALSE}
insert_cassette("wait-for-extract")
```

```{r}
usa_extract_complete <- wait_for_extract(usa_extract_submitted)
```

```{r}
usa_extract_complete$status
```

```{r}
# `download_links` should be populated if the extract is ready for download
names(usa_extract_complete$download_links)
```

```{r, echo=FALSE, results="hide", message=FALSE}
eject_cassette()
```

Note that `wait_for_extract()` will tie up your R session until your
extract is ready to download. While this is fine in a strictly
programmatic workflow, it may be frustrating when working interactively,
especially for large extracts or when the IPUMS servers are busy.

In these cases, you can manually check whether an extract is ready for
download with `is_extract_ready()`. As long as this returns `TRUE`, you
should be able to download your extract's files.

```{r, echo=FALSE, results="hide", message=FALSE}
insert_cassette("extract-ready")
```

```{r}
is_extract_ready(usa_extract_submitted)
```

```{r, echo=FALSE, results="hide", message=FALSE}
eject_cassette()
```

For a more detailed status check, provide the extract's collection and
number to `get_extract_info()`. This returns an `ipums_extract` object
reflecting the requested extract definition with the most current
status. The `status` of a submitted extract will be one of `"queued"`,
`"started"`, `"produced"`, `"canceled"`, `"failed"`, or `"completed"`.

```{r, echo=FALSE, results="hide", message=FALSE}
insert_cassette("check-extract-info")
```

```{r}
usa_extract_submitted <- get_extract_info(usa_extract_submitted)

usa_extract_submitted$status
```

```{r, echo=FALSE, results="hide", message=FALSE}
eject_cassette()
```

Note that extracts are removed from the IPUMS servers after a set period
of time (72 hours for microdata collections, 2 weeks for IPUMS NHGIS).
Therefore, an extract that has a `"completed"` status may still be
unavailable for download.

`is_extract_ready()` will alert you if the extract has expired and needs
to be resubmitted. Simply use `submit_extract()` to resubmit an extract
request. Note that this will produce a *new* extract (with a new extract
number), even if the extract definition is identical.

## Download an extract {#download}

Once your extract has finished processing, use `download_extract()` to
download the extract's data files to your local machine. This will
return the path to the downloaded file(s) required to load the data into
R.

For microdata collections, this will be the path to the DDI codebook
(.xml) file, which can be used to read the associated data (contained in
a .dat.gz file).

For NHGIS, this will be a path to the .zip archive containing the
requested data files and/or shapefiles.

```{r, eval=FALSE}
# By default, downloads to your current working directory
filepath <- download_extract(usa_extract_submitted)
```

The files produced by `download_extract()` can be passed directly into
the reader functions provided by ipumsr. For instance, for microdata
projects:

```{r, eval=FALSE}
ddi <- read_ipums_ddi(filepath)
micro_data <- read_ipums_micro(ddi)
```

If instead you're working with an aggregate data extract, use 
`read_ipums_agg()` (or `read_ipums_sf()` for spatial data from IPUMS NHGIS
extracts).

See the [associated vignette](ipums-read.html) for more information
about loading IPUMS data into R.

## Get info on past extracts {#recent}

To retrieve the definition corresponding to a particular extract,
provide its collection and number to `get_extract_info()`. These can be
provided either as a single string of the form `"collection:number"` or
as a length-2 vector: `c("collection", number)`. Several other API
functions support this syntax as well.

```{r, echo=FALSE, results="hide", message=FALSE}
insert_cassette("check-extract-history")
```

```{r}
usa_extract <- get_extract_info("usa:47")

# Alternatively:
usa_extract <- get_extract_info(c("usa", 47))

usa_extract
```

If you know you made a specific extract definition in the past, but you
can't remember the exact number, you can use `get_extract_history()` to
peruse your recent extract requests for a particular collection.

By default, this returns your 10 most recent extract requests as a list
of `ipums_extract` objects. You can adjust how many requests to retrieve
with the `how_many` argument:

```{r}
usa_extracts <- get_extract_history("usa", how_many = 3)

usa_extracts
```

Because this is a list of `ipums_extract` objects, you can operate on
them with the API functions that have been introduced already.

```{r}
is_extract_ready(usa_extracts[[2]])
```

You can also iterate through your extract history to find extracts with
particular characteristics. For instance, we can use `purrr::keep()` to
find all extracts that contain a certain variable or are ready for
download:

```{r}
purrr::keep(usa_extracts, ~ "MARST" %in% names(.x$variables))

purrr::keep(usa_extracts, is_extract_ready)
```

Or we can use the `purrr::map()` family to browse certain values:

```{r}
purrr::map_chr(usa_extracts, ~ .x$description)
```

If you regularly use only a single IPUMS collection, you can save
yourself some typing by setting that collection as your default.
`set_ipums_default_collection()` will save a specified collection to the
value of the `IPUMS_DEFAULT_COLLECTION` environment variable. If you
have a default collection set, API functions will use that collection in
all requests, assuming no other collection is specified.

```{r, eval=FALSE}
# Set `save = TRUE` to store across sessions
set_ipums_default_collection("usa")
```

```{r, echo=FALSE, results="hide", message=FALSE}
set_ipums_default_collection("usa")
```

```{r}
# Check the default collection:
Sys.getenv("IPUMS_DEFAULT_COLLECTION")
```

```{r}
# Most recent USA extract:
usa_last <- get_last_extract_info()

# Request info on extract request "usa:10"
usa_extract_10 <- get_extract_info(10)

# You can still request other collections as usual:
cps_extract_10 <- get_extract_info("cps:10")
```


```{r, echo=FALSE, results="hide", message=FALSE}
eject_cassette()
```

## Share an extract definition {#share}

One exciting feature enabled by the IPUMS API is the ability to share a
standardized extract definition with other IPUMS users so that they can
create an identical extract request themselves. The terms of use for
most IPUMS collections prohibit the public redistribution of IPUMS data, but
don't prohibit the sharing of data extract definitions.

ipumsr facilitates this type of sharing with `save_extract_as_json()`
and `define_extract_from_json()`, which write and read `ipums_extract`
objects to and from a standardized JSON-formatted file.

```{r, eval=FALSE}
usa_extract_10 <- get_extract_info("usa:10")
save_extract_as_json(usa_extract_10, file = "usa_extract_10.json")
```

At this point, you can send `"usa_extract_10.json"` to another user to
allow them to create a duplicate `ipums_extract` object, which they can
load and submit to the API themselves (as long as they have [API
access](#set-key)).

```{r, eval=FALSE}
clone_of_usa_extract_10 <- define_extract_from_json("usa_extract_10.json")
usa_extract_10_resubmitted <- submit_extract(clone_of_usa_extract_10)
```

Note that the code in the previous chunk assumes that the file is saved
in the current working directory. If it's saved somewhere else, replace
`"usa_extract_10.json"` with the full path to the file.

## Revise a previous extract request

Occasionally, you may want to modify an existing extract definition
(e.g. to update an analysis with new data). The easiest way to do so is
to add the new specifications to the `define_extract_*()` code that
produced the original extract definition. This is why we highly
recommend that you save this code somewhere where it can be accessed and
updated in the future.

However, there are cases where the original extract definition code does
not exist (e.g. if the extract was created using the online IPUMS
extract system). In this case, the best approach is to view the extract
definition with `get_extract_info()` and create a new extract definition
(using a `define_extract_*()` function) that reproduces that definition
along with the desired modifications. While this may be a bit tedious
for complex extract definitions, it is a one-time investment that will
make any future updates to the extract definition much easier.

Previously, we encouraged users to use the helpers `add_to_extract()`
and `remove_from_extract()` when modifying extracts. We now encourage
you to re-write extract definitions because they improve
reproducibility: extract definition code will always be more clear and
stable if it is written explicitly, rather than based only on an old
extract number. These two functions may be retired in the future.

## Putting it all together

The core API functions in ipumsr are compatible with one another such
that they can be combined into a single pipeline that requests,
downloads, and reads your extract data into an R data frame:

```{r, eval=FALSE}
usa_data <- define_extract_micro(
  "usa",
  "USA extract for API vignette",
  samples = c("us2018a", "us2019a"),
  variables = c("AGE", "SEX", "RACE", "STATEFIP")
) %>%
  submit_extract() %>%
  wait_for_extract() %>%
  download_extract() %>%
  read_ipums_micro()
```

Note that for NHGIS extracts that contain both data and shapefiles, a
single file will need to be selected before reading, as
`download_extract()` will return the path to each file. For instance,
for a hypothetical `nhgis_extract` that contains both tabular and
spatial data:

```{r, eval=FALSE}
nhgis_data <- download_extract(nhgis_extract) %>%
  purrr::pluck("data") %>% # Select only the tabular data file to read
  read_ipums_agg()
```

Not only does this API workflow allow you to obtain IPUMS data without
ever leaving your R environment, but it also allows you to retain a
reproducible record of your process. This makes it much easier to
document your workflow, collaborate with other researchers, and update
your analysis in the future.
