<!-- Generated by galley: do not edit by hand -->

Multiple, linked tables are common within computer science. Because many
R users have backgrounds in other disciplines, we present **six
important terms in relational data modeling** to help you to jump-start
working with {dm}. These terms are:

1.  [Data Frames and Tables](#tables)
2.  [Data Model](#model)
3.  [Primary Keys](#pk)
4.  [Foreign Keys](#fk)
5.  [Referential Integrity](#referential-integrity)
6.  [Normalization](#normalization)
7.  [Relational Databases](#relational-databases)

## 1. Data Frames and Tables

A data frame is a fundamental data structure in R. Columns represent
variables, rows represent observations. In more technical terms: a data
frame is a list of variables of identical length and unique row names.
If you imagine it visually, the result is a typical table structure.
That is why working with data from spreadsheets is so convenient and the
users of the popular [{dplyr}](https://dplyr.tidyverse.org) package for
data wrangling mainly rely on data frames.

The downside is that data frames and flat file systems like spreadsheets
can result in bloated tables because they hold many repetitive values.
In the worst case, a data frame can contain multiple columns with only a
single value different in each row.

This calls for better data organization by utilizing the resemblance
between data frames and database tables, which also consist of columns
and rows. The elements are just named differently:

| Data Frame | Table     |
|------------|-----------|
| Column     | Attribute |
| Row        | Tuple     |

Relational databases, unlike data frames, do not keep all data in one
large table but instead split it into multiple smaller tables. That
separation into sub-tables has several advantages:

-   all information is stored only once, avoiding redundancy and
    conserving memory
-   all information needs to be updated only once and in one place,
    improving consistency and avoiding errors that may result from
    updating the same value in multiple locations
-   all information is organized by topic and segmented into smaller
    tables that are easier to handle

It is for these reasons that separation of data helps with data quality,
and they explain the popularity of relational databases in
production-level data management.

The downside of this approach is that it is harder to merge together
information from different data sources and to identify which entities
refer to the same object, a common task when modeling or plotting data.

Thus, to take full advantage of the relational database approach, an
associated data model is needed to overcome the challenges that arise
when working with multiple tables.

Let’s illustrate this challenge with the data from the [`nycflights13`
dataset](https://github.com/tidyverse/nycflights13) that contains
detailed information about the 336776 flights that departed from New
York City in 2013. The information is stored in five tables.

Details like the full name of an airport are not available immediately;
these can only be obtained by joining or merging the constituent tables,
which can result in long and inflated pipe chains full of `left_join()`,
`anti_join()` and other forms of data merging.

In classical {dplyr} notation, you will need three `left_join()` calls
to merge the `flights` table gradually to `airlines`, `planes`,
`airports` and `weather` tables to create one wide data frame.

``` r
library(tidyverse)
#> ── Attaching packages ────────────────────────────────── tidyverse 1.3.1 ──
#> ✔ ggplot2 3.3.3     ✔ purrr   0.3.4
#> ✔ tibble  3.1.1     ✔ dplyr   1.0.5
#> ✔ tidyr   1.1.3     ✔ stringr 1.4.0
#> ✔ readr   1.4.0     ✔ forcats 0.5.1
#> ── Conflicts ───────────────────────────────────── tidyverse_conflicts() ──
#> ✖ purrr::%@%()         masks dm::%@%()
#> ✖ purrr::as_function() masks dm::as_function()
#> ✖ tidyr::extract()     masks dm::extract()
#> ✖ dplyr::filter()      masks dm::filter(), stats::filter()
#> ✖ purrr::flatten()     masks dm::flatten()
#> ✖ purrr::flatten_chr() masks dm::flatten_chr()
#> ✖ purrr::flatten_dbl() masks dm::flatten_dbl()
#> ✖ purrr::flatten_int() masks dm::flatten_int()
#> ✖ purrr::flatten_lgl() masks dm::flatten_lgl()
#> ✖ purrr::flatten_raw() masks dm::flatten_raw()
#> ✖ purrr::invoke()      masks dm::invoke()
#> ✖ purrr::is_null()     masks dm::is_null(), testthat::is_null()
#> ✖ dplyr::lag()         masks dm::lag(), stats::lag()
#> ✖ purrr::list_along()  masks dm::list_along()
#> ✖ dplyr::matches()     masks tidyr::matches(), dm::matches(), testthat::matches()
#> ✖ purrr::modify()      masks dm::modify()
#> ✖ purrr::prepend()     masks dm::prepend()
#> ✖ purrr::splice()      masks dm::splice()
library(dm)
library(nycflights13)

flights %>%
  left_join(airlines, by = "carrier") %>%
  left_join(planes, by = "tailnum") %>%
  left_join(airports, by = c("origin" = "faa")) %>%
  left_join(weather, by = c("origin", "time_hour"))
#> # A tibble: 336,776 x 48
#>    year.x month.x day.x dep_time sched_dep_time dep_delay arr_time
#>     <int>   <int> <int>    <int>          <int>     <dbl>    <int>
#>  1   2013       1     1      517            515         2      830
#>  2   2013       1     1      533            529         4      850
#>  3   2013       1     1      542            540         2      923
#>  4   2013       1     1      544            545        -1     1004
#>  5   2013       1     1      554            600        -6      812
#>  6   2013       1     1      554            558        -4      740
#>  7   2013       1     1      555            600        -5      913
#>  8   2013       1     1      557            600        -3      709
#>  9   2013       1     1      557            600        -3      838
#> 10   2013       1     1      558            600        -2      753
#> # … with 336,766 more rows, and 41 more variables: sched_arr_time <int>,
#> #   arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
#> #   origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
#> #   hour.x <dbl>, minute <dbl>, time_hour <dttm>, name.x <chr>,
#> #   year.y <int>, type <chr>, manufacturer <chr>, model <chr>,
#> #   engines <int>, seats <int>, speed <int>, engine <chr>, name.y <chr>,
#> #   lat <dbl>, lon <dbl>, alt <dbl>, tz <dbl>, dst <chr>, tzone <chr>,
#> #   year <int>, month.y <int>, day.y <int>, hour.y <int>, temp <dbl>,
#> #   dewp <dbl>, humid <dbl>, wind_dir <dbl>, wind_speed <dbl>,
#> #   wind_gust <dbl>, precip <dbl>, pressure <dbl>, visib <dbl>
```

{dm} offers a more elegant and shorter way to combine tables while
augmenting {dplyr}/{dbplyr} workflows.

It is possible to have the best of both worlds: manage your data with
{dm} as linked tables, then flatten multiple tables into one for your
analysis with {dplyr} on an as-needed basis.

The next step is to create a [data model](#model) based on multiple
tables:

## 2. Data Model

A data model shows the structure between multiple tables that can be
linked together.

The `nycflights13` relations can be transferred into the following
graphical representation:

``` r
dm <- dm_nycflights13(cycle = TRUE)

dm %>%
  dm_draw()
```

![](/home/kirill/git/R/dm/vignettes/out/howto-dm-theory_files/figure-gfm/unnamed-chunk-2-1.png)<!-- -->

The `flights` table is linked to three other tables: `airlines`,
`planes` and `airports`. By using directed arrows, the visualization
shows explicitly the connection between different columns (they are
called attributes in the relational data sphere).

For example: The column `carrier` in `flights` can be joined with the
column `carrier` from the `airlines` table.

The links between the tables are established through [primary keys](#pk)
and [foreign keys](#fk).

Further Reading: The {dm} methods for [visualizing data
models](https://cynkra.github.io/dm/articles/tech-dm-draw.html).

## 3. Primary Keys

In a relational data model, each table should have **one or several
columns that uniquely identifies a row**. These columns define the
*primary key* (abbreviated with “pk”). If the key consists of a single
column, it is called *simple key*. A key consisting of more than one
column is called a *compound key*.

Example: In the `airlines` table of `nycflights13` the column `carrier`
is the primary key, a simple key. The `weather` table has the
combination of `origin` and `time_hour` as primary key, a compound key.

You can get all primary keys in a `dm` by calling `dm_get_all_pks()`:

``` r
dm %>%
  dm_get_all_pks()
#> # A tibble: 4 x 2
#>   table    pk_col           
#>   <chr>    <keys>           
#> 1 airlines carrier          
#> 2 airports faa              
#> 3 planes   tailnum          
#> 4 weather  origin, time_hour
```

`dm_enum_pk_candidates()` checks all columns if they are suitable as a
simple primary key:

``` r
dm %>%
  dm_enum_pk_candidates(airports)
#> # A tibble: 8 x 3
#>   columns candidate why                                                    
#>   <keys>  <lgl>     <chr>                                                  
#> 1 faa     TRUE      ""                                                     
#> 2 lon     TRUE      ""                                                     
#> 3 name    FALSE     "has duplicate values: Municipal Airport (5), All Airp…
#> 4 lat     FALSE     "has duplicate values: 38.88944 (2), 40.63975 (2)"     
#> 5 alt     FALSE     "has duplicate values: 0 (51), 13 (13), 14 (12), 15 (1…
#> 6 tz      FALSE     "has duplicate values: -5 (521), -6 (342), -9 (240), -…
#> 7 dst     FALSE     "has duplicate values: A (1388), U (47), N (23)"       
#> 8 tzone   FALSE     "has duplicate values: America/New_York (519), America…
```

Further Reading: The {dm} package offers several functions for dealing
with [primary
keys](https://cynkra.github.io/dm/articles/tech-dm-class.html#pk).

## 4. Foreign Keys

The **counterpart of a primary key in one table is the foreign key in
another table**. In order to join two tables, the primary key of the
first table needs to be referenced from the second table. This column or
these columns are called the *foreign key* (abbreviated with “fk”).

For example, if you want to link the `airlines` table to the `flights`
table, the primary key in `airlines` needs to match the foreign key in
`flights`. This condition is satisfied because the column `carrier` is
present as a primary key in the `airlines` table as well as a foreign
key in the `flights` table. In the case of compound keys, the `origin`
and `time_hour` columns (which form the primary key of the `weather`
table) are also present in the `flights` table.

You can find foreign key candidates for simple keys with the function
`dm_enum_fk_candidates()`, they are marked with `TRUE` in the
`candidate` column.

``` r
dm %>%
  dm_enum_fk_candidates(flights, airlines)
#> # A tibble: 19 x 3
#>    columns      candidate why                                              
#>    <keys>       <lgl>     <chr>                                            
#>  1 carrier      TRUE      ""                                               
#>  2 year         FALSE     "Can't combine `value1` <integer> and `value1` <…
#>  3 month        FALSE     "Can't combine `value1` <integer> and `value1` <…
#>  4 day          FALSE     "Can't combine `value1` <integer> and `value1` <…
#>  5 dep_time     FALSE     "Can't combine `value1` <integer> and `value1` <…
#>  6 sched_dep_t… FALSE     "Can't combine `value1` <integer> and `value1` <…
#>  7 dep_delay    FALSE     "Can't combine `value1` <double> and `value1` <c…
#>  8 arr_time     FALSE     "Can't combine `value1` <integer> and `value1` <…
#>  9 sched_arr_t… FALSE     "Can't combine `value1` <integer> and `value1` <…
#> 10 arr_delay    FALSE     "Can't combine `value1` <double> and `value1` <c…
#> 11 flight       FALSE     "Can't combine `value1` <integer> and `value1` <…
#> 12 tailnum      FALSE     "values of `flights$tailnum` not in `airlines$ca…
#> 13 origin       FALSE     "values of `flights$origin` not in `airlines$car…
#> 14 dest         FALSE     "values of `flights$dest` not in `airlines$carri…
#> 15 air_time     FALSE     "Can't combine `value1` <double> and `value1` <c…
#> 16 distance     FALSE     "Can't combine `value1` <double> and `value1` <c…
#> 17 hour         FALSE     "Can't combine `value1` <double> and `value1` <c…
#> 18 minute       FALSE     "Can't combine `value1` <double> and `value1` <c…
#> 19 time_hour    FALSE     "Can't combine `value1` <datetime<America/New_Yo…
```

Further Reading: All {dm} functions for working with [foreign
keys](https://cynkra.github.io/dm/articles/tech-dm-class.html#foreign-keys).

## 5. Referential Integrity

A data set has referential integrity if all relations between tables are
valid. That is, every foreign key holds a primary key that is present in
the parent table. If a foreign key contains a reference where the
corresponding row in the parent table is not availabe, that row is an
orphan row and the database no longer has referential integrity.

{dm} allows checking referential integrity with the
`dm_examine_constraints()` function. The following conditions are
checked:

-   All primary key values must be unique.
-   All primary key values must be set, `NA` values are not allowed.
-   Each foreign key value must have a corresponding primary key value.

In the example data model, for a substantial share of the flights,
detailed information for the corresponding airplane is not available:

``` r
dm %>%
  dm_examine_constraints()
#> ! Unsatisfied constraints:
#> ● Table `flights`: foreign key dest into table `airports`: values of `flights$dest` not in `airports$faa`: SJU (187), BQN (28), STT (15), PSE (12)
#> ● Table `flights`: foreign key tailnum into table `planes`: values of `flights$tailnum` not in `planes$tailnum`: N722MQ (27), N725MQ (20), N520MQ (19), N723MQ (19), N508MQ (16), …
```

Establishing referential integrity is important for providing clean data
for analysis or downstream users. See `vignette("howto-dm-rows")` for
more information on adding, deleting or updating individual rows, and
`vignette("tech-dm-zoom")` for operations on the data in a data model.

## 6. Normalization

Normalization is a technical term that describes the **central design
principle of a relational data model**: splitting data into multiple
tables.

A normalized data schema consists of several relations (tables) that are
linked with attributes (columns). The relations can be joined together
by means of [primary](#pk) and [foreign keys](#fk). The main goal of
normalization is to keep data organization as clean and simple as
possible by avoiding redundant data entries.

For example, if you want to change the name of one airport in the
`nycflights13` dataset, you will only need to update a single data
value. This principle is sometimes called the *single point of truth*.

``` r
#  Update in one single location...
airlines[airlines$carrier == "UA", "name"] <- "United broke my guitar"

airlines %>%
  filter(carrier == "UA")
#> # A tibble: 1 x 2
#>   carrier name                  
#>   <chr>   <chr>                 
#> 1 UA      United broke my guitar

# ...propagates to all related records
flights %>%
  left_join(airlines) %>%
  select(flight, name)
#> Joining, by = "carrier"
#> # A tibble: 336,776 x 2
#>    flight name                    
#>     <int> <chr>                   
#>  1   1545 United broke my guitar  
#>  2   1714 United broke my guitar  
#>  3   1141 American Airlines Inc.  
#>  4    725 JetBlue Airways         
#>  5    461 Delta Air Lines Inc.    
#>  6   1696 United broke my guitar  
#>  7    507 JetBlue Airways         
#>  8   5708 ExpressJet Airlines Inc.
#>  9     79 JetBlue Airways         
#> 10    301 American Airlines Inc.  
#> # … with 336,766 more rows
```

Another way to demonstrate normalization is splitting a table into two
parts.

Let’s look at the `planes` table, which consists of 3322 individual tail
numbers and corresponding information for the specific airplane, like
the year it was manufactured or the average cruising speed.

The function `decompose_table()` extracts two new tables and creates a
new key `model_id`, that links both tables.

This results in a `parent_table` and a `child_table` that differ
massively in the number of rows:

``` r
planes %>%
  decompose_table(model_id, model, manufacturer, type, engines, seats, manufacturer, speed)
#> $child_table
#> # A tibble: 3,322 x 4
#>    tailnum  year engine    model_id
#>    <chr>   <int> <chr>        <int>
#>  1 N10156   2004 Turbo-fan      120
#>  2 N102UW   1998 Turbo-fan       93
#>  3 N103US   1999 Turbo-fan       93
#>  4 N104UW   1999 Turbo-fan       93
#>  5 N10575   2002 Turbo-fan      119
#>  6 N105UW   1999 Turbo-fan       93
#>  7 N107US   1999 Turbo-fan       93
#>  8 N108UW   1999 Turbo-fan       93
#>  9 N109UW   1999 Turbo-fan       93
#> 10 N110UW   1999 Turbo-fan       93
#> # … with 3,312 more rows
#> 
#> $parent_table
#> # A tibble: 147 x 7
#>    model_id model      manufacturer   type              engines seats speed
#>       <int> <chr>      <chr>          <chr>               <int> <int> <int>
#>  1      120 EMB-145XR  EMBRAER        Fixed wing multi…       2    55    NA
#>  2       93 A320-214   AIRBUS INDUST… Fixed wing multi…       2   182    NA
#>  3      119 EMB-145LR  EMBRAER        Fixed wing multi…       2    55    NA
#>  4       39 737-824    BOEING         Fixed wing multi…       2   149    NA
#>  5       68 767-332    BOEING         Fixed wing multi…       2   330    NA
#>  6       52 757-224    BOEING         Fixed wing multi…       2   178    NA
#>  7       94 A320-214   AIRBUS         Fixed wing multi…       2   182    NA
#>  8      112 CL-600-2D… BOMBARDIER INC Fixed wing multi…       2    95    NA
#>  9       30 737-724    BOEING         Fixed wing multi…       2   149    NA
#> 10       27 737-524    BOEING         Fixed wing multi…       2   149    NA
#> # … with 137 more rows
```

While `child_table` contains 3322 unique `tailnum` rows and therefore
consists of 3322 rows, just like the original `planes` table, the
`parent_table` shrunk to just 147 rows, enough to store all relevant
combinations and avoid storing redundant information.

Further Reading: See the [Simple English Wikipedia article on database
normalization](https://simple.wikipedia.org/wiki/Database_normalisation)
for more details.

## 7. Relational Databases

{dm} is built upon relational data models but it is not a database
itself. Databases are systems for data management and many of them are
constructed as relational databases, e.g. SQLite, MySQL, MSSQL,
Postgres. As you can guess from the names of the databases, SQL, short
for Structured Querying Language, plays an important role: it was
invented for the purpose of querying relational databases.

In production, the data is stored in a relational database and {dm} is
used to work with the data.

Therefore, {dm} can copy data [from and to
databases](https://cynkra.github.io/dm/articles/dm.html#copy), and works
transparently with both in-memory data and with relational database
systems.

For example, let’s create a local SQLite database and copy the `dm`
object to it:

``` r
con_sqlite <- DBI::dbConnect(RSQLite::SQLite())
con_sqlite
#> <SQLiteConnection>
#>   Path: 
#>   Extensions: TRUE
DBI::dbListTables(con_sqlite)
#> character(0)

copy_dm_to(con_sqlite, dm)
DBI::dbListTables(con_sqlite)
#> [1] "airlines_2020_08_28_07_13_03_12345_1"
#> [2] "airports_2020_08_28_07_13_03_12345_1"
#> [3] "flights_2020_08_28_07_13_03_12345_1" 
#> [4] "planes_2020_08_28_07_13_03_12345_1"  
#> [5] "sqlite_stat1"                        
#> [6] "sqlite_stat4"                        
#> [7] "weather_2020_08_28_07_13_03_12345_1"
```

In the opposite direction, `dm` can also be populated with data from a
DB. Unfortunately, keys currently can be learned only for MSSQL and
Postgres, but not for SQLite. Therefore, the dm contains the tables but
not the keys:

``` r
dm_from_con(con_sqlite)
#> Keys could not be queried, use `learn_keys = FALSE` to mute this message.
#> ── Table source ───────────────────────────────────────────────────────────
#> src:  sqlite 3.35.5 []
#> ── Metadata ───────────────────────────────────────────────────────────────
#> Tables: `airlines_2020_08_28_07_13_03_12345_1`, `airports_2020_08_28_07_13_03_12345_1`, `flights_2020_08_28_07_13_03_12345_1`, `planes_2020_08_28_07_13_03_12345_1`, `sqlite_stat1`, … (7 total)
#> Columns: 62
#> Primary keys: 0
#> Foreign keys: 0
```

Remember to terminate the database connection:

``` r
DBI::dbDisconnect(con_sqlite)
```
