---
title: "Standardised Classification of Effects Based on Their Uncertainty"
author: Thierry Onkelinx
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Standardised Classification of Effects Based on Their Uncertainty}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
library(knitr)
opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
library(ggplot2)
```

## A Simple, Stable and Transparent Classification of Effects

The `effectclass` package provides functions to classify effects in a stable and transparent manner.
We base the classification on the lower and upper bounds of a confidence (or credible) interval, focussing on the uncertainty rather than the point estimates.
First, we compare the confidence interval with a reference.
This reference typically refers to the null hypothesis.
An effect with confidence interval above the reference is a significant (important) effect.
Let's call this a `positive effect`.
Likewise, a `negative effect` has a confidence interval below the reference.

```{r base-effect, fig.cap = "Comparing confidence intervals to the reference.", echo = FALSE}
ds <- data.frame(
  effect = factor(1:2, labels = c("positive\neffect", "negative\neffect")
  ),
  lcl = c(0.1, -0.5),
  ucl = c(0.5, -0.1)
)
ggplot(ds, aes(ymin = lcl, ymax = ucl, x = effect)) +
  geom_errorbar() +
  geom_hline(yintercept = 0, linetype = 2) +
  scale_y_continuous(
    breaks = 0,
    labels = "reference"
  ) +
  theme(
    axis.title = element_blank()
  )
```

But what if the confidence interval contains the reference?
Then the width of the confidence interval plays an important role.
Suppose the confidence interval is wide.
The value of the effect can range from a strong negative effect over no effect to a strong positive effect.
Basically, we have no information on the effect.
On the other hand, suppose we have a narrow confidence interval.
The effect still is either positive or negative, but we know that it won't be a strong positive nor strong negative effect.
We could define a lower^[below the reference] and upper^[above the reference] threshold and use them to separate confidence intervals that contain the reference.
If the confidence interval is between both the upper and lower threshold, we can state that the effect does not exceed these thresholds.
We set the thresholds at a level where we can consider the effect to be not relevant so we can call this `no effect`.
As soon as the confidence interval contains the reference and at least one of the thresholds, we consider it an `unknown effect`.

```{r base-effect2, fig.cap = "Comparing confidence intervals to the reference and thresholds.", echo = FALSE}
ds <- data.frame(
  effect = factor(1:2, labels = c("no\neffect", "unknown\neffect")
  ),
  lcl = c(-0.2, -0.5),
  ucl = c(0.2, 0.5)
)
ggplot(ds, aes(ymin = lcl, ymax = ucl, x = effect)) +
  geom_errorbar() +
  geom_hline(yintercept = c(0, -0.25, 0.25), linetype = c(2, 3, 3)) +
  scale_y_continuous(
    breaks = c(-0.25, 0, 0.25),
    labels = c("lower\nthreshold", "reference", "upper\nthreshold")
  ) +
  theme(
    axis.title = element_blank()
  )
```

Comparing the confidence interval with a reference and two thresholds, gives us four distinct categories.
The table below lists the rules for each category and suggests a symbol and standardised wording for effects and trends.

| Symbol | Effect          | Trend    | Rules                                  |
| :----: | --------------- | -------- | -------------------------------------- |
| `+`    | positive effect | increase | confidence interval above reference    |
| `-`    | negative effect | decrease | confidence interval below reference    |
| `~`    | no effect       | stable   | confidence interval between thresholds |
| `?`    | unknown effect  | unknown  | confidence interval contains reference and at least one threshold |

Table: Summary of the simple classification into four categories.

## A Detailed Classification

The upper threshold divides the zone of a `positive effect`.
Suppose the confidence interval is between the reference and the upper threshold.
The confidence interval is above the reference so there is a `positive effect`.
At the same time, the confidence interval is also below the upper threshold which would lead to `no effect` if the confidence interval contains the reference.
We suggest the term `moderate positive effect` for this situation.
A confidence interval above the upper threshold is outside of the `no effect` zone.
We suggest the term `strong positive effect` for this situation.
We keep the term `positive effect` when the confidence interval is above the reference and contains the upper threshold.

```{r detailed-effect, fig.cap = "Comparing confidence intervals to the reference and thresholds for positive effects.", echo = FALSE}
ds <- data.frame(
  effect = factor(
    1:3,
    labels = c(
      "positive\neffect", "moderate\npositive\neffect",
      "strong\npositive\neffect"
    )
  ),
  lcl = c(0.1, 0.1, 0.3),
  ucl = c(0.5, 0.2, 0.5)
)
ggplot(ds, aes(ymin = lcl, ymax = ucl, x = effect)) +
  geom_errorbar() +
  geom_hline(yintercept = c(0, -0.25, 0.25), linetype = c(2, 3, 3)) +
  scale_y_continuous(
    breaks = c(-0.25, 0, 0.25),
    labels = c("lower\nthreshold", "reference", "upper\nthreshold")
  ) +
  theme(
    axis.title = element_blank()
  )
```

We split the `negative effect` in a `strong negative effect`, `moderate negative effect` and `negative effect` using analogous rules as for the `positve effect`.
Except that we use the lower threshold instead of the upper threshold.
The `no effect` class remains as is.

When the `unknown effect` case contains the reference and upper threshold but not the lower threshold, we known that we can exclude a `strong negative effect`.
We classify such an effect as a `potential positive effect`.
Similarly, when the confidence interval contains the reference and the lower threshold but not the upper threshold, we call it a `potential negative effect`.
If the confidence interval contains both the lower and upper threshold, we call it an `unknown effect`.

```{r detailed-effect2, fig.cap = "Comparing confidence intervals to the reference and thresholds for unknown effects.", echo = FALSE}
ds <- data.frame(
  effect = factor(
    1:3,
    labels = c(
      "potential positive\neffect", "potential\nnegative\neffect",
      "unknown\neffect"
    )
  ),
  lcl = c(-0.1, -0.5, -0.5),
  ucl = c(0.5, 0.1, 0.5)
)
ggplot(ds, aes(ymin = lcl, ymax = ucl, x = effect)) +
  geom_errorbar() +
  geom_hline(yintercept = c(0, -0.25, 0.25), linetype = c(2, 3, 3)) +
  scale_y_continuous(
    breaks = c(-0.25, 0, 0.25),
    labels = c("lower\nthreshold", "reference", "upper\nthreshold")
  ) +
  theme(
    axis.title = element_blank()
  )
```

The table below gives an overview of the ten classes that we can distinct based on the reference, lower and upper threshold.
Note that four classes get the same symbol and terminology as in the four class classification.
Those classes have wide confidence intervals and we can't refine the classification.
The six other classes get a two digit symbol.
They inherit their first symbol from the four class classification.
The second digit refines the classification, using the same symbols as a kind of modifier.
The terminology of these six classes uses in principle their four class classification plus a modifying adjective (`strong` or `moderate`).
The `unknown` class is an exception.
Here we use the `potential` adjective in combination with `positive effect` or `negative effect`.

| Symbol | Effect          | Trend    | Rules                                  | Thresholds in the interval |
| :----: | --------------- | -------- | -------------------------------------- | :-: |
| `++`    | strong positive effect | strong increase | confidence interval above the upper threshold | 0 |
| `+`    | positive effect | increase | confidence interval above reference and contains the upper threshold | 1 |
| `+~`    | moderate positive effect | moderate increase | confidence interval between reference and the upper threshold | 0 |
| `~`    | no effect       | stable   | confidence interval between thresholds and contains reference | 1 |
| `-~`    | moderate negative effect | moderate decrease | confidence interval between reference and the lower threshold | 0 |
| `-`    | negative effect | decrease | confidence interval below reference and contains the lower threshold | 1 |
| `--`    | strong negative effect | strong decrease | confidence interval below the lower threshold | 0 |
| `?+`    | potential positive effect  | potential increase  | confidence interval contains reference and the upper threshold | 2 |
| `?-`    | potential negative effect  | potential decrease  | confidence interval contains reference and the lower threshold | 2 |
| `?`    | unknown effect  | unknown  | confidence interval contains the lower and upper threshold | 3 |

Table: Summary of the simple classification into ten categories.

```{r detailed-effect3, fig.cap = "Graphical overview of the ten class classification split by the number of thresholds in the confidence interval.", echo = FALSE}
library(effectclass, warn.conflicts = FALSE)
ds <- data.frame(
  thresholds =  c(3,  2,  1,  0,  2,  1,  0, 1, 0, 0),
  lcl =        c(-5, -5, -5, -5, -2, -2, -2, 1, 1, 3),
  ucl =         c(5,  2, -1, -3,  5,  2, -1, 5, 2, 5)
)
ds$effect <- classification(ds$lcl, ds$ucl, threshold = 2.5)
ds$coarse <- coarse_classification(ds$effect)
ds$thresholds <- sprintf("interval contains\n%i thresholds", ds$thresholds)
ggplot(ds, aes(ymin = lcl, ymax = ucl, x = effect)) +
  geom_errorbar() +
  geom_hline(yintercept = 0, linetype = 2) +
  geom_hline(yintercept = c(-2.5, 2.5), linetype = 3) +
  facet_wrap(~thresholds, scales = "free_x") +
  scale_y_continuous(
    breaks = c(-2.5, 0, 2.5),
    labels = c("lower\nthreshold", "reference", "upper\nthreshold")
  ) +
  theme(
    axis.title = element_blank()
  )
```

## Classification Using `effectclass`

The `effectclass` package provides `classification()` for classifying confidence intervals.
The function requires a vector of lower bounds of the confidence intervals (`ucl`) and a vector of upper bounds (`ucl`).
It also requires the lower and upper thresholds (`threshold`) as a two-element vector.
In case the thresholds are symmetric around the `reference`, you can provide a single value which is the absolute difference between the reference and both thresholds.
The default `reference` is zero.
`classification()` returns a ten level factor using the two digits symbols mentioned above.
The levels range from `strong positive effect` over `no effect` to `strong negative effect`.
The `potential` and `unknown` levels end the row.

```{r}
ds <- data.frame(
  lcl =        c(-5, -5, -5, -5, -2, -2, -2, 1, 1, 4),
  ucl =         c(5,  2, -1, -4,  5,  2, -1, 5, 2, 5)
)
library(effectclass)
ds$effect <- classification(lcl = ds$lcl, ucl = ds$ucl, threshold = 3)
ds <- ds[order(ds$effect), ]
kable(ds, row.names = FALSE)
```

`coarse_classification()` converts the ten classes into the corresponding four classes.
The factor has the four single digit levels in a similar order as the ten class factor.

```{r}
ds$coarse <- coarse_classification(ds$effect)
kable(ds, row.names = FALSE)
```

In some cases we are not interested in the sign of the effect.
`remove_sign()` recodes the ten classes into six and the four classes into three.
It replaces `+` and `-` by `*`.

```{r}
ds$unsigned_10 <- remove_sign(ds$effect)
ds$unsigned_4 <- remove_sign(ds$coarse)
kable(ds, row.names = FALSE)
```

## Suggestion for Displaying Effects in a Tabular Format

We recommend to publish at least the confidence interval and its classification in a table.
You could add the point estimate too, but never without the confidence interval.
Think about a relevant sorting order of the rows in the table.
The example below displays the effect in decreasing order with uncertain effects at the bottom.
Note that we first sort on the classification and then on the point estimate.
We added a `format()` function for your convenience.
A double hyphen `--` is rendered in Markdown as a so-called em-dash '--'.
Use the `type = "markdown"` argument of `format()` to convert the classification as verbatim text, which preserves the double hyphen when rendering the document.

```{r}
set.seed(20191126)
n <- 50
ds <- data.frame(
  effect = paste0(
    sample(LETTERS, n, replace = TRUE),
    sample(LETTERS, n, replace = TRUE),
    sample(LETTERS, n, replace = TRUE)
  ),
  point_estimate = runif(n, min = -2, max = 2),
  se = runif(n)
)
ds$lcl <- qnorm(p = 0.025, mean = ds$point_estimate, sd = ds$se)
ds$ucl <- qnorm(p = 0.975, mean = ds$point_estimate, sd = ds$se)
ds$estimate <- sprintf("%.2f (%.2f; %.2f)", ds$point_estimate, ds$lcl, ds$ucl)
ds$class <- classification(lcl = ds$lcl, ucl = ds$ucl, threshold = 1)
tab <- ds[order(ds$class, -ds$point_estimate), c("effect", "estimate", "class")]
tab$markdown <- format(tab$class, type = "markdown")
kable(
  tab,
  caption = "Example table with effects, their confidences interval and
  classification using different formatting styles.",
  row.names = FALSE,
  align = "c"
)
```