{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Posterior Predictive Checks\n",
    "\n",
    "Posterior predictive checks (PPCs) are a great way to validate a model. The idea is to generate data from the model using parameters from draws from the posterior. \n",
    "\n",
    "Elaborating slightly, one can say that PPCs analyze the degree to which data generated from the model deviate from data generated from the true distribution. So often you will want to know if, for example, your posterior distribution is approximating your underlying distribution. The visualization aspect of this model evaluation method is also great for a 'sense check' or explaining your model to others and getting criticism. \n",
    "\n",
    "Here we will implement a general routine to draw samples from the observed nodes of a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running with PyMC3 version v.3.3\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from collections import defaultdict\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pymc3 as pm\n",
    "import scipy.stats as st\n",
    "\n",
    "plt.style.use('seaborn-darkgrid')\n",
    "print('Running with PyMC3 version v.{}'.format(pm.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets generate a very simple model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Auto-assigning NUTS sampler...\n",
      "Initializing NUTS using jitter+adapt_diag...\n",
      "Multiprocess sampling (2 chains in 2 jobs)\n",
      "NUTS: [sd_log__, mu]\n",
      "100%|██████████| 5500/5500 [00:04<00:00, 1189.84it/s]\n"
     ]
    }
   ],
   "source": [
    "data = np.random.randn(100)\n",
    "\n",
    "with pm.Model() as model: \n",
    "    mu = pm.Normal('mu', mu=0, sd=1, testval=0)\n",
    "    sd = pm.HalfNormal('sd', sd=1)\n",
    "    n = pm.Normal('n', mu=mu, sd=sd, observed=data)\n",
    "    \n",
    "    trace = pm.sample(5000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 79228 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105cb3630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pm.traceplot(trace);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function will randomly draw 500 samples of parameters from the trace. Then, for each sample, it will draw 100 random numbers from a normal distribution specified by the values of `mu` and `std` in that sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 500/500 [00:00<00:00, 2120.72it/s]\n"
     ]
    }
   ],
   "source": [
    "ppc = pm.sample_ppc(trace, samples=500, model=model, size=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, `ppc` contains 500 generated data sets (containing 100 samples each), each using a different parameter setting from the posterior:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(500, 100)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.asarray(ppc['n']).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One common way to visualize is to look if the model can reproduce the patterns observed in the real data. For example, how close are the inferred means to the actual sample mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 13776 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1c1020eb70>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(12, 6))\n",
    "ax.hist([n.mean() for n in ppc['n']], bins=19, alpha=0.5)\n",
    "ax.axvline(data.mean())\n",
    "ax.set(title='Posterior predictive of the mean', xlabel='mean(x)', ylabel='Frequency');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison between PPC and other model evaluation methods. \n",
    "\n",
    "An excellent introduction to this was given in the [Edward](http://edwardlib.org) documentation:\n",
    "\n",
    "> PPCs are an excellent tool for revising models, simplifying or expanding the current model as one examines how well it fits the data. They are inspired by prior checks and classical hypothesis testing, under the philosophy that models should be criticized under the frequentist perspective of large sample assessment.\n",
    "\n",
    "> PPCs can also be applied to tasks such as hypothesis testing, model comparison, model selection, and model averaging. It’s important to note that while they can be applied as a form of Bayesian hypothesis testing, hypothesis testing is generally not recommended: binary decision making from a single test is not as common a use case as one might believe. We recommend performing many PPCs to get a holistic understanding of the model fit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "\n",
    "The same pattern can be used for prediction. Here we're building a logistic regression model. Note that since we are dealing the full posterior, we are also getting uncertainty in our predictions for free."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use a theano shared variable to be able to exchange the data the model runs on\n",
    "from theano import shared"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def invlogit(x):\n",
    "    return np.exp(x) / (1 + np.exp(x))\n",
    "\n",
    "n = 4000\n",
    "coeff = 1.\n",
    "\n",
    "predictors = np.random.normal(size=n)\n",
    "# Turn predictor into a shared var so that we can change it later\n",
    "predictors_shared = shared(predictors)\n",
    "\n",
    "outcomes = np.random.binomial(1, invlogit(coeff * predictors))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 0, ..., 0, 0, 1])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outcomes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictors_out_of_sample = np.random.normal(size=50)\n",
    "outcomes_out_of_sample = np.random.binomial(1, invlogit(coeff * predictors_out_of_sample))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Auto-assigning NUTS sampler...\n",
      "Initializing NUTS using jitter+adapt_diag...\n",
      "Multiprocess sampling (2 chains in 2 jobs)\n",
      "NUTS: [coeff]\n",
      "100%|██████████| 5500/5500 [00:05<00:00, 1047.23it/s]\n",
      "The acceptance probability does not match the target. It is 0.8893164887134138, but should be close to 0.8. Try to increase the number of tuning steps.\n"
     ]
    }
   ],
   "source": [
    "with pm.Model() as model:\n",
    "    coeff = pm.Normal('coeff', mu=0, sd=1)\n",
    "    outcome = pm.Bernoulli('outcome', logit_p=coeff*predictors_shared, observed=outcomes)    \n",
    "    trace = pm.sample(5000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Changing values here will also change values in the model\n",
    "predictors_shared.set_value(predictors_out_of_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:00<00:00, 1616.89it/s]\n"
     ]
    }
   ],
   "source": [
    "# Simply running PPC will use the updated values and do prediction\n",
    "ppc = pm.sample_ppc(trace, model=model, samples=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mean predicted values plus error bars to give sense of uncertainty in prediction\n",
    "\n",
    "Note that these are uncertainty caused by MCMC estimation, and not from the underlying model.  If we had taken more samples in `sample_ppc`, the error bars would get smaller quite quickly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29348 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1c0ca4c358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "β = st.beta((ppc['outcome'] == 1).sum(axis=0), (ppc['outcome'] == 0).sum(axis=0))\n",
    "\n",
    "# estimated probability\n",
    "ax.scatter(x=predictors_out_of_sample, y=β.mean())\n",
    "\n",
    "# error bars on the estimate\n",
    "plt.vlines(predictors_out_of_sample, *β.interval(0.95))\n",
    "\n",
    "# actual outcomes\n",
    "ax.scatter(x=predictors_out_of_sample, \n",
    "           y=outcomes_out_of_sample, marker='x')\n",
    "\n",
    "# True probabilities\n",
    "x = np.linspace(predictors_out_of_sample.min(), predictors_out_of_sample.max())\n",
    "ax.plot(x, invlogit(x), linestyle='-')\n",
    "\n",
    "\n",
    "ax.set_xlabel('predictor')\n",
    "ax.set_ylabel('outcome');"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "pymc33.6",
   "language": "python",
   "name": "pymc33_6"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
