{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use(['seaborn-darkgrid'])\n",
    "import pymc3 as pm\n",
    "import numpy as np\n",
    "import  pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model averaging\n",
    "\n",
    "When confronted with more than one model we have several options. One of them is to perform model selection, using for example a given Information Criterion as exemplified [in this notebook](model_comparsion.ipynb) and this other [example](model_comparsion.ipynb). Model selection is appealing for its simplicity, but we are discarding information about the uncertainty in our models. This is somehow similar to computing the full posterior and then just keep a point-estimate like the posterior mean; we may become overconfident of what we really know.\n",
    "\n",
    "One alternative is to perform model selection but discuss all the different models together with the computed values of a given Information Criterion. It is important to put all these numbers and tests in the context of our problem so that we and our audience can have a better feeling of the possible limitations and shortcomings of our methods. If you are in the academic world you can use this approach to add elements to the discussion section of a paper, presentation, thesis, and so on.\n",
    "\n",
    "Yet another approach is to perform model averaging. The idea now is to generate a meta-model (and meta-predictions) using a weighted average of the models. There are several ways to do this and PyMC3 includes 3 of them that we are going to briefly discuss, you will find a more thorough explanation in the work by [Yuling Yao et. al.](https://arxiv.org/abs/1704.02030)\n",
    "\n",
    "## Pseudo Bayesian model averaging\n",
    "\n",
    "Bayesian models can be weighted by their marginal likelihood, this is known as Bayesian Model Averaging. While this is theoretically appealing, is problematic in practice: on the one hand the marginal likelihood is highly sensible to the specification of the prior, in a way that parameter estimation is not, and on the other computing the marginal likelihood is usually a challenging task. An alternative route is to use the values of WAIC (Widely Applicable Information Criterion) or LOO (pareto-smoothed importance sampling Leave-One-Out cross-validation), which we will call generically IC, to estimate weights. We can do this by using the following formula:\n",
    "\n",
    "$$w_i = \\frac {e^{ \\frac{1}{2} dIC_i }} {\\sum_j^M e^{ - \\frac{1}{2} dIC_j }}$$\n",
    "\n",
    "Where $dIC_i$ is the difference between the i-esim information criterion value and the lowest one. Remember that the lowest the value of the IC, the better. We can use any information criterion we want to compute a set of weights, but, of course, we cannot mix them. \n",
    "\n",
    "This approach is called pseudo Bayesian model averaging, or Akaike-like weighting and is an heuristic way to compute the relative probability of each model (given a fixed set of models) from the information criteria values. Look how the denominator is just a normalization term to ensure that the weights sum up to one.\n",
    "\n",
    "## Pseudo Bayesian model averaging with Bayesian Bootstrapping\n",
    "\n",
    "The above formula for computing weights is a very nice and simple approach, but with one major caveat it does not take into account the uncertainty in the computation of the IC. We could compute the standard error of the IC (assuming a Gaussian approximation) and modify the above formula accordingly. Or we can do something more robust, like using a [Bayesian Bootstrapping](http://www.sumsar.net/blog/2015/04/the-non-parametric-bootstrap-as-a-bayesian-model/) to estimate, and incorporate this uncertainty.\n",
    "\n",
    "## Stacking\n",
    "\n",
    "The third approach implemented in PyMC3 is know as _stacking of predictive distributions_ and it has been recently [proposed](https://arxiv.org/abs/1704.02030). We want to combine several models in a metamodel in order to minimize the diverge between the meta-model and the _true_ generating model, when using a logarithmic scoring rule this is equivalently to:\n",
    "\n",
    "$$\\max_{n} \\frac{1}{n} \\sum_{i=1}^{n}log\\sum_{k=1}^{K} w_k p(y_i|y_{-i}, M_k)$$\n",
    "\n",
    "Where $n$ is the number of data points and $K$ the number of models. To enforce a solution we constrain $w$ to be $w_k \\ge 0$ and  $\\sum_{k=1}^{K} w_k = 1$. \n",
    "\n",
    "The quantity $p(y_i|y_{-i}, M_k)$ is the leave-one-out predictive distribution for the $M_k$ model. Computing it requires fitting each model $n$ times, each time leaving out one data point. Fortunately we can approximate the exact leave-one-out predictive distribution using LOO (or even WAIC), and that is what we do in practice.\n",
    "\n",
    "## Weighted posterior predictive samples\n",
    "\n",
    "Once we have computed the weights, using any of the above 3 methods,  we can use them to get a weighted posterior predictive samples. PyMC3 offers functions to perform these steps in a simple way, so let see them in action using an example.\n",
    "\n",
    "The following example is taken from the superb book [Statistical Rethinking](http://xcelab.net/rm/statistical-rethinking/) by Richard McElreath. You will find more PyMC3 examples from this book in this [repository](https://github.com/aloctavodia/Statistical-Rethinking-with-Python-and-PyMC3). We are going to explore a simplified version of it. Check the book for the whole example and a more thorough discussion of both, the biological motivation for this problem and a theoretical/practical discussion of using Information Criteria to compare, select and average models.\n",
    "\n",
    "Briefly, our problem is as follows: We want to explore the composition of milk across several primate species, it is hypothesized that females from species of primates with larger brains produce more _nutritious_ milk (loosely speaking this is done _in order to_ support the development of such big brains). This is an important question for evolutionary biologists and try to give and answer we will use 3 variables, two predictor variables: the proportion of neocortex compare to the total mass of the brain and the logarithm of the body mass of the mothers. And for predicted variable, the kilocalories per gram of milk. With these variables we are going to build 3 different linear models:\n",
    " \n",
    "1. A model using only the neocortex variable\n",
    "2. A model using only the logarithm of the mass variable\n",
    "3. A model using both variables\n",
    "\n",
    "Let start by uploading the data and centering the `neocortex` and `log mass` variables, for better sampling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>kcal.per.g</th>\n",
       "      <th>neocortex</th>\n",
       "      <th>log_mass</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.49</td>\n",
       "      <td>-0.123706</td>\n",
       "      <td>-0.831353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.47</td>\n",
       "      <td>-0.030706</td>\n",
       "      <td>0.158647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.56</td>\n",
       "      <td>-0.030706</td>\n",
       "      <td>0.181647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.89</td>\n",
       "      <td>0.000294</td>\n",
       "      <td>-0.579353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.92</td>\n",
       "      <td>0.012294</td>\n",
       "      <td>-1.885353</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   kcal.per.g  neocortex  log_mass\n",
       "0        0.49  -0.123706 -0.831353\n",
       "1        0.47  -0.030706  0.158647\n",
       "2        0.56  -0.030706  0.181647\n",
       "3        0.89   0.000294 -0.579353\n",
       "4        0.92   0.012294 -1.885353"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = pd.read_csv('../data/milk.csv')\n",
    "d.iloc[:,1:] = d.iloc[:,1:] - d.iloc[:,1:].mean()\n",
    "d.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the data we are going to build our first model using only the `neocortex`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Auto-assigning NUTS sampler...\n",
      "Initializing NUTS using jitter+adapt_diag...\n",
      "Multiprocess sampling (2 chains in 2 jobs)\n",
      "NUTS: [sigma_log__, beta, alpha]\n",
      "100%|██████████| 2500/2500 [00:03<00:00, 735.00it/s] \n"
     ]
    }
   ],
   "source": [
    "with pm.Model() as model_0:\n",
    "    alpha = pm.Normal('alpha', mu=0, sd=10)\n",
    "    beta = pm.Normal('beta', mu=0, sd=10)\n",
    "    sigma = pm.HalfNormal('sigma', 10)\n",
    "    \n",
    "    mu = alpha + beta * d['neocortex']\n",
    "    \n",
    "    kcal = pm.Normal('kcal', mu=mu, sd=sigma, observed=d['kcal.per.g'])\n",
    "    trace_0 = pm.sample(2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second model is exactly the same as the first one, except we now use the logarithm of the mass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Auto-assigning NUTS sampler...\n",
      "Initializing NUTS using jitter+adapt_diag...\n",
      "Multiprocess sampling (2 chains in 2 jobs)\n",
      "NUTS: [sigma_log__, beta, alpha]\n",
      " 97%|█████████▋| 2429/2500 [00:03<00:00, 745.05it/s]/Users/fonnescj/Repos/pymc3/pymc3/step_methods/hmc/nuts.py:452: UserWarning: The acceptance probability in chain 0 does not match the target. It is 0.880482254108, but should be close to 0.8. Try to increase the number of tuning steps.\n",
      "  % (self._chain_id, mean_accept, target_accept))\n",
      "100%|██████████| 2500/2500 [00:03<00:00, 719.76it/s]\n"
     ]
    }
   ],
   "source": [
    "with pm.Model() as model_1:\n",
    "    alpha = pm.Normal('alpha', mu=0, sd=10)\n",
    "    beta = pm.Normal('beta', mu=0, sd=1)\n",
    "    sigma = pm.HalfNormal('sigma', 10)\n",
    "    \n",
    "    mu = alpha + beta * d['log_mass']\n",
    "    \n",
    "    kcal = pm.Normal('kcal', mu=mu, sd=sigma, observed=d['kcal.per.g'])\n",
    "    \n",
    "    trace_1 = pm.sample(2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally the third model using the `neocortex` and `log_mass` variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Auto-assigning NUTS sampler...\n",
      "Initializing NUTS using jitter+adapt_diag...\n",
      "Multiprocess sampling (2 chains in 2 jobs)\n",
      "NUTS: [sigma_log__, beta, alpha]\n",
      " 97%|█████████▋| 2429/2500 [00:03<00:00, 741.00it/s]/Users/fonnescj/Repos/pymc3/pymc3/step_methods/hmc/nuts.py:452: UserWarning: The acceptance probability in chain 0 does not match the target. It is 0.890938172071, but should be close to 0.8. Try to increase the number of tuning steps.\n",
      "  % (self._chain_id, mean_accept, target_accept))\n",
      "100%|██████████| 2500/2500 [00:03<00:00, 690.59it/s]\n"
     ]
    }
   ],
   "source": [
    "with pm.Model() as model_2:\n",
    "    alpha = pm.Normal('alpha', mu=0, sd=10)\n",
    "    beta = pm.Normal('beta', mu=0, sd=1, shape=2)\n",
    "    sigma = pm.HalfNormal('sigma', 10)\n",
    "\n",
    "    mu = alpha + pm.math.dot(beta, d[['neocortex','log_mass']].T)\n",
    "\n",
    "    kcal = pm.Normal('kcal', mu=mu, sd=sigma, observed=d['kcal.per.g'])\n",
    "\n",
    "    trace_2 = pm.sample(2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have sampled the posterior for the 3 models, we are going to compare them visually. One option is to use the `forestplot` function that supports plotting more than one trace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 17008 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10e06d128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "traces = [trace_0, trace_1, trace_2]\n",
    "pm.forestplot(traces);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another option is to plot several traces in a same plot is to use `densityplot`. This plot is somehow similar to a forestplot, but we get truncated KDE plots (by default 95% credible intervals) grouped by variable names together with a point estimate (by default the mean)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 46316 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11ccf0cc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pm.densityplot(traces);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have sampled the posterior for the 3 models, we are going to use WAIC (Widely applicable information criterion) to compare the 3 models. We can do this using the `compare` function included with PyMC3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>WAIC</th>\n",
       "      <th>pWAIC</th>\n",
       "      <th>dWAIC</th>\n",
       "      <th>weight</th>\n",
       "      <th>SE</th>\n",
       "      <th>dSE</th>\n",
       "      <th>warning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-15.45</td>\n",
       "      <td>2.64</td>\n",
       "      <td>0</td>\n",
       "      <td>0.89</td>\n",
       "      <td>4.71</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-8.97</td>\n",
       "      <td>2.01</td>\n",
       "      <td>6.48</td>\n",
       "      <td>0.05</td>\n",
       "      <td>3.81</td>\n",
       "      <td>2.16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-7.17</td>\n",
       "      <td>1.93</td>\n",
       "      <td>8.28</td>\n",
       "      <td>0.06</td>\n",
       "      <td>3.05</td>\n",
       "      <td>4.21</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    WAIC pWAIC dWAIC weight    SE   dSE warning\n",
       "2 -15.45  2.64     0   0.89  4.71     0       0\n",
       "1  -8.97  2.01  6.48   0.05  3.81  2.16       0\n",
       "0  -7.17  1.93  8.28   0.06  3.05  4.21       1"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_dict = dict(zip([model_0, model_1, model_2], traces))\n",
    "comp = pm.compare(model_dict, method='BB-pseudo-BMA')\n",
    "comp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the best model is `model_2`, the one with both predictor variables. Notice the DataFrame is ordered from lowest to highest WAIC (_i.e_ from _better_ to _worst_ model). Check [this notebook](model_comparison.ipynb) for a more detailed discussing on model comparison.\n",
    "\n",
    "We can also see that we get a column with the relative `weight` for each model (according to the first equation at the beginning of this notebook). This weights can be _vaguely_ interpreted as the probability that each model will make the correct predictions on future data. Of course this interpretation is conditional on the models used to compute the weights, if we add or remove models the weights will change. And also is dependent on the assumptions behind WAIC (or any other Information Criterion used). So try to do not overinterpret these `weights`. \n",
    "\n",
    "Now we are going to use copmuted `weights` to generate predictions based not on a single model but on the weighted set of models. This is one way to perform model averaging. Using PyMC3 we can call the `sample_ppc_w` function as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ppc_w = pm.sample_ppc_w(traces, 1000, models,\n",
    "                        weights=comp.weight.sort_index(ascending=True),\n",
    "                        progressbar=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we are passing the weights ordered by their index. We are doing this because we pass `traces` and `models` ordered from model 0 to 2, but the computed weights are ordered from lowest to highest WAIC (or equivalently from larger to lowest weight). In summary, we must be sure that we are correctly pairing the weights and models.\n",
    "\n",
    "We are also going to compute PPCs for the lowest-WAIC model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ppc_2 = pm.sample_ppc(trace_2, 1000, model_2,\n",
    "                     progressbar=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple way to compare both kind of predictions is to plot their mean and hpd interval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAEBCAYAAADbxHY7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFxdJREFUeJzt3X1Y1fXBx/HP4fAQCoIWmZqCoqym\nhlL5cO9CXVpqZoUP4G2ealmZ5j1bOkNRM2eWmrPNZpPygcvMZTZSNytTM101p4GWpmND5FYsSgTl\nyYNHfvcfznOPwAMifI9nvV9/5fl5fufjuejNuQ7ww2ZZliUAgBF+3h4AAD8kRBcADCK6AGAQ0QUA\ng4guABhEdAHAIH9PB7/7rrjWE4SEBKmkxNlggxobexsXexsXextXQ+2NiAi95LErfqXr72+/0lMY\nxd7Gxd7Gxd7GZWIvby8AgEFEFwAMIroAYBDRBQCDiC4AGER0AcAgogsABhFdwAe9dyhfQ1N3q8ei\nnRqaulvvHcr39iQ5nU6NGDH0ksczMvbq2WenVbv9d7/7jcaN+5keffRBbdyY3pgTrwoefyINwNXn\nvUP5mrflHzrrqpQkfVPs1Lwt/5AkDb65pTenXbaMjL06fvyYli1bqYqKCjkcierXr7+aNWvm7WmN\nhugCPmLcW/vl7++nzGNFOne+6i98Oeuq1K8+yNK7X3yjZUmxl33uzZs36ZNPdsrpdKqg4KRGjvxv\n7dr1sXJysvXkk5MUH99PW7a8p3Xr1iogIEBt27bT1Kkpqqio0Jw5M1RcXKw2bW50ny87+596+eWF\nstv91LRpiKZNe7bGx+3cuas6doyRJNlsNlVWVsrf/z87S//Z/zrgP9D3g1vb7XVVVlamxYt/p61b\nP9Bbb72p1NRVysz8XG+/vVa33NJNy5cv08qVa9SkSVP99reLtGHDO5Kk9u2jNW7ckzp48IAyMvZK\nkubPn6tp02ape/cueuONN7VmTZpuv71ntccMCgpSUFCQXC6X5s59Vvfem6AmTZpc0b/jakd0AR+x\nLClW4eFNFL/gI31TXP2iLDeEBtXrVe5FnTr9SJIUEhKqqKj2stlsCg0NldNZoRMn8tS+fQc1adJU\nkhQbG6c9e/4qSerZs7ckqXPnLu5Xqbm5OVq06EX5+9vldDrVtm3kJR/3zJkzmjnzGXXvfqscjp/V\ne7+vILqAj5kQH1XlPV1JusbfTxPio67ovDab7ZLHWrVqo6NHc1ReXq7g4GDt25ehtm3byWbz04ED\nXyo+vp+ysg7L5XJJktq1i9SMGXN0000dtHPnpyooOFnjeZ3Os3rqqfEaNWqM7rpr8BXt9xVEF/Ax\nF79YtnTXUeUXO9UyNEgT4qMa9Yto4eHheuSRcfr5z8fJZvPTjTe21RNPTJTdbtcLLzyn8ePHKjIy\nSgEBAZKkyZOnae7cWfLzk1yuSiUnz9TJk99VO++7776jEyfytHFjuvs7F6ZPf1atW7dptH+Lt9k8\n/Qr2ulxPNzy8iYqKyhp0VGNib+Nib+Nib+NqqL2Nej1dAEDdEV0AMIjoAoBBRBcADCK6AGAQ0QUA\ng4gu4IOCstLVIq2nrvtdW7VI66mgLPNX51q9epW++urAJY9PnPi4jhw5Uu32d955q86P8eqrS7R5\n86Z67fMkN/eoJk58/JLHN2/epFdfXdLgjysRXcDnBGWlK/SjqbKX5MkmS/aSPIV+NNV4eB2Oh/Xj\nH3e57Pulpa1ohDW+g59IA3xEWPoI2f3tCs3bI9v5iirHbK5yhW6frGsOrtHphPWXfe5HHnlAixYt\nUWhoM919d3+98soyxcTcpEceeUC///1Kbdz4R3344Qey2Wzq3/8ujRw5Ss8/P1v9+9+l7t3j9Ktf\nPauCgu90/fUttW9fpjZseF+S9OqrS5Wf/63Ky8s1e/bz2rr1A505c1ovvfSinnpqihYunKfjx4+p\nsrJSjz02XnFxt2nHjm1KS1uu8PDmOnfunCIjo6psXb58mfLyjquoqEjFxaeVkDBSO3Zs17FjuUpJ\neU5dunTV2rVvaNu2LbLb7YqN7a4JE36ukydPas6cGbIsSy1aXOs+X2bm50pNXSq73a6oqEg99dQz\n7mNOp1OzZiWrtLRUTudZjR//c8XF3XbZz++/I7qAr/lecGu9vQ7i4/tp9+7PdP31LdWqVWvt2bNb\nAQGBatu2nfLyjmvbtg+1dOnrstlseuqpCerZs5f7vhs2pKt169aaO3e+cnOPyuFIdB/r06eP4uMH\naPnyZdqxY5seemis3nlnnaZMSVZ6+nqFhYVr2rRZOn26SE8++bjeeGOdli79rV57LU3NmoXpl7+c\nVOPeoKAg/frXS7R69Sp99tknWrBgsf78543atm2LgoODtX37h/r971fIbrcrJWWqPvlklzIzP9eA\nAQN1770J2rZti9LT18uyLM2f/7xeffV1NW/eQqtXv67Nmze5L9yTl3dcp04V6OWXl6qwsFDHjuXW\n+zm+iOgCPuJ0wnqFhzeR32+6yl6SV+14ZUiber3KlaS+fX+qtLQVatnyBj3++AStX/8HVVZa6tu3\nv44cyVZ+/jeaNGm8JKm4uFjHjx933zc3N0c9e/6XJCkyMkrh4c3dx378486SpGuvvVYFBQVVHjM7\n+5/64otM9/vC58+7dOpUgZo2baqwsHBJUpcut9S4NybmJklSaGiIoqLa/+u/m6miwqnc3KPq3Lmr\nO5yxsd2Uk5OtnJwjGjjwbklS166xSk9fr6KiQhUUnNTMmcn/2nBOcXE93NcG7tAhWsOGJWr27BS5\nXC6NGDGqXs/vv+M9XcDHlPZOluUfXOU2yz9Ypb2T633ODh066uuvT+jQoYPq3fsnKi8v11/+8rF6\n9fovtWsXqaioDlqyZJleeSVVd999jzp06Phv943WgQNfSLrwyvD06SL3sZquXHbxci+RkVEaMGCg\nXnklVYsW/VY//ekAhYY2U0lJqQoLCyVJhw9/VeNeDxdEU2RklL766oBcLpcsy9K+fZlq2zZSkZGR\nOnjwws5Dhy6cNywsXNdff71efPHXeuWVVD3++Lgqbx9kZ/9TZWWlWrjwN0pJeU4vv7ywLk+nR7zS\nBXyMMyZBktT0sxflV3JClSGtVdo72X17fXXrFqevvz4hPz8/desWp6NHj6hJkybq1ClGt912uyZM\nGKuKinO6+ebOioiIcN/vnnvu0/PPP6cnn3xMN9xwgwIDAz0+TlRUe82ZM1PJyTM1f/5cTZz4uEpL\nS5SQMFIBAQGaPn2WJk+eqNDQsHr9Fono6I66444BGj9+rCzL0i23xKpPn366/faeevbZadq6dYv7\nKmZ+fn6aNGmKfvnLSbIsS2FhzZScPEv5+d9Ikm68sa1WrkzV++//Wf7+ARo7dtxl7/k+rjJ2lWNv\n42Lvlfvyy/0qLy9Xjx69dOzY/2ry5P/RunUbJF2dez0xcZUxXukCuCKtW7fR7NkpWrkyVS6XS08/\n/Uztd/oBI7oArsi1116nJUuWeXuGz+ALaQBgENEFAIOILgAYRHQBwCCiCwAGEV0AMIjoAoBBRBcA\nDCK6AGAQ0QUAg4guABhEdAHAIKILAAYRXQAwiOgCgEFEFwAMIroAYBDRBQCDiC4AGER0AcAgogsA\nBhFdADCI6AKAQUQXAAwiugBgENEFAIOILgAYRHQBwCCiCwAGEV0AMIjoAoBBRBcADCK6AGAQ0QUA\ng4guABhEdAHAIH9vDwDq6r1D+Vq666jyi51qGRqkCfFRGnxzS2/PAi4L0YVPeO9QvuZt+YfOuiol\nSd8UOzVvyz8kifDCp9gsy7IudfC774prPUF4eBMVFZXVe8C4t/bX+7714e/vJ9e//sf1Bey94Muv\nz+jc+eofqgF2m7q2albv8/L8Nq6rZe+ypNg6/b0r7dlFERGhlzzGe7rwCTUF19PtwNXK628v1PUz\nUENpqM9kprD3gqGpu/VNsbPa7TeEBl3RxxDPb+Pytb0m8EoXPmFCfJSu8a/64XqNv58mxEd5ZxBQ\nT15/pQvUxcUvlvHdC/B1RBc+Y/DNLYksfB5vLwCAQUQXAAwiugBgENEFAIOILgAYRHQBwCCiCwAG\nEV0AMIjoAoBBRBcADCK6AGAQ0QUAg4guABhEdAHAIKILAAYRXQAwiOgCgEFEFwAMIroAYBDRBQCD\niC4AGER0AcAgogsABhFdADCI6AKAQUQXAAwiugBgENEFAIOILgAYRHQBwCCiCwAGEV0AMIjoAoBB\nRBcADCK6AGAQ0QUAg4guABhEdAHAIKILAAYRXQAwiOgCgEFEFwAMIroAYBDRBQCDiC4AGER0AcAg\nogsABhFdADCI6AKAQUQXAAwiugBgENEFAIOILgAYRHQBwCCiCwAGEV0AMIjoAoBBRBcADCK6AGAQ\n0QUAg4guABhEdAHAIKILAAYRXQAwiOgCgEH+3h4AXK6grHQ1/exF+ZWcUGVIa5X2TpYzJsHbs4A6\nIbrwKUFZ6Qr9aKpsrnJJkr0kT6EfTZUkwgufYLMsy7rUwe++K671BOHhTVRUVFbt9rD0EVe2rJH4\n+9vlcp339ow6Y29VAfkZsp2vqHa7ZQ/UuZZxl30+nt/G5ct7Tyesr/d5IiJCL3mM93ThW2oIrsfb\ngatMo729cCWfJRpTeHgTna7hlfnVir1VtUjrKXtJXrXbK0Pa1Otjjue3cbG3Ol7pwqeU9k6W5R9c\n5TbLP1ilvZO9tAi4PHwhDT7l4hfL+O4F+CqiC5/jjEkgsvBZvL0AAAYRXQAwiOgCgEFEFwAMIroA\nYBDRBQCDiC4AGER0AcAgogsABhFdADCI6AKAQUQXAAwiugBgENEFAIOILgAYRHQBwCCiCwAGEV0A\nMIjoAoBBRBcADCK6AGAQ0QUAg4guABhEdAHAIKILAAYRXQAwiOgCgEFEFwAMIroAYBDRBQCDiC4A\nGER0AcAgogsABhFdADCI6AKAQUQXAAwiugBgENEFAIOILgAYRHQBwCCiCwAGEV0AMIjoAoBBRBcA\nDCK6AGAQ0QUAg4guABhEdAHAIKILAAYRXQAwiOgCgEFEFwAMIroAYBDRBQCDiC4AGER0AcAgogsA\nBhFdADCI6AKAQUQXAAwiugBgENEFAIOILgAYRHQBwCCiCwAGEV0AMIjoAoBBNsuyLG+PAIAfCl7p\nAoBBRBcADCK6AGBQnaJbWVmpWbNmKSkpSQ6HQ7m5uVWOr1mzRsOHD9eIESP00UcfNcrQy1Hb3ot/\n59FHH9XatWu9sLC62jbPnTtXw4YNk8PhkMPhUHFxsZeWXlDb3o8//liJiYlKTEzU7Nmz5e0vHXja\ne+jQIffz6nA41LVrV+3cudOLa2t/fpcvX65hw4Zp+PDh+vDDD7208v/Vtjc1NVX33XefHnjggaui\nERft379fDoej2u3bt2/X8OHDlZSUpHXr1jXsg1p18MEHH1jPPPOMZVmWlZmZaT3xxBPuYwUFBdbd\nd99tVVRUWMXFxVafPn2sysrKupy20Xjae9GiRYusESNGWG+++abpeTWqbfOoUaOsgoICb0yrkae9\nxcXF1pAhQ9x7U1NTvb69Lh8TlmVZmzdvtp5++mmT02rkae/p06etvn37Wk6n0yoqKrL69evnrZlu\nnvYePnzYGjp0qHX27Fnr7Nmz1v3332+VlZV5a6pbamqqdc8991gjR46scntFRYU1YMAAq6ioyHI6\nndawYcOsb7/9tsEet06vdD///HPFx8dLkrp166YDBw64j7Vo0UIbNmxQQECATp48qWbNmslmszXs\nZ4bL5GmvJL3//vuy2Wzq06ePN+bVyNPmyspK5ebmatasWRo1apTWr1/vrZlunvZmZmYqJiZG8+fP\n1+jRo3XdddepRYsW3poqqfaPCUkqKyvTkiVLlJKSYnpeNZ72BgcHq3Xr1iovL1d5ebnX/3+TPO/N\nzs5Wjx49FBQUpKCgIEVGRurvf/+7t6a6tWvXTkuWLKl2e3Z2ttq1a6ewsDAFBgbq1ltv1d69exvs\ncesU3ZKSEoWEhLj/bLfb5XK53H/29/fXG2+8oaSkJA0cOLDBxtWXp71ZWVn605/+pEmTJnlrXo08\nbS4rK9OYMWO0cOFCvf7663rzzTd1+PBhb02V5HlvYWGhdu/erSlTpui1115TWlqacnJyvDVVUu0f\nw5K0fv16DRo0yOufIKTa97Zq1UpDhgxRQkKCHnzwQW9MrMLT3h/96Efau3evSkpKVFhYqMzMTJWX\nl3trqtvAgQPl7+9f7faSkhKFhoa6/9y0aVOVlJQ02OPWKbohISEqLS11/7mysrLa2DFjxmjXrl3a\ns2eP/vrXvzbYwPrwtPfdd99Vfn6+HnroIaWnp2vVqlVef/9O8rw5ODhYDz74oIKDgxUSEqJevXp5\nPbqe9oaHh6tr166KiIhQ06ZNddttt+nQoUPemiqpbh/DmzZt0siRI01Pq5GnvTt37tS3336rbdu2\naceOHdq6dau++OILb02V5HlvdHS0HnjgAT322GOaP3++YmNj1bx5c29NrdX3/y2lpaVVInyl6hTd\nuLg4d5j27dunmJgY97EjR45o4sSJsixLAQEBCgwMlJ+fd78pwtPeqVOn6u2339bq1auVkJCghx9+\n+Kp4m8HT5qNHj2r06NE6f/68zp07p4yMDHXu3NlbUyV53tulSxdlZWXp1KlTcrlc2r9/vzp27Oit\nqZI875Wk4uJiVVRUqFWrVt6YV42nvWFhYbrmmmsUGBiooKAghYaG6syZM96aKsnz3lOnTqmwsFBr\n165VSkqKvv76a3Xq1MlbU2sVHR2t3NxcFRUVqaKiQnv37lX37t0b7PzVX1vX4M4779Qnn3yiUaNG\nybIszZs3TytXrlS7du3Uv39/3XTTTUpKSpLNZlN8fLx69OjRYAPro7a9V6PaNg8dOlSJiYkKCAjQ\nfffd5/UP2tr2Tp48WY8++qgkadCgQdUid7XtzcnJUZs2bby68d/VtvfTTz9VYmKi/Pz8FBcXp5/8\n5CdX7d477rhDx48f1/DhwxUQEKCpU6fKbrd7dW9NNm3apLKyMiUlJSk5OVljx46VZVkaPny4WrZs\n2WCPw48BA4BB/HAEABhEdAHAIKILAAYRXQAwiOgCgEFEF43uj3/8o1566aV639/hcCg7O7sBFwHe\nQ3QBwKA6/XAE0BBOnTqlCRMmaNKkSerevbumTZumEydO6Ny5c5o5c6Y6deqklJQUFRcXq7CwUCNH\njtTo0aNrPJfD4VD79u2Vk5Mjy7K0ePFiRUREaNGiRdqzZ48sy9LDDz+swYMHy+FwqHnz5jpz5oyW\nL1/u/sb83NxcJScny9/fX23atFFeXp5Wr15t8inBDxDRhREFBQUaP368pk+frtjYWK1atUpt2rTR\n4sWLlZWVpU8//VSBgYEaMmSI7rrrLuXn58vhcFwyutKFHz2dM2eO1qxZo2XLlik+Pl7Hjx/XH/7w\nBzmdTiUmJrp/Umvo0KG68847q9x/wYIFeuKJJ9S3b1+tW7dOeXl5jfocABLRhSG7du1SRESEKisr\nJV24ZsfFa17ExMQoJiZG+fn5SktL05YtWxQSElLtKmDf16tXL0kX4rt9+3a1bNlSBw8edF+U2uVy\n6cSJE5Kk9u3bV7t/dna2+2fqb731Vm3atKlh/rGAB7ynCyPuv/9+LVy4UDNmzFBZWZmio6P15Zdf\nSpKOHTumyZMna8WKFerWrZteeuklDRo0qNbfNnHxmq0ZGRnq2LGjOnTooJ49e2r16tVKS0vT4MGD\ndeONN0pSjdecjYmJUWZmpqQLv0EAMIFXujCmY8eOuvfee/XCCy9oxowZmj59usaMGaPz589r+vTp\nKi0t1ezZs7Vp0yaFh4fLbreroqLikue7eGnO4OBgLViwQOHh4frb3/6m0aNHq6ysTAMGDKhyjdfv\nmzJliqZPn64VK1YoNDS0xmurAg2NC97AJzkcDs2ePVvR0dH1PsfGjRsVGxuryMhIvf3228rIyNAL\nL7zQgCuB6vjUjh+sVq1a6Re/+IWCg4Pl5+enefPmeXsSfgB4pQsABvGFNAAwiOgCgEFEFwAMIroA\nYBDRBQCDiC4AGPR/qMTzZuFZxJgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8539f53dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mean_w = ppc_w['kcal'].mean()\n",
    "hpd_w = pm.hpd(ppc_w['kcal']).mean(0)\n",
    "\n",
    "mean = ppc_2['kcal'].mean()\n",
    "hpd = pm.hpd(ppc_2['kcal']).mean(0)\n",
    "\n",
    "plt.errorbar(mean, 1, xerr=[[mean - hpd]], fmt='o', label='model 2')\n",
    "plt.errorbar(mean_w, 0, xerr=[[mean_w - hpd_w]], fmt='o', label='weighted models')\n",
    "\n",
    "plt.yticks([])\n",
    "plt.ylim(-1, 2)\n",
    "plt.xlabel('kcal per g')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see the mean value is almost the same for both predictions but the uncertainty in the weighted model is larger. We have effectively propagated the uncertainty about which model we should select to the posterior predictive samples. You can now try with the other two methods for computing weights `stacking` (the default and recommended method) and `pseudo-BMA`.\n",
    "\n",
    "**Final notes:** \n",
    "\n",
    "There are other ways to average models such as, for example, explicitly building a meta-model that includes all the models we have. We then perform parameter inference while jumping between the models. One problem with this approach is that jumping between models could hamper the proper sampling of the posterior.\n",
    "\n",
    "Besides averaging discrete models we can sometimes think of continuous versions of them. A toy example is to imagine that we have a coin and we want to estimated it's degree of bias, a number between 0 and 1 being 0.5 equal chance of head and tails. We could think of two separated models one with a prior biased towards heads and one towards tails. We could fit both separate models and then average them using, for example, IC-derived weights. An alternative, is to build a hierarchical model to estimate the prior distribution, instead of contemplating two discrete models we will be computing a continuous model that includes these the discrete ones as particular cases. Which approach is better? That depends on our concrete problem. Do we have good reasons to think about two discrete models, or is our problem better represented with a continuous bigger model?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
