PK @pO? i , plot_nonnegative_linear_ridge_variance.ipynb{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n# NonnegativeLinear Least Squares and NonnegativeRidge Regression Variance\n\n\n
Note
This example is a copy of ``plot_ols_ridge_variance.py`` by Gael Varoquaux\n and Jaques Grobler in the package Scikit-learn, using\n NonnegativeLinear and NonnegativeRidge.
\n\n.. currentmodule:: mcmodels.regressors.nonnegative_linear\n\nDue to the few points in each dimension and the straight\nline that linear regression uses to follow these points\nas well as it can, noise on the observations will cause\ngreat variance as shown in the first plot. Every line's slope\ncan vary quite a bit for each prediction due to the noise\ninduced in the observations.\n\nRidge regression is basically minimizing a penalised version\nof the least-squared function. The penalising `shrinks` the\nvalue of the regression coefficients.\nDespite the few data points in each dimension, the slope\nof the prediction is much more stable and the variance\nin the line itself is greatly reduced, in comparison to that\nof the standard linear regression\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Authors: Joseph Knox \n# License: Allen Institute Software License\n\n# NOTE: modified from plot_ols_ridge_variance.py by Gael Varoquaux and Jaques Grobler\n# from the package Scikit-Learn licensed under the 3 clause BSD License\n# reproduced below:\n#\n# New BSD License\n#\n# Copyright (c) 2007\u20132018 The scikit-learn developers.\n# All rights reserved.\n#\n#\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions are met:\n#\n# a. Redistributions of source code must retain the above copyright notice,\n# this list of conditions and the following disclaimer.\n# b. Redistributions in binary form must reproduce the above copyright\n# notice, this list of conditions and the following disclaimer in the\n# documentation and/or other materials provided with the distribution.\n# c. Neither the name of the Scikit-learn Developers nor the names of\n# its contributors may be used to endorse or promote products\n# derived from this software without specific prior written\n# permission.\n#\n#\n# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n# ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR\n# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\n# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\n# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY\n# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH\n# DAMAGE.\n\n\nfrom __future__ import print_function\n\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nfrom mcmodels.regressors import NonnegativeLinear, NonnegativeRidge\n\n\nprint(__doc__)\n\nn_datasets = 5\n\nX_train = np.c_[.5, 1].T\ny_train = [.5, 1]\nX_test = np.c_[0, 2].T\n\nnp.random.seed(0)\n\nregressors = dict(NonnegativeLinear=NonnegativeLinear(),\n NonnegativeRidge=NonnegativeRidge(alpha=0.1))\n\nfig, axes = plt.subplots(1, 2, figsize=(8, 3))\n\nfor ax, (name, reg) in zip(axes, regressors.items()):\n\n for _ in range(n_datasets):\n this_X = .15 * np.random.normal(size=(2, 1)) + X_train\n reg.fit(this_X, y_train)\n\n ax.plot(X_test, reg.predict(X_test), color='.5')\n ax.scatter(this_X, y_train, s=3, c='.5', marker='o', zorder=10)\n\n reg.fit(X_train, y_train)\n ax.plot(X_test, reg.predict(X_test), linewidth=2, color='blue')\n ax.scatter(X_train, y_train, s=30, c='r', marker='+', zorder=10)\n\n ax.set_xlim(0, 2)\n ax.set_ylim((0, 1.6))\n\n ax.set_xlabel('X')\n ax.set_ylabel('y')\n\n ax.set_title(name)\n\nplt.show()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK @pO6 plot_nadaraya_watson.ipynb{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n# Nadaraya-Watson Regression\n\n\nNote
This example is in part a copy of ``plot_kernel_ridge_regressions`` by\n Jan Hendrik Metzen found in the package Scikit-Learn.
\n\n.. currentmodule:: mcmodels.regressors.nonparametric\n\nNadaraya-Watos (NW) regression learns a non-linear function by using a kernel-\nweighted average of the data. Fitting NW can be done in closed-form and is\ntypically very fast. However, the learned model is non-sparse and thus suffers\nat prediction-time.\n\nTALK ABOUT EFFICIENT LOOCV!!\n\nThis example illustrates NW on an artificial dataset, which\nconsists of a sinusoidal target function and strong noise added to every fifth\ndatapoint.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Authors: Joseph Knox \n# License: Allen Institute Software License\n\n# NOTE: modified from plot_kernel_ridge_regression.py by Jan Hendrik Metzen\n# from the package Scikit-Learn licensed under the 3 clause BSD License\n# reproduced below:\n#\n# New BSD License\n#\n# Copyright (c) 2007\u20132018 The scikit-learn developers.\n# All rights reserved.\n#\n#\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions are met:\n#\n# a. Redistributions of source code must retain the above copyright notice,\n# this list of conditions and the following disclaimer.\n# b. Redistributions in binary form must reproduce the above copyright\n# notice, this list of conditions and the following disclaimer in the\n# documentation and/or other materials provided with the distribution.\n# c. Neither the name of the Scikit-learn Developers nor the names of\n# its contributors may be used to endorse or promote products\n# derived from this software without specific prior written\n# permission.\n#\n#\n# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n# ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR\n# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\n# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\n# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY\n# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH\n# DAMAGE.\n\n\nfrom __future__ import division, print_function\nimport time\n\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nfrom sklearn.model_selection import GridSearchCV\n\nfrom mcmodels.regressors import NadarayaWatson, NadarayaWatsonCV\n\n\nprint(__doc__)\n\nrng = np.random.RandomState(0)\n\n# #############################################################################\n# Generate sample data\nX = 5 * rng.rand(10000, 1)\ny = np.sin(X).ravel()\n\n# Add noise to targets\ny[::5] += 3 * (0.5 - rng.rand(X.shape[0] // 5))\n\nX_plot = np.linspace(0, 5, 1e3)[:, None]\n\n# #############################################################################\n# Fit regression model\ntrain_size = 100\nparam_grid=dict(kernel=[\"rbf\"], gamma=np.logspace(-2, 2, 25))\n\nnw_gs = GridSearchCV(NadarayaWatson(), cv=5, param_grid=param_grid)\nnw_cv = NadarayaWatsonCV(param_grid)\n\n# fit 5-fold using GridSearch\nt0 = time.time()\nnw_gs.fit(X[:train_size], y[:train_size])\ngs_fit = time.time() - t0\nprint(\"GridSearchCV 5 fold cross validation fitted in %.2f s\" % gs_fit)\nprint(\"\\toptimal bandwidth found: %.2f\" % nw_gs.best_estimator_.gamma)\n\n# fit leave-one-out using NadarayaWatsonCV\nt0 = time.time()\nnw_cv.fit(X[:train_size], y[:train_size])\ncv_fit = time.time() - t0\nprint(\"NadarayaWatsonCV leave-one-out cross validation fitted in %.2f s\" % cv_fit)\nprint(\"\\toptimal bandwidth found: %.3f\" % nw_cv.gamma)\n\n# predict\ny_gs = nw_gs.predict(X_plot)\ny_cv = nw_cv.predict(X_plot)\n\n\n# #############################################################################\n# Look at the results\nplt.scatter(X[:100], y[:100], c='k', label='data', zorder=1,\n edgecolors=(0, 0, 0))\nplt.plot(X_plot, y_gs, 'c-', lw=3, label='5-Fold GridSearchCV')\nplt.plot(X_plot, y_cv, 'r:', lw=3, label='LOO NadarayaWatsonCV')\nplt.xlabel('data')\nplt.ylabel('target')\nplt.title('Nadaraya-Watson Regression')\nplt.legend()\n\nplt.show()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK @pO)o o ! plot_nonnegative_ridge_path.ipynb{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n# Plot NonnegativeRidge coefficients as a function of regularization\n\n\nNote
This example is a copy of ``plot_ridge_path.py`` by Fabian Pedregosa\n in the package Scikit-learn, using NonnegativeRidge.
\n\n.. currentmodule:: mcmodels.regressors.nonnegative_linear\n\n:class:`NonnegativeRidge` Regression is the estimator used in this example.\nEach color represents a different feature of the\ncoefficient vector, and this is displayed as a function of the\nregularization parameter.\n\nThis example also shows the usefulness of applying Ridge regression\nto highly ill-conditioned matrices. For such matrices, a slight\nchange in the target variable can cause huge variances in the\ncalculated weights. In such cases, it is useful to set a certain\nregularization (alpha) to reduce this variation (noise).\n\nWhen alpha is very large, the regularization effect dominates the\nsquared loss function and the coefficients tend to zero.\nAt the end of the path, as alpha tends toward zero\nand the solution tends towards the ordinary least squares, coefficients\nexhibit big oscillations. In practise it is necessary to tune alpha\nin such a way that a balance is maintained between both.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Authors: Joseph Knox \n# License: Allen Institute Software License\n\n# NOTE: modified from plot_ridge_path.py by Fabian Pedregosa\n# from the package Scikit-Learn licensed under the 3 clause BSD License\n# reproduced below:\n#\n# New BSD License\n#\n# Copyright (c) 2007\u20132018 The scikit-learn developers.\n# All rights reserved.\n#\n#\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions are met:\n#\n# a. Redistributions of source code must retain the above copyright notice,\n# this list of conditions and the following disclaimer.\n# b. Redistributions in binary form must reproduce the above copyright\n# notice, this list of conditions and the following disclaimer in the\n# documentation and/or other materials provided with the distribution.\n# c. Neither the name of the Scikit-learn Developers nor the names of\n# its contributors may be used to endorse or promote products\n# derived from this software without specific prior written\n# permission.\n#\n#\n# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n# ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR\n# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\n# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\n# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY\n# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH\n# DAMAGE.\n\n\nfrom __future__ import division, print_function\n\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nfrom mcmodels.regressors import NonnegativeRidge\n\n\nprint(__doc__)\n\n# X is the n x n Hilbert matrix\nn = 6\nX = 1. / (np.arange(1, n + 1) + np.arange(n)[:, np.newaxis])\ny = np.ones(n)\n\n# #############################################################################\n# Compute paths\n\nn_alphas = 41\nalphas = np.logspace(-2, 2, n_alphas)\n\ncoefs = []\nfor a in alphas:\n ridge = NonnegativeRidge(alpha=a)\n ridge.fit(X, y)\n coefs.append(ridge.coef_)\n\n# #############################################################################\n# Display results\n\nfig, axes = plt.subplots(1, 2, figsize=(8, 3))\n\nfor ax in axes:\n ax.plot(alphas, coefs, lw=2)\n ax.set_xscale('log')\n ax.set_xlabel('alpha')\n ax.set_ylabel('weights')\n\n# trim subplots\naxes[0].set_xlim(1e-2, 1e2)\naxes[1].set_xlim(1e-1, 1e1)\n\naxes[0].set_ylim(0, 8)\naxes[1].set_ylim(0, 1)\n\nplt.suptitle('Ridge coefficients as a function of the regularization')\nplt.show()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}PK @pO? i , plot_nonnegative_linear_ridge_variance.ipynbPK @pO6 L plot_nadaraya_watson.ipynbPK @pO)o o ! W* plot_nonnegative_ridge_path.ipynbPK ?