New Pull request

.jekyll-cache/Jekyll/Cache/Jekyll--Cache/b7/9606fb3afea5bd1609ed40b622142f1c98125abcfe89a76a661b0e8e343910

@@ -0,0 +1 @@
+I"1{"source"=>"/Users/ashutoshkushwaha/Desktop/Website/ashkush.github.io", "destination"=>"/Users/ashutoshkushwaha/Desktop/Website/ashkush.github.io/_site", "collections_dir"=>"", "cache_dir"=>".jekyll-cache", "plugins_dir"=>"_plugins", "layouts_dir"=>"_layouts", "data_dir"=>"_data", "includes_dir"=>"_includes", "collections"=>{"posts"=>{"output"=>true, "permalink"=>"/:categories/:year/:month/:day/:title:output_ext"}}, "safe"=>false, "include"=>[".htaccess"], "exclude"=>[".sass-cache", ".jekyll-cache", "gemfiles", "Gemfile", "Gemfile.lock", "node_modules", "vendor/bundle/", "vendor/cache/", "vendor/gems/", "vendor/ruby/"], "keep_files"=>[".git", ".svn"], "encoding"=>"utf-8", "markdown_ext"=>"markdown,mkdown,mkdn,mkd,md", "strict_front_matter"=>false, "show_drafts"=>nil, "limit_posts"=>0, "future"=>false, "unpublished"=>false, "whitelist"=>[], "plugins"=>["jekyll-feed", "jekyll-paginate", "jekyll-sitemap"], "markdown"=>"kramdown", "highlighter"=>"rouge", "lsi"=>false, "excerpt_separator"=>"\n\n", "incremental"=>false, "detach"=>false, "port"=>"4000", "host"=>"127.0.0.1", "baseurl"=>"", "show_dir_listing"=>false, "permalink"=>"date", "paginate_path"=>"/posts/page:num/", "timezone"=>nil, "quiet"=>false, "verbose"=>false, "defaults"=>[], "liquid"=>{"error_mode"=>"warn", "strict_filters"=>false, "strict_variables"=>false}, "kramdown"=>{"auto_ids"=>true, "toc_levels"=>[1, 2, 3, 4, 5, 6], "entity_output"=>"as_char", "smart_quotes"=>"lsquo,rsquo,ldquo,rdquo", "input"=>"GFM", "hard_wrap"=>false, "guess_lang"=>true, "footnote_nr"=>1, "show_warnings"=>false, "syntax_highlighter"=>"rouge", "syntax_highlighter_opts"=>{:default_lang=>"plaintext", :guess_lang=>true}, "coderay"=>{}}, "title"=>"Ashutosh Kushwaha", "email"=>"akushwaha9808@gmail.com", "description"=>"Data Scientist | IIT Bombay", "previous_companies"=>"HDFC Life | Flytxt", "author"=>"Ashutosh Kushwaha", "url"=>"http://localhost:4000", "github_username"=>"ashkush", "medium_username"=>"ashkush", "linkedin_username"=>"ashkush", "paginate"=>5, "livereload"=>true, "livereload_port"=>35729, "serving"=>true, "watch"=>true}:ET

.jekyll-cache/Jekyll/Cache/Jekyll--Converters--Markdown/a4/4139e11c347bc521761027f2035ed9e709fd0b44f5775020eb0aafa6863960

@@ -0,0 +1,226 @@
+I"�6<p>Variational inference methods in Bayesian inference and machine learning are techniques which are involved in approximating intractable integrals. Most of the machine learning techniques involves finding the point estimates (Maximum likelihood estimation (MLE), Maximum a posteriori estimate (MAP)) of the latent variables, parameters which certainly do not account for any uncertainty. For taking account of uncertainty in point estimates, an entire posterior distribution over the latent variables and parameters is approximated.</p>
+
+<p>    </p>
+
+<p><img src="/img/posts/Gaussian/2.png" alt="" height="250px" width="700px" /></p>
+
+<p>    </p>
+
+<p>For approximation of posterior distribution, which involves intractable integrals variational inference methods are applied for making the calculation tractable (having some kind of close form mathematically !!) i.e. which does not involve calculation of marginal distribution of the observed variables P(X).</p>
+
+<p>    </p>
+
+<p><img src="/img/posts/Gaussian/3.png" alt="2" height="150px" width="700px" /></p>
+
+<p>    </p>
+
+<p>The marginalisation over Z to calculate P(X) is intractable, because of the search space of Z is combinatorially large, thus we seek an approximation, using variational distribution Q(Z) with it’s own variational parameters.</p>
+
+<p>    </p>
+
+<h2 id="variational-inference-as-an-optimisation-problem">Variational Inference as an Optimisation problem</h2>
+
+<p>Since we are trying to approximate a true posterior distribution p(Z|X) with Q(Z), a good choice of measure for measuring the dissimilarity between the true posterior and approximated posterior is Kullback–Leibler divergence (<strong>KL-divergence</strong>),</p>
+
+<p><img src="/img/posts/Gaussian/4.png" alt="2" height="100px" width="700px" /></p>
+
+<p>   </p>
+
+<p>which is basically expectation of difference in log of two probability distribution with respect to approximated distribution. Considering KL divergence as an optimisation function, it needs to be minimised with respect to variational parameters of the variational distribution Q(Z). Since calculating KL divergence itself involves the calculation of posterior distribution P(Z|X), the calculation of dissimilarity in-terms of KL divergence needs to be expressed where we come across <strong>Evidence lower bound</strong> (ELBO). The above expression KL divergence can be written in terms of ELBO as</p>
+
+<p><img src="/img/posts/Gaussian/5.png" alt="2" height="25px" width="700px" /></p>
+
+<p>in which log evidence log P(X) is constant with respect to Q(Z), maximising the term L(Q) (Evidence lower bound) minimises KL divergence. By choosing an appropriate choice of Q(Z), L(Q) becomes maximizable and tractable to compute.</p>
+
+<h2 id="mean-field-approximation">Mean field approximation</h2>
+
+<p>The mean field approximation assumes that all the hidden variables are independent of each other, which simplifies the joint distribution of hidden variables as the product of marginal distribution of each hidden variable.</p>
+
+<p><img src="/img/posts/Gaussian/6.png" alt="2" height="150px" width="700px" /></p>
+
+<p>where N is the no. of the hidden variables. It can be shown that best distribution qⱼ(Zⱼ) for maximising ELBO can be estimated as</p>
+
+<p><img src="/img/posts/Gaussian/7.png" alt="2" height="50px" width="700px" /></p>
+
+<p>which is expectation of log joint probability distribution of observed and latent variables taken over variables which do not include Zⱼ. This type above equations can be simplified into functions of hyper-parameters of the prior distribution over the latent variable and expectation of the latent variables which is not being calculated. These type of equation create circular between the parameters of the distribution over the variables being calculated and expectation of variables not being calculated, this section of circular dependencies will be more clear in the following section. This circular dependence of distribution of parameters of the variational distribution suggest an iterative update procedure same as <strong>Expectation-Maximisation</strong> algorithm.</p>
+
+<h1 id="variational-inference-in-gaussian-mixture-model">Variational inference in Gaussian mixture model</h1>
+
+<!-- <img align="middle" width="300" height="300" src="/img/posts/Gaussian/8.png"> -->
+
+<p align="center">
+  <img width="460" height="300" src="/img/posts/Gaussian/8.png" />
+</p>
+<p>Consider the above Bayesian Gaussian mixture model in plate notation, where square plates denotes the hyper-parameters, large circular plates denotes latent variables and filled-in objects denotes known values.</p>
+
+<p>Here we are considering a finite Bayesian Gaussian mixture model with K components for a data-set of N X D dimension data-set with unknown parameters (mean vectors and precision matrix). For a graphical model to be in Bayesian setting, we need to define some prior (initial believes) on the unknown parameters for component weights (π), mean vectors (μ<em>{i=1…K}), precision matrix (Λ</em>{i=1…K})and latent variable (Z = z₁,…zn) for each of the observation present in the data, representing to which component does the observation belong to¹.</p>
+
+<p align="center">
+  <img width="460" height="300" src="/img/posts/Gaussian/9.jpeg" />
+</p>
+
+<!-- ![2](/img/posts/Gaussian/9.jpeg){: aligh ='center'} -->
+
+<ul>
+  <li>K-dimension Symmetric Dirichlet distribution prior for components weights, with each hyper-parameter set to α₀.</li>
+  <li>Wishart distribution prior on the precision matrix with known hyper-parameters (W₀, ν₀).</li>
+  <li>Categorical distribution, for each of the latent variable with K-dimensional vector in which only one of the element is 1, rest are 0.</li>
+  <li>Multivariate Normal Gaussian distribution on the means of components on of the mixture model, with known hyper parameter μ₀,β₀.</li>
+</ul>
+
+<p>The joint probability distribution of the variables can be written as</p>
+
+<p><img src="/img/posts/Gaussian/10.png" alt="2" height="40px" width="700px" /></p>
+
+<p>where the factorisation of joint probability distribution into individual distribution is dependent upon the way the graphical model is described. Individual factors can be written as:</p>
+
+<!-- ![2](/img/posts/Gaussian/11.jpeg){:height="300px" } -->
+<p align="center">
+  <img width="460" height="300" src="/img/posts/Gaussian/11.jpeg" />
+</p>
+
+<p>where,</p>
+
+<p><img src="/img/posts/Gaussian/12.png" alt="2" height="200px" width="700px" /></p>
+
+<p>where ν is the degree of freedom and D is the dimension of the scale matrix for Wishart distribution and the dimensionality of each data point.</p>
+
+<p>Since we have to approximate true posterior distribution <strong>P(Z,μ,Λ,π | X)</strong> with a variational distribution of latent variables i.e. with <strong>Q(Z,μ,Λ,π)</strong>. The variational distribution can be factorised into individual factors according to the assumption of mean-field approximation.</p>
+
+<!-- ![2](/img/posts/Gaussian/13.png) -->
+<p align="center">
+  <img width="400" height="20" src="/img/posts/Gaussian/13.png" />
+</p>
+
+<p>Now using the equation for the best variation distribution for each parameters, first let us calculate for Q*(Z), we focus only on the terms containing Z and take expectation over the other latent variable which is as follows:</p>
+
+<p><img src="/img/posts/Gaussian/14.png" alt="2" height="100px" width="700px" /></p>
+
+<p>where, ⟨ ⟩ symbol denotes the expectation over the latent variables³ which are in the subscript of it. Substituting the expression for <strong>p(X | Z, μ, Λ)</strong>, <strong>p(Z | π)</strong> in the above expression we get</p>
+
+<p><img src="/img/posts/Gaussian/15.gif" alt="2" /></p>
+
+<p><img src="/img/posts/Gaussian/16.png" alt="2" height="200px" width="700px" /></p>
+
+<p><img src="/img/posts/Gaussian/17.gif" alt="2" /></p>
+
+<!-- ![2](/img/posts/Gaussian/18.gif){:height="50px" width="500px" } -->
+<p align="center">
+  <img width="150" height="100" src="/img/posts/Gaussian/18.gif" />
+</p>
+
+<p>where rₙₖ is expressed as</p>
+
+<p align="center">
+  <img width="150" height="50" src="/img/posts/Gaussian/19.gif" />
+</p>
+
+<p>Thus the best q*(Z) is product of categorical distribution for each of latent variables with parameters rₙₖ for k = 1…K.</p>
+
+<p>Now similarly we evaluate best variational distribution for of the other variables and according to our graphical model we can express <strong>Q(π,μ,Λ)</strong> as product of <strong>Q(π)</strong> and <strong>Q(μ,Λ)</strong>. Then we can write,</p>
+
+<p align="center">
+  <img width="400" height="150" src="/img/posts/Gaussian/20.png" />
+</p>
+
+<p align="center">
+  <img width="400" height="150" src="/img/posts/Gaussian/21.gif" />
+</p>
+
+<p>taking exponential on the both sides of the above equation, this formulation is the form of Dirichlet distribution q*(π) ~ Dir(α), with parameters</p>
+
+<p align="center">
+  <img width="200" height="25" src="/img/posts/Gaussian/22.gif" />
+</p>
+
+<p>Finally, we can derive best variational distribution parameters for <strong>q*(μₖ,Λₖ)</strong></p>
+
+<p align="center">
+  <img width="500" height="100" src="/img/posts/Gaussian/23.png" />
+</p>
+
+<p>after some mathematical manipulations, the variation distribution assumes a form of Gaussian-Wishart distribution which is given by:</p>
+
+<p align="center">
+  <img width="400" height="50" src="/img/posts/Gaussian/24.png" />
+</p>
+
+<p>with parameters <strong>mₖ, βₖ, Wₖ, νₖ</strong> which are defined as:</p>
+
+<p align="center">
+  <img width="400" height="300" src="/img/posts/Gaussian/25.png" />
+</p>
+
+<p>All the parameters defined above <strong>( mₖ, βₖ, Wₖ, νₖ)</strong> are somehow dependent upon the categorical distribution parameter rₙₖ, which in turn is also dependent upon the above defined parameters. For evaluation rₙₖ, certain expression need to expressed which are as follows:</p>
+
+<p align="center">
+  <img width="500" height="200" src="/img/posts/Gaussian/26.png" />
+</p>
+
+<p>where <strong>ψ</strong> is log derivative of multinomial gamma function. The circular dependence of these parameters of variational distribution leads iterative update procedure of these parameters that alternates between two steps and <strong>is guaranteed to converge with respect to maximisation of ELBO or the minimisation of KL divergence between the true posterior</strong> and the approximate posterior over the latent variables. The iterative update procedure which alternate between two steps are as follows:</p>
+
+<ul>
+  <li>An E-step (Estimation step) that computes the value of rₙₖ using the current values of all other parameters.</li>
+  <li>An M-step (Update step) that uses new rₙₖ value to update all other parameters.</li>
+</ul>
+
+<h2 id="evidence-lower-bound-calculation">Evidence lower bound calculation</h2>
+
+<p>For the variational mixture of Gaussian the lower bound is given by²:</p>
+
+<p align="center">
+  <img width="500" height="75" src="/img/posts/Gaussian/27.png" />
+</p>
+
+<p>The various terms involved in the calculation of bounds are as follows:</p>
+
+<p align="center">
+  <img width="500" height="200" src="/img/posts/Gaussian/28.png" />
+</p>
+
+<p align="center">
+  <img width="500" height="150" src="/img/posts/Gaussian/29.png" />
+</p>
+
+<p align="center">
+  <img width="500" height="200" src="/img/posts/Gaussian/30.png" />
+</p>
+
+<p>where C(α) and B(W,ν) were defined earlier and H[q(Λₖ)] is the entropy of the Wishart distribution. For the convergence of the algorithm Evidence lower bound is to be taken as the convergence criterion, i.e. the algorithm is set to converge when there is no significant improvement in the convergence criterion.</p>
+
+<h2 id="implementation-of-the-above-algorithm-to-a-real-life-data-set">Implementation of the above algorithm to a real life data-set</h2>
+
+<p>Considering the scaled version of <a href="https://www.stat.cmu.edu/~larry/all-of-statistics/=data/faithful.dat"><strong>Old faithful data-set</strong></a>, which when graphically plotted, depicts only two cluster present within the data-set.</p>
+
+<p align="center">
+    <img width="500" height="400" src="/img/posts/Gaussian/31.png" />
+</p>
+
+<p>On application of the above <strong>Variational Expectation-Maximisation</strong> algorithm on the data-set with K=6 components, with suitable values of hyper-parameters and proper initialisation of the parameters of variational distribution, we see that the algorithm converges with only two components as expected.</p>
+
+<p align="center">
+    <img width="600" height="1000" src="/img/posts/Gaussian/32.png" />
+</p>
+
+<p>The above plots depicts cluster assignment of data points to cluster at various iteration, in which no. of colours present within a plot depicts number of total cluster. From the last plot it is evident that the algorithm converges with cluster only. For the below plot we can see that the evidence lower bound converges after 100 more iterations being close to zero, which shows that algorithm was able to approximate the true posterior distribution of the latent variables with variational distribution.</p>
+
+<p align="center">
+    <img width="600" height="400" src="/img/posts/Gaussian/33.png" />
+</p>
+
+<p>The code for the above implementation is available at: <a href="https://github.com/ashkush/Variational-inference-Gaussian-mixture-model">https://github.com/ashkush/Variational-inference-Gaussian-mixture-model</a></p>
+
+<h1 id="references">References</h1>
+
+<ul>
+  <li>
+    <p><a href="https://en.wikipedia.org/wiki/Variational_Bayesian_methods">Variational Bayes methods</a></p>
+  </li>
+  <li>
+    <p>C. Bishop: Pattern Recognition and Machine Learning, Springer, 2010 (Ch. 9–11 &amp; Appendix B)</p>
+  </li>
+  <li>
+    <p><a href="http://sap.ist.i.kyoto-u.ac.jp/members/yoshii/lectures/pattern_recognition/2017/20170606-npb-gmm.pdf">http://sap.ist.i.kyoto-u.ac.jp/members/yoshii/lectures/pattern_recognition/2017/20170606-npb-gmm.pdf</a></p>
+  </li>
+</ul>
+:ET

.jekyll-cache/Jekyll/Cache/Jekyll--Converters--Markdown/d2/e25426c8d7d9390af64eb48b2fcb48b67e31db9c9ed22a904956986dc460ea

@@ -0,0 +1,2 @@
+I"<p>Variational inference methods in Bayesian inference and machine learning are techniques which are involved in approximating intractable integrals. Most of the machine learning techniques involves finding the point estimates (Maximum likelihood estimation (MLE), Maximum a posteriori estimate (MAP)) of the latent variables, parameters which certainly do not account for any uncertainty. For taking account of uncertainty in point estimates, an entire posterior distribution over the latent variables and parameters is approximated.</p>
+:ET

.jekyll-cache/Jekyll/Cache/Jekyll--Converters--Markdown/e3/b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

@@ -0,0 +1,2 @@
+I"
+:ET

Gemfile

@@ -13,3 +13,5 @@ gem "tzinfo-data", platforms: [:mingw, :mswin, :x64_mingw, :jruby]
 
 # Performance-booster for watching directories on Windows
 gem "wdm", "~> 0.1.0" if Gem.win_platform?
+
+gem "webrick", "~> 1.7"

_config.yml

@@ -1,24 +1,24 @@
-title:              Clean Blog
-email:              your-email@example.com
-description:        A Blog Theme by Start Bootstrap
-author:             Start Bootstrap
-baseurl:            "/startbootstrap-clean-blog-jekyll"
-url:                "https://startbootstrap.github.io"
+title: Ashutosh Kushwaha
+email: akushwaha9808@gmail.com
+description: Data Scientist | IIT Bombay
+previous_companies: HDFC Life | Flytxt
+author: Ashutosh Kushwaha
+baseurl: "" #empty
+url: "https://ashkush.github.io"
 
 # Social Profiles
-twitter_username:   SBootstrap
-github_username:    StartBootstrap
-facebook_username:  StartBootstrap
-instagram_username:  
-linkedin_username:
+
+github_username: "ashkush"
+medium_username: "ashkush"
+linkedin_username: "ashkush"
 
 # Add your google-analytics ID here to activate google analytics
-google_analytics:   UA-XXXXXXXXX-X # out your google-analytics code
+# google_analytics:   UA-XXXXXXXXX-X # out your google-analytics code
 
 # Build settings
-markdown:           kramdown
-paginate:           5
-paginate_path:      "/posts/page:num/"
+markdown: kramdown
+paginate: 5
+paginate_path: "/posts/page:num/"
 plugins:
   - jekyll-feed
   - jekyll-paginate