Posts

• 

  Recurrent Neural Networks are Turing-Complete

  In this article, I will review recurrent neural networks. I will also review the concept of Turing machines, and I will then move on to show one of the most fascinating properties of recurrent neural networks: they may be Turing-Complete.

  Nov 21, 2020

• 

  Kolgomorov Complexity

  This article gives a brief overview of a fascinating theory, which is complementary to Information Theory, and which, one can argue, even more general than the latter. .

  Nov 18, 2020

• Generalized Linear Models

  Nov 18, 2020

• Rmarkdown.nb

  Nov 18, 2020

• 

  The Blahut-Arimoto Algorithms and their Use in the Information Bottleneck Problem

  This article discusses the Blahut-Arimoto algorithms. We will begin with a geometric and functional interpretation of the algorithm. We will then discuss the capacity maximization and the Rate-Distortion minimization algorithms. We will dedicate more time to the latter, as we will see that it can be used to compute the Information Bottleneck problema.

  Nov 15, 2020

• 

  Generalized Linear Models: A Statistical View of Data

  This article explores the concept of Generalized Linear Models. We present in this post John Nelder's theory, which informs us that behind many regression and classification problems there lies a common framework. We will explore the commonalities between many different problems, including linear regression, logistic regression, and Poisson regression.

  Nov 13, 2020

• 

  Information Theoretical Pruning of Convolutional Neural Networks

  This article has three stand-alone pieces. It gives a brief overview of Convolutional Neural Networks. It presents an overview of the Information Bottleneck Theory. And it presents an architecture that is gaining a lot of traction: statistical convolutional neural networks. Think of them as convolutional networks with a taste of recurrent networks or with a taste of traditional time convolutional, where we have some memory.

  Nov 9, 2020

• 

  Cross Entropy and its Significance to Machine Learning

  In this article, I will discuss what Cross Entropy means from an information theoretical point of view and how it relates to key concepts in Machine Learning, including the log-loss cost function in logistic regression classification and parameter estimation.

  Nov 3, 2020