Brief Overview of G-Ramp - Simple AR Model
Note:- Since G-Ramps platform is under development, you will find missing steps or missing parts of already mentioned steps. That is, you will find some arguments which are either not proved mathematically or not supported by experimentation. With time, there should be a lesser and lesser number of such arguments. If there are such arguments, then I should at least fix them temporarily by providing a good reference for each such argument.
Between any two adjacent steps, either it will be explicitly mention that there are missing steps between these two steps. Or there may not be such mention of missing steps. Similarly, there may or may not be explicit mention of missing parts of any already mentioned steps. Text in square brackets ‘[]’ is used to mention about missing steps or about missing parts of already mentioned steps.
A step title below starts with “LS” meaning logical step. Steps should be numbered like “LS#1 …”. But putting numbers during development creates difficult because it is not straightforward how to increment the numbers when steps are added in between. So, until a g-ramp is complete and contain all the steps, steps are not numbered.
Brief Overview
- Prerequisite
- It is assumed that you have gone through the first ramp.
- If not, you should atleast know that neural networks (NNs) are universal function approximators and how they are trained.
- Main Problem
- Humans have imagination power. We can imagine literally anything, be it some known or novel concept or physical object. Let’s denote whatever concept or physical object we can imagine as \(x\) for simplicity.
- Novel \(x\) is some \(x\) which is atleast according to you does not exist yet in the world but also it may not really exist for anyone. We create useful stuff with the help of our imagination power.
- We can do two kind of imaginations - “wild” and “problem-driven”. We use problem-driven imagination when we want to solve a problem. Given a problem, we only think and imagine those \(x\) which can atleast solve a problem to some extent instead of not solving it at all. Let’s call such viable but not necessarily perfect \(x\) “candidate”. So, problem-driven imagination can be defined as an imagination where space of imagination is constrained to only candidates of a given problem. We do not think of those \(x\) which are not candidate.
- E.g. a problem is that you want to buy a shirt and you are not sure what kind of shirt you are looking for. But ofcourse you have some preferences. Like you do not want to repeat the shirts you have in your wardrobe. And atleast you also have size or fitting preferences. Let’s say you have just these preferences. What you will do? You will go to online or offline stores. But before going there you will think and imagine many shirts on you one after another which follow these preferences. Here any shirt which follow the preferences is a candidate but a car is not a candidate or any other shirt which does not follow the preferences is not a candidate. This explains what we mean by “candidate”. So, the space of imagination contain only shirts and all kinds of shirts which follow preferences. By imagining, you are trying to find what next shirt you should buy.
- On the other hand, one does “wild” imagination when someone does not want to solve a problem but still imagines. Then in this case there is no constrain in the space of imagination. It is borderless. One can imagine literally any \(x\).
- Wild imagination is not dependent on one particular problem. It is unhinged. The space of imagination has no boundaries. So we are free to imagine anything. And since space of imagination is just infinite, there are high chances that whatever we are imagining may not lead to solving one of many real problems out there. Example of such imagination is dreaming, be it night or day dreaming.
- Relative to wild imagination, problem-driven imagination is very clearly useful because there is a problem which we are focused on solving. And there is space of imagination which is constrained or fixed. Since, we are focused on solving and space is fixed, we can easily imagine a candidate which will atleast help us solve a problem instead of not helping at all. For easier problems, problem-driven imagination can actually solve the problem. E.g. designers of all sorts be it physical product designers, architects, web designers etc. use imagination power to solve a problem.
- On the other hand, scientists deals with problems which are extremely difficult. For such problems, problem driven imagination is very difficult to do it effectively and hence one do not form a complete solution via imagination. But atleast imagination help them to propose a theory and hypothesis which they can test whether it is solving a problem or not. Einstein once imagined riding on a beam of light to better understand light.
- But [As far as I know. May edit this para later if required] there are many problems like drug discovery, material discovery, biological discovery etc. where problem-driven imagination is only barely more useful than wild imagination. Because such problems have exponentially large space of imagination. Take an example of drug discovery problem. Given some disease, there are exponential number of candidates. And one cannot sweep all the candidates and think of a novel molecule which is better than existing molecules. It is impossible for a human to do that because of limited time or capacity. This is just an example. Many real-world problems like mentioned above which requires imagination are very difficult to solve because space of imagination is exponential.
- As mentioned, wild imagination is not clearly useful because of infinite space of imagination. But atleast problem-driven imagination has not infinite space. So let’s first focus on problem-driven imagination and let’s see if we can do problem-driven imagination when it is hard to do. Then later on, we can talk about improving wild imagination too in some other g-ramp.
- What if we can let computer do problem-driven imagination and create novel things which are useful. Why computer? Because of its speed, accuracy, and scalability (in term of space of imagination).
- When we let computer imagine, the algorithm which does it is called “Generative AI” algorithm or “gen AI” algorithm in short. And “generate” term is used when a computer imagine something.
-
We can let gen AI algorithm generate \(x\) like molecules etc. in terms of data structures like images, graph, tree etc.
- Now, we have come to our “main problem” of this g-ramp. “Main problem” which this g-ramp is for, is that we want a computer to do “problem-driven imagination”.
- We can make the “main problem” more concrete so that it is easy for us to think of solution. But our solution should be for the “main problem” and not just for its concrete version.
- To make it concrete, we can think of some particular problem for which we want a computer to do problem-driven imagination. For simplicity, let’s choose a problem where we want a computer to imagine or generate digits in different handwriting.
- LS - Two Functions of Intelligence
- Since we want to let computer do imagination, first obvious step should be that we should first understand how humans learn to do imagination?
- [Explain that we mentioned in previous ramp that intelligence meaning having two functions - learning and action. And imagination is one kind of action function of human intelligence. And there should be a learning function which outputs an action function for imagination]
- We also mentioned that, to simulate these two functions in a computer we need to implement them via mathematics.
- Let’s first try to understand these two functions with respect to the human action - imagination or more specifically problem-driven imagination because that is what our “main problem” demands. That is, how a human learn to do problem-driven imagination and how they do this imagination, once they have learned to do it.
- And then we can think of formulating them in mathematics.
- LS - How a human learn to imagine?
- [Explain that we do not learn it consciously. It is inherent and already there naturally.]
- But one thing, we can say that imagination becomes mature or more controlled with time, as we gain more knowledge of the world.
- Why? Because we understand more and more regularity in the world. We understand that there are representations or description of the world at different level. We will be using the term “representation” to mean description, just like in previous ramp.
- [mention that we have explained representation in previous ramp]
- And as we see the world around, representation at every level changes at different rate. As we move from highest level to lowest level, the rate of change of representation increases from 0 to highest.
- As toddlers grow, they understand more and more things. With time, they understand basic physics, shape, texture, color, material, position, posture, pose, light etc.
- [Explain that we can represent or describe at any level literally anything in the world. We represent whatever we focus on. We can describe a scene containing multiple objects or just single object]
- [give examples explaining this]
- Scene here is nothing but whatever we focus on.
-
[explain that more higher-level the representation is of, more number of different scenes can be described by single representation]
- E.g. we are focusing on a shirt. So, scene is nothing but that shirt. We describe that shirt. Lowest level of representation describes it in most detail which includes everything what we are seeing including the exact posture, background description etc.
- A little higher representation will exclude the posture and background description. Such representation just describes that shirt. It is independent of its posture and background. Such representation let’s imagine that shirt in different scenes which we haven’t seen before, i.e., in different postures and background.
-
As we see more and more shirts, we notice that they come in different color, material, pattern, collar, fitting, sleeves etc. And we notice that all these descriptions of a shirt does not define shirt. And we figure out a description or representation which describes all the shirts in any posture, background, color, material, pattern etc. This is the highest level of representation if we only want to describe a single object, here shirt, of the world.
- [talk about how this explains problem-driven imagination?]
- [Give the example of shirt problem. We have worn and seen a lot of shirts.]
-
Another example for understanding different levels of representation is imagining a particular cup on a table. After you observe this cup at different positions and in different orientation on a table, you will know that highest level of representation only describing cup and table is that “there is this cup on a table”. And lower level representations describes cup’s different orientations and different position on the table.
-
[now, explain different levels of representation for a drug design problem and how understanding this will help us generating all possible drugs]
- [conclude that learning to imagine means understanding different levels of representations]
- [conclude that for problem-driven imagination, one learns a high-level representation, let’s call it “problem-level representation”, which describes all the candidates. Also explain that this representation is just high-enough that it only describes all candidates and nothing else apart from this.]
- Different problems will have different problem-level representation.
- LS - How a human imagine?
- When we try to solve a problem via problem-driven imagination, we do not just imagine one candidate. But we imagine multiple candidates that we think could solve the problem because we do not really know what is the best solution. We may think, there could be even better candidate which we have not imagined yet and this is the reason we keep imagining.
- [given a problem, explain that to imagine a candidate, one need to create lower levels of representation until the lowest level representation by “adding” some more description on top of problem-level representation]
- Let’s call lowest level representation as “candidate-level representation”.
- [Explain that candidate-level representation describes a particular candidate]
- [and explain that the description is chosen first randomly and then added to problem-level representation]
- [give examples]
- In shirt example, we can choose color, pattern, fitting randomly out of many options.
-
Given a problem, action function of imagination just brings to your mind the candidate \(x\) one by one. And there is no fixed order. Whenever you try to solve same problem by problem-driven imagination you will imagine its candidates in different order every time. As if, a candidate to imagine is chosen randomly. And when you keep imagining, you are keep picking next candidate randomly from space of imagination.
-
After imagining many candidates, you know which candidates are potentially better than other candidates. But only after rigourously testing these potential candidates, we select the candidate or candidates which solve the problem in a better manner. Testing the candidates would involve manifesting them in some form so that one can consciously think about them.
- [give examples]
- Let’s continue our shirt example from “Main Problem” section. We would like to imagine many shirts. We know space of imagination is large. And we know there is possibility that we can imagine a shirt which is better than the already imagined shirts. When we try to imagine a shirt, it seems like we just randomly choose a candidate to imagine out of all candidates. Same process happens whenever we try to change our current imagination and try to imagine another shirt.
- After imaginging many shirts one after another, you have in mind some shirts which are potentally better than other shirts. So now you would like to test these shirts properly by finding these shirts or similar shirts in online or offline stores. And try them on. And then only you can finally decide whether to buy which shirts out of these potential shirts.
-
Another example is web designing. You imagined some web designs. But to decide finally whether which design is good, you create a mockup and test it.
- [now explain that similar process happens in drug design. Drug testing would involve manifesting them first, which is very expensive but also accurate. So, one need to first search potential drugs with the help of some cheap proxy which tells in some way that a molecule could be a potential drug. A proxy should be such that it should be able to filter out safely those molecules which can’t be a drug for a given disease. By cheap screening first, we do not do expensive test on those molecules which can’t be a drug for a given disease. We only rigourously test molecules which has potential to be a drug.]
- [explain that this very much sound like a problem which can be solved via problem-driven imagination but it is just that humans can’t do that as explained in “main problem” section. So, we decided to let computer do problem-driven imagination.]
- LS - Mathematical Formulation of Functions
- We have understood two functions - action and learning with respect to human action - problem-driven imagination.
- Now we should try to formulate them in mathematics.
- [As explained in previous ramp, to mathematically formulate the two functions of intelligence, we have to first formulate action function and then we can formulate learning function accordingly.]
- When we think of action function described in previous step, it immediately reminds us a problem of picking a card from shuffled deck. One can pick any card out of \(52\). Each card have \(1/52\) probability of getting picked.
-
Similarly, one can imagine any candidate from space of imagination. It means, each next candidate have \(1/\vert S \vert\) probability of getting picked for imagination, where \(\vert S \vert\) is size of space of imagination.
- Let’s represent probability over \(x\) as \(p(x)\). In problem-driven imagination, \(x\) is a candidate. While in wild imagination, \(x\) is just \(x\), i.e., there is no constrain.
- [define “random variable”]
- So mathematically, we can say, action function is nothing but sampling from this probability distribution.
- [define what we mean by “probability distribution” and “sampling” for a refresher. This is covered in school-level maths.]
-
For a given problem, a computer does not know the probability distribution over candidates because it does not know all the candidates. Let’s call this unknown target probability distribution as \(T\).
- When we try to approximate \(T\) in a computer or in mathematical form, that mathematical form is called “model” (similarly explained in first ramp).
- [And may be we can explain more about what do we mean by word “model”]
- We can denote model by \(M\).
- If we have a model of \(T\), then we can let a computer sample from \(M\) because we know this distirbution unlike \(T\). And this is how we can implement action function.
- [explain how to sample from given probability distribution?]
- [also mention and explain the term “data generating process”]
- LS - Mathematical Formulation of Learning Function
- We know action function is nothing but sampling from \(M\).
- So, we need \(M\) to implement action function in a computer.
- It means learning function or algorithm should output \(M\).
- The process of creating a model of \(T\) is called modeling, but more popularly called either learning or training.
- Given any problem e.g. handwritten digit image generation, how do we create its learning algorithm.
- Given a problem, how do we model \(T\)?
- LS - Need Dataset
- Next and only obvious step is to try to mimic how human learn to imagine, because that is what inspired the “main problem”.
- We explained how human learn above:
- They see many candidates.
- And learn all levels of representations.
-
So, given some problem, we need a computer to see enough candidates so that somehow it can learn all levels of representations.
- Collection of examples or candidates given to a computer to learn is called a “dataset” or simply “data”.
- [mention that this is generally called “training dataset” or “training data”]
-
So, learning algorithm will be taking the dataset as input. And the algorithm should somehow let a computer learn all levels of representations from a given dataset and somehow model the \(T\).
- For the handwritten digit generation problem, what dataset we should have?
- [explain first that candidate should be in a form of “image” for this problem]
- \(x\) needs to be represented in a computer so that a computer can generate it.
- [about image: an image is 3 square matrices corresponding to three color channels]
- We have to have a dataset which contain only digit images, i.e., images of only 0 to 9.
- [explain that we need multiple images of each digit but with different handwriting. So, there could be variation due to different hand-writing.]
- [explain MNIST dataset]
- [explain first that candidate should be in a form of “image” for this problem]
- For drug design problem, we can let a computer to see a lot of molecules.
- How can we store molecules in a computer?
- [explain that a molecule can be represented (lowest-level representation) as a graph and stored in a computer. And a graph can be stored in a computer in many ways like adjacency list, adjacency matrix etc.]
- For this problem, we can see the advantage of letting a computer do problem-driven imagination over a human doing it.
- As we mentioned that for this problem, space of imagination is exponentially large. But computer can ingest a lot of data, process it quickly (compared to humans) and based on this data, it can hopefully imagine novel and more useful molecule in comparatively much less time.
- LS - Naive Model
- We have defined the output and input of a learning algorithm.
- [explain that before going on understanding all levels of representations through dataset, we can think of a very naive model]
- [explain how? We need a model of \(T\). We have observations in terms of dataset. We know from school that prob = obs/total]
- Such way of modeling leads to spiky distribution because such way requires all candidates, i.e. whole space of imagination and we don’t have that.
- If we have all possible candidates then why would we need to generate in the first place.
- Or we can say this way of modeling the \(T\) is not a good approximation of \(T\).
- LS - Analyze MNIST Dataset
- We have a dataset. Now we need to mimic how humans learn all levels of representation by seeing many candidates.
- This will let’s create a good model of \(T\).
- [explain how?]
- A good model should be able to generate all the candidates which are possible apart from what is available in the dataset.
- [ Use the term “generalize”]
- In the “main problem” section, we defined the “novelty” for humans. Now let’s define “novelty” for computer.
-
Any \(x\) or candidate is “novel” for a computer if the computer has not seen it during learning.
- In order to mimic, let’s recall what we do when we see a lot of examples.
- We just need to recall “what” we learned to do such imagination.
- [explain that “how” to learn is not important]
- Given a problem, we understand different levels of representations.
- Since, representations are different for different problem.
- Let’s understand what problem-level and all lower level representations until candidate-level representation, we will learn to do imagination for handwritten digit generation problem.
- [explain that we will be looking at candidates in a form which a computer will process, i.e. in the form of images. Conclude that we will be analyzing MNIST dataset by ourselves]
- After looking at MNIST dataset, problem-level representation we will learn is that image contains a handwritten digit in a center in white color with black background.
- With digits having more or less same size.
- All images in the \(T\) should follow this description.
- Problem-level representation does not specify digit.
- To create candidate-level representation, we add description to problem-level representation. Every single addition of description create next lower level representation. This addition continues until we get lowest level of representation which we are also calling candidate-level representation.
- [Talk about order in which description is added. E.g. local randomness is added on top of global structure. Because it depends on global structure.]
- Second level describes which particular digit out of 10 digits does an image contain and global structure of each digit, which is same for all images in a digit class.
- [Can we say that class label of each image can act as second-level representation?]
- We can select any second-level representation out of total 10. E.g. let’s select “7”.
- Second-level representation is nothing but a description of a “wave” with a particular global structure (talked about it in previous ramp). So, it does not describe that particular pixel should have such and such value. But is only about the particular set of differences in values which could occur over any set of pixels in an image.
- Any two waves of same class will have same global structure and only varying locally.
- Any two waves of different class will also have different global structure along with different local structure.
- [and also explain that since it is only about the differences, it leads to any color image]
- But since our problem-level representation describes a white color.
- [it means only a particular kind (corresponding to white color and black background) of difference out of many possible differences is allowed]
- This representation contain information about all possible handwriting.
- Also since the wave could occur anywhere, but problem-level representation describes that digit only occurs at the center and are of similar sizes.
-
It means wave could only occur in a center and can not vary much in size.
-
Now we have a wave with a particular global structure that is only occurring in center and in white.
- This let us imagine all the possible handwritten seven digit in white in center.
- [how? Explain via movement of wave]
- Or we can say this is last-second representation and describes all possible lowest-level or last-level representation.
- Last-level representation is nothing but pixel values of a particular image.
- [explain that we get last-level representation when we take snapshot of a moving wave. Snapshot could be explained as pausing the wave]
- [lowest-level representation varies the most from one image to another]
-
- Novelty will come from randomness in handwriting on top of this regularity.
- This will lead to novel handwriting of one of a digit and hence novel object.
-
These novel objects have actually non-zero probability and hence part of the \(T\) we are trying to model.
- it is the lowest level of representation, i.e., no two images in a dataset have same such representation unless they are identical.
- it means it varies from one image to other. And this variation leads to all possible objects with non-zero probability in \(T\).
- Let’s divide the area into central and non-central area. Wave can start from any pixel lying near the boundary of central area.
- If we focus on one particular pixel near the boundary of central area and if we imagine different 7 digit candidates. Then we will see that sometimes this pixel is bright and sometime it is dark, depending on wave which is moving randomly.
- That means, when we take snapshot without looking at wave, this pixel could be bright or dark. This again looks like we are sampling from some probability distribution over two outcomes bright and dark.
- [also explain that this is not binary distribution but categorical distribution over pixel values. Bright pixel means having higher pixel value like between 200 to 255. And dark pixel means having lower pixel value like between 0 to 50.]
- It means at each pixel near boundary of central area, there is some probability that a wave could start.
- LS - Joint And Conditional Probability
- We can also notice that once the wave starts at some pixel, there are high chances that a pixel adjacent to this pixel is also bright. Because bright pixels occur together.
- [explain what do we mean by “adjacent” here. We have to explain order in which we are iterating over pixels, like from left to right and top to bottom. If we have this order than adjacent means a pixel on the right side or bottom or diagonally bottom and right side]
- This is at least true for diagonally adjacent pixel.
- If we do not fix the starting of a wave at some pixel then probability that adjacent pixel is bright is low. But if we fix, then probability of adjacent pixel being bright is very high.
-
This immediately reminds us of conditional probability. And independent and dependent events. We have studied about these in school.
- [Explain this. When probability of an event changes when we know some other event has occurred then these two events are dependent. Otherwise independent.]
- [also mention the idea of dependency or independency of two random variables]
- A pixel value is a kind of random variable.
-
Here a pixel near a border of central area and its adjacent pixel are dependent.
- Now if we continue, we can say same thing about the third pixel in the iteration.
- Numbering the pixels starting from start of the wave. And iterating in the order defined above.
-
Third pixel also has high chances of being bright if we know previous two pixels are bright.
- [now calculate these probabilities for 7 digit]
- [show that this let you generate different handwritten 7 digits]
- [explain that similar dependency of pixels is true for rest of the digits]
- We can also notice that once the wave starts at some pixel, there are high chances that a pixel adjacent to this pixel is also bright. Because bright pixels occur together.
- LS - Back To Naive Model
- [explain that to generate other digits too, we can calculate probabilities over pixels by considering all the digits]
- We will see there is some probability distribution over first pixel values. And every next pixel has a different probability distribution over its values depending on values of previous pixels, i.e., which digit wave has started if started.
- [now explain that this leads to naive model]
-
This whole analysis realizes you that probability of \(x\) can be thought as probability of multiple events occurring together (“joint probability”) like $$p(x) = p(ab) = p(a).p(b a)$$ - You can break probability of \(x\) into multiplication of probability of its constituents, here pixels.
-
- [as we notice probability distributions of more pixels. We will notice that conditional probability of pixels away from the border, conditioned on pixel values coming from a particular digit, is approximately independent of these previous pixel values when we know that digit.]
- [now explain “conditional independence”]
- Because global structure approximately determine the probability distributions of pixels.
- Dependency is only there because we do not know the class up front and condition determine the class as it becomes bigger by sequential sampling. Once the condition becomes big enough and class is determined then one can just use probabilities conditioned on that class because in digit dataset, class determines the global structure.
- [explain that this is a one trail of steps which we can take to improve upon naive model. But it will be very different from our current naive model. We will take this trail in next ramp. Can we now think of a small tweak in naive model so that we can improve upon it without changing it much.]
- [explain that to generate other digits too, we can calculate probabilities over pixels by considering all the digits]
- LS - Add Some Error
- To improve upon naive model, we should first understand what it is doing.
- It constructs one of an image in a dataset exactly pixel by pixel.
- As we understood above that we should only be caring about second-level representations.
- As long as, we have this representation being followed, we are solving the “main problem” which is letting a computer do problem-driven imagination, where problem here is handwritten digit generation.
- It means, we have to focus on moving a wave randomly which is fixed in center, and is in white color.
- [explain that fixing a color does not mean, using some particular set of values for pixels. But it means that with naked eyes this color should like white but can have very subtle differences from one image to another, i.e. +5, -5 in values.]
-
This tells us that what we can at least do to create novel candidates which may not be too novel but very subtly novel is to have non-zero probabilities for those pixel values which are close to modes of current distributions.
- It means, we are adding some error and hence making the distribution to not “overfit” the data.
- [we will be explaining “overfit” term in first ramp]
- Or we can say we are making the modes of conditional distribution less spiky by adding some error.
- [explain that these less spiky distributions lets you sample novel candidates. And also what kind of novel candidates.]
- LS - Use Neural Network
- How do we do this?
- We can manually do this but it seems laborious.
- [explain that even if we manually do this, we will have conditions which we haven’t seen in dataset therefore we don’t have corresponding conditional probability distributions]
- We need probability distributions corresponding to these conditions.
- And we can very easily think of these distributions. It is just that we can not calculate them via a computer.
- What we currently need is that we want to map unseen conditions (which are not present in a dataset) to appropriate less spiky distributions.
- “Appropriate” here means output probability distribution should be such that the overall novel object should follow the second-level representation.
-
It means we need a function which can do such mapping.
- This looks like a same problem we dealt in first ramp where we needed to find a function that would map an image, be it seen or unseen, to its correct label.
-
And we learned that “neural networks” are universal function approximators and they can be trained to approximate any arbitrary function given the appropriate dataset.
-
So here, we are again doing “machine learning”, but this time we are letting the machine to learn to imagine.
- — function should output prob dist similar to dataset given a condition from the dataset.
- LS - Softmax Function
- We decided that we are using NN but how?
- We need the NN to output probability distribution.
- [now introduce “softmax” layer and come to conclusion that we can use it]
- LS - Training Data
- How to train this neural network?
- [Call this neural network as “internal” model. Explain that there is another “modeling” problem inside a larger modeling problem of modeling \(T\). And explain that this internal model indirectly models \(T\).]
- This internal model models the mapping between condition and conditional probability distribution explained above.
- To let computer learn this neural network, we have to have a dataset just like in previous ramp.
-
[explain the creation of training data for internal model]
- This internal model is simple “Auto Regressive (AR)” model.
- [explain more about this term in detail]
- LS - Cost Function
- Now we need cost function.
- [explain that this cost function will come from comparing two probability distribution]
- [derive “KL divergence”]
- ref: https://gregorygundersen.com/blog/2019/01/22/kld/
-
[explain that KL divergence is differentiable and hence can be used as a cost function.]
- [From this cost function derive log likelihood and talk about “MLE”]
- About likelihood
- About log likelihood
- About Maximum Likelihood Estimate (MLE)
- [Explain all these terms. Explaining why these terms are called this way, will explain the “statistical derivation” of this cost function, which we should be talking about in other ramp like in some bayesian based algorithm ramp].
- LS - SGD
- [try optimizing via SGD]
- [turns out, it will be very slow optimization process]
- Since it is slow, we will keep stopping the training and check our model.
- We will find that our model is very correctly outputting the probability distributions regarding the black area around a digit. Why? [explain that - because all candidates agree with it].
- [explain that “surrounding” distributions will converge way faster than “central” distributions. Here “surrounding” means black area surrounding a digit and “central” already defined above]
- [Now explain that probability distributions near central area are very different for every unique conditions. Even if they are very subtly difference for us but for model they are totally different. Because of categorical distribution and not something like Gaussian distribution]
- This gives us an idea that this itself lead us to less spiky distribution and we do not have to make it less spiky ourselves by adding appropriate errors manually.
-
It means we have to stop training before convergence. And we have to train it enough such that surrounding distributions are converged and central distributions are close to convergence but not converged.
- [also mention that full convergence means overfitting to given dataset and it would just become a naive model because NNs are too flexible]
[Steps about making SGD convergence faster. And fixing manual tuning of learning rate. Which includes steps about momentum, nesterov etc. to Adam]
- LS - Adam
- [derive Adam here]
- Adam optimizer train the model faster than SGD.
- LS - Evaluation
- [mention about quantitative and qualitative evaluation]
- [show that AR model indeed generates some novel candidates, but also generates “non-candidate” \(x\)]
Note:- To be honest, I haven’t read much about autoregressive model. I have put autoregressive model in this ramp because it is very intuitive to understand and also very easy to implement. Therefore, it works as a good but non-trivial (because NN is involved) baseline to compare with VAE. Being intuitive does not mean that I should be able to logically derive it easily. Not sure if I how derived autoregressive model in the best way possible. I have to go through its resources to get better inspiration. I will update this ramp when I feel that there is a better way to derive it.
- Take Away
- Mastered a simple AR model
- Which lead us to introduction and mastery of these topics
- generation problem and generative algorithm
- probability distribution
- joint distribution
- conditional probability
- and conditional independence
- Sampling process
- How to model probability distribution given a dataset.
- Softmax activation function
- KL divergence
- MLE
- Adam
- etc.