Daily Archives: December 6, 2016

Logistic Regression

Logistic Regression answer the YES/NO question. For example, giving a set of size of tumor, it answers if it is a tumor. Giving height and weight of a person, answer if it is a man.

Hypothesis

We have hypothesis function logistic_regression1, and it ranges  logistic_regression2. And we define the answer is YES when hypothesis is greater than 0.5, else is NO.
logistic_regression3.

First, let’s take a look at the shape of logistic function(sigmoid function) g(z).
logistic_regression7

It is like:
logistic_regression8

We found that this function can satisfy our need. g(z) ranges from (0, 1).
We have z>=0  –>  g(z)>=0.5  –>  predict y=1,
And we have  z<0  –>  g(z)<0.5  –> predict y=0

Having chose the logistic function, let’s define hypothesis for logistic regression:
logistic_regression4

Let’s do some analysis. When logistic_regression5, we have logistic_regression6.
For this hypothesis, we know when 1+x1-x2>=0, hypothesis is equal to or greater than 0.5, prediction is YES; else hypothesis is less than 0.5, prediction is NO. Take a look at below picture, we know when the point is from region A, the answer is YES; when the point is from region B, the answer is NO.

logistic_regression9

Another example, when logistic_regression10, we have logistic_regression11.
For this hypothesis, we know when x1^2 + x2^2>=1, hypothesis is equal to or greater than 0.5, prediction is YES; else hypothesis is less than 0.5, prediction is NO. Take a look at below picture, we know when the point is from region A, the answer is YES; when the point is from region B, the answer is NO.
logistic_regression12

Cost function

Cost function for logistic regression is:
logistic_regression13
For y=1 case, if the hypothesis is is close to 0, we know the cost function will be close to positive infinity. If hypothesis is close to 1, then cost function will be close to 0.
For y=0 case, if the hypothesis is is close to 0, we know the cost function will be close to 0. If hypothesis is close to 1, then cost function will be close to positive infinity.

Furthermore, cost function can be rewritten as:

logistic_regression14

And J of theta can be written as:
logistic_regression15

Since we have the cost function, we can solve the logistic regression by gradient descent, repeating below loop until theta becomes to a value which is small enough.
logistic_regression16

Or logistic regression can be solved by some optimization algorithms(Conjugate gradient, BFGS or L-BFGS).

Learning materials: Andrew NG Machine Learning course.