# Using Logistic Regression for Image Classification

Machine Learning Project

# SYSTEM REQUIREMENTS

- Basic python programming skills
- Basic knowledge of logistic regression
- Numpy python library
- Pandas python library
- Matplotlib python library

Before anything else add this lines of code at the top of your file

`import pandas as pd`

import numpy as np

import matplotlib.pyplot as plt

In case you need it you can use the command “pip install *package_name” *to set up your work environment.

# PROBLEM STATEMENT

The idea of this project is to develop and train a model that is able to take the pixel values of a digit and identify if it is an image of the digit one or not.

# SET UP OF DATA

Here you can download the dataset

Each row of the dataset represents the flattened pixel values of a digit, it goes from zero to nine but we will not use all of them because the goal of the binary classifiction model is return 1 if the digit is one and 0 otherwise.

To retrieve the digits data (X) and labels (Y) we must add in our code the following lines:

df_x= pd.read_excel('dataset.xlsx', 'X',header=None)

df_x.head()df_y= pd.read_excel('dataset.xlsx', 'Y',header=None)

df_y.head()

Because this model will identify the digit 1 only, in the labels we need to convert the rest of digits that are not one to zero.

`y = df_y[0]`

for i in range(len(y)):

if y[i] != 1:

y[i] = 0

y = pd.DataFrame(y)

From the 5000 rows of data, 4000 will be used to train the model and the remaining 1000 will be used to test it, because is important for the model to be tested by unseen data. To do this we must add the following lines to our code:

x_train = df_x.iloc[0:4000].T

x_test = df_x.iloc[4000:].T

x_train = np.array(x_train)

x_test = np.array(x_test)y_train = y.iloc[0:4000].T

y_test = y.iloc[4000:].T

y_train = np.array(y_train)

y_test = np.array(y_test)

These training and test datasets are in DataFrame form so they need to be in an array format for the convenience of calculation. Also it is worth noting that we are using .T to take the transpose of each dataset.

Once the training and test datasets are ready to be used it is time to focus on the model itself. To give it form we must define some formulas.

**FORMULAS**

We start from the basic linear regression formula:

Where Y is the output, X is the independent variable, A is the slope and B is the intercept. But in logistic regression variables are expressed diferrently:

Where:

- z is the output variable
- x is the input variable
- w and b will be initialized as zeros and they will be modified while training the model.

Z should be passed through a non-linear function, this time we will use the sigmoid function that returns a value between 0 and 1.

It is worth saying that this *a* will be the final output that is the value in the *y_train* or *y_test*.

To define the sigmoid function in our code we add at the top:

`def sigmoid(z):`

s = 1/(1 + np.exp(-z))

return s

About *w* and *b* being initialized as zeros let’s define a function to do it:

*def* initialize_with_zeros(*dim*):

w = np.zeros(*shape*=(dim, 1))

b = 0

return w, b

Cost function and Gradient descent

The cost function is a measure of a model that reflects how much the predicted output differs from the original output. The model aim to lower the cost function value and for that we use the following formula for all the rows.

Furthermore, we need to update the values of *w* and *b* in the formula 2. These values would be initialized as zeros but they will need more appropriate values and gradient descent will help us with that.

In Formula 4, the cost function is expressed as a function of *a* and *y,* but it can also be expressed as a function of *w* and *b* as well.

The differential *w* and *b* will be derived by taking the partial differentiation of cost function with respect to *w* and *b*.

# CODING THE MODEL

Once all the formulas are set, let’s put it all together in a function called *propagate* by adding the next lines in our code

defpropagate(w,b,X,Y):

#Find the number of training data

m = X.shape[1] #Calculate the predicted output

A = sigmoid(np.dot(w.T, X) + b) #Calculate the cost function

cost = -1/m * np.sum(Y*np.log(A) + (1-Y) * np.log(1-A)) #Calculate the gradients

dw = 1/m * np.dot(X, (A-Y).T)

db = 1/m * np.sum(A-Y)

grads = {"dw": dw, "db": db}

return grads, cost

This function calculates:

- A: Predicted output
- cost: Cost function
- dw and db: Gradients

By using this function now we are able to update *w* and *b* in Formula 2.

Optimize the parameters

To best fit the training data we will update the parameters which are the core of this model. The *propagate* function will be run through a number of iterations. In each iteration, *w* and *b* will be updated.

defoptimize(w,b,X,Y,num_iterations,learning_rate):

costs = []

#propagate function will run for a number of iterations

for i in range(num_iterations):

grads, cost = propagate(w, b, X, Y)

dw = grads["dw"]

db = grads["db"]

#Updating w and b by deducting the dw and db

#times learning rate from the previous w and b

w = w - learning_rate * dw

b = b - learning_rate * db #Record the cost function value for each 100 iterations

if i % 100 == 0:

costs.append(cost) #The final updated parameters

params = {"w": w,"b": b}

#The final updated gradients

grads = {"dw": dw,"db": db}

return params, grads, costs

The term *learning rate* was introduced in this function. It’s not a calculated value. It can vary depending the different machine learning algorithms. Later in the conclusions I will show the impact it has.

Predict the output

By now we can optimize the parameters, reduce the cost function and update the values of gradients, so is the time to predict the output.

defpredict(w,b,X):

m = X.shape[1]

w = w.reshape(X.shape[0], 1) #Initializing an aray of zeros which has a size of the input

#These zeros will be replaced by the predicted output

Y_prediction = np.zeros((1, m))

#Calculating the predicted output using the sigmoid function

#This will return the values from 0 to 1

A = sigmoid(np.dot(w.T, X) + b) #Iterating through A and predict an 1 if the value of A

#is greater than 0.5 and zero otherwise

for i in range(A.shape[1]):

Y_prediction[:, i] = (A[:, i] > 0.5) * 1 return Y_prediction

Model

By putting all the functions together the final model will look like this.

defmodel(X_train,Y_train,X_test,Y_test,num_iterations,learning_rate):

#Initializing the w and b as zeros

w, b = initialize_with_zeros(X_train.shape[0]) #Best fit the training data

parameters, grads, costs = optimize(w, b, X_train, Y_train, num_iterations, learning_rate)

w = parameters["w"]

b = parameters["b"] # Predicting the output for both test and training set

Y_prediction_test = predict(w, b, X_test)

Y_prediction_train = predict(w, b, X_train) #Calculating the training and test set accuracy by comparing

#the predicted output and the original output

print("train accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100))

print("test accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100)) d = {"costs": costs,

"Y_prediction_test": Y_prediction_test,

"Y_prediction_train" : Y_prediction_train,

"w" : w,

"b" : b,

"learning_rate" : learning_rate,

"num_iterations": num_iterations} return d

Using the model

To use the model and see how well it works let us pass our data that we prepared at the beginning by adding the next lines of code

`ni = 500 # num iterations`

lr = 0.005 # learning rate

d = model(x_train, y_train, x_test, y_test, ni, lr)

Once the code it is executed it will appear on the console something like this

`train accuracy: XX%`

test accuracy: XX%

Percentages may vary depending on the number of iterations and learning rate you use.

Plotting data

Is worth noting the *model* function returns a dictionary that contains:

- Costs
- Final parameters
- Predicted Outputs
- Learning Rate
- Number of iterarions used

So, let us see how cost function changed with each updated ‘w’s and ‘b’s by adding the next lines of code.

`#Plot how cost function changed each updated w's and b's`

plt.figure(*figsize*=(7,5))

plt.scatter(*x* = range(len(d['costs'])), *y* = d['costs'], *color*='black')

plt.title('Scatter Plot of Cost Functions', *fontsize*=18)

plt.ylabel('Costs', *fontsize*=12)

plt.show()

After doing some tests changing values I got the following results.

# CONCLUSIONS

- With each iteration, the cost function went down as it should, that means the parameters
*w*and*b*kept refining towards perfection. - It is worth noting that increasing the value of the learning rate will get us a better % at training accuracy but it will be lower on test accuracy. For this particular case I recommend using 0.015 that had both percentages more even.
- To change the model for recognizing another digit apart from 1 you can go to the line where it changes all numbers that are not 1 to zero, but instead of 1 use the digit of your preference.
- Reducing the number of rows for training the model will underfit it and if we increase the number it will overfit it. But you can try and check how it behaves.

# REFERENCES

- https://towardsdatascience.com/logistic-regression-detailed-overview-46c4da4303bc
- https://www.statisticssolutions.com/what-is-logistic-regression/
- https://towardsdatascience.com/a-complete-logistic-regression-algorithm-from-scratch-in-python-step-by-step-ce33eae7d703
- https://medium.com/swlh/logistic-regression-for-image-classification-e15d0ae59ce9