Otdfbt

I am trying to make a CNN in Keras, and to test the validity of my model i am trying to get it to train on MNIST dataset, so i am sure that everything is working fine, but unfortunately model is barely training and i suspect that nothing updating.
My model is :

model=Sequential()

#conv1_1
model.add(Conv2D(128,kernel_size=3, strides=1,
 padding='SAME', use_bias=False, 
 activation='relu',name='conv1_1',input_shape=(28,28,1)))
#conv1_2
model.add(Conv2D(128, kernel_size=3, strides=1,
 padding='SAME', use_bias=False, 
 activation='relu',name='conv1_2'))
model.add(MaxPooling2D(pool_size=2,strides=2))

#conv2_1
model.add(Conv2D(64, kernel_size=3, strides=1,
 padding='SAME', use_bias=False, 
 activation='relu',name="conv2_1"))

#conv2_2
model.add(Conv2D(64, kernel_size=3, strides=1,
 padding='SAME', use_bias=False, 
 activation='relu',name='conv2_2'))
model.add(MaxPooling2D(pool_size=2,strides=2))

model.add(Flatten())
model.add(Dense(1024, activation='relu',name='Dense1'))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu',name='Dense2'))
model.add(Dense(10, activation='softmax',name='output'))

Compiled with:

model.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=['accuracy'])

model.fit(X_train,y_train,batch_size=10,validation_split=0.2,epochs=10)

My X_train and y_train look like:

plt.imshow(X_train[0].reshape(28,28))
plt.show()

y_train[0]
array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.])

Here are the Results of first 3 epochs:

Epoch 1/10
48000/48000 [==============================] - 45s 927us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 2/10
48000/48000 [==============================] - 44s 915us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 3/10
48000/48000 [==============================] - 44s 924us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 4/10
48000/48000 [==============================] - 45s 930us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060

This is my first Keras Model, and i think i am missing something important here.

There are two things I can suspect. First, the dropout rate at the last layer seems way to high. Its better to have a lower dropout rate after each CNN layer. Secondly, you should use a bias in your CNN layers.

Try out this code as a starting point and then you can start tuning your model from here.

Load the data

from keras.datasets import mnist
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.


print('Training data shape: ', x_train.shape)
print('Testing data shape : ', x_test.shape)

Import Keras stuff

import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

Now we reshape the data such that it can fit with the tensorflow backend. This requires the channel to be the last dimension. We will also set up our one-hot encoded outputs

# The known number of output classes.
num_classes = 10

# Input image dimensions
img_rows, img_cols = 28, 28

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test_reshaped = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

Define the model

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
 activation='relu',
 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
 optimizer=keras.optimizers.Adadelta(),
 metrics=['accuracy'])

Train the model

epochs = 10
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
 batch_size=batch_size,
 epochs=epochs,
 verbose=1,
 validation_data=(x_test_reshaped, y_test_binary))

Evaluate the model

score = model.evaluate(x_test_reshaped, y_test_binary, verbose=0)
print('Model accuracy:')
print('Test loss:', score[0])
print('Test accuracy:', score[1])

I have implemented your model to the astonishment, there is a very minute error that is hard to notice.

The way, I was able to get better accuracy is by changing the optimizer to "SGD" or "ADAM".

As you have used "ADADELTA" which is an extension of "ADAGRAD" optimizer. In "ADAGRAD" has good performs on sparse data & while training a large scale neural network. Its monotonic learning rate usually proves too aggressive, stops learning too early.

Refer to this link for understanding on optimizers