Otdfbt

I'm working on my first NN following a tensorflow tut and trying to use my own data.
After about 80 attempts of formatting my data and trying to load it into a dataset to train I'm throwing the towel.

Here is how my data currently looks

syslog_data = [
[302014,0,0,63878,30,3,1], [302014,0,0,3891,0,0,0], [302014,0,0,15928,0,0,2], [305013,5,0,123,99999,0,3],
[302014,0,0,5185,0,0,0], [305013,5,0,123,99999,0,3], [302014,0,0,56085,0,0,0], [110002,4,2,50074,99999,0,4],

In this the last item in each list is the label.
If you can tell me if I need to reformat my data and how or just how to get it loaded into a dataset properly.

Thanks for any help or advice you can give

Here is the full code:

import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from . import syslog

print(tf.VERSION)
print(tf.keras.__version__)

model = tf.keras.Sequential()
# Adds a densely-connected layer with 64 units to the model:
model.add(layers.Dense(64, activation='relu'))
# Add another:
model.add(layers.Dense(64, activation='relu'))
# Add a softmax layer with 10 output units:
model.add(layers.Dense(10, activation='softmax'))

model.compile(optimizer=tf.train.AdamOptimizer(0.001),
 loss='categorical_crossentropy',
 metrics=['accuracy'])

dataset = tf.data.dataset.from_tensor_slices(syslog)

model.fit(dataset, epochs=10, steps_per_epoch=30)

There are a couple of problems and things you might want to add to your existing script.

Below I separate your example data into two NumPy arrays:

• input values x
  


• labels y
  

It is also important to make sure they are of type float32, because Tensorflow will complain if you pass it integers (as they otherwise would be interpreted).

The following works for me, the model trains to completion:

import numpy as np
import tensorflow as tf
from tensorflow.keras import layers

syslog_data = [
 [302014, 0, 0, 63878, 30, 3, 1],
 [302014, 0, 0, 3891, 0, 0, 0],
 [302014, 0, 0, 15928, 0, 0, 2],
 [305013, 5, 0, 123, 99999, 0, 3],
 [302014, 0, 0, 5185, 0, 0, 0],
 [305013, 5, 0, 123, 99999, 0, 3],
 [302014, 0, 0, 56085, 0, 0, 0],
 [110002, 4, 2, 50074, 99999, 0, 4],
]

print(tf.VERSION)
print(tf.keras.__version__)

x = np.array([arr[:-1] for arr in syslog_data], dtype=np.float32)
y = np.array([arr[-1:] for arr in syslog_data], dtype=np.float32)

model = tf.keras.Sequential()
# Adds a densely-connected layer with 64 units to the model:
model.add(layers.Dense(64, activation="relu"))
# Add another:
model.add(layers.Dense(64, activation="relu"))
# Add a softmax layer with 10 output units:
model.add(layers.Dense(10, activation="softmax"))

model.compile(optimizer=tf.train.AdamOptimizer(0.001), loss="categorical_crossentropy", metrics=["accuracy"])

model.fit(x, y, epochs=10, steps_per_epoch=30)

import keras
import numpy as np
full_data = np.array(syslog_data)
X = full_data[:,:6]
Y = full_data[:,6]
# Convert labels to categorical one-hot encoding
one_hot_labels = keras.utils.to_categorical(Y, num_classes=10)

model.fit(X,Y, epochs=10, steps_per_epoch=30)

Does this work? I think I might be misunderstanding the problem.