1
$\begingroup$

I am trying to reproduce results from one paper, where authors minimized the following loss function \begin{align} \min_{w \in R^d} \frac{1}{n} \sum_{i \in [n]} log(1 + exp(-y_ix_i^Tw))+\frac{\lambda}{2}\|w\|^2, \end{align} where $w$ are weights are $\lambda$ is regularization parameter for ijcnn1 dataset.

This dataset is specific by unbalanced data (90%-0, 10%-1). As a preprocessing step, I applied MinMaxScaler and StandardScaler.

I use model written in keras, which looks simply as:

model = keras.Sequential([
    keras.layers.Dense(1, activation="sigmoid", kernel_initializer='uniform', kernel_regularizer=regularizers.l2(1e-4), use_bias=True)
])
sgd = optimizers.SGD(lr=0.05, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(optimizer=sgd, loss='binary_crossentropy', metrics=['accuracy'])
model.fit(train_dataset, epochs=1000)

However, I manage to obtain only 91% accuracy at the best. Looking at the prediction, I observe that my model learns to predicts almost everything to be zero. I also tried using class_weight, but it did not seem to help. Does anybody have any suggestions, how to obtain better results?

Improve this question
$\endgroup$
2
  • $\begingroup$ Are your classes balanced? Or do you have a disproportionate amount of zeros? $\endgroup$ Commented Mar 30, 2020 at 10:00
  • $\begingroup$ no, classes are not balanced, there is disproportionate amount of zeros $\endgroup$ Commented Mar 30, 2020 at 10:15

1 Answer 1

Reset to default
1
$\begingroup$

If you have a disproportionate amount of zeros, it means you models doesn't have enough data in order to learn how to correctly classify observations. Because it sees zeros almost all the time, it probably has learned to output zeros.

The main thing you can do to solve this problem is to train your model using mini-batches, and build them by sampling 0- and 1-observations with equal probability - in other words, attributing greater weight to 1-observations compared to zeros. In that way, your model will be fed with balanced data and will learn to classify them correctly.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi, thanks for your recommendation. It helped but I still can not reach higher accuracy than 93%. Are there any other tricks, which I could use? $\endgroup$ Commented Mar 30, 2020 at 14:26
  • $\begingroup$ Ensembles can usually boost performance. Train more than one model, possibly with different features/architectures, and combine their prediction (simple average should do the job). $\endgroup$ Commented Mar 30, 2020 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.