0
$\begingroup$

So I'm working with a rather large dataset (perhaps not really by ML standards - but too big to fit into my computer's RAM at any rate). And so, I train the model by successively loading a subsample of the dataset and calling model.fit using that data. This means I call model.fit quite regularly and it seems to take significantly more time to initialize model.fit than it does to perform 10 epochs of training (on the data subset).

I'm wondering if there any solutions or tricks to improve initialization time? Should I input the data in a certain way (say tensorflow dataset object instead of numpy arrays) or employ model.train_on_batch in some fashion instead?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I think in your use-case it would make more sense to use a create a custom data generator (or an existing one, such as ImageDataGenerator, depending on your data) which will load data into memory as needed. $\endgroup$ Commented Apr 9, 2024 at 16:21
  • $\begingroup$ See for instance this question $\endgroup$ Commented Apr 9, 2024 at 18:47

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.