Can transfer learning be applied after learning using homomorphic encryption to obfuscate dataset source?

Context

Suppose one has a public dataset plant_labels with:

• Input: plant pictures
• Labels: plant names

And a larger public model: object for object recognition with:

• Input: object picturesmodel
• Labels: object names

which has been trained on various public datasets, except for plant_labels.

Scenarios

Suppose 2 different scenarios:

1. One applies homomorphic encryption to the plant_labels dataset and trains a model HE_plant_I on it. Next, one applies transfer learning to improve the larger model object with it.
2. One first trains model plant_II without homomorphic encryption, then applies transfer learning to towards a HE_plant_II model with homomorphic encryption, and then applies transfer learning again from the HE_plant model towards the object model to improve it.

Assumptions

I assume either 1 and/or 2 is possible, if this assumption is invalid, feel free to clarify that.

Question

Assuming either 1 and/or 2 is/are possible, is it possible to do in such a way that others* cannot determine, or prove, on which dataset you have trained the HE_plant_I (or HE_plant_II?

Others who also have access to all the datasets on which object is trained, as well as access to dataset plant, (but e.g. not to your random seed used for training).