I have two computers over which I would like to distribute some large-scale computations. On each computer there is a TCP server, written in another language, that listens for requests and return the results.
The first strategy I tried was the following:
(* Connect to socket on computer 1 *)
sockOnComp1 = SocketConnect[{"ip-addr-1", portnum}, "TCP"];
(* Connect to socket on computer 2 *)
sockOnComp2 = SocketConnect[{"ip-addr-2", portnum}, "TCP"];
largeLst = Range[1000];
(* Master function *)
parFunc[sock_, lst_] := "return result of long computation on lst";
DistributeDefinitions[parFunc];
res = WaitAll[{
ParallelSubmit[
{sockOnComp1, largeLst},
parFunc[sockOnComp1, largeLst[[1;;500]]]
],
ParallelSubmit[
{sockOnComp2, largeLst},
parFunc[sockOnComp2, largeLst[[501;;-1]]]
]
}];
But this strategy does not parallelise the calculations over the two computers. It always first finishes the calculations on computer 1, and only then starts the computations on computer 2, whereas I expect the two sets of calculations to proceed in parallel on the two different computers.
I assumed that this failure to parallelise was the result of the sockets themselves having been created on the main kernel, not on the parallel kernels. So, I tried a different strategy:
(* New master function *)
parFuncNew[{ipAddr_, portnum_}, lst_] := With[{
sock = SocketConnect[{ipAddr, portnum}, "TCP"]
},
"return result of long computation on lst"
];
But this strategy never triggers the computations, because SocketConnect always returns an inactive SocketObject when called within ParallelSubmit. SocketConnect appears to return a valid, active SocketObject only when it is called from the main kernel, as can be seen from the image below:
What is the correct way to parallelise computations that need socket connections?
