1
$\begingroup$

I have two computers over which I would like to distribute some large-scale computations. On each computer there is a TCP server, written in another language, that listens for requests and return the results.

The first strategy I tried was the following:

(* Connect to socket on computer 1 *)
sockOnComp1 = SocketConnect[{"ip-addr-1", portnum}, "TCP"];
(* Connect to socket on computer 2 *)
sockOnComp2 = SocketConnect[{"ip-addr-2", portnum}, "TCP"];

largeLst = Range[1000];

(* Master function *)
parFunc[sock_, lst_] := "return result of long computation on lst";

DistributeDefinitions[parFunc];

res = WaitAll[{
    ParallelSubmit[
        {sockOnComp1, largeLst},
        parFunc[sockOnComp1, largeLst[[1;;500]]]
    ],
    ParallelSubmit[
        {sockOnComp2, largeLst},
        parFunc[sockOnComp2, largeLst[[501;;-1]]]
    ]
}];

But this strategy does not parallelise the calculations over the two computers. It always first finishes the calculations on computer 1, and only then starts the computations on computer 2, whereas I expect the two sets of calculations to proceed in parallel on the two different computers.

I assumed that this failure to parallelise was the result of the sockets themselves having been created on the main kernel, not on the parallel kernels. So, I tried a different strategy:

(* New master function *)
parFuncNew[{ipAddr_, portnum_}, lst_] := With[{
    sock = SocketConnect[{ipAddr, portnum}, "TCP"]
},
    "return result of long computation on lst"
];

But this strategy never triggers the computations, because SocketConnect always returns an inactive SocketObject when called within ParallelSubmit. SocketConnect appears to return a valid, active SocketObject only when it is called from the main kernel, as can be seen from the image below:

Socket objects

What is the correct way to parallelise computations that need socket connections?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.