Pickling errors when using multiprocessing

Question

I want to control and run two pieces of lab equipment at the same time (at least starts them simultaneously), so I use multiprocessing module.

from myLabModule import Newport1936R
from myLabModule import DCx_camera
from multiprocessing import Process, Queue

def multiprocess_camera(camera, shots_per_img, queue, delay):
    time.sleep(delay)
    img, camera_time = camera.capture(shots_per_img, np.float64, True) #img is a 2x2 ndarray
    results = [camera_time[0], camera_time[1], img, 'img']
    queue.put(results)

def multiprocess_power(newport, interval, N, queue, delay):
    time.sleep(delay)
    power_reading, newport_time = newport.get_power(interval, N, True)
    results = [newport_time[0], newport_time[1], power_reading, 'power']
    queue.put(results)

if __name__ == "__main__":
    thorcam = DCx_camera()
    newport = Newport1936R()
    queue = Queue()
    p1 = Process(target=multiprocess_camera, args=(thorcam,10,queue,0))
    p2 = Process(target=multiprocess_power, args=(newport,1,10,queue,0))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    results = []
    results.append(queue.get())
    results.append(queue.get())

When I run the code, I get the following error: AttributeError: Can't pickle local object 'CDLL.__init__.<locals>._FuncPtr'. I read this post, it seems that it has to do with the scope of the variables (I am passing two classess defined in another module to the functions, and the functions basically run the classes' methods). What I'm not sure exactly which part is causing the problem. How should I modify the code to make it work?

why do you have to create instances and then forward them just to call their methods with forwarded arguments? Looks like that is the issue, and if that is not a strict requirement, why don't you make instances in functions? — Ilija
– Ilija, Commented Feb 16, 2018 at 18:13
@ikac Sorry I don't quite understand what you mean. As I said what I wanna achieve at the end is to execute thorcam.capture() and newport.get_power() at the same time (using multiprocessing) and also get their return values. I just checked a few relevant posts and it seems that making a separate functions is the way to do it, of course I'm open to other better methods. — Physicist
– Physicist, Commented Feb 17, 2018 at 9:00
Well why don't you create instances in target methods? If real use case is similar to this example, you do not have code referencing instances thorcam and newport bellow processes are started. — Ilija
– Ilija, Commented Feb 17, 2018 at 11:57
Do mean defining thorcam and newport in the current script? I can do so, the only problem is the module I have written for thorcam and newport are a few hundred lines long, so I think it looks cleaner if I keep them as separate files. Is it possible to just redefine the classes in a few lines and referring the actual code to the separate files? — Physicist
– Physicist, Commented Feb 18, 2018 at 9:57
I mean to not have to worry about pickling at all. Here, pickling is performed when you pass instance as an argument for target, e.g. args=(thorcam,10,queue,0). thorcam has to be pickled and I'm not sure how DCx_camera is implemented. — Ilija
– Ilija, Commented Feb 18, 2018 at 12:47

Ilija · Accepted Answer · 2018-02-18 16:24:21Z

In this case pickling is performed when passing arguments to target method that will be executed as new (child) process.

To be precise, I'm not sure if pickling instances of DCx_camera and Newport1936R is possible at all. Depends on how those classes are defined, but looking at the error message I suppose you can't. That is why you can try to not pass those references at all. It seems, by the example, that you do not need to.

Here is what I expect to work:

from myLabModule import Newport1936R
from myLabModule import DCx_camera
from multiprocessing import Process, Queue

def multiprocess_camera(shots_per_img, queue, delay):
    camera = DCx_camera()
    time.sleep(delay)
    img, camera_time = camera.capture(shots_per_img, np.float64, True) #img is a 2x2 ndarray
    results = [camera_time[0], camera_time[1], img, 'img']
    queue.put(results)

def multiprocess_power(interval, N, queue, delay):
    newport = Newport1936R()
    time.sleep(delay)
    power_reading, newport_time = newport.get_power(interval, N, True)
    results = [newport_time[0], newport_time[1], power_reading, 'power']
    queue.put(results)

if __name__ == "__main__":
    queue = Queue()
    p1 = Process(target=multiprocess_camera, args=(10,queue,0))
    p2 = Process(target=multiprocess_power, args=(1,10,queue,0))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    results = []
    results.append(queue.get())
    results.append(queue.get())

Is this something that can get you desired results?

There is an alternative, according to this link.

Better to inherit than pickle/unpickle

However, one should generally avoid sending shared objects to other processes using pipes or queues. Instead you should arrange the program so that a process which needs access to a shared resource created elsewhere can inherit it from an ancestor process.

That is why I think you can even have global variables that reference instances of DCx_camera and Newport1936R, and then use those in both target methods.

Collectives™ on Stack Overflow

Pickling errors when using multiprocessing

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related