python multiprocessing:如何从子流程修改在主流程中创建的字典?

问题描述

此问题与以下内容有关:多重处理:如何在多个进程之间共享一个字典?

This question is related to: multiprocessing: How do I share a dict among multiple processes?

我在一个多处理Dict中存储了多个numpy数组.声明了多处理字典，并在预定义键处填充了numpy数组.每个子进程仅在dict的单个键中写入和修改数据.子进程甚至不认为子进程似乎在做某些事情(在主进程中声明的字典的存储位置上不应该将字典修改为"inplace"吗?).

I have multiple numpy arrays stored in a multiprocessing Dict. The multiprocessing dict is declared and populated with numpy arrays at predefined keys. each subprocess only writes and modifies data in a single key of the dict. The dictionary isnt updated by the subprocesses even thought the subprocesses seem to do something(shouldn't the dictionary be modified "inplace", at the memory location of the dictionary declared in the main process?).

我不明白为什么它不起作用；字典中包含的数据被复制到每个子流程，然后在其中进行修改，而不返回主流程吗?如果是这样，是否有一种方法可以修改数据而不将其复制到其他地方?在多处理中，当多个进程尝试写入同一地址时，可能会出现不想要的数据删除的问题，在我的情况下，由于每个子进程仅写入特定的键，这种不想要的数据删除会成为问题吗?

I do not understand why it isn't working; is the data contained in the dict copied to each subprocess then modified in it and not returned to the main process? if that's the case, is there a way to modify the data without copying it somewhere else? In multiprocessing there may be a problem with unwanted data deletion when multiple processes try to write to the same adress, in my case, since each subprocess only writes to a specific key, will this unwanted data deletion be a problem?

示例代码:

    
import datetime
import numpy as np
import random
from multiprocessing import Process,Manager

class nbrgen(object):
    def __init__(self,ticker,TBA,delay):
        self.delay=delay
        self.value=100.00
        self.volume=50
        self.ticker=ticker
        self.TBA=TBA

    def generate_value(self):
        self.value=round (self.value + random.gauss(0,1)*self.delay + 0.01 ,2)
        self.volume=random.randint(1,100)

    def __next__(self):
        return self.next()

    def next(self):
        self.generate_value()
        t=datetime.datetime.now(tz=datetime.timezone.utc)
        return np.array([t,self.ticker,self.TBA,self.value,self.volume])

    def apenddict(D, tik,gnr):
      for i in range(8):
        print(tik)
        D[tik][:-1] = D[tik][1:]
        D[tik][-1:, :] = gnr.next()


    if __name__ =="__main__":
     manager=Manager()
     d=manager.dict()
     d["TOK"] = np.zeros((10, 5), dtype="O")
     d["TIK"] = np.zeros((10, 5), dtype="O")
 
     p1=Process(target=apenddict,args=(d,"TIK",nbrgen("TIK","T",0.1)))
     p2=Process(target=apenddict,args=(d,"TOK",nbrgen("TOK","T",0.1)))

     p1.start()
     p2.start()
     p1.join()
     p2.join()

     print(d)

打印:TIK和TOK随机(按预期)和

prints: TIK and TOK randomly (as expected) and

{'TOK': array([[0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0]], dtype=object), 'TIK': array([[0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0],
   [0, 0, 0, 0, 0]], dtype=object)}

已返回

推荐答案

这是由于 Manager()管理 dict 的方式引起的.仅管理 dict 本身，而不管理其条目(除非条目碰巧是代理对象).因此，当使用可变字典条目时，管理器不会处理(甚至注册)对象的更新.

This is due to the way the Manager() manages the dict. Only the dict itself is managed, not its entries (unless the entries happen to be Proxy Objects). Thus when working with mutable dict entries, updates to the objects are not handled (or even registered) by the manager.

但是，可以更新字典本身，例如通过替换完整的 dict 条目.因此，一种解决上述限制的方法是，获取条目的本地引用，对其进行更新，然后将本地引用重新分配给托管的 dict .

However, it is possible to update the dict itself, e.g. by replacing a full dict entry. Thus, a way to get around the above limitation, is to get a local reference to the entry, update it, and then re-assign the local reference to the managed dict.

在这种情况下，这意味着 appenddict 函数必须稍作修改:

In this particular case, it would mean that the appenddict function has to be modified slightly:

def apenddict(D, tik,gnr):
    tik_array = D[tik]  # Create local reference
    for i in range(8):
        print(tik)
        tik_array[:-1] = D[tik][1:]  # Update local reference
        tik_array[-1:, :] = gnr.next()  # Update local reference
    D[tik] = tik_array  # Assign local reference to managed dict

有了这些更改，您的程序应该可以按预期的方式工作.

With these changes, your program ought to work more as you expect it to.

这篇关于python multiprocessing:如何从子流程修改在主流程中创建的字典?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！