multiprocessing.pool.MapResult._number_left没有给出我期望的结果

问题描述

我对应该返回的_number_left感到困惑.我假设这是池中剩余的任务数量，但是它似乎无法在我的代码中提供正确的值.例如，如果我有一个由10个工人组成的池，它们的数量为1000，那么我希望result._number_left从1000开始倒数​​.但是，它只告诉我在代码即将完成之前我还有40个工人.我在这里想念什么吗?

I am confused as to what _number_left is supposed to return. I assumed it was the number tasks remaining in the pool, but it does not appear to provide the correct value in my code. For example, if I have a pool of 10 workers counting to the number 1000, I would expect result._number_left to countdown from 1000. However, it only tells me I have 40 left until the code is about finished. Am I missing something here?

代码:

import multiprocessing
import time


def do_something(x):
    print x
    time.sleep(1)
    return


def main():
    pool = multiprocessing.Pool(10)

    result = pool.map_async(do_something, [x for x in range(1000)])
    while not result.ready():
        print("num left: {}".format(result._number_left))
        result.wait(timeout=1)


if __name__ == "__main__":
    main()

推荐答案

首先，_number_left是未公开类的未公开私有属性.您没有理由期望它具有任何特殊含义.

First, _number_left is an undocumented private attribute of an undocumented class. There's no reason you should expect it to have any particular meaning.

如果您查看未公开文档的来源MapResult 类，您可以看到它在CPython 3.6中的使用方式.

If you look at the source for the undocumented MapResult class, you can see how it's used in CPython 3.6 in particular.

首先，它被初始化:

self._number_left = length//chunksize + bool(length % chunksize)

因此，已经很明显，它永远不会是您迭代的长度.这将是映射整个可迭代项所需的块的预期数量.然后，只要调用_set，它就会从那里开始递减计数，这很复杂，但是显然每个值都不是一次.

So, already, it's pretty clear that it's never going to be the length of your iterable; it's going to be the expected number of chunks needed to map the whole iterable. Then, it counts down from there whenever _set is called, which… well, that's pretty complicated, but it's clearly not once per value.

无论如何，无论您要做什么，都可能有一种实际的方法，而无需查看私有属性并猜测它们可能意味着什么.例如，如果您只想获得一些进展，但是不能使用imap_unordered，因为您需要将结果放在顺序表的末尾，则很容易从中构建顺序表:只需传递enumerate(iterable)在其中进行修改或包装func以返回索引和值，然后对返回的结果进行排序.

At any rate, whatever you're trying to do, there probably is a way to actually do it, without peeking at private attributes and guessing at what they might mean. For example, if you just want to get some progress, but can't use imap_unordered because you need the results in an ordered list at the end, it's pretty easy to build an ordered list out of it: just pass enumerate(iterable) in, modify or wrap func to return the index along with the value, and then sort the results that come back.

这篇关于multiprocessing.pool.MapResult._number_left没有给出我期望的结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！