问题描述
在2011年3月左右，我测试了GAE（Java版本）作为大规模并行计算的潜在平台。这个日期是相关的，因为GAE一直在发展。我发现应用程序在43.2X的计算吞吐量下被有效地扼制。 是否有人成功地将GAE用于大规模并行计算或获得了更高的计算增益？为了这个问题的目的，我将任意定义大规模并行计算，意味着大于1000倍的计算吞吐量。
我使用桌面客户端实例化了多个线程来访问URL。我正在使用GAE任务队列。该应用程序所需的输入非常少，产生的输出非常少，无论是数据存储还是HTML，因为它旨在评估计算吞吐量。
由于通常建议保持GAE任务不到1秒（尽管这个建议是否适用于任务队列任务并不清楚），我尝试了各种排列。我的一些结果包括在这里。正如您所看到的，即使在0.8秒的任务中，与低于1秒的建议一致，吞吐量也达到了43.2X。
已完成的任务秒数总收益
请求的秒数WorkPerTask工作
FLT（几个大任务）
15 72 1 72 4.9
103 72 20 1440 14.0
1524 72 400 28800 18.9
MST（许多小任务）
53 1000 0.8 800 15.1
63 2000 0.8 1600 25.4
127 4000 0.8 3200 25.2
313 4000 0.8 3200 10.2
258 8000 0.8 6400 24.8
177 8000 0.8 6400 36.2 （有5％的任务什么都不做）
49 2000 0.8 1600 32.7（有1％的任务不做任何事情）
37 2000 0.8 1600 43.2（有5％的任务不做任何事情）
42 2000 0.8 1600 38.1（有10％的任务不做任何事情）
249 2000 0.8 1600 6.4（有50％的任务什么也不做）
MLT（许多大任务）
6373 1000 200 200000 31.4
380 200 60 12000 31.6
请注意，对于任务队列任务超过600秒是不可取的，因此最高的I去了400秒，只是为了留下一个安全的边缘。某些任务无所作为的情况是为了降低每项任务为了影响整个Google会计所做的平均工作量。因此，每个说2000个任务的人有0.8秒的工作时间，但是额外的222个任务没有任何工作，这意味着10％的任务没有工作。我正在测量吞吐量增益，它是totalWorkInSeconds除以elapsedTimeInSeconds，这是在客户端测量的。客户做出请求并测量经过的时间，直到所有GAE任务完成，每个任务发送一个微不足道的小响应。我试图找出目前形式的GAE是否可用于创建实现吞吐量增益高值的应用。 2011年3月似乎不太可能。今天怎么样？以及它将如何完成或你是如何做到的？吞吐量增益达到了什么水平？正如我所说的，数据存储的使用非常少，每个任务由一项任务完成时写入一个细小的对象组成。每个任务循环到与secondsOfWorkPerTask成比例的整数。 GAE旋转实例是问题的一部分。谷歌通过告诉人们他们更喜欢次秒级的任务而恶化了这个问题。如果我的任务很大，问题就会得到缓解，因为实例化的使用周期数是所用周期数的一小部分。
App Engine实际上并不是设计用作大型计算作业的后端 - 它旨在为可扩展站点（和API）提供快速有效的服务。它所做的并不是围绕你要实现的目标进行优化。
In approximately March 2011 I tested GAE (the Java version) as a potential platform for a massively parallel computation. The date is relevant because GAE is evolving all the time. I found that the application was effectively being throttled at about 43.2X computational throughput. Has anybody successfully used GAE for massively parallel computation or achieved a much higher computational gain? For the purpose of this question, I will arbitrarily define massively parallel computation to mean greater than 1000x computational throughput.
I used a desktop client that instantiated multiple threads to hit the URL. I was using GAE Task Queues. The application required very little input and produced very little output, whether Datastore or HTML, as it was designed to evaluate computational throughput.
Since it is often advised to keep GAE tasks under 1 second (although it is not clear as to whether this recommendation applies to Task Queue tasks) I tried various permutations. Some of my results are included here. As you can see, even with 0.8 second tasks, consistent with the sub 1 second recommendation, throughput peaked at 43.2X.
Elapsed Tasks SecondsOf Total Gain
Seconds Requested WorkPerTask Work
FLT (FEW LARGE TASKS)
15 72 1 72 4.9
103 72 20 1440 14.0
1524 72 400 28800 18.9
MST (MANY SMALL TASKS)
53 1000 0.8 800 15.1
63 2000 0.8 1600 25.4
127 4000 0.8 3200 25.2
313 4000 0.8 3200 10.2
258 8000 0.8 6400 24.8
177 8000 0.8 6400 36.2 (Have 5% of tasks do nothing.)
49 2000 0.8 1600 32.7 (Have 1% of tasks do nothing.)
37 2000 0.8 1600 43.2 (Have 5% of tasks do nothing.)
42 2000 0.8 1600 38.1 (Have 10% of tasks do nothing.)
249 2000 0.8 1600 6.4 (Have 50% of tasks do nothing.)
MLT (MANY LARGE TASKS)
6373 1000 200 200000 31.4
380 200 60 12000 31.6
Note that it was inadvisable to go above 600 seconds for Task Queue tasks so the highest I went was 400 seconds just to leave a margin of safety. The cases where some tasks do nothing was to lower the average amount of work each task had to do in order to influence the overall Google "accounting". So each of, say 2000 tasks, have 0.8 seconds of work but an extra 222 tasks have no work, meaning 10% have no work.
Edit: @PeterRecore, I am measuring the throughput gain and it is totalWorkInSeconds divided by elapsedTimeInSeconds and this is measured at the client. The client makes the requests and measures the elapsed time until all the GAE tasks finish which is indicated by each sending a trivially small response. I am trying to find out if GAE in its current form can be used to create an application that achieves high values of throughput gain. In March 2011 it seemed not likely. What about today? and how would it be done or how did you actually do it? what level of throughput gain was achieved? As I said Datastore use is minimal and consists of each task writing a single trivially small object when a task is done. Each task loops to an integer proportional to secondsOfWorkPerTask. GAE spinning up instances is part of the problem. Google sort of worsens this problem by telling people that they prefer sub-second tasks. The problem is mitigated if I have large tasks because then instantiation is a smaller percentage of the number of cycles used.
App Engine really isn't designed for use as a backend for huge computing jobs - it's designed for fast efficient serving of scalable sites (and APIs, for that matter). What it does isn't optimized around what you're trying to achieve.
这篇关于Google App Engine可以用于大规模并行计算吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！