问题描述
我正在尝试使用通过Azure App Service托管的NodeJS Express服务器将响应流式传输到客户端.但是,我注意到它不是真正的流式传输,而是尝试整体发送响应.当响应大小很大(> 50MB)时,客户端会收到内部服务器错误
,但服务器不会引发错误.
I am trying to stream responses to my client using a NodeJS Express server hosted using Azure App Service. However, I noticed that it is not really streaming but tries to send the response as a whole. When the response size is huge (>50MB), the client gets an Internal Server Error
, but the server does not throw an error.
此外,当我在Docker内部运行服务器(节点映像: 10.22.0-alpine3.9
)时,我看到即使是巨大的响应,客户端也将响应作为流获得.(这是我真正需要的行为)
Further, when I run the server inside a Docker (Node Image: 10.22.0-alpine3.9
), I see that the client gets the response as a stream even for huge responses. (This is the behavior I actually need)
我的 web.config
文件如下.
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<handlers>
<add name="iisnode" path="server.js" verb="*" modules="iisnode" responseBufferLimit="0"/>
</handlers>
<iisnode flushResponse="true" />
...
</system.webServer>
</configuration>
这只是我程序的简单解释.
This is a small explanation as to what my program does.
我有一个外部API,该API返回类似于以下内容的对象.
I have an external API that returns an object similar to the following.
{
"title":"Test Title",
"lastBuildDate":"1597981114347",
"items":[
{
id: 'item1',
value: 'value1'
},
{
id: 'item2',
value: 'value2'
},
...
[
}
我只想过滤 items
数组中的元素,然后将它们发送给客户端.客户应收到如下响应.
I want to filter only the elements in items
array and send those to the client. The client should get a response like below.
[
{
id: 'item1',
value: 'value1'
},
{
id: 'item2',
value: 'value2'
},
...
[
有时该对象太大(大于50MB),因此,我将响应作为流发送,以避免在服务器中使用过多的缓冲内存.下面是我用于流式传输响应的代码.
Sometimes this object is too large (>50MB) and because of that, I am sending the response as a stream to avoid using too much buffer memory in my server. Below here is the code I used to stream the response.
const https = require('https');
const { withParser } = require('stream-json/filters/Pick');
const { streamArray } = require('stream-json/streamers/StreamArray');
const { chain } = require('stream-chain');
exports.getStreamResponse = async function (req, res) {
const options = {
hostname,
port,
path,
method: 'GET',
};
return new Promise((resolve, reject) => {
https.request(options, (dataStream) => {
const pipeline = chain([
dataStream,
withParser({ filter: 'items' }),
streamArray()
]);
res.write("[");
let separator = '';
pipeline.on('data', data => {
res.write(separator + JSON.stringify(data.value));
if (!separator) {
separator = ',';
}
});
pipeline.on('end', () => {
res.write("]");
res.end();
resolve();
});
pipeline.on('error', (error) => {
reject(error);
});
});
})
};
我还注意到,如果我像下面那样编写代码,我总是会得到一个流响应.但是,响应的格式不正确.
I also noticed that if I write the code like below, I always get a stream response. However, the response is not in the correct format as needed.
https.request(options, (dataStream) => {
dataStream.pipe(res);
});
推荐答案
就像我在问题的后半部分描述的那样,直接将 res
(我对客户端的响应)传递给dataStream
(我从外部API获取的数据流)允许流传输而没有任何问题.
Like I described in the latter part of my question, directly piping the res
(my response to the client) to dataStream
(the data stream I got from the external API) allowed to stream without any issues.
扩展相同的行为,我创建了一个 Readable
流,该流等效于应发送给我的客户端的响应.然后,我将其通过管道传输到 res
,并且可以正常工作.
Extending the same behavior, I created a Readable
stream which is equivalent to the response I should send to my client. Then I piped it to res
and it worked.
这是我的解决方法.
const https = require('https');
const { withParser } = require('stream-json/filters/Pick');
const { streamArray } = require('stream-json/streamers/StreamArray');
const { chain } = require('stream-chain');
const { Readable } = require('stream');
exports.getStreamResponse = async function (req, res) {
const options = {
hostname,
port,
path,
method: 'GET',
};
return new Promise((resolve, reject) => {
https.request(options, (dataStream) => {
const pipeline = chain([
dataStream,
withParser({ filter: 'items' }),
streamArray()
]);
// create a readable stream to collect data from response
const readable = new Readable({
// this empty method is to avoid 'ERR_METHOD_NOT_IMPLEMENTED'
// error when read method is called while there is no data in the
// readable stream
read(size) { }
});
let separator = '';
readable.pipe(res);
readable.push("[");
pipeline.on('data', data => {
readable.push(separator + JSON.stringify(data.value));
if (!separator) {
separator = ',';
}
});
pipeline.on('end', () => {
readable.push("]");
readable.push(null);
resolve();
});
pipeline.on('error', reject);
});
})
};
但是,我注意到此解决方案比我遇到的解决方案需要更多的内存.可能是因为我正在创建一个多余的可读流.
However, I noticed this solution requires more memory than the solution I had issues with. Probably because I am creating a readable stream that is redundant.
这篇关于Express中的响应流在Azure App Service中不起作用的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!