有没有一种方法可以将数据直接从python请求流式传输到minio bucket

问题描述

我正在尝试向服务器发出GET请求以检索tiff图像.然后，我想使用MinIO python SDK中的put_object方法将其直接流式传输到MinIO.

I am trying to make a GET request to a server to retrieve a tiff image. I then want to stream it directly to MinIO using the put_object method in the MinIO python SDK.

我知道我可以通过将图像保存到临时文件中然后上传来完成此操作，但是我想看看是否可以跳过该步骤.

I know I could do this by saving the image to a temp file, then uploading but I wanted to see if I could skip that step.

我尝试直接插入字节响应并使用BytesIO对其进行包装，但我认为我缺少了一些东西.

I've tried inserting the byte response directly and using BytesIO to wrap it but I think I am missing something.

r = requests.get(url_to_download, stream=True)
Minio_client.put_object("bucket_name", "stream_test.tiff", r.content, r.headers['Content-length'])

我得到以下错误

非常感谢您的帮助！

推荐答案

您可以像这样直接将文件流式传输到minio存储桶中:

You can stream your file directly into a minio bucket like this:

import requests

from pathlib import Path
from urllib.parse import urlparse

from django.conf import settings
from django.core.files.storage import default_storage

client = default_storage.client
object_name = Path(urlparse(response.url).path).name
bucket_name = settings.MINIO_STORAGE_MEDIA_BUCKET_NAME

with requests.get(url_to_download, stream=True) as r:
    content_length = int(r.headers["Content-Length"])
    result = client.put_object(bucket_name, object_name, r.raw, content_length)

或者您可以直接使用Django文件字段:

Or you can use a django file field directly:

with requests.get(url_to_download, stream=True) as r:
    # patch the stream to make django-minio-storage belief
    # it's about to read from a legit file
    r.raw.seek = lambda x: 0
    r.raw.size = int(r.headers["Content-Length"])
    model = MyModel()
    model.file.save(object_name, r.raw, save=True)

Dinko Pehar的RawIOBase提示确实很有帮助，非常感谢.但是您必须使用response.raw而不是response.content，它会立即下载文件，并且在尝试存储较大的视频时确实非常不便.

The RawIOBase hint from Dinko Pehar was really helpful, thanks a lot. But you have to use response.raw not response.content which would download your file immediately and be really inconvenient when trying to store a large video for example.

这篇关于有没有一种方法可以将数据直接从python请求流式传输到minio bucket的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！