如何将数据从S3加载到PostgreSQL RDS

问题描述

我需要将数据从S3加载到Postgres RDS（大约50-100 GB），我无法选择使用AWS Data Pipeline，并且正在寻找与使用COPY命令加载数据类似的内容。通过S3进入Amazon Redshift。

我将对如何实现此目标提出任何建议。

解决方案

更新

另一种选择是安装s3并使用 COPY 命令使用直接到csv的路径。我不确定它是否可以有效容纳100GB，但是值得尝试。 。 / p>

另一种选择是逐段解析 s3文件复制到文件中，并从命名管道中 COPY ，

最明显的选择就是下载文件保存到本地存储并使用 COPY 我根本不介绍

值得一提的是（状态不稳定）。 非常简洁，但是我认为您可以创建一个指向s3文件的外部表。这本身意味着您可以将数据加载到其他关系中...

I have a need to load data from S3 to Postgres RDS (around 50-100 GB) I don't have the option to use AWS Data Pipeline and I am looking for something similar to using the COPY command to load data in S3 into Amazon Redshift.

I would appreciate any suggestions on how I can accomplish this.

解决方案

http://docs.aws.amazon.com/redshift/latest/dg/t_loading-tables-from-s3.html

update

Another option is to mount s3 and use direct path to the csv with COPY command. I'm not sure If it will hold 100GB effectively, but worth of trying. Here is some list of options on software.

Yet another option would be "parsing" s3 file part by part with something described here to a file and COPY from named pipe, described here

And the most obvious option to just download file to local storage and use COPY I don't cover at all

Also worth of mentioning would be s3_fdw (status unstable). Readme is very laconic, but I assume you could create a foreign table leading to s3 file. Which itself means you can load data to other relation...

这篇关于如何将数据从S3加载到PostgreSQL RDS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！