带有Java的Google云存储gsutil工具

问题描述

根据google docs，如果每天大约需要30G文件(从50MB到4GB不等)上载到Google Cloud Storage，gsutil可能是唯一合适的选择，不是吗?

If we have around 30G files (ranged from 50MB to 4GB) need to be uploaded to Google Cloud Storage everyday, according to google docs, gsutil might be the only fitted choice, isn't it?

我想通过Java调用gsutil命令，现在下面的代码可以工作了.但是，如果我删除了while循环，该程序将在runtime.exec(command)之后立即停止，但是python进程已启动但没有上载，并且很快就会被杀死.我不知道为什么.

I want to call gsutil command by Java, now the code below can work. But If I delete that while loop, the program will stop immediately after the runtime.exec(command) but python process was started but doing no uploading and it will soon be killed. I wonder why.

我从sterr流中读取内容的原因是受将gsutil输出到文件的管道

The reason I read from sterr stream is inspired by Pipe gsutil output to file

我决定gsutil是否通过read util的状态输出的最后一行完成执行，但这是一种可靠的方法吗?有没有更好的方法来检测gsutil执行是否在Java中结束了?

I decide whether gsutil finish executing by read util the last line of its status output, but is it a reliable way? Is there any better ways to detect whether gsutil execution is end in Java?

String command="python c:/gsutil/gsutil.py cp C:/SFC_Data/gps.txt"
            + " gs://getest/gps.txt";
 try {
        Process process = Runtime.getRuntime().exec(command);
        System.out.println("the output stream is "+process.getErrorStream());
        BufferedReader reader=new BufferedReader(new InputStreamReader(process.getErrorStream()));
        String s;
        while ((s = reader.readLine()) != null){
            System.out.println("The inout stream is " + s);
        }
    } catch (IOException e) {
        e.printStackTrace();
    }

推荐答案

肯定有不止一种方式，每天可以将价值30G的数据上传到GCS.由于您使用的是Java，因此您是否考虑过使用Cloud Storage API Java客户端库? https://developers.google.com/api-client-library/java/apis/storage/v1

There are certainly more than one way to upload 30G worth of data per day to GCS. Since you are working in Java, have you considered to use the Cloud Storage API Java client library?https://developers.google.com/api-client-library/java/apis/storage/v1

关于使用Runtime.exec()从Java调用gsutil的具体问题，我怀疑当没有while循环时，该程序将在创建子进程后立即退出，从而导致"process"变量成为GC'ed，这可能会杀死该子进程.

As for the specific questions about calling gsutil from Java using Runtime.exec(), I suspect when there is no while loop, the program will exit immediately after creating the sub-process, causing the "process" variable to be GC'ed, which might kill the sub-process.

我认为您应该等待子过程完成，这实际上是while循环正在做的事情.或者，如果您不关心输出，则可以调用waitFor()并检查existValue(): http://docs.oracle.com/javase/7/docs/api/java/lang/Process.html

I think you should wait for the sub-process to complete, which is effectively what the while loop is doing. Or you can just call waitFor() and check the existValue() if you don't care about the output:http://docs.oracle.com/javase/7/docs/api/java/lang/Process.html

这篇关于带有Java的Google云存储gsutil工具的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！