java - MapReduce Avro输出正在创建文本文件

我有一个MapReduce作业，该作业读取avro数据，然后应该输出avro数据。但是，当我在作业成功时检查输出文件时，它们没有.avro扩展名，并且可以使用简单的文本编辑器查看它们。

我将驱动程序配置为输出avro，所以我不确定问题出在哪里，任何帮助将不胜感激。

这是我的驱动程序类:

public class Driver extends Configured implements Tool{

public static void main(String[] args) throws Exception {
    int res =
            ToolRunner.run(new Configuration(), new Driver(), args);
    System.exit(res);
}

@Override
public int run(String[] args) throws Exception {
    Job job = new Job(getConf());
    job.setJarByClass(Driver.class);
    job.setJobName("nearestpatient");


    AvroJob.setOutputKeySchema(job, Pair.getPairSchema(Schema.create(Schema.Type.LONG), Schema.create(Schema.Type.STRING)));
    job.setOutputValueClass(NullWritable.class);

    job.setMapperClass(PatientMapper.class);
    job.setReducerClass(PatientReducer.class);

    job.setInputFormatClass(AvroKeyInputFormat.class);
    AvroJob.setInputKeySchema(job, PatientAvro.getClassSchema());

    job.setMapOutputKeyClass(LongWritable.class);
    job.setMapOutputValueClass(LongWritable.class);


    FileInputFormat.setInputPaths(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.waitForCompletion(true);

    return 0;
}
}

这是我的Reducer类:

public class PatientReducer extends Reducer<LongWritable, LongWritable, AvroWrapper<Pair<Long, String>>, NullWritable> {

    @Override
    public void reduce(LongWritable providerKey, Iterable<LongWritable> patients, Context context) throws IOException, InterruptedException {

        String outputList = "[";
   `enter code here` List<Long> patientList = new ArrayList<>();
    for (LongWritable patientKey : patients) {
        outputList += new LongWritable(patientKey.get()) + ", ";
    }
    outputList = outputList.substring(0, outputList.length() - 2);
    outputList += "]";
    context.write(new AvroWrapper<Pair<Long, String>>(new Pair<Long, String> (providerKey.get(), outputList)), NullWritable.get());
}
}

最佳答案

在您的run()方法中，您需要添加以下内容

job.setOutputFormatClass(AvroKeyValueOutputFormat.class);

关于java - MapReduce Avro输出正在创建文本文件，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/30808218/