MapReduce job does give ANY output and does not run. Why?

Moderator: NorbertKrupa

Post Reply
Expressions_Galore
Newbie
Newbie
Posts: 3
Joined: Fri Feb 22, 2013 12:49 pm
Contact:

MapReduce job does give ANY output and does not run. Why?

Post by Expressions_Galore » Mon Mar 18, 2013 1:09 pm

I have written a MapReduce job to connect to the Vertica Database. I used this document as a reference: http://www.vertica.com/wp-content/uploa ... -11-10.pdf

I don't know if it a problem with Vertica or Hadoop but even though my code successfully compiles, it does not run when triggered and does not giv ANY output at all. I cannot understand what the problem is and am stumped. Any help would be really appreciated. The code is:

Code: Select all

import java.io.IOException;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import com.vertica.hadoop.VerticaConfiguration;
import com.vertica.hadoop.VerticaInputFormat;
import com.vertica.hadoop.VerticaRecord;

public class MapRVertica {

    public static class VerticaMapper extends 
        Mapper<LongWritable, VerticaRecord, LongWritable, Text> {

        @Override
        public void map(LongWritable key, VerticaRecord value, Context context) 
                throws IOException, InterruptedException {

            context.write(key, new Text((String) value.get(0)));
        }
    }

    public static void main(String[] args) throws IOException {
        if (args.length != 1) {
            System.err.println("Usage: MapRVertica <output_path>");
            System.exit(-1);
        }

        Job job = new Job();
        job.setJarByClass(MapRVertica.class);
        job.setJobName("MapReduce Vertica");

        job.setInputFormatClass(VerticaInputFormat.class);
        VerticaConfiguration.configureVertica(job.getConfiguration(), 
                new String[]{"hostname"}, 
                "dbname", "port", "username", "password");
        VerticaInputFormat.setInput(job, "select now()");

        FileOutputFormat.setOutputPath(job, new Path(args[0]));

        job.setMapperClass(VerticaMapper.class);
        job.setNumReduceTasks(0);

        job.setOutputKeyClass(LongWritable.class);
        job.setOutputValueClass(Text.class);
    }

}

Expressions_Galore
Newbie
Newbie
Posts: 3
Joined: Fri Feb 22, 2013 12:49 pm
Contact:

Re: MapReduce job does give ANY output and does not run. Why

Post by Expressions_Galore » Mon Mar 18, 2013 1:57 pm

The answer was really stupid actually. I missed this in the main method.

Code: Select all

job.waitForCompletion(true);

User avatar
JimKnicely
Site Admin
Site Admin
Posts: 1804
Joined: Sat Jan 21, 2012 4:58 am
Contact:

Re: MapReduce job does give ANY output and does not run. Why

Post by JimKnicely » Tue Mar 19, 2013 4:08 pm

Thanks for sharing!!!
Jim Knicely

Image

Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.

Post Reply

Return to “Hadoop Connector”