2017-02-19 02:25:29,811 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-02-19 02:26:23,282 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-02-19 02:26:27,134 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2017-02-19 02:26:27,155 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2017-02-19 02:26:29,463 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2017-02-19 02:26:29,621 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3
2017-02-19 02:26:30,236 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3
2017-02-19 02:26:32,316 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local816132518_0001
2017-02-19 02:26:34,621 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/
2017-02-19 02:26:34,622 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local816132518_0001
2017-02-19 02:26:34,646 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null
2017-02-19 02:26:34,722 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:26:34,739 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2017-02-19 02:26:35,249 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks
2017-02-19 02:26:35,251 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local816132518_0001_m_000000_0
2017-02-19 02:26:35,554 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:26:35,625 INFO org.apache.hadoop.mapreduce.Job: Job job_local816132518_0001 running in uber mode : false
2017-02-19 02:26:35,662 INFO org.apache.hadoop.mapreduce.Job:  map 0% reduce 0%
2017-02-19 02:26:35,741 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 02:26:35,759 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935
2017-02-19 02:26:37,229 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2017-02-19 02:26:37,229 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100
2017-02-19 02:26:37,229 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2017-02-19 02:26:37,229 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600
2017-02-19 02:26:37,230 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2017-02-19 02:26:37,253 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-02-19 02:26:37,283 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it
2017-02-19 02:26:41,675 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:26:42,674 INFO org.apache.hadoop.mapreduce.Job:  map 1% reduce 0%
2017-02-19 02:26:44,694 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:26:45,679 INFO org.apache.hadoop.mapreduce.Job:  map 7% reduce 0%
2017-02-19 02:26:47,696 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:26:48,682 INFO org.apache.hadoop.mapreduce.Job:  map 13% reduce 0%
2017-02-19 02:26:50,698 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:26:51,685 INFO org.apache.hadoop.mapreduce.Job:  map 18% reduce 0%
2017-02-19 02:26:51,724 INFO org.apache.hadoop.mapred.MapTask: Spilling map output
2017-02-19 02:26:51,742 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 42611793; bufvoid = 104857600
2017-02-19 02:26:51,744 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 15895828(63583312); length = 10318569/6553600
2017-02-19 02:26:51,747 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 53097537 kvi 13274380(53097520)
2017-02-19 02:26:52,846 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:26:52,847 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2017-02-19 02:26:53,699 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:26:54,688 INFO org.apache.hadoop.mapreduce.Job:  map 22% reduce 0%
2017-02-19 02:26:56,700 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:26:59,704 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:27:02,706 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:27:05,707 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:27:08,708 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:27:11,710 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:27:14,711 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:27:17,712 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:27:20,713 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:27:23,714 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:27:26,715 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:27:38,940 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2017-02-19 02:27:38,946 INFO org.apache.hadoop.mapred.MapTask: (RESET) equator 53097537 kv 13274380(53097520) kvi 12424484(49697936)
2017-02-19 02:27:38,946 INFO org.apache.hadoop.mapred.MapTask: Spilling map output
2017-02-19 02:27:38,946 INFO org.apache.hadoop.mapred.MapTask: bufstart = 53097537; bufend = 56605708; bufvoid = 104857600
2017-02-19 02:27:38,947 INFO org.apache.hadoop.mapred.MapTask: kvstart = 13274380(53097520); kvend = 12424488(49697952); length = 849893/6553600
2017-02-19 02:27:41,788 INFO org.apache.hadoop.mapred.MapTask: Finished spill 1
2017-02-19 02:27:42,148 INFO org.apache.hadoop.mapred.Merger: Merging 2 sorted segments
2017-02-19 02:27:42,273 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 51704192 bytes
2017-02-19 02:27:44,719 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 
2017-02-19 02:27:44,746 INFO org.apache.hadoop.mapreduce.Job:  map 23% reduce 0%
2017-02-19 02:27:47,720 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 
2017-02-19 02:27:47,749 INFO org.apache.hadoop.mapreduce.Job:  map 24% reduce 0%
2017-02-19 02:27:50,721 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 
2017-02-19 02:27:50,753 INFO org.apache.hadoop.mapreduce.Job:  map 25% reduce 0%
2017-02-19 02:27:53,722 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 
2017-02-19 02:27:53,755 INFO org.apache.hadoop.mapreduce.Job:  map 27% reduce 0%
2017-02-19 02:27:56,723 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 
2017-02-19 02:27:56,758 INFO org.apache.hadoop.mapreduce.Job:  map 28% reduce 0%
2017-02-19 02:27:59,724 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 
2017-02-19 02:27:59,760 INFO org.apache.hadoop.mapreduce.Job:  map 29% reduce 0%
2017-02-19 02:28:02,725 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 
2017-02-19 02:28:02,763 INFO org.apache.hadoop.mapreduce.Job:  map 31% reduce 0%
2017-02-19 02:28:05,726 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 
2017-02-19 02:28:05,766 INFO org.apache.hadoop.mapreduce.Job:  map 32% reduce 0%
2017-02-19 02:28:08,727 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 
2017-02-19 02:28:08,769 INFO org.apache.hadoop.mapreduce.Job:  map 33% reduce 0%
2017-02-19 02:28:09,211 INFO org.apache.hadoop.mapred.Task: Task:attempt_local816132518_0001_m_000000_0 is done. And is in the process of committing
2017-02-19 02:28:09,234 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:28:09,239 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local816132518_0001_m_000000_0' done.
2017-02-19 02:28:09,242 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local816132518_0001_m_000000_0
2017-02-19 02:28:09,245 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local816132518_0001_m_000001_0
2017-02-19 02:28:09,258 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:28:09,258 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 02:28:09,268 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889
2017-02-19 02:28:09,770 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 0%
2017-02-19 02:28:09,800 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2017-02-19 02:28:09,806 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100
2017-02-19 02:28:09,807 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2017-02-19 02:28:09,808 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600
2017-02-19 02:28:09,808 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2017-02-19 02:28:09,817 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-02-19 02:28:10,087 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it
2017-02-19 02:28:14,459 INFO org.apache.hadoop.mapred.LocalJobRunner: 
2017-02-19 02:28:14,463 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2017-02-19 02:28:14,466 INFO org.apache.hadoop.mapred.MapTask: Spilling map output
2017-02-19 02:28:14,467 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 13999099; bufvoid = 104857600
2017-02-19 02:28:14,467 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 22598156(90392624); length = 3616241/6553600
2017-02-19 02:28:14,773 INFO org.apache.hadoop.mapreduce.Job:  map 33% reduce 0%
2017-02-19 02:28:15,270 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:28:15,775 INFO org.apache.hadoop.mapreduce.Job:  map 56% reduce 0%
2017-02-19 02:28:18,271 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:28:21,273 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:28:24,273 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:28:27,274 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:28:29,312 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2017-02-19 02:28:29,339 INFO org.apache.hadoop.mapred.Task: Task:attempt_local816132518_0001_m_000001_0 is done. And is in the process of committing
2017-02-19 02:28:29,352 INFO org.apache.hadoop.mapred.LocalJobRunner: map
2017-02-19 02:28:29,359 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local816132518_0001_m_000001_0' done.
2017-02-19 02:28:29,361 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local816132518_0001_m_000001_0
2017-02-19 02:28:29,364 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local816132518_0001_m_000002_0
2017-02-19 02:28:29,386 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:28:29,388 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 02:28:29,418 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050
2017-02-19 02:28:29,791 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 0%
2017-02-19 02:28:29,980 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2017-02-19 02:28:29,988 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100
2017-02-19 02:28:29,990 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2017-02-19 02:28:29,991 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600
2017-02-19 02:28:29,991 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2017-02-19 02:28:30,009 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-02-19 02:28:30,044 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it
2017-02-19 02:28:33,034 INFO org.apache.hadoop.mapred.LocalJobRunner: 
2017-02-19 02:28:33,044 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2017-02-19 02:28:33,044 INFO org.apache.hadoop.mapred.MapTask: Spilling map output
2017-02-19 02:28:33,044 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 13719018; bufvoid = 104857600
2017-02-19 02:28:33,044 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 23093504(92374016); length = 3120893/6553600
2017-02-19 02:28:33,822 INFO org.apache.hadoop.mapreduce.Job:  map 67% reduce 0%
2017-02-19 02:28:35,399 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:28:35,825 INFO org.apache.hadoop.mapreduce.Job:  map 89% reduce 0%
2017-02-19 02:28:38,401 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:28:41,402 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:28:44,405 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:28:44,759 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2017-02-19 02:28:44,778 INFO org.apache.hadoop.mapred.Task: Task:attempt_local816132518_0001_m_000002_0 is done. And is in the process of committing
2017-02-19 02:28:44,788 INFO org.apache.hadoop.mapred.LocalJobRunner: map
2017-02-19 02:28:44,796 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local816132518_0001_m_000002_0' done.
2017-02-19 02:28:44,799 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local816132518_0001_m_000002_0
2017-02-19 02:28:44,801 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete.
2017-02-19 02:28:44,832 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 0%
2017-02-19 02:28:44,858 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks
2017-02-19 02:28:44,859 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local816132518_0001_r_000000_0
2017-02-19 02:28:44,955 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:28:44,957 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 02:28:45,011 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6cc6a82c
2017-02-19 02:28:45,223 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2017-02-19 02:28:45,273 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local816132518_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2017-02-19 02:28:45,694 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local816132518_0001_m_000002_0 decomp: 15279468 len: 15279472 to MEMORY
2017-02-19 02:28:46,039 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 15279468 bytes from map-output for attempt_local816132518_0001_m_000002_0
2017-02-19 02:28:46,067 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 15279468, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->15279468
2017-02-19 02:28:46,271 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local816132518_0001_m_000001_0 decomp: 15807223 len: 15807227 to MEMORY
2017-02-19 02:28:46,643 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 15807223 bytes from map-output for attempt_local816132518_0001_m_000001_0
2017-02-19 02:28:46,644 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 15807223, inMemoryMapOutputs.size() -> 2, commitMemory -> 15279468, usedMemory ->31086691
2017-02-19 02:28:47,428 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local816132518_0001_m_000000_0 decomp: 51704200 len: 51704204 to MEMORY
2017-02-19 02:28:50,792 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 51704200 bytes from map-output for attempt_local816132518_0001_m_000000_0
2017-02-19 02:28:50,800 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 51704200, inMemoryMapOutputs.size() -> 3, commitMemory -> 31086691, usedMemory ->82790891
2017-02-19 02:28:50,812 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning
2017-02-19 02:28:50,814 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied.
2017-02-19 02:28:50,815 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs
2017-02-19 02:28:50,835 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments
2017-02-19 02:28:50,836 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 82790873 bytes
2017-02-19 02:28:50,970 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort
2017-02-19 02:28:51,556 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 33%
2017-02-19 02:28:53,971 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort
2017-02-19 02:28:54,559 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 36%
2017-02-19 02:28:56,973 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort
2017-02-19 02:28:57,563 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 40%
2017-02-19 02:28:59,974 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort
2017-02-19 02:29:00,566 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 44%
2017-02-19 02:29:02,975 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort
2017-02-19 02:29:03,570 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 48%
2017-02-19 02:29:05,977 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort
2017-02-19 02:29:06,573 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 52%
2017-02-19 02:29:08,978 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort
2017-02-19 02:29:09,576 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 56%
2017-02-19 02:29:11,980 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort
2017-02-19 02:29:12,580 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 60%
2017-02-19 02:29:14,980 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort
2017-02-19 02:29:15,584 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 64%
2017-02-19 02:29:17,099 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 82790891 bytes to disk to satisfy reduce memory limit
2017-02-19 02:29:17,100 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 82790891 bytes from disk
2017-02-19 02:29:17,151 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2017-02-19 02:29:17,152 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments
2017-02-19 02:29:17,152 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 82790883 bytes
2017-02-19 02:29:17,153 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort
2017-02-19 02:29:17,206 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete.
2017-02-19 02:29:17,382 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local816132518_0001
java.lang.Exception: java.lang.RuntimeException: java.lang.NoSuchMethodException: Question2.InvIndex$Reduce.<init>()
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:556)
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: Question2.InvIndex$Reduce.<init>()
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:346)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodException: Question2.InvIndex$Reduce.<init>()
	at java.lang.Class.getConstructor0(Class.java:2849)
	at java.lang.Class.getDeclaredConstructor(Class.java:2053)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
	... 8 more
2017-02-19 02:29:17,589 INFO org.apache.hadoop.mapreduce.Job: Job job_local816132518_0001 failed with state FAILED due to: NA
2017-02-19 02:29:17,982 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-02-19 02:29:18,019 INFO org.apache.hadoop.mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=379344960
		FILE: Number of bytes written=568985052
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=507535
		Map output records=4476402
		Map output bytes=73838081
		Map output materialized bytes=82790903
		Input split bytes=351
		Combine input records=0
		Combine output records=0
		Reduce input groups=0
		Reduce shuffle bytes=82790903
		Reduce input records=0
		Reduce output records=0
		Spilled Records=7268519
		Shuffled Maps =3
		Failed Shuffles=0
		Merged Map outputs=3
		GC time elapsed (ms)=2048
		Total committed heap usage (bytes)=773603328
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=26057874
	File Output Format Counters 
		Bytes Written=0
2017-02-19 02:34:50,772 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-02-19 02:34:53,857 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2017-02-19 02:34:53,891 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2017-02-19 02:34:54,187 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/home/cloudera/workspace/bpa/Assign1/output_Q2 already exists
2017-02-19 02:35:58,581 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-02-19 02:36:02,113 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2017-02-19 02:36:02,152 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2017-02-19 02:36:04,722 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2017-02-19 02:36:04,880 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3
2017-02-19 02:36:05,691 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3
2017-02-19 02:36:08,773 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local2101611055_0001
2017-02-19 02:36:11,894 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/
2017-02-19 02:36:11,896 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local2101611055_0001
2017-02-19 02:36:11,937 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null
2017-02-19 02:36:12,058 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:36:12,077 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2017-02-19 02:36:12,909 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks
2017-02-19 02:36:12,911 INFO org.apache.hadoop.mapreduce.Job: Job job_local2101611055_0001 running in uber mode : false
2017-02-19 02:36:12,910 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local2101611055_0001_m_000000_0
2017-02-19 02:36:12,917 INFO org.apache.hadoop.mapreduce.Job:  map 0% reduce 0%
2017-02-19 02:36:13,386 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:36:13,593 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 02:36:13,615 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935
2017-02-19 02:36:14,991 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2017-02-19 02:36:14,992 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100
2017-02-19 02:36:14,992 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2017-02-19 02:36:14,992 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600
2017-02-19 02:36:14,993 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2017-02-19 02:36:15,044 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-02-19 02:36:15,116 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it
2017-02-19 02:36:19,508 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:22,512 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:25,515 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:26,322 INFO org.apache.hadoop.mapreduce.Job:  map 1% reduce 0%
2017-02-19 02:36:28,517 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:31,518 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:32,333 INFO org.apache.hadoop.mapreduce.Job:  map 3% reduce 0%
2017-02-19 02:36:34,522 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:35,338 INFO org.apache.hadoop.mapreduce.Job:  map 4% reduce 0%
2017-02-19 02:36:37,523 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:38,342 INFO org.apache.hadoop.mapreduce.Job:  map 6% reduce 0%
2017-02-19 02:36:40,527 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:41,348 INFO org.apache.hadoop.mapreduce.Job:  map 7% reduce 0%
2017-02-19 02:36:43,529 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:44,350 INFO org.apache.hadoop.mapreduce.Job:  map 8% reduce 0%
2017-02-19 02:36:46,546 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:47,353 INFO org.apache.hadoop.mapreduce.Job:  map 10% reduce 0%
2017-02-19 02:36:49,547 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:50,360 INFO org.apache.hadoop.mapreduce.Job:  map 11% reduce 0%
2017-02-19 02:36:52,548 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:36:53,368 INFO org.apache.hadoop.mapreduce.Job:  map 12% reduce 0%
2017-02-19 02:37:27,675 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-02-19 02:37:31,166 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2017-02-19 02:37:31,167 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2017-02-19 02:37:33,297 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2017-02-19 02:37:33,426 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3
2017-02-19 02:37:33,976 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3
2017-02-19 02:37:35,755 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local1344261413_0001
2017-02-19 02:37:37,731 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/
2017-02-19 02:37:37,739 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local1344261413_0001
2017-02-19 02:37:37,756 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null
2017-02-19 02:37:37,817 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:37:37,827 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2017-02-19 02:37:38,291 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks
2017-02-19 02:37:38,292 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1344261413_0001_m_000000_0
2017-02-19 02:37:38,585 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:37:38,702 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 02:37:38,730 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935
2017-02-19 02:37:38,743 INFO org.apache.hadoop.mapreduce.Job: Job job_local1344261413_0001 running in uber mode : false
2017-02-19 02:37:38,745 INFO org.apache.hadoop.mapreduce.Job:  map 0% reduce 0%
2017-02-19 02:37:39,652 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2017-02-19 02:37:39,653 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100
2017-02-19 02:37:39,653 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2017-02-19 02:37:39,653 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600
2017-02-19 02:37:39,653 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2017-02-19 02:37:39,715 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-02-19 02:37:39,774 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it
2017-02-19 02:37:44,673 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:37:47,687 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:37:47,766 INFO org.apache.hadoop.mapreduce.Job:  map 1% reduce 0%
2017-02-19 02:37:50,689 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:37:53,691 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:37:53,779 INFO org.apache.hadoop.mapreduce.Job:  map 3% reduce 0%
2017-02-19 02:37:56,693 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:37:56,784 INFO org.apache.hadoop.mapreduce.Job:  map 5% reduce 0%
2017-02-19 02:37:59,695 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:37:59,788 INFO org.apache.hadoop.mapreduce.Job:  map 6% reduce 0%
2017-02-19 02:38:02,696 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:02,791 INFO org.apache.hadoop.mapreduce.Job:  map 8% reduce 0%
2017-02-19 02:38:05,699 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:05,798 INFO org.apache.hadoop.mapreduce.Job:  map 10% reduce 0%
2017-02-19 02:38:08,700 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:08,803 INFO org.apache.hadoop.mapreduce.Job:  map 12% reduce 0%
2017-02-19 02:38:11,702 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:11,808 INFO org.apache.hadoop.mapreduce.Job:  map 13% reduce 0%
2017-02-19 02:38:14,705 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:14,816 INFO org.apache.hadoop.mapreduce.Job:  map 15% reduce 0%
2017-02-19 02:38:17,707 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:17,826 INFO org.apache.hadoop.mapreduce.Job:  map 17% reduce 0%
2017-02-19 02:38:20,709 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:20,830 INFO org.apache.hadoop.mapreduce.Job:  map 18% reduce 0%
2017-02-19 02:38:23,710 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:23,886 INFO org.apache.hadoop.mapreduce.Job:  map 20% reduce 0%
2017-02-19 02:38:26,712 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:26,892 INFO org.apache.hadoop.mapreduce.Job:  map 22% reduce 0%
2017-02-19 02:38:28,141 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:28,142 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2017-02-19 02:38:28,142 INFO org.apache.hadoop.mapred.MapTask: Spilling map output
2017-02-19 02:38:28,142 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5624480; bufvoid = 104857600
2017-02-19 02:38:28,142 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25005312(100021248); length = 1209085/6553600
2017-02-19 02:38:29,717 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:38:32,720 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 02:38:34,142 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2017-02-19 02:38:34,172 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1344261413_0001_m_000000_0 is done. And is in the process of committing
2017-02-19 02:38:34,181 INFO org.apache.hadoop.mapred.LocalJobRunner: map
2017-02-19 02:38:34,189 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1344261413_0001_m_000000_0' done.
2017-02-19 02:38:34,192 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1344261413_0001_m_000000_0
2017-02-19 02:38:34,194 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1344261413_0001_m_000001_0
2017-02-19 02:38:34,210 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:38:34,211 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 02:38:34,226 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889
2017-02-19 02:38:34,698 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2017-02-19 02:38:34,699 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100
2017-02-19 02:38:34,700 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2017-02-19 02:38:34,700 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600
2017-02-19 02:38:34,701 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2017-02-19 02:38:34,713 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-02-19 02:38:34,758 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it
2017-02-19 02:38:34,907 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 0%
2017-02-19 02:38:40,215 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:40,913 INFO org.apache.hadoop.mapreduce.Job:  map 41% reduce 0%
2017-02-19 02:38:43,219 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:43,915 INFO org.apache.hadoop.mapreduce.Job:  map 45% reduce 0%
2017-02-19 02:38:46,221 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:46,920 INFO org.apache.hadoop.mapreduce.Job:  map 50% reduce 0%
2017-02-19 02:38:49,226 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:49,925 INFO org.apache.hadoop.mapreduce.Job:  map 54% reduce 0%
2017-02-19 02:38:50,382 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:50,386 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2017-02-19 02:38:50,388 INFO org.apache.hadoop.mapred.MapTask: Spilling map output
2017-02-19 02:38:50,389 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2143719; bufvoid = 104857600
2017-02-19 02:38:50,391 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25715252(102861008); length = 499145/6553600
2017-02-19 02:38:52,077 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2017-02-19 02:38:52,092 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1344261413_0001_m_000001_0 is done. And is in the process of committing
2017-02-19 02:38:52,111 INFO org.apache.hadoop.mapred.LocalJobRunner: map
2017-02-19 02:38:52,114 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1344261413_0001_m_000001_0' done.
2017-02-19 02:38:52,116 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1344261413_0001_m_000001_0
2017-02-19 02:38:52,117 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1344261413_0001_m_000002_0
2017-02-19 02:38:52,126 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:38:52,127 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 02:38:52,135 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050
2017-02-19 02:38:52,600 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2017-02-19 02:38:52,610 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100
2017-02-19 02:38:52,612 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2017-02-19 02:38:52,614 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600
2017-02-19 02:38:52,615 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2017-02-19 02:38:52,624 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-02-19 02:38:52,633 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it
2017-02-19 02:38:52,928 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 0%
2017-02-19 02:38:58,142 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:38:58,941 INFO org.apache.hadoop.mapreduce.Job:  map 79% reduce 0%
2017-02-19 02:39:01,145 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:39:01,967 INFO org.apache.hadoop.mapreduce.Job:  map 85% reduce 0%
2017-02-19 02:39:02,945 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 02:39:02,948 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2017-02-19 02:39:02,950 INFO org.apache.hadoop.mapred.MapTask: Spilling map output
2017-02-19 02:39:02,950 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1596794; bufvoid = 104857600
2017-02-19 02:39:02,951 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25892496(103569984); length = 321901/6553600
2017-02-19 02:39:04,074 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2017-02-19 02:39:04,079 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1344261413_0001_m_000002_0 is done. And is in the process of committing
2017-02-19 02:39:04,101 INFO org.apache.hadoop.mapred.LocalJobRunner: map
2017-02-19 02:39:04,107 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1344261413_0001_m_000002_0' done.
2017-02-19 02:39:04,111 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1344261413_0001_m_000002_0
2017-02-19 02:39:04,112 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete.
2017-02-19 02:39:04,153 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks
2017-02-19 02:39:04,154 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1344261413_0001_r_000000_0
2017-02-19 02:39:04,257 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 02:39:04,258 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 02:39:04,301 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5fa2ac60
2017-02-19 02:39:04,505 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2017-02-19 02:39:04,532 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local1344261413_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2017-02-19 02:39:04,971 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 0%
2017-02-19 02:39:05,029 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1344261413_0001_m_000002_0 decomp: 1757748 len: 1757752 to MEMORY
2017-02-19 02:39:05,171 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1757748 bytes from map-output for attempt_local1344261413_0001_m_000002_0
2017-02-19 02:39:05,249 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1757748, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->1757748
2017-02-19 02:39:05,295 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1344261413_0001_m_000000_0 decomp: 6229026 len: 6229030 to MEMORY
2017-02-19 02:39:05,583 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 6229026 bytes from map-output for attempt_local1344261413_0001_m_000000_0
2017-02-19 02:39:05,612 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 6229026, inMemoryMapOutputs.size() -> 2, commitMemory -> 1757748, usedMemory ->7986774
2017-02-19 02:39:05,615 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1344261413_0001_m_000001_0 decomp: 2393295 len: 2393299 to MEMORY
2017-02-19 02:39:05,698 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 2393295 bytes from map-output for attempt_local1344261413_0001_m_000001_0
2017-02-19 02:39:05,699 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2393295, inMemoryMapOutputs.size() -> 3, commitMemory -> 7986774, usedMemory ->10380069
2017-02-19 02:39:05,699 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning
2017-02-19 02:39:05,700 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied.
2017-02-19 02:39:05,707 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs
2017-02-19 02:39:05,846 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments
2017-02-19 02:39:05,849 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 10380058 bytes
2017-02-19 02:39:09,888 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 10380069 bytes to disk to satisfy reduce memory limit
2017-02-19 02:39:09,904 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 10380069 bytes from disk
2017-02-19 02:39:09,920 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2017-02-19 02:39:09,920 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments
2017-02-19 02:39:09,921 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10380062 bytes
2017-02-19 02:39:09,932 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied.
2017-02-19 02:39:10,102 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2017-02-19 02:39:10,288 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-02-19 02:39:10,975 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 67%
2017-02-19 02:39:13,292 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-02-19 02:39:13,982 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 69%
2017-02-19 02:39:16,313 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-02-19 02:39:16,986 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 90%
2017-02-19 02:39:17,442 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1344261413_0001_r_000000_0 is done. And is in the process of committing
2017-02-19 02:39:17,465 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-02-19 02:39:17,473 INFO org.apache.hadoop.mapred.Task: Task attempt_local1344261413_0001_r_000000_0 is allowed to commit now
2017-02-19 02:39:17,487 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local1344261413_0001_r_000000_0' to file:/home/cloudera/workspace/bpa/Assign1/output_Q2/_temporary/0/task_local1344261413_0001_r_000000
2017-02-19 02:39:17,492 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-02-19 02:39:17,504 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1344261413_0001_r_000000_0' done.
2017-02-19 02:39:17,505 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1344261413_0001_r_000000_0
2017-02-19 02:39:17,505 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete.
2017-02-19 02:39:17,683 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local1344261413_0001
java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpUriRequest
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:573)
Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpUriRequest
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 1 more
2017-02-19 02:39:17,987 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 100%
2017-02-19 02:39:17,987 INFO org.apache.hadoop.mapreduce.Job: Job job_local1344261413_0001 failed with state FAILED due to: NA
2017-02-19 02:39:18,244 INFO org.apache.hadoop.mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=110497367
		FILE: Number of bytes written=49152988
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=507535
		Map output records=507535
		Map output bytes=9364993
		Map output materialized bytes=10380081
		Input split bytes=351
		Combine input records=0
		Combine output records=0
		Reduce input groups=87026
		Reduce shuffle bytes=10380081
		Reduce input records=507535
		Reduce output records=87026
		Spilled Records=1015070
		Shuffled Maps =3
		Failed Shuffles=0
		Merged Map outputs=3
		GC time elapsed (ms)=1104
		Total committed heap usage (bytes)=773603328
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=26057874
	File Output Format Counters 
		Bytes Written=2051650
2017-02-19 04:44:24,115 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-02-19 04:44:26,229 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2017-02-19 04:44:26,254 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2017-02-19 04:44:27,594 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2017-02-19 04:44:27,671 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3
2017-02-19 04:44:28,071 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3
2017-02-19 04:44:29,250 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local937133229_0001
2017-02-19 04:44:30,550 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/
2017-02-19 04:44:30,551 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local937133229_0001
2017-02-19 04:44:30,573 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null
2017-02-19 04:44:30,615 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 04:44:30,621 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2017-02-19 04:44:30,931 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks
2017-02-19 04:44:30,939 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local937133229_0001_m_000000_0
2017-02-19 04:44:31,136 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 04:44:31,241 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 04:44:31,247 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935
2017-02-19 04:44:31,687 INFO org.apache.hadoop.mapreduce.Job: Job job_local937133229_0001 running in uber mode : false
2017-02-19 04:44:31,697 INFO org.apache.hadoop.mapreduce.Job:  map 0% reduce 0%
2017-02-19 04:44:32,062 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2017-02-19 04:44:32,063 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100
2017-02-19 04:44:32,063 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2017-02-19 04:44:32,064 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600
2017-02-19 04:44:32,065 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2017-02-19 04:44:32,078 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-02-19 04:44:32,102 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it
2017-02-19 04:44:37,228 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:44:37,724 INFO org.apache.hadoop.mapreduce.Job:  map 1% reduce 0%
2017-02-19 04:44:40,240 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:44:40,743 INFO org.apache.hadoop.mapreduce.Job:  map 3% reduce 0%
2017-02-19 04:44:43,245 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:44:43,755 INFO org.apache.hadoop.mapreduce.Job:  map 6% reduce 0%
2017-02-19 04:44:46,252 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:44:46,768 INFO org.apache.hadoop.mapreduce.Job:  map 8% reduce 0%
2017-02-19 04:44:49,255 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:44:49,783 INFO org.apache.hadoop.mapreduce.Job:  map 11% reduce 0%
2017-02-19 04:44:52,265 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:44:52,817 INFO org.apache.hadoop.mapreduce.Job:  map 14% reduce 0%
2017-02-19 04:44:55,276 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:44:55,828 INFO org.apache.hadoop.mapreduce.Job:  map 17% reduce 0%
2017-02-19 04:44:58,277 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:44:58,846 INFO org.apache.hadoop.mapreduce.Job:  map 20% reduce 0%
2017-02-19 04:45:00,846 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:45:00,848 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2017-02-19 04:45:00,849 INFO org.apache.hadoop.mapred.MapTask: Spilling map output
2017-02-19 04:45:00,849 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5624480; bufvoid = 104857600
2017-02-19 04:45:00,849 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25005312(100021248); length = 1209085/6553600
2017-02-19 04:45:01,286 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort
2017-02-19 04:45:01,853 INFO org.apache.hadoop.mapreduce.Job:  map 22% reduce 0%
2017-02-19 04:45:04,131 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2017-02-19 04:45:04,159 INFO org.apache.hadoop.mapred.Task: Task:attempt_local937133229_0001_m_000000_0 is done. And is in the process of committing
2017-02-19 04:45:04,165 INFO org.apache.hadoop.mapred.LocalJobRunner: map
2017-02-19 04:45:04,168 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local937133229_0001_m_000000_0' done.
2017-02-19 04:45:04,169 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local937133229_0001_m_000000_0
2017-02-19 04:45:04,170 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local937133229_0001_m_000001_0
2017-02-19 04:45:04,177 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 04:45:04,178 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 04:45:04,180 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889
2017-02-19 04:45:04,474 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2017-02-19 04:45:04,476 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100
2017-02-19 04:45:04,476 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2017-02-19 04:45:04,476 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600
2017-02-19 04:45:04,477 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2017-02-19 04:45:04,484 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-02-19 04:45:04,491 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it
2017-02-19 04:45:04,865 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 0%
2017-02-19 04:45:10,187 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:45:10,893 INFO org.apache.hadoop.mapreduce.Job:  map 47% reduce 0%
2017-02-19 04:45:13,189 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:45:13,639 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:45:13,640 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2017-02-19 04:45:13,641 INFO org.apache.hadoop.mapred.MapTask: Spilling map output
2017-02-19 04:45:13,641 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2143719; bufvoid = 104857600
2017-02-19 04:45:13,642 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25715252(102861008); length = 499145/6553600
2017-02-19 04:45:13,920 INFO org.apache.hadoop.mapreduce.Job:  map 54% reduce 0%
2017-02-19 04:45:14,652 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2017-02-19 04:45:14,658 INFO org.apache.hadoop.mapred.Task: Task:attempt_local937133229_0001_m_000001_0 is done. And is in the process of committing
2017-02-19 04:45:14,665 INFO org.apache.hadoop.mapred.LocalJobRunner: map
2017-02-19 04:45:14,669 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local937133229_0001_m_000001_0' done.
2017-02-19 04:45:14,670 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local937133229_0001_m_000001_0
2017-02-19 04:45:14,670 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local937133229_0001_m_000002_0
2017-02-19 04:45:14,677 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 04:45:14,678 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 04:45:14,680 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050
2017-02-19 04:45:14,928 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 0%
2017-02-19 04:45:14,970 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2017-02-19 04:45:14,979 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100
2017-02-19 04:45:14,980 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2017-02-19 04:45:14,986 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600
2017-02-19 04:45:14,987 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2017-02-19 04:45:14,993 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-02-19 04:45:14,994 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it
2017-02-19 04:45:20,687 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:45:20,949 INFO org.apache.hadoop.mapreduce.Job:  map 88% reduce 0%
2017-02-19 04:45:21,079 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map
2017-02-19 04:45:21,083 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2017-02-19 04:45:21,083 INFO org.apache.hadoop.mapred.MapTask: Spilling map output
2017-02-19 04:45:21,084 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1596794; bufvoid = 104857600
2017-02-19 04:45:21,084 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25892496(103569984); length = 321901/6553600
2017-02-19 04:45:21,616 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2017-02-19 04:45:21,640 INFO org.apache.hadoop.mapred.Task: Task:attempt_local937133229_0001_m_000002_0 is done. And is in the process of committing
2017-02-19 04:45:21,647 INFO org.apache.hadoop.mapred.LocalJobRunner: map
2017-02-19 04:45:21,650 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local937133229_0001_m_000002_0' done.
2017-02-19 04:45:21,650 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local937133229_0001_m_000002_0
2017-02-19 04:45:21,651 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete.
2017-02-19 04:45:21,658 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks
2017-02-19 04:45:21,658 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local937133229_0001_r_000000_0
2017-02-19 04:45:21,696 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-02-19 04:45:21,697 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-02-19 04:45:21,712 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@38981d47
2017-02-19 04:45:21,822 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2017-02-19 04:45:21,872 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local937133229_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2017-02-19 04:45:21,958 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 0%
2017-02-19 04:45:22,058 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local937133229_0001_m_000001_0 decomp: 2393295 len: 2393299 to MEMORY
2017-02-19 04:45:22,107 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 2393295 bytes from map-output for attempt_local937133229_0001_m_000001_0
2017-02-19 04:45:22,130 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2393295, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2393295
2017-02-19 04:45:22,154 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local937133229_0001_m_000002_0 decomp: 1757748 len: 1757752 to MEMORY
2017-02-19 04:45:22,166 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1757748 bytes from map-output for attempt_local937133229_0001_m_000002_0
2017-02-19 04:45:22,198 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1757748, inMemoryMapOutputs.size() -> 2, commitMemory -> 2393295, usedMemory ->4151043
2017-02-19 04:45:22,202 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local937133229_0001_m_000000_0 decomp: 6229026 len: 6229030 to MEMORY
2017-02-19 04:45:22,333 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 6229026 bytes from map-output for attempt_local937133229_0001_m_000000_0
2017-02-19 04:45:22,343 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 6229026, inMemoryMapOutputs.size() -> 3, commitMemory -> 4151043, usedMemory ->10380069
2017-02-19 04:45:22,346 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning
2017-02-19 04:45:22,347 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied.
2017-02-19 04:45:22,347 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs
2017-02-19 04:45:22,371 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments
2017-02-19 04:45:22,381 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 10380058 bytes
2017-02-19 04:45:25,071 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 10380069 bytes to disk to satisfy reduce memory limit
2017-02-19 04:45:25,073 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 10380069 bytes from disk
2017-02-19 04:45:25,074 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2017-02-19 04:45:25,074 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments
2017-02-19 04:45:25,075 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10380062 bytes
2017-02-19 04:45:25,075 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied.
2017-02-19 04:45:25,158 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2017-02-19 04:45:27,728 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-02-19 04:45:27,985 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 68%
2017-02-19 04:45:30,735 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-02-19 04:45:31,013 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 69%
2017-02-19 04:45:33,737 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-02-19 04:45:34,024 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 85%
2017-02-19 04:45:34,842 INFO org.apache.hadoop.mapred.Task: Task:attempt_local937133229_0001_r_000000_0 is done. And is in the process of committing
2017-02-19 04:45:34,848 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-02-19 04:45:34,848 INFO org.apache.hadoop.mapred.Task: Task attempt_local937133229_0001_r_000000_0 is allowed to commit now
2017-02-19 04:45:34,849 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local937133229_0001_r_000000_0' to file:/home/cloudera/workspace/bpa/Assign1/output_Q2/_temporary/0/task_local937133229_0001_r_000000
2017-02-19 04:45:34,858 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-02-19 04:45:34,861 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local937133229_0001_r_000000_0' done.
2017-02-19 04:45:34,861 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local937133229_0001_r_000000_0
2017-02-19 04:45:34,862 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete.
2017-02-19 04:45:34,935 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local937133229_0001
java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpUriRequest
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:573)
Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpUriRequest
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 1 more
2017-02-19 04:45:35,032 INFO org.apache.hadoop.mapreduce.Job:  map 100% reduce 100%
2017-02-19 04:45:35,032 INFO org.apache.hadoop.mapreduce.Job: Job job_local937133229_0001 failed with state FAILED due to: NA
2017-02-19 04:45:35,147 INFO org.apache.hadoop.mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=110497367
		FILE: Number of bytes written=49147132
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=507535
		Map output records=507535
		Map output bytes=9364993
		Map output materialized bytes=10380081
		Input split bytes=351
		Combine input records=0
		Combine output records=0
		Reduce input groups=87026
		Reduce shuffle bytes=10380081
		Reduce input records=507535
		Reduce output records=87026
		Spilled Records=1015070
		Shuffled Maps =3
		Failed Shuffles=0
		Merged Map outputs=3
		GC time elapsed (ms)=785
		Total committed heap usage (bytes)=773603328
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=26057874
	File Output Format Counters 
		Bytes Written=2051650