2017-02-19 02:25:29,811 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 02:26:23,282 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 02:26:27,134 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 02:26:27,155 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 02:26:29,463 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2017-02-19 02:26:29,621 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3 2017-02-19 02:26:30,236 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3 2017-02-19 02:26:32,316 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local816132518_0001 2017-02-19 02:26:34,621 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/ 2017-02-19 02:26:34,622 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local816132518_0001 2017-02-19 02:26:34,646 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null 2017-02-19 02:26:34,722 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:26:34,739 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2017-02-19 02:26:35,249 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks 2017-02-19 02:26:35,251 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local816132518_0001_m_000000_0 2017-02-19 02:26:35,554 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:26:35,625 INFO org.apache.hadoop.mapreduce.Job: Job job_local816132518_0001 running in uber mode : false 2017-02-19 02:26:35,662 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0% 2017-02-19 02:26:35,741 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 02:26:35,759 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935 2017-02-19 02:26:37,229 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 02:26:37,229 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 02:26:37,229 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 02:26:37,229 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 02:26:37,230 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 02:26:37,253 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 02:26:37,283 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 02:26:41,675 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:26:42,674 INFO org.apache.hadoop.mapreduce.Job: map 1% reduce 0% 2017-02-19 02:26:44,694 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:26:45,679 INFO org.apache.hadoop.mapreduce.Job: map 7% reduce 0% 2017-02-19 02:26:47,696 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:26:48,682 INFO org.apache.hadoop.mapreduce.Job: map 13% reduce 0% 2017-02-19 02:26:50,698 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:26:51,685 INFO org.apache.hadoop.mapreduce.Job: map 18% reduce 0% 2017-02-19 02:26:51,724 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 02:26:51,742 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 42611793; bufvoid = 104857600 2017-02-19 02:26:51,744 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 15895828(63583312); length = 10318569/6553600 2017-02-19 02:26:51,747 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 53097537 kvi 13274380(53097520) 2017-02-19 02:26:52,846 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:26:52,847 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 02:26:53,699 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:26:54,688 INFO org.apache.hadoop.mapreduce.Job: map 22% reduce 0% 2017-02-19 02:26:56,700 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:26:59,704 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:27:02,706 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:27:05,707 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:27:08,708 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:27:11,710 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:27:14,711 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:27:17,712 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:27:20,713 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:27:23,714 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:27:26,715 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:27:38,940 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 02:27:38,946 INFO org.apache.hadoop.mapred.MapTask: (RESET) equator 53097537 kv 13274380(53097520) kvi 12424484(49697936) 2017-02-19 02:27:38,946 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 02:27:38,946 INFO org.apache.hadoop.mapred.MapTask: bufstart = 53097537; bufend = 56605708; bufvoid = 104857600 2017-02-19 02:27:38,947 INFO org.apache.hadoop.mapred.MapTask: kvstart = 13274380(53097520); kvend = 12424488(49697952); length = 849893/6553600 2017-02-19 02:27:41,788 INFO org.apache.hadoop.mapred.MapTask: Finished spill 1 2017-02-19 02:27:42,148 INFO org.apache.hadoop.mapred.Merger: Merging 2 sorted segments 2017-02-19 02:27:42,273 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 51704192 bytes 2017-02-19 02:27:44,719 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 2017-02-19 02:27:44,746 INFO org.apache.hadoop.mapreduce.Job: map 23% reduce 0% 2017-02-19 02:27:47,720 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 2017-02-19 02:27:47,749 INFO org.apache.hadoop.mapreduce.Job: map 24% reduce 0% 2017-02-19 02:27:50,721 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 2017-02-19 02:27:50,753 INFO org.apache.hadoop.mapreduce.Job: map 25% reduce 0% 2017-02-19 02:27:53,722 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 2017-02-19 02:27:53,755 INFO org.apache.hadoop.mapreduce.Job: map 27% reduce 0% 2017-02-19 02:27:56,723 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 2017-02-19 02:27:56,758 INFO org.apache.hadoop.mapreduce.Job: map 28% reduce 0% 2017-02-19 02:27:59,724 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 2017-02-19 02:27:59,760 INFO org.apache.hadoop.mapreduce.Job: map 29% reduce 0% 2017-02-19 02:28:02,725 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 2017-02-19 02:28:02,763 INFO org.apache.hadoop.mapreduce.Job: map 31% reduce 0% 2017-02-19 02:28:05,726 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 2017-02-19 02:28:05,766 INFO org.apache.hadoop.mapreduce.Job: map 32% reduce 0% 2017-02-19 02:28:08,727 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort > 2017-02-19 02:28:08,769 INFO org.apache.hadoop.mapreduce.Job: map 33% reduce 0% 2017-02-19 02:28:09,211 INFO org.apache.hadoop.mapred.Task: Task:attempt_local816132518_0001_m_000000_0 is done. And is in the process of committing 2017-02-19 02:28:09,234 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:28:09,239 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local816132518_0001_m_000000_0' done. 2017-02-19 02:28:09,242 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local816132518_0001_m_000000_0 2017-02-19 02:28:09,245 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local816132518_0001_m_000001_0 2017-02-19 02:28:09,258 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:28:09,258 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 02:28:09,268 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889 2017-02-19 02:28:09,770 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 02:28:09,800 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 02:28:09,806 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 02:28:09,807 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 02:28:09,808 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 02:28:09,808 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 02:28:09,817 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 02:28:10,087 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 02:28:14,459 INFO org.apache.hadoop.mapred.LocalJobRunner: 2017-02-19 02:28:14,463 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 02:28:14,466 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 02:28:14,467 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 13999099; bufvoid = 104857600 2017-02-19 02:28:14,467 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 22598156(90392624); length = 3616241/6553600 2017-02-19 02:28:14,773 INFO org.apache.hadoop.mapreduce.Job: map 33% reduce 0% 2017-02-19 02:28:15,270 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:28:15,775 INFO org.apache.hadoop.mapreduce.Job: map 56% reduce 0% 2017-02-19 02:28:18,271 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:28:21,273 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:28:24,273 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:28:27,274 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:28:29,312 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 02:28:29,339 INFO org.apache.hadoop.mapred.Task: Task:attempt_local816132518_0001_m_000001_0 is done. And is in the process of committing 2017-02-19 02:28:29,352 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 02:28:29,359 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local816132518_0001_m_000001_0' done. 2017-02-19 02:28:29,361 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local816132518_0001_m_000001_0 2017-02-19 02:28:29,364 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local816132518_0001_m_000002_0 2017-02-19 02:28:29,386 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:28:29,388 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 02:28:29,418 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050 2017-02-19 02:28:29,791 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 02:28:29,980 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 02:28:29,988 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 02:28:29,990 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 02:28:29,991 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 02:28:29,991 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 02:28:30,009 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 02:28:30,044 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 02:28:33,034 INFO org.apache.hadoop.mapred.LocalJobRunner: 2017-02-19 02:28:33,044 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 02:28:33,044 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 02:28:33,044 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 13719018; bufvoid = 104857600 2017-02-19 02:28:33,044 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 23093504(92374016); length = 3120893/6553600 2017-02-19 02:28:33,822 INFO org.apache.hadoop.mapreduce.Job: map 67% reduce 0% 2017-02-19 02:28:35,399 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:28:35,825 INFO org.apache.hadoop.mapreduce.Job: map 89% reduce 0% 2017-02-19 02:28:38,401 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:28:41,402 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:28:44,405 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:28:44,759 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 02:28:44,778 INFO org.apache.hadoop.mapred.Task: Task:attempt_local816132518_0001_m_000002_0 is done. And is in the process of committing 2017-02-19 02:28:44,788 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 02:28:44,796 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local816132518_0001_m_000002_0' done. 2017-02-19 02:28:44,799 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local816132518_0001_m_000002_0 2017-02-19 02:28:44,801 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete. 2017-02-19 02:28:44,832 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 02:28:44,858 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks 2017-02-19 02:28:44,859 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local816132518_0001_r_000000_0 2017-02-19 02:28:44,955 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:28:44,957 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 02:28:45,011 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6cc6a82c 2017-02-19 02:28:45,223 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2017-02-19 02:28:45,273 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local816132518_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2017-02-19 02:28:45,694 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local816132518_0001_m_000002_0 decomp: 15279468 len: 15279472 to MEMORY 2017-02-19 02:28:46,039 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 15279468 bytes from map-output for attempt_local816132518_0001_m_000002_0 2017-02-19 02:28:46,067 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 15279468, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->15279468 2017-02-19 02:28:46,271 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local816132518_0001_m_000001_0 decomp: 15807223 len: 15807227 to MEMORY 2017-02-19 02:28:46,643 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 15807223 bytes from map-output for attempt_local816132518_0001_m_000001_0 2017-02-19 02:28:46,644 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 15807223, inMemoryMapOutputs.size() -> 2, commitMemory -> 15279468, usedMemory ->31086691 2017-02-19 02:28:47,428 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local816132518_0001_m_000000_0 decomp: 51704200 len: 51704204 to MEMORY 2017-02-19 02:28:50,792 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 51704200 bytes from map-output for attempt_local816132518_0001_m_000000_0 2017-02-19 02:28:50,800 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 51704200, inMemoryMapOutputs.size() -> 3, commitMemory -> 31086691, usedMemory ->82790891 2017-02-19 02:28:50,812 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning 2017-02-19 02:28:50,814 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 02:28:50,815 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs 2017-02-19 02:28:50,835 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments 2017-02-19 02:28:50,836 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 82790873 bytes 2017-02-19 02:28:50,970 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort 2017-02-19 02:28:51,556 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 33% 2017-02-19 02:28:53,971 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort 2017-02-19 02:28:54,559 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 36% 2017-02-19 02:28:56,973 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort 2017-02-19 02:28:57,563 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 40% 2017-02-19 02:28:59,974 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort 2017-02-19 02:29:00,566 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 44% 2017-02-19 02:29:02,975 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort 2017-02-19 02:29:03,570 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 48% 2017-02-19 02:29:05,977 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort 2017-02-19 02:29:06,573 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 52% 2017-02-19 02:29:08,978 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort 2017-02-19 02:29:09,576 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 56% 2017-02-19 02:29:11,980 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort 2017-02-19 02:29:12,580 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 60% 2017-02-19 02:29:14,980 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort 2017-02-19 02:29:15,584 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 64% 2017-02-19 02:29:17,099 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 82790891 bytes to disk to satisfy reduce memory limit 2017-02-19 02:29:17,100 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 82790891 bytes from disk 2017-02-19 02:29:17,151 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2017-02-19 02:29:17,152 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2017-02-19 02:29:17,152 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 82790883 bytes 2017-02-19 02:29:17,153 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > sort 2017-02-19 02:29:17,206 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete. 2017-02-19 02:29:17,382 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local816132518_0001 java.lang.Exception: java.lang.RuntimeException: java.lang.NoSuchMethodException: Question2.InvIndex$Reduce.<init>() at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:556) Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: Question2.InvIndex$Reduce.<init>() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:346) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoSuchMethodException: Question2.InvIndex$Reduce.<init>() at java.lang.Class.getConstructor0(Class.java:2849) at java.lang.Class.getDeclaredConstructor(Class.java:2053) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 8 more 2017-02-19 02:29:17,589 INFO org.apache.hadoop.mapreduce.Job: Job job_local816132518_0001 failed with state FAILED due to: NA 2017-02-19 02:29:17,982 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 02:29:18,019 INFO org.apache.hadoop.mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=379344960 FILE: Number of bytes written=568985052 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=507535 Map output records=4476402 Map output bytes=73838081 Map output materialized bytes=82790903 Input split bytes=351 Combine input records=0 Combine output records=0 Reduce input groups=0 Reduce shuffle bytes=82790903 Reduce input records=0 Reduce output records=0 Spilled Records=7268519 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=2048 Total committed heap usage (bytes)=773603328 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=26057874 File Output Format Counters Bytes Written=0 2017-02-19 02:34:50,772 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 02:34:53,857 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 02:34:53,891 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 02:34:54,187 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/home/cloudera/workspace/bpa/Assign1/output_Q2 already exists 2017-02-19 02:35:58,581 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 02:36:02,113 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 02:36:02,152 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 02:36:04,722 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2017-02-19 02:36:04,880 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3 2017-02-19 02:36:05,691 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3 2017-02-19 02:36:08,773 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local2101611055_0001 2017-02-19 02:36:11,894 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/ 2017-02-19 02:36:11,896 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local2101611055_0001 2017-02-19 02:36:11,937 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null 2017-02-19 02:36:12,058 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:36:12,077 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2017-02-19 02:36:12,909 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks 2017-02-19 02:36:12,911 INFO org.apache.hadoop.mapreduce.Job: Job job_local2101611055_0001 running in uber mode : false 2017-02-19 02:36:12,910 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local2101611055_0001_m_000000_0 2017-02-19 02:36:12,917 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0% 2017-02-19 02:36:13,386 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:36:13,593 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 02:36:13,615 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935 2017-02-19 02:36:14,991 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 02:36:14,992 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 02:36:14,992 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 02:36:14,992 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 02:36:14,993 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 02:36:15,044 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 02:36:15,116 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 02:36:19,508 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:22,512 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:25,515 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:26,322 INFO org.apache.hadoop.mapreduce.Job: map 1% reduce 0% 2017-02-19 02:36:28,517 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:31,518 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:32,333 INFO org.apache.hadoop.mapreduce.Job: map 3% reduce 0% 2017-02-19 02:36:34,522 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:35,338 INFO org.apache.hadoop.mapreduce.Job: map 4% reduce 0% 2017-02-19 02:36:37,523 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:38,342 INFO org.apache.hadoop.mapreduce.Job: map 6% reduce 0% 2017-02-19 02:36:40,527 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:41,348 INFO org.apache.hadoop.mapreduce.Job: map 7% reduce 0% 2017-02-19 02:36:43,529 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:44,350 INFO org.apache.hadoop.mapreduce.Job: map 8% reduce 0% 2017-02-19 02:36:46,546 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:47,353 INFO org.apache.hadoop.mapreduce.Job: map 10% reduce 0% 2017-02-19 02:36:49,547 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:50,360 INFO org.apache.hadoop.mapreduce.Job: map 11% reduce 0% 2017-02-19 02:36:52,548 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:36:53,368 INFO org.apache.hadoop.mapreduce.Job: map 12% reduce 0% 2017-02-19 02:37:27,675 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 02:37:31,166 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 02:37:31,167 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 02:37:33,297 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2017-02-19 02:37:33,426 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3 2017-02-19 02:37:33,976 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3 2017-02-19 02:37:35,755 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local1344261413_0001 2017-02-19 02:37:37,731 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/ 2017-02-19 02:37:37,739 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local1344261413_0001 2017-02-19 02:37:37,756 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null 2017-02-19 02:37:37,817 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:37:37,827 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2017-02-19 02:37:38,291 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks 2017-02-19 02:37:38,292 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1344261413_0001_m_000000_0 2017-02-19 02:37:38,585 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:37:38,702 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 02:37:38,730 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935 2017-02-19 02:37:38,743 INFO org.apache.hadoop.mapreduce.Job: Job job_local1344261413_0001 running in uber mode : false 2017-02-19 02:37:38,745 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0% 2017-02-19 02:37:39,652 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 02:37:39,653 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 02:37:39,653 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 02:37:39,653 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 02:37:39,653 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 02:37:39,715 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 02:37:39,774 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 02:37:44,673 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:37:47,687 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:37:47,766 INFO org.apache.hadoop.mapreduce.Job: map 1% reduce 0% 2017-02-19 02:37:50,689 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:37:53,691 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:37:53,779 INFO org.apache.hadoop.mapreduce.Job: map 3% reduce 0% 2017-02-19 02:37:56,693 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:37:56,784 INFO org.apache.hadoop.mapreduce.Job: map 5% reduce 0% 2017-02-19 02:37:59,695 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:37:59,788 INFO org.apache.hadoop.mapreduce.Job: map 6% reduce 0% 2017-02-19 02:38:02,696 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:02,791 INFO org.apache.hadoop.mapreduce.Job: map 8% reduce 0% 2017-02-19 02:38:05,699 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:05,798 INFO org.apache.hadoop.mapreduce.Job: map 10% reduce 0% 2017-02-19 02:38:08,700 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:08,803 INFO org.apache.hadoop.mapreduce.Job: map 12% reduce 0% 2017-02-19 02:38:11,702 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:11,808 INFO org.apache.hadoop.mapreduce.Job: map 13% reduce 0% 2017-02-19 02:38:14,705 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:14,816 INFO org.apache.hadoop.mapreduce.Job: map 15% reduce 0% 2017-02-19 02:38:17,707 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:17,826 INFO org.apache.hadoop.mapreduce.Job: map 17% reduce 0% 2017-02-19 02:38:20,709 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:20,830 INFO org.apache.hadoop.mapreduce.Job: map 18% reduce 0% 2017-02-19 02:38:23,710 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:23,886 INFO org.apache.hadoop.mapreduce.Job: map 20% reduce 0% 2017-02-19 02:38:26,712 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:26,892 INFO org.apache.hadoop.mapreduce.Job: map 22% reduce 0% 2017-02-19 02:38:28,141 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:28,142 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 02:38:28,142 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 02:38:28,142 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5624480; bufvoid = 104857600 2017-02-19 02:38:28,142 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25005312(100021248); length = 1209085/6553600 2017-02-19 02:38:29,717 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:38:32,720 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 02:38:34,142 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 02:38:34,172 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1344261413_0001_m_000000_0 is done. And is in the process of committing 2017-02-19 02:38:34,181 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 02:38:34,189 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1344261413_0001_m_000000_0' done. 2017-02-19 02:38:34,192 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1344261413_0001_m_000000_0 2017-02-19 02:38:34,194 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1344261413_0001_m_000001_0 2017-02-19 02:38:34,210 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:38:34,211 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 02:38:34,226 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889 2017-02-19 02:38:34,698 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 02:38:34,699 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 02:38:34,700 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 02:38:34,700 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 02:38:34,701 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 02:38:34,713 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 02:38:34,758 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 02:38:34,907 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 02:38:40,215 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:40,913 INFO org.apache.hadoop.mapreduce.Job: map 41% reduce 0% 2017-02-19 02:38:43,219 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:43,915 INFO org.apache.hadoop.mapreduce.Job: map 45% reduce 0% 2017-02-19 02:38:46,221 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:46,920 INFO org.apache.hadoop.mapreduce.Job: map 50% reduce 0% 2017-02-19 02:38:49,226 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:49,925 INFO org.apache.hadoop.mapreduce.Job: map 54% reduce 0% 2017-02-19 02:38:50,382 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:50,386 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 02:38:50,388 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 02:38:50,389 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2143719; bufvoid = 104857600 2017-02-19 02:38:50,391 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25715252(102861008); length = 499145/6553600 2017-02-19 02:38:52,077 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 02:38:52,092 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1344261413_0001_m_000001_0 is done. And is in the process of committing 2017-02-19 02:38:52,111 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 02:38:52,114 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1344261413_0001_m_000001_0' done. 2017-02-19 02:38:52,116 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1344261413_0001_m_000001_0 2017-02-19 02:38:52,117 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1344261413_0001_m_000002_0 2017-02-19 02:38:52,126 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:38:52,127 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 02:38:52,135 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050 2017-02-19 02:38:52,600 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 02:38:52,610 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 02:38:52,612 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 02:38:52,614 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 02:38:52,615 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 02:38:52,624 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 02:38:52,633 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 02:38:52,928 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 02:38:58,142 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:38:58,941 INFO org.apache.hadoop.mapreduce.Job: map 79% reduce 0% 2017-02-19 02:39:01,145 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:39:01,967 INFO org.apache.hadoop.mapreduce.Job: map 85% reduce 0% 2017-02-19 02:39:02,945 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 02:39:02,948 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 02:39:02,950 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 02:39:02,950 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1596794; bufvoid = 104857600 2017-02-19 02:39:02,951 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25892496(103569984); length = 321901/6553600 2017-02-19 02:39:04,074 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 02:39:04,079 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1344261413_0001_m_000002_0 is done. And is in the process of committing 2017-02-19 02:39:04,101 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 02:39:04,107 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1344261413_0001_m_000002_0' done. 2017-02-19 02:39:04,111 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1344261413_0001_m_000002_0 2017-02-19 02:39:04,112 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete. 2017-02-19 02:39:04,153 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks 2017-02-19 02:39:04,154 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1344261413_0001_r_000000_0 2017-02-19 02:39:04,257 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 02:39:04,258 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 02:39:04,301 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5fa2ac60 2017-02-19 02:39:04,505 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2017-02-19 02:39:04,532 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local1344261413_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2017-02-19 02:39:04,971 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 02:39:05,029 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1344261413_0001_m_000002_0 decomp: 1757748 len: 1757752 to MEMORY 2017-02-19 02:39:05,171 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1757748 bytes from map-output for attempt_local1344261413_0001_m_000002_0 2017-02-19 02:39:05,249 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1757748, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->1757748 2017-02-19 02:39:05,295 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1344261413_0001_m_000000_0 decomp: 6229026 len: 6229030 to MEMORY 2017-02-19 02:39:05,583 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 6229026 bytes from map-output for attempt_local1344261413_0001_m_000000_0 2017-02-19 02:39:05,612 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 6229026, inMemoryMapOutputs.size() -> 2, commitMemory -> 1757748, usedMemory ->7986774 2017-02-19 02:39:05,615 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1344261413_0001_m_000001_0 decomp: 2393295 len: 2393299 to MEMORY 2017-02-19 02:39:05,698 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 2393295 bytes from map-output for attempt_local1344261413_0001_m_000001_0 2017-02-19 02:39:05,699 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2393295, inMemoryMapOutputs.size() -> 3, commitMemory -> 7986774, usedMemory ->10380069 2017-02-19 02:39:05,699 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning 2017-02-19 02:39:05,700 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 02:39:05,707 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs 2017-02-19 02:39:05,846 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments 2017-02-19 02:39:05,849 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 10380058 bytes 2017-02-19 02:39:09,888 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 10380069 bytes to disk to satisfy reduce memory limit 2017-02-19 02:39:09,904 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 10380069 bytes from disk 2017-02-19 02:39:09,920 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2017-02-19 02:39:09,920 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2017-02-19 02:39:09,921 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10380062 bytes 2017-02-19 02:39:09,932 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 02:39:10,102 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2017-02-19 02:39:10,288 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 02:39:10,975 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 67% 2017-02-19 02:39:13,292 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 02:39:13,982 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 69% 2017-02-19 02:39:16,313 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 02:39:16,986 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 90% 2017-02-19 02:39:17,442 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1344261413_0001_r_000000_0 is done. And is in the process of committing 2017-02-19 02:39:17,465 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 02:39:17,473 INFO org.apache.hadoop.mapred.Task: Task attempt_local1344261413_0001_r_000000_0 is allowed to commit now 2017-02-19 02:39:17,487 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local1344261413_0001_r_000000_0' to file:/home/cloudera/workspace/bpa/Assign1/output_Q2/_temporary/0/task_local1344261413_0001_r_000000 2017-02-19 02:39:17,492 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 02:39:17,504 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1344261413_0001_r_000000_0' done. 2017-02-19 02:39:17,505 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1344261413_0001_r_000000_0 2017-02-19 02:39:17,505 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete. 2017-02-19 02:39:17,683 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local1344261413_0001 java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpUriRequest at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:573) Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpUriRequest at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more 2017-02-19 02:39:17,987 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 100% 2017-02-19 02:39:17,987 INFO org.apache.hadoop.mapreduce.Job: Job job_local1344261413_0001 failed with state FAILED due to: NA 2017-02-19 02:39:18,244 INFO org.apache.hadoop.mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=110497367 FILE: Number of bytes written=49152988 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=507535 Map output records=507535 Map output bytes=9364993 Map output materialized bytes=10380081 Input split bytes=351 Combine input records=0 Combine output records=0 Reduce input groups=87026 Reduce shuffle bytes=10380081 Reduce input records=507535 Reduce output records=87026 Spilled Records=1015070 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=1104 Total committed heap usage (bytes)=773603328 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=26057874 File Output Format Counters Bytes Written=2051650 2017-02-19 04:44:24,115 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 04:44:26,229 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 04:44:26,254 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 04:44:27,594 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2017-02-19 04:44:27,671 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3 2017-02-19 04:44:28,071 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3 2017-02-19 04:44:29,250 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local937133229_0001 2017-02-19 04:44:30,550 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/ 2017-02-19 04:44:30,551 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local937133229_0001 2017-02-19 04:44:30,573 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null 2017-02-19 04:44:30,615 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 04:44:30,621 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2017-02-19 04:44:30,931 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks 2017-02-19 04:44:30,939 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local937133229_0001_m_000000_0 2017-02-19 04:44:31,136 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 04:44:31,241 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 04:44:31,247 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935 2017-02-19 04:44:31,687 INFO org.apache.hadoop.mapreduce.Job: Job job_local937133229_0001 running in uber mode : false 2017-02-19 04:44:31,697 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0% 2017-02-19 04:44:32,062 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 04:44:32,063 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 04:44:32,063 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 04:44:32,064 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 04:44:32,065 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 04:44:32,078 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 04:44:32,102 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 04:44:37,228 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:44:37,724 INFO org.apache.hadoop.mapreduce.Job: map 1% reduce 0% 2017-02-19 04:44:40,240 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:44:40,743 INFO org.apache.hadoop.mapreduce.Job: map 3% reduce 0% 2017-02-19 04:44:43,245 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:44:43,755 INFO org.apache.hadoop.mapreduce.Job: map 6% reduce 0% 2017-02-19 04:44:46,252 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:44:46,768 INFO org.apache.hadoop.mapreduce.Job: map 8% reduce 0% 2017-02-19 04:44:49,255 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:44:49,783 INFO org.apache.hadoop.mapreduce.Job: map 11% reduce 0% 2017-02-19 04:44:52,265 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:44:52,817 INFO org.apache.hadoop.mapreduce.Job: map 14% reduce 0% 2017-02-19 04:44:55,276 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:44:55,828 INFO org.apache.hadoop.mapreduce.Job: map 17% reduce 0% 2017-02-19 04:44:58,277 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:44:58,846 INFO org.apache.hadoop.mapreduce.Job: map 20% reduce 0% 2017-02-19 04:45:00,846 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:45:00,848 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 04:45:00,849 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 04:45:00,849 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5624480; bufvoid = 104857600 2017-02-19 04:45:00,849 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25005312(100021248); length = 1209085/6553600 2017-02-19 04:45:01,286 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 04:45:01,853 INFO org.apache.hadoop.mapreduce.Job: map 22% reduce 0% 2017-02-19 04:45:04,131 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 04:45:04,159 INFO org.apache.hadoop.mapred.Task: Task:attempt_local937133229_0001_m_000000_0 is done. And is in the process of committing 2017-02-19 04:45:04,165 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 04:45:04,168 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local937133229_0001_m_000000_0' done. 2017-02-19 04:45:04,169 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local937133229_0001_m_000000_0 2017-02-19 04:45:04,170 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local937133229_0001_m_000001_0 2017-02-19 04:45:04,177 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 04:45:04,178 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 04:45:04,180 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889 2017-02-19 04:45:04,474 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 04:45:04,476 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 04:45:04,476 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 04:45:04,476 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 04:45:04,477 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 04:45:04,484 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 04:45:04,491 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 04:45:04,865 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 04:45:10,187 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:45:10,893 INFO org.apache.hadoop.mapreduce.Job: map 47% reduce 0% 2017-02-19 04:45:13,189 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:45:13,639 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:45:13,640 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 04:45:13,641 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 04:45:13,641 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2143719; bufvoid = 104857600 2017-02-19 04:45:13,642 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25715252(102861008); length = 499145/6553600 2017-02-19 04:45:13,920 INFO org.apache.hadoop.mapreduce.Job: map 54% reduce 0% 2017-02-19 04:45:14,652 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 04:45:14,658 INFO org.apache.hadoop.mapred.Task: Task:attempt_local937133229_0001_m_000001_0 is done. And is in the process of committing 2017-02-19 04:45:14,665 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 04:45:14,669 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local937133229_0001_m_000001_0' done. 2017-02-19 04:45:14,670 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local937133229_0001_m_000001_0 2017-02-19 04:45:14,670 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local937133229_0001_m_000002_0 2017-02-19 04:45:14,677 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 04:45:14,678 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 04:45:14,680 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050 2017-02-19 04:45:14,928 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 04:45:14,970 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 04:45:14,979 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 04:45:14,980 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 04:45:14,986 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 04:45:14,987 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 04:45:14,993 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 04:45:14,994 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 04:45:20,687 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:45:20,949 INFO org.apache.hadoop.mapreduce.Job: map 88% reduce 0% 2017-02-19 04:45:21,079 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 04:45:21,083 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 04:45:21,083 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 04:45:21,084 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1596794; bufvoid = 104857600 2017-02-19 04:45:21,084 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25892496(103569984); length = 321901/6553600 2017-02-19 04:45:21,616 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 04:45:21,640 INFO org.apache.hadoop.mapred.Task: Task:attempt_local937133229_0001_m_000002_0 is done. And is in the process of committing 2017-02-19 04:45:21,647 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 04:45:21,650 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local937133229_0001_m_000002_0' done. 2017-02-19 04:45:21,650 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local937133229_0001_m_000002_0 2017-02-19 04:45:21,651 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete. 2017-02-19 04:45:21,658 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks 2017-02-19 04:45:21,658 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local937133229_0001_r_000000_0 2017-02-19 04:45:21,696 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 04:45:21,697 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 04:45:21,712 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@38981d47 2017-02-19 04:45:21,822 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2017-02-19 04:45:21,872 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local937133229_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2017-02-19 04:45:21,958 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 04:45:22,058 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local937133229_0001_m_000001_0 decomp: 2393295 len: 2393299 to MEMORY 2017-02-19 04:45:22,107 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 2393295 bytes from map-output for attempt_local937133229_0001_m_000001_0 2017-02-19 04:45:22,130 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2393295, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2393295 2017-02-19 04:45:22,154 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local937133229_0001_m_000002_0 decomp: 1757748 len: 1757752 to MEMORY 2017-02-19 04:45:22,166 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1757748 bytes from map-output for attempt_local937133229_0001_m_000002_0 2017-02-19 04:45:22,198 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1757748, inMemoryMapOutputs.size() -> 2, commitMemory -> 2393295, usedMemory ->4151043 2017-02-19 04:45:22,202 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local937133229_0001_m_000000_0 decomp: 6229026 len: 6229030 to MEMORY 2017-02-19 04:45:22,333 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 6229026 bytes from map-output for attempt_local937133229_0001_m_000000_0 2017-02-19 04:45:22,343 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 6229026, inMemoryMapOutputs.size() -> 3, commitMemory -> 4151043, usedMemory ->10380069 2017-02-19 04:45:22,346 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning 2017-02-19 04:45:22,347 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 04:45:22,347 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs 2017-02-19 04:45:22,371 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments 2017-02-19 04:45:22,381 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 10380058 bytes 2017-02-19 04:45:25,071 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 10380069 bytes to disk to satisfy reduce memory limit 2017-02-19 04:45:25,073 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 10380069 bytes from disk 2017-02-19 04:45:25,074 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2017-02-19 04:45:25,074 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2017-02-19 04:45:25,075 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10380062 bytes 2017-02-19 04:45:25,075 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 04:45:25,158 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2017-02-19 04:45:27,728 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 04:45:27,985 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 68% 2017-02-19 04:45:30,735 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 04:45:31,013 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 69% 2017-02-19 04:45:33,737 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 04:45:34,024 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 85% 2017-02-19 04:45:34,842 INFO org.apache.hadoop.mapred.Task: Task:attempt_local937133229_0001_r_000000_0 is done. And is in the process of committing 2017-02-19 04:45:34,848 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 04:45:34,848 INFO org.apache.hadoop.mapred.Task: Task attempt_local937133229_0001_r_000000_0 is allowed to commit now 2017-02-19 04:45:34,849 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local937133229_0001_r_000000_0' to file:/home/cloudera/workspace/bpa/Assign1/output_Q2/_temporary/0/task_local937133229_0001_r_000000 2017-02-19 04:45:34,858 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 04:45:34,861 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local937133229_0001_r_000000_0' done. 2017-02-19 04:45:34,861 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local937133229_0001_r_000000_0 2017-02-19 04:45:34,862 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete. 2017-02-19 04:45:34,935 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local937133229_0001 java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpUriRequest at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:573) Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpUriRequest at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more 2017-02-19 04:45:35,032 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 100% 2017-02-19 04:45:35,032 INFO org.apache.hadoop.mapreduce.Job: Job job_local937133229_0001 failed with state FAILED due to: NA 2017-02-19 04:45:35,147 INFO org.apache.hadoop.mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=110497367 FILE: Number of bytes written=49147132 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=507535 Map output records=507535 Map output bytes=9364993 Map output materialized bytes=10380081 Input split bytes=351 Combine input records=0 Combine output records=0 Reduce input groups=87026 Reduce shuffle bytes=10380081 Reduce input records=507535 Reduce output records=87026 Spilled Records=1015070 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=785 Total committed heap usage (bytes)=773603328 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=26057874 File Output Format Counters Bytes Written=2051650 2017-02-19 05:44:00,142 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 05:44:03,627 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 05:44:03,629 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 05:44:05,788 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2017-02-19 05:44:05,917 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3 2017-02-19 05:44:06,422 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3 2017-02-19 05:44:08,041 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local633248467_0001 2017-02-19 05:44:10,090 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/ 2017-02-19 05:44:10,103 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local633248467_0001 2017-02-19 05:44:10,110 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null 2017-02-19 05:44:10,180 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 05:44:10,197 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2017-02-19 05:44:10,707 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks 2017-02-19 05:44:10,713 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local633248467_0001_m_000000_0 2017-02-19 05:44:11,039 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 05:44:11,109 INFO org.apache.hadoop.mapreduce.Job: Job job_local633248467_0001 running in uber mode : false 2017-02-19 05:44:11,160 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0% 2017-02-19 05:44:11,217 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 05:44:11,245 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935 2017-02-19 05:44:12,284 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 05:44:12,299 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 05:44:12,300 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 05:44:12,305 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 05:44:12,305 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 05:44:12,329 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 05:44:12,370 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 05:44:17,133 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:20,142 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:20,179 INFO org.apache.hadoop.mapreduce.Job: map 1% reduce 0% 2017-02-19 05:44:23,149 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:26,150 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:26,186 INFO org.apache.hadoop.mapreduce.Job: map 3% reduce 0% 2017-02-19 05:44:29,152 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:29,188 INFO org.apache.hadoop.mapreduce.Job: map 5% reduce 0% 2017-02-19 05:44:32,153 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:32,191 INFO org.apache.hadoop.mapreduce.Job: map 7% reduce 0% 2017-02-19 05:44:35,175 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:35,193 INFO org.apache.hadoop.mapreduce.Job: map 8% reduce 0% 2017-02-19 05:44:38,177 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:38,195 INFO org.apache.hadoop.mapreduce.Job: map 10% reduce 0% 2017-02-19 05:44:41,187 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:41,198 INFO org.apache.hadoop.mapreduce.Job: map 12% reduce 0% 2017-02-19 05:44:44,189 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:44,200 INFO org.apache.hadoop.mapreduce.Job: map 14% reduce 0% 2017-02-19 05:44:47,191 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:47,202 INFO org.apache.hadoop.mapreduce.Job: map 16% reduce 0% 2017-02-19 05:44:50,192 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:50,204 INFO org.apache.hadoop.mapreduce.Job: map 18% reduce 0% 2017-02-19 05:44:53,196 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:53,216 INFO org.apache.hadoop.mapreduce.Job: map 19% reduce 0% 2017-02-19 05:44:56,197 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:56,218 INFO org.apache.hadoop.mapreduce.Job: map 21% reduce 0% 2017-02-19 05:44:57,953 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:44:57,960 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 05:44:57,962 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 05:44:57,963 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5624480; bufvoid = 104857600 2017-02-19 05:44:57,963 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25005312(100021248); length = 1209085/6553600 2017-02-19 05:44:59,198 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 05:44:59,220 INFO org.apache.hadoop.mapreduce.Job: map 22% reduce 0% 2017-02-19 05:45:02,201 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 05:45:03,576 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 05:45:03,611 INFO org.apache.hadoop.mapred.Task: Task:attempt_local633248467_0001_m_000000_0 is done. And is in the process of committing 2017-02-19 05:45:03,621 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 05:45:03,629 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local633248467_0001_m_000000_0' done. 2017-02-19 05:45:03,632 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local633248467_0001_m_000000_0 2017-02-19 05:45:03,633 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local633248467_0001_m_000001_0 2017-02-19 05:45:03,654 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 05:45:03,655 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 05:45:03,661 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889 2017-02-19 05:45:04,158 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 05:45:04,163 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 05:45:04,164 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 05:45:04,165 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 05:45:04,166 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 05:45:04,176 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 05:45:04,189 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 05:45:04,224 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 05:45:09,662 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:45:10,229 INFO org.apache.hadoop.mapreduce.Job: map 41% reduce 0% 2017-02-19 05:45:12,664 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:45:13,232 INFO org.apache.hadoop.mapreduce.Job: map 46% reduce 0% 2017-02-19 05:45:15,665 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:45:16,236 INFO org.apache.hadoop.mapreduce.Job: map 50% reduce 0% 2017-02-19 05:45:18,668 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:45:19,017 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:45:19,022 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 05:45:19,023 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 05:45:19,024 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2143719; bufvoid = 104857600 2017-02-19 05:45:19,025 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25715252(102861008); length = 499145/6553600 2017-02-19 05:45:19,239 INFO org.apache.hadoop.mapreduce.Job: map 55% reduce 0% 2017-02-19 05:45:20,661 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 05:45:20,668 INFO org.apache.hadoop.mapred.Task: Task:attempt_local633248467_0001_m_000001_0 is done. And is in the process of committing 2017-02-19 05:45:20,685 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 05:45:20,700 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local633248467_0001_m_000001_0' done. 2017-02-19 05:45:20,702 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local633248467_0001_m_000001_0 2017-02-19 05:45:20,703 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local633248467_0001_m_000002_0 2017-02-19 05:45:20,712 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 05:45:20,713 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 05:45:20,729 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050 2017-02-19 05:45:21,222 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 05:45:21,231 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 05:45:21,232 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 05:45:21,234 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 05:45:21,235 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 05:45:21,241 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 05:45:21,245 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 05:45:21,252 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 05:45:26,726 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:45:27,250 INFO org.apache.hadoop.mapreduce.Job: map 79% reduce 0% 2017-02-19 05:45:29,727 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:45:30,252 INFO org.apache.hadoop.mapreduce.Job: map 86% reduce 0% 2017-02-19 05:45:31,116 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 05:45:31,121 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 05:45:31,122 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 05:45:31,123 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1596794; bufvoid = 104857600 2017-02-19 05:45:31,124 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25892496(103569984); length = 321901/6553600 2017-02-19 05:45:32,095 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 05:45:32,108 INFO org.apache.hadoop.mapred.Task: Task:attempt_local633248467_0001_m_000002_0 is done. And is in the process of committing 2017-02-19 05:45:32,124 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 05:45:32,131 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local633248467_0001_m_000002_0' done. 2017-02-19 05:45:32,131 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local633248467_0001_m_000002_0 2017-02-19 05:45:32,134 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete. 2017-02-19 05:45:32,150 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks 2017-02-19 05:45:32,152 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local633248467_0001_r_000000_0 2017-02-19 05:45:32,254 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 05:45:32,258 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 05:45:32,266 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 05:45:32,279 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@30e303ad 2017-02-19 05:45:32,425 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2017-02-19 05:45:32,477 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local633248467_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2017-02-19 05:45:32,759 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local633248467_0001_m_000002_0 decomp: 1757748 len: 1757752 to MEMORY 2017-02-19 05:45:32,811 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1757748 bytes from map-output for attempt_local633248467_0001_m_000002_0 2017-02-19 05:45:32,839 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1757748, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->1757748 2017-02-19 05:45:32,864 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local633248467_0001_m_000001_0 decomp: 2393295 len: 2393299 to MEMORY 2017-02-19 05:45:32,930 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 2393295 bytes from map-output for attempt_local633248467_0001_m_000001_0 2017-02-19 05:45:32,940 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2393295, inMemoryMapOutputs.size() -> 2, commitMemory -> 1757748, usedMemory ->4151043 2017-02-19 05:45:32,964 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local633248467_0001_m_000000_0 decomp: 6229026 len: 6229030 to MEMORY 2017-02-19 05:45:33,090 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 6229026 bytes from map-output for attempt_local633248467_0001_m_000000_0 2017-02-19 05:45:33,100 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 6229026, inMemoryMapOutputs.size() -> 3, commitMemory -> 4151043, usedMemory ->10380069 2017-02-19 05:45:33,107 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning 2017-02-19 05:45:33,109 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 05:45:33,110 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs 2017-02-19 05:45:33,167 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments 2017-02-19 05:45:33,171 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 10380058 bytes 2017-02-19 05:45:36,533 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 10380069 bytes to disk to satisfy reduce memory limit 2017-02-19 05:45:36,540 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 10380069 bytes from disk 2017-02-19 05:45:36,544 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2017-02-19 05:45:36,555 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2017-02-19 05:45:36,557 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10380062 bytes 2017-02-19 05:45:36,562 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 05:45:36,656 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2017-02-19 05:45:38,273 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 05:45:39,260 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 67% 2017-02-19 05:45:41,276 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 05:45:42,264 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 68% 2017-02-19 05:45:44,280 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 05:45:45,270 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 69% 2017-02-19 05:45:47,281 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 05:45:48,274 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 70% 2017-02-19 05:45:50,282 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 05:45:51,276 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 89% 2017-02-19 05:45:51,617 INFO org.apache.hadoop.mapred.Task: Task:attempt_local633248467_0001_r_000000_0 is done. And is in the process of committing 2017-02-19 05:45:51,631 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 05:45:51,640 INFO org.apache.hadoop.mapred.Task: Task attempt_local633248467_0001_r_000000_0 is allowed to commit now 2017-02-19 05:45:51,644 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local633248467_0001_r_000000_0' to file:/home/cloudera/workspace/bpa/Assign1/output_Q3/_temporary/0/task_local633248467_0001_r_000000 2017-02-19 05:45:51,652 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 05:45:51,656 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local633248467_0001_r_000000_0' done. 2017-02-19 05:45:51,657 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local633248467_0001_r_000000_0 2017-02-19 05:45:51,659 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete. 2017-02-19 05:45:52,015 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local633248467_0001 java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpUriRequest at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:573) Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpUriRequest at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more 2017-02-19 05:45:52,278 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 100% 2017-02-19 05:45:52,278 INFO org.apache.hadoop.mapreduce.Job: Job job_local633248467_0001 failed with state FAILED due to: NA 2017-02-19 05:45:52,450 INFO org.apache.hadoop.mapreduce.Job: Counters: 31 File System Counters FILE: Number of bytes read=110497367 FILE: Number of bytes written=48509622 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=507535 Map output records=507535 Map output bytes=9364993 Map output materialized bytes=10380081 Input split bytes=351 Combine input records=0 Combine output records=0 Reduce input groups=87026 Reduce shuffle bytes=10380081 Reduce input records=507535 Reduce output records=68697 Spilled Records=1015070 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=1157 Total committed heap usage (bytes)=773603328 Question3.InvertedIndex_Q3$MyCount UNIQUE_WORDS=68697 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=26057874 File Output Format Counters Bytes Written=1414068 2017-02-19 06:07:12,173 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 06:07:14,620 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 06:07:14,640 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 06:07:16,223 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2017-02-19 06:07:16,294 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3 2017-02-19 06:07:16,662 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3 2017-02-19 06:07:18,067 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local1094101727_0001 2017-02-19 06:07:19,791 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/ 2017-02-19 06:07:19,793 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local1094101727_0001 2017-02-19 06:07:19,806 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null 2017-02-19 06:07:19,881 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:07:19,899 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2017-02-19 06:07:20,438 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks 2017-02-19 06:07:20,439 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1094101727_0001_m_000000_0 2017-02-19 06:07:20,688 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:07:20,771 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:07:20,776 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935 2017-02-19 06:07:20,798 INFO org.apache.hadoop.mapreduce.Job: Job job_local1094101727_0001 running in uber mode : false 2017-02-19 06:07:20,799 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0% 2017-02-19 06:07:21,506 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:07:21,537 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:07:21,538 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:07:21,538 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:07:21,538 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:07:21,557 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:07:21,580 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:07:26,814 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:29,818 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:29,819 INFO org.apache.hadoop.mapreduce.Job: map 1% reduce 0% 2017-02-19 06:07:32,819 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:32,825 INFO org.apache.hadoop.mapreduce.Job: map 2% reduce 0% 2017-02-19 06:07:35,820 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:35,828 INFO org.apache.hadoop.mapreduce.Job: map 3% reduce 0% 2017-02-19 06:07:38,821 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:38,831 INFO org.apache.hadoop.mapreduce.Job: map 5% reduce 0% 2017-02-19 06:07:41,822 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:41,838 INFO org.apache.hadoop.mapreduce.Job: map 7% reduce 0% 2017-02-19 06:07:44,824 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:44,842 INFO org.apache.hadoop.mapreduce.Job: map 8% reduce 0% 2017-02-19 06:07:47,826 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:47,845 INFO org.apache.hadoop.mapreduce.Job: map 10% reduce 0% 2017-02-19 06:07:50,827 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:50,848 INFO org.apache.hadoop.mapreduce.Job: map 11% reduce 0% 2017-02-19 06:07:53,828 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:53,850 INFO org.apache.hadoop.mapreduce.Job: map 13% reduce 0% 2017-02-19 06:07:56,829 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:56,855 INFO org.apache.hadoop.mapreduce.Job: map 15% reduce 0% 2017-02-19 06:07:59,831 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:07:59,860 INFO org.apache.hadoop.mapreduce.Job: map 16% reduce 0% 2017-02-19 06:08:02,832 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:02,867 INFO org.apache.hadoop.mapreduce.Job: map 18% reduce 0% 2017-02-19 06:08:05,834 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:05,871 INFO org.apache.hadoop.mapreduce.Job: map 19% reduce 0% 2017-02-19 06:08:08,835 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:08,873 INFO org.apache.hadoop.mapreduce.Job: map 21% reduce 0% 2017-02-19 06:08:10,841 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:10,846 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:08:10,847 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:08:10,848 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5624480; bufvoid = 104857600 2017-02-19 06:08:10,850 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25005312(100021248); length = 1209085/6553600 2017-02-19 06:08:11,836 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:08:11,876 INFO org.apache.hadoop.mapreduce.Job: map 22% reduce 0% 2017-02-19 06:08:14,837 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:08:17,026 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:08:17,068 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1094101727_0001_m_000000_0 is done. And is in the process of committing 2017-02-19 06:08:17,088 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:08:17,092 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1094101727_0001_m_000000_0' done. 2017-02-19 06:08:17,094 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1094101727_0001_m_000000_0 2017-02-19 06:08:17,096 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1094101727_0001_m_000001_0 2017-02-19 06:08:17,116 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:08:17,117 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:08:17,119 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889 2017-02-19 06:08:17,607 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:08:17,632 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:08:17,633 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:08:17,633 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:08:17,634 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:08:17,645 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:08:17,655 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:08:17,882 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:08:23,124 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:23,904 INFO org.apache.hadoop.mapreduce.Job: map 40% reduce 0% 2017-02-19 06:08:26,128 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:26,909 INFO org.apache.hadoop.mapreduce.Job: map 44% reduce 0% 2017-02-19 06:08:29,130 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:29,911 INFO org.apache.hadoop.mapreduce.Job: map 48% reduce 0% 2017-02-19 06:08:32,131 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:32,913 INFO org.apache.hadoop.mapreduce.Job: map 52% reduce 0% 2017-02-19 06:08:34,434 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:34,438 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:08:34,439 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:08:34,440 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2143719; bufvoid = 104857600 2017-02-19 06:08:34,441 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25715252(102861008); length = 499145/6553600 2017-02-19 06:08:35,132 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:08:35,916 INFO org.apache.hadoop.mapreduce.Job: map 56% reduce 0% 2017-02-19 06:08:36,166 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:08:36,181 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1094101727_0001_m_000001_0 is done. And is in the process of committing 2017-02-19 06:08:36,190 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:08:36,199 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1094101727_0001_m_000001_0' done. 2017-02-19 06:08:36,201 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1094101727_0001_m_000001_0 2017-02-19 06:08:36,202 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1094101727_0001_m_000002_0 2017-02-19 06:08:36,214 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:08:36,215 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:08:36,227 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050 2017-02-19 06:08:36,745 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:08:36,752 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:08:36,754 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:08:36,755 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:08:36,756 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:08:36,765 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:08:36,819 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:08:36,917 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:08:42,230 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:42,924 INFO org.apache.hadoop.mapreduce.Job: map 78% reduce 0% 2017-02-19 06:08:45,232 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:45,926 INFO org.apache.hadoop.mapreduce.Job: map 83% reduce 0% 2017-02-19 06:08:48,066 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:08:48,071 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:08:48,073 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:08:48,074 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1596794; bufvoid = 104857600 2017-02-19 06:08:48,074 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25892496(103569984); length = 321901/6553600 2017-02-19 06:08:48,233 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:08:48,928 INFO org.apache.hadoop.mapreduce.Job: map 89% reduce 0% 2017-02-19 06:08:49,179 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:08:49,193 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1094101727_0001_m_000002_0 is done. And is in the process of committing 2017-02-19 06:08:49,202 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:08:49,210 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1094101727_0001_m_000002_0' done. 2017-02-19 06:08:49,212 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1094101727_0001_m_000002_0 2017-02-19 06:08:49,215 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete. 2017-02-19 06:08:49,262 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks 2017-02-19 06:08:49,263 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1094101727_0001_r_000000_0 2017-02-19 06:08:49,354 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:08:49,355 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:08:49,402 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@52e0b7ad 2017-02-19 06:08:49,579 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2017-02-19 06:08:49,641 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local1094101727_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2017-02-19 06:08:49,929 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:08:49,942 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1094101727_0001_m_000001_0 decomp: 2393295 len: 2393299 to MEMORY 2017-02-19 06:08:50,025 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 2393295 bytes from map-output for attempt_local1094101727_0001_m_000001_0 2017-02-19 06:08:50,052 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2393295, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2393295 2017-02-19 06:08:50,082 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1094101727_0001_m_000002_0 decomp: 1757748 len: 1757752 to MEMORY 2017-02-19 06:08:50,133 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1757748 bytes from map-output for attempt_local1094101727_0001_m_000002_0 2017-02-19 06:08:50,145 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1757748, inMemoryMapOutputs.size() -> 2, commitMemory -> 2393295, usedMemory ->4151043 2017-02-19 06:08:50,167 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1094101727_0001_m_000000_0 decomp: 6229026 len: 6229030 to MEMORY 2017-02-19 06:08:50,306 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 6229026 bytes from map-output for attempt_local1094101727_0001_m_000000_0 2017-02-19 06:08:50,314 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 6229026, inMemoryMapOutputs.size() -> 3, commitMemory -> 4151043, usedMemory ->10380069 2017-02-19 06:08:50,317 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning 2017-02-19 06:08:50,318 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 06:08:50,320 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs 2017-02-19 06:08:50,381 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments 2017-02-19 06:08:50,386 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 10380058 bytes 2017-02-19 06:08:53,992 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 10380069 bytes to disk to satisfy reduce memory limit 2017-02-19 06:08:53,998 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 10380069 bytes from disk 2017-02-19 06:08:53,999 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2017-02-19 06:08:53,999 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2017-02-19 06:08:53,999 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10380062 bytes 2017-02-19 06:08:54,000 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 06:08:54,192 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2017-02-19 06:08:55,381 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:08:55,933 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 67% 2017-02-19 06:08:58,383 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:08:58,937 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 77% 2017-02-19 06:09:01,380 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1094101727_0001_r_000000_0 is done. And is in the process of committing 2017-02-19 06:09:01,385 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:09:01,398 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:09:01,409 INFO org.apache.hadoop.mapred.Task: Task attempt_local1094101727_0001_r_000000_0 is allowed to commit now 2017-02-19 06:09:01,417 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local1094101727_0001_r_000000_0' to file:/home/cloudera/workspace/bpa/Assign1/output_Q3/_temporary/0/task_local1094101727_0001_r_000000 2017-02-19 06:09:01,424 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:09:01,428 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1094101727_0001_r_000000_0' done. 2017-02-19 06:09:01,430 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1094101727_0001_r_000000_0 2017-02-19 06:09:01,437 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete. 2017-02-19 06:09:01,539 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local1094101727_0001 java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpUriRequest at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:573) Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpUriRequest at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more 2017-02-19 06:09:01,939 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 100% 2017-02-19 06:09:01,940 INFO org.apache.hadoop.mapreduce.Job: Job job_local1094101727_0001 failed with state FAILED due to: NA 2017-02-19 06:09:02,087 INFO org.apache.hadoop.mapreduce.Job: Counters: 31 File System Counters FILE: Number of bytes read=110497367 FILE: Number of bytes written=48515478 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=507535 Map output records=507535 Map output bytes=9364993 Map output materialized bytes=10380081 Input split bytes=351 Combine input records=0 Combine output records=0 Reduce input groups=87026 Reduce shuffle bytes=10380081 Reduce input records=507535 Reduce output records=68697 Spilled Records=1015070 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=982 Total committed heap usage (bytes)=773603328 Question3.InvertedIndex_Q3$MyCount UNIQUE_WORDS=68697 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=26057874 File Output Format Counters Bytes Written=1414068 2017-02-19 06:24:25,789 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 06:24:28,926 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 06:24:28,969 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 06:24:31,230 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2017-02-19 06:24:31,381 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3 2017-02-19 06:24:31,959 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3 2017-02-19 06:24:33,848 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local1035144387_0001 2017-02-19 06:24:35,746 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/ 2017-02-19 06:24:35,760 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local1035144387_0001 2017-02-19 06:24:35,781 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null 2017-02-19 06:24:35,867 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:24:35,874 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2017-02-19 06:24:36,426 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks 2017-02-19 06:24:36,427 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1035144387_0001_m_000000_0 2017-02-19 06:24:36,732 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:24:36,786 INFO org.apache.hadoop.mapreduce.Job: Job job_local1035144387_0001 running in uber mode : false 2017-02-19 06:24:36,813 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0% 2017-02-19 06:24:36,911 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:24:36,954 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935 2017-02-19 06:24:38,146 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:24:38,146 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:24:38,146 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:24:38,146 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:24:38,147 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:24:38,238 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:24:38,275 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:24:42,846 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:24:45,871 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:24:46,848 INFO org.apache.hadoop.mapreduce.Job: map 1% reduce 0% 2017-02-19 06:24:48,873 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:24:51,874 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:24:52,859 INFO org.apache.hadoop.mapreduce.Job: map 2% reduce 0% 2017-02-19 06:24:54,875 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:24:55,863 INFO org.apache.hadoop.mapreduce.Job: map 4% reduce 0% 2017-02-19 06:24:57,879 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:24:58,870 INFO org.apache.hadoop.mapreduce.Job: map 5% reduce 0% 2017-02-19 06:25:00,881 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:01,873 INFO org.apache.hadoop.mapreduce.Job: map 7% reduce 0% 2017-02-19 06:25:03,883 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:04,877 INFO org.apache.hadoop.mapreduce.Job: map 9% reduce 0% 2017-02-19 06:25:06,884 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:07,883 INFO org.apache.hadoop.mapreduce.Job: map 11% reduce 0% 2017-02-19 06:25:09,886 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:09,886 INFO org.apache.hadoop.mapreduce.Job: map 13% reduce 0% 2017-02-19 06:25:12,887 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:12,891 INFO org.apache.hadoop.mapreduce.Job: map 15% reduce 0% 2017-02-19 06:25:15,892 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:15,896 INFO org.apache.hadoop.mapreduce.Job: map 17% reduce 0% 2017-02-19 06:25:18,893 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:18,899 INFO org.apache.hadoop.mapreduce.Job: map 18% reduce 0% 2017-02-19 06:25:21,896 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:21,903 INFO org.apache.hadoop.mapreduce.Job: map 20% reduce 0% 2017-02-19 06:25:24,755 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:24,760 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:25:24,764 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:25:24,765 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5624480; bufvoid = 104857600 2017-02-19 06:25:24,766 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25005312(100021248); length = 1209085/6553600 2017-02-19 06:25:24,899 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:25:24,909 INFO org.apache.hadoop.mapreduce.Job: map 22% reduce 0% 2017-02-19 06:25:27,900 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:25:30,902 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:25:33,157 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:25:33,208 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1035144387_0001_m_000000_0 is done. And is in the process of committing 2017-02-19 06:25:33,225 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:25:33,232 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1035144387_0001_m_000000_0' done. 2017-02-19 06:25:33,237 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1035144387_0001_m_000000_0 2017-02-19 06:25:33,238 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1035144387_0001_m_000001_0 2017-02-19 06:25:33,252 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:25:33,253 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:25:33,277 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889 2017-02-19 06:25:33,729 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:25:33,742 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:25:33,743 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:25:33,744 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:25:33,744 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:25:33,752 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:25:33,771 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:25:33,933 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:25:39,259 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:39,941 INFO org.apache.hadoop.mapreduce.Job: map 42% reduce 0% 2017-02-19 06:25:42,261 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:42,949 INFO org.apache.hadoop.mapreduce.Job: map 46% reduce 0% 2017-02-19 06:25:45,263 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:45,953 INFO org.apache.hadoop.mapreduce.Job: map 51% reduce 0% 2017-02-19 06:25:48,043 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:48,048 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:25:48,050 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:25:48,051 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2143719; bufvoid = 104857600 2017-02-19 06:25:48,051 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25715252(102861008); length = 499145/6553600 2017-02-19 06:25:48,265 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:25:48,973 INFO org.apache.hadoop.mapreduce.Job: map 56% reduce 0% 2017-02-19 06:25:50,123 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:25:50,143 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1035144387_0001_m_000001_0 is done. And is in the process of committing 2017-02-19 06:25:50,152 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:25:50,159 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1035144387_0001_m_000001_0' done. 2017-02-19 06:25:50,163 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1035144387_0001_m_000001_0 2017-02-19 06:25:50,165 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1035144387_0001_m_000002_0 2017-02-19 06:25:50,172 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:25:50,173 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:25:50,192 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050 2017-02-19 06:25:50,662 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:25:50,667 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:25:50,668 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:25:50,668 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:25:50,669 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:25:50,678 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:25:50,693 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:25:50,977 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:25:56,181 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:56,990 INFO org.apache.hadoop.mapreduce.Job: map 80% reduce 0% 2017-02-19 06:25:59,183 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:25:59,993 INFO org.apache.hadoop.mapreduce.Job: map 87% reduce 0% 2017-02-19 06:26:00,254 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:26:00,257 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:26:00,260 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:26:00,261 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1596794; bufvoid = 104857600 2017-02-19 06:26:00,261 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25892496(103569984); length = 321901/6553600 2017-02-19 06:26:01,183 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:26:01,196 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1035144387_0001_m_000002_0 is done. And is in the process of committing 2017-02-19 06:26:01,214 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:26:01,220 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1035144387_0001_m_000002_0' done. 2017-02-19 06:26:01,222 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1035144387_0001_m_000002_0 2017-02-19 06:26:01,224 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete. 2017-02-19 06:26:01,276 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks 2017-02-19 06:26:01,277 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1035144387_0001_r_000000_0 2017-02-19 06:26:01,355 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:26:01,356 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:26:01,398 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@639e075e 2017-02-19 06:26:01,562 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2017-02-19 06:26:01,592 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local1035144387_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2017-02-19 06:26:01,927 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1035144387_0001_m_000000_0 decomp: 1483680 len: 1483684 to MEMORY 2017-02-19 06:26:01,973 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1483680 bytes from map-output for attempt_local1035144387_0001_m_000000_0 2017-02-19 06:26:01,997 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:26:02,023 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1483680, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->1483680 2017-02-19 06:26:02,056 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1035144387_0001_m_000002_0 decomp: 467007 len: 467011 to MEMORY 2017-02-19 06:26:02,071 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 467007 bytes from map-output for attempt_local1035144387_0001_m_000002_0 2017-02-19 06:26:02,082 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 467007, inMemoryMapOutputs.size() -> 2, commitMemory -> 1483680, usedMemory ->1950687 2017-02-19 06:26:02,106 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1035144387_0001_m_000001_0 decomp: 692669 len: 692673 to MEMORY 2017-02-19 06:26:02,131 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 692669 bytes from map-output for attempt_local1035144387_0001_m_000001_0 2017-02-19 06:26:02,131 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 692669, inMemoryMapOutputs.size() -> 3, commitMemory -> 1950687, usedMemory ->2643356 2017-02-19 06:26:02,132 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning 2017-02-19 06:26:02,133 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 06:26:02,138 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs 2017-02-19 06:26:02,192 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments 2017-02-19 06:26:02,193 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 2643345 bytes 2017-02-19 06:26:03,600 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 2643356 bytes to disk to satisfy reduce memory limit 2017-02-19 06:26:03,609 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 2643356 bytes from disk 2017-02-19 06:26:03,612 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2017-02-19 06:26:03,619 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2017-02-19 06:26:03,622 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2643349 bytes 2017-02-19 06:26:03,630 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 06:26:03,728 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2017-02-19 06:26:07,366 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:26:08,009 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 70% 2017-02-19 06:26:10,368 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:26:11,028 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 74% 2017-02-19 06:26:13,370 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:26:14,035 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 81% 2017-02-19 06:26:14,865 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1035144387_0001_r_000000_0 is done. And is in the process of committing 2017-02-19 06:26:14,878 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:26:14,883 INFO org.apache.hadoop.mapred.Task: Task attempt_local1035144387_0001_r_000000_0 is allowed to commit now 2017-02-19 06:26:14,886 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local1035144387_0001_r_000000_0' to file:/home/cloudera/workspace/bpa/Assign1/output_Q4/_temporary/0/task_local1035144387_0001_r_000000 2017-02-19 06:26:14,895 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:26:14,900 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1035144387_0001_r_000000_0' done. 2017-02-19 06:26:14,902 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1035144387_0001_r_000000_0 2017-02-19 06:26:14,903 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete. 2017-02-19 06:26:15,001 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local1035144387_0001 java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpUriRequest at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:573) Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpUriRequest at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more 2017-02-19 06:26:15,036 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 100% 2017-02-19 06:26:15,037 INFO org.apache.hadoop.mapreduce.Job: Job job_local1035144387_0001 failed with state FAILED due to: NA 2017-02-19 06:26:15,204 INFO org.apache.hadoop.mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=95023941 FILE: Number of bytes written=15199671 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=507535 Map output records=507535 Map output bytes=9364993 Map output materialized bytes=2643368 Input split bytes=351 Combine input records=507535 Combine output records=110746 Reduce input groups=87026 Reduce shuffle bytes=2643368 Reduce input records=110746 Reduce output records=87026 Spilled Records=221492 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=1111 Total committed heap usage (bytes)=773603328 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=26057874 File Output Format Counters Bytes Written=2498094 2017-02-19 06:35:49,512 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 06:35:52,739 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 06:35:52,777 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 06:35:54,929 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2017-02-19 06:35:55,030 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3 2017-02-19 06:35:55,626 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3 2017-02-19 06:35:57,497 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local1683744245_0001 2017-02-19 06:35:59,406 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/ 2017-02-19 06:35:59,419 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local1683744245_0001 2017-02-19 06:35:59,435 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null 2017-02-19 06:35:59,501 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:35:59,517 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2017-02-19 06:35:59,966 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks 2017-02-19 06:35:59,968 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1683744245_0001_m_000000_0 2017-02-19 06:36:00,274 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:36:00,401 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:36:00,406 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935 2017-02-19 06:36:00,474 INFO org.apache.hadoop.mapreduce.Job: Job job_local1683744245_0001 running in uber mode : false 2017-02-19 06:36:00,475 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0% 2017-02-19 06:36:01,405 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:36:01,405 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:36:01,406 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:36:01,406 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:36:01,406 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:36:01,452 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:36:01,542 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:36:06,417 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:09,435 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:09,501 INFO org.apache.hadoop.mapreduce.Job: map 1% reduce 0% 2017-02-19 06:36:12,443 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:12,505 INFO org.apache.hadoop.mapreduce.Job: map 2% reduce 0% 2017-02-19 06:36:15,447 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:15,511 INFO org.apache.hadoop.mapreduce.Job: map 4% reduce 0% 2017-02-19 06:36:18,451 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:18,517 INFO org.apache.hadoop.mapreduce.Job: map 6% reduce 0% 2017-02-19 06:36:21,454 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:21,521 INFO org.apache.hadoop.mapreduce.Job: map 8% reduce 0% 2017-02-19 06:36:24,457 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:24,526 INFO org.apache.hadoop.mapreduce.Job: map 10% reduce 0% 2017-02-19 06:36:27,461 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:27,531 INFO org.apache.hadoop.mapreduce.Job: map 11% reduce 0% 2017-02-19 06:36:30,462 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:30,538 INFO org.apache.hadoop.mapreduce.Job: map 13% reduce 0% 2017-02-19 06:36:33,465 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:33,542 INFO org.apache.hadoop.mapreduce.Job: map 15% reduce 0% 2017-02-19 06:36:36,468 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:36,545 INFO org.apache.hadoop.mapreduce.Job: map 17% reduce 0% 2017-02-19 06:36:39,469 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:39,553 INFO org.apache.hadoop.mapreduce.Job: map 19% reduce 0% 2017-02-19 06:36:42,473 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:42,557 INFO org.apache.hadoop.mapreduce.Job: map 21% reduce 0% 2017-02-19 06:36:43,880 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:43,883 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:36:43,884 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:36:43,884 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5624480; bufvoid = 104857600 2017-02-19 06:36:43,884 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25005312(100021248); length = 1209085/6553600 2017-02-19 06:36:45,475 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:36:45,562 INFO org.apache.hadoop.mapreduce.Job: map 22% reduce 0% 2017-02-19 06:36:48,480 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:36:51,483 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:36:53,117 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:36:53,150 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1683744245_0001_m_000000_0 is done. And is in the process of committing 2017-02-19 06:36:53,161 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:36:53,169 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1683744245_0001_m_000000_0' done. 2017-02-19 06:36:53,170 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1683744245_0001_m_000000_0 2017-02-19 06:36:53,182 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1683744245_0001_m_000001_0 2017-02-19 06:36:53,195 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:36:53,196 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:36:53,209 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889 2017-02-19 06:36:53,580 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:36:53,646 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:36:53,653 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:36:53,654 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:36:53,654 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:36:53,655 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:36:53,663 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:36:53,676 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:36:59,229 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:36:59,598 INFO org.apache.hadoop.mapreduce.Job: map 42% reduce 0% 2017-02-19 06:37:02,239 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:37:02,603 INFO org.apache.hadoop.mapreduce.Job: map 47% reduce 0% 2017-02-19 06:37:05,242 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:37:05,607 INFO org.apache.hadoop.mapreduce.Job: map 52% reduce 0% 2017-02-19 06:37:07,505 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:37:07,508 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:37:07,508 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:37:07,508 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2143719; bufvoid = 104857600 2017-02-19 06:37:07,508 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25715252(102861008); length = 499145/6553600 2017-02-19 06:37:08,245 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:37:08,628 INFO org.apache.hadoop.mapreduce.Job: map 56% reduce 0% 2017-02-19 06:37:09,282 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:37:09,308 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1683744245_0001_m_000001_0 is done. And is in the process of committing 2017-02-19 06:37:09,319 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:37:09,327 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1683744245_0001_m_000001_0' done. 2017-02-19 06:37:09,328 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1683744245_0001_m_000001_0 2017-02-19 06:37:09,328 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1683744245_0001_m_000002_0 2017-02-19 06:37:09,342 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:37:09,343 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:37:09,361 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050 2017-02-19 06:37:09,657 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:37:09,804 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:37:09,816 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:37:09,817 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:37:09,818 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:37:09,819 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:37:09,828 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:37:09,835 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:37:15,352 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:37:15,677 INFO org.apache.hadoop.mapreduce.Job: map 80% reduce 0% 2017-02-19 06:37:18,354 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:37:18,683 INFO org.apache.hadoop.mapreduce.Job: map 88% reduce 0% 2017-02-19 06:37:18,890 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:37:18,894 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:37:18,895 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:37:18,896 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1596794; bufvoid = 104857600 2017-02-19 06:37:18,896 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25892496(103569984); length = 321901/6553600 2017-02-19 06:37:19,859 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:37:19,869 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1683744245_0001_m_000002_0 is done. And is in the process of committing 2017-02-19 06:37:19,880 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:37:19,888 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1683744245_0001_m_000002_0' done. 2017-02-19 06:37:19,889 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1683744245_0001_m_000002_0 2017-02-19 06:37:19,890 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete. 2017-02-19 06:37:19,902 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks 2017-02-19 06:37:19,903 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1683744245_0001_r_000000_0 2017-02-19 06:37:19,974 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:37:19,975 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:37:20,008 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@12f21700 2017-02-19 06:37:20,148 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2017-02-19 06:37:20,180 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local1683744245_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2017-02-19 06:37:20,452 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1683744245_0001_m_000001_0 decomp: 692669 len: 692673 to MEMORY 2017-02-19 06:37:20,484 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 692669 bytes from map-output for attempt_local1683744245_0001_m_000001_0 2017-02-19 06:37:20,501 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 692669, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->692669 2017-02-19 06:37:20,532 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1683744245_0001_m_000002_0 decomp: 467007 len: 467011 to MEMORY 2017-02-19 06:37:20,543 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 467007 bytes from map-output for attempt_local1683744245_0001_m_000002_0 2017-02-19 06:37:20,552 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 467007, inMemoryMapOutputs.size() -> 2, commitMemory -> 692669, usedMemory ->1159676 2017-02-19 06:37:20,560 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1683744245_0001_m_000000_0 decomp: 1483680 len: 1483684 to MEMORY 2017-02-19 06:37:20,607 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1483680 bytes from map-output for attempt_local1683744245_0001_m_000000_0 2017-02-19 06:37:20,613 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1483680, inMemoryMapOutputs.size() -> 3, commitMemory -> 1159676, usedMemory ->2643356 2017-02-19 06:37:20,616 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning 2017-02-19 06:37:20,618 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 06:37:20,619 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs 2017-02-19 06:37:20,665 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments 2017-02-19 06:37:20,679 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 2643345 bytes 2017-02-19 06:37:20,686 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:37:21,848 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 2643356 bytes to disk to satisfy reduce memory limit 2017-02-19 06:37:21,854 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 2643356 bytes from disk 2017-02-19 06:37:21,855 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2017-02-19 06:37:21,855 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2017-02-19 06:37:21,855 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2643349 bytes 2017-02-19 06:37:21,869 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 06:37:21,934 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2017-02-19 06:37:26,011 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:37:26,776 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 72% 2017-02-19 06:37:29,013 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:37:29,785 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 80% 2017-02-19 06:37:30,227 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1683744245_0001_r_000000_0 is done. And is in the process of committing 2017-02-19 06:37:30,242 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:37:30,244 INFO org.apache.hadoop.mapred.Task: Task attempt_local1683744245_0001_r_000000_0 is allowed to commit now 2017-02-19 06:37:30,256 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local1683744245_0001_r_000000_0' to file:/home/cloudera/workspace/bpa/Assign1/output_Q4/_temporary/0/task_local1683744245_0001_r_000000 2017-02-19 06:37:30,259 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:37:30,262 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1683744245_0001_r_000000_0' done. 2017-02-19 06:37:30,265 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1683744245_0001_r_000000_0 2017-02-19 06:37:30,268 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete. 2017-02-19 06:37:30,366 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local1683744245_0001 java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpUriRequest at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:573) Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpUriRequest at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more 2017-02-19 06:37:30,787 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 100% 2017-02-19 06:37:30,788 INFO org.apache.hadoop.mapreduce.Job: Job job_local1683744245_0001 failed with state FAILED due to: NA 2017-02-19 06:37:30,918 INFO org.apache.hadoop.mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=95023941 FILE: Number of bytes written=15199671 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=507535 Map output records=507535 Map output bytes=9364993 Map output materialized bytes=2643368 Input split bytes=351 Combine input records=507535 Combine output records=110746 Reduce input groups=87026 Reduce shuffle bytes=2643368 Reduce input records=110746 Reduce output records=87026 Spilled Records=221492 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=1032 Total committed heap usage (bytes)=773603328 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=26057874 File Output Format Counters Bytes Written=2498094 2017-02-19 06:39:22,054 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 06:39:25,215 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 06:39:25,250 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 06:39:27,050 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2017-02-19 06:39:27,147 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3 2017-02-19 06:39:27,618 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3 2017-02-19 06:39:29,201 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local818542887_0001 2017-02-19 06:39:31,058 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/ 2017-02-19 06:39:31,060 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local818542887_0001 2017-02-19 06:39:31,073 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null 2017-02-19 06:39:31,142 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:39:31,143 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2017-02-19 06:39:31,596 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks 2017-02-19 06:39:31,598 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local818542887_0001_m_000000_0 2017-02-19 06:39:31,906 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:39:32,019 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:39:32,032 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935 2017-02-19 06:39:32,081 INFO org.apache.hadoop.mapreduce.Job: Job job_local818542887_0001 running in uber mode : false 2017-02-19 06:39:32,082 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0% 2017-02-19 06:39:32,940 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:39:32,941 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:39:32,941 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:39:32,941 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:39:32,941 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:39:32,976 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:39:33,038 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:39:38,053 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:39:41,057 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:39:41,104 INFO org.apache.hadoop.mapreduce.Job: map 1% reduce 0% 2017-02-19 06:39:44,060 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:39:44,107 INFO org.apache.hadoop.mapreduce.Job: map 2% reduce 0% 2017-02-19 06:39:47,063 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:39:47,116 INFO org.apache.hadoop.mapreduce.Job: map 4% reduce 0% 2017-02-19 06:39:50,065 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:39:50,121 INFO org.apache.hadoop.mapreduce.Job: map 6% reduce 0% 2017-02-19 06:39:53,067 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:39:53,124 INFO org.apache.hadoop.mapreduce.Job: map 8% reduce 0% 2017-02-19 06:39:56,068 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:39:56,127 INFO org.apache.hadoop.mapreduce.Job: map 10% reduce 0% 2017-02-19 06:39:59,070 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:39:59,136 INFO org.apache.hadoop.mapreduce.Job: map 11% reduce 0% 2017-02-19 06:40:02,072 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:02,142 INFO org.apache.hadoop.mapreduce.Job: map 13% reduce 0% 2017-02-19 06:40:05,076 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:05,147 INFO org.apache.hadoop.mapreduce.Job: map 15% reduce 0% 2017-02-19 06:40:08,077 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:08,151 INFO org.apache.hadoop.mapreduce.Job: map 17% reduce 0% 2017-02-19 06:40:11,083 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:11,155 INFO org.apache.hadoop.mapreduce.Job: map 19% reduce 0% 2017-02-19 06:40:14,086 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:14,175 INFO org.apache.hadoop.mapreduce.Job: map 21% reduce 0% 2017-02-19 06:40:16,740 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:16,747 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:40:16,748 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:40:16,748 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5624480; bufvoid = 104857600 2017-02-19 06:40:16,749 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25005312(100021248); length = 1209085/6553600 2017-02-19 06:40:17,087 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:40:17,185 INFO org.apache.hadoop.mapreduce.Job: map 22% reduce 0% 2017-02-19 06:40:20,089 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:40:23,091 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 06:40:25,972 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:40:26,007 INFO org.apache.hadoop.mapred.Task: Task:attempt_local818542887_0001_m_000000_0 is done. And is in the process of committing 2017-02-19 06:40:26,020 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:40:26,033 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local818542887_0001_m_000000_0' done. 2017-02-19 06:40:26,038 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local818542887_0001_m_000000_0 2017-02-19 06:40:26,039 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local818542887_0001_m_000001_0 2017-02-19 06:40:26,050 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:40:26,051 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:40:26,053 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889 2017-02-19 06:40:26,351 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:40:26,512 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:40:26,518 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:40:26,520 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:40:26,520 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:40:26,520 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:40:26,530 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:40:26,539 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:40:32,090 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:32,357 INFO org.apache.hadoop.mapreduce.Job: map 41% reduce 0% 2017-02-19 06:40:35,092 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:35,369 INFO org.apache.hadoop.mapreduce.Job: map 46% reduce 0% 2017-02-19 06:40:38,093 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:38,374 INFO org.apache.hadoop.mapreduce.Job: map 51% reduce 0% 2017-02-19 06:40:41,095 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:41,130 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:41,132 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:40:41,133 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:40:41,134 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2143719; bufvoid = 104857600 2017-02-19 06:40:41,134 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25715252(102861008); length = 499145/6553600 2017-02-19 06:40:41,381 INFO org.apache.hadoop.mapreduce.Job: map 56% reduce 0% 2017-02-19 06:40:42,832 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:40:42,872 INFO org.apache.hadoop.mapred.Task: Task:attempt_local818542887_0001_m_000001_0 is done. And is in the process of committing 2017-02-19 06:40:42,878 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:40:42,883 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local818542887_0001_m_000001_0' done. 2017-02-19 06:40:42,895 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local818542887_0001_m_000001_0 2017-02-19 06:40:42,895 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local818542887_0001_m_000002_0 2017-02-19 06:40:42,901 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:40:42,902 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:40:42,911 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050 2017-02-19 06:40:43,372 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 06:40:43,379 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 06:40:43,380 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 06:40:43,380 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 06:40:43,384 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:40:43,385 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 06:40:43,390 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 06:40:43,392 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 06:40:48,909 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:49,404 INFO org.apache.hadoop.mapreduce.Job: map 79% reduce 0% 2017-02-19 06:40:51,911 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:52,409 INFO org.apache.hadoop.mapreduce.Job: map 86% reduce 0% 2017-02-19 06:40:53,146 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 06:40:53,150 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 06:40:53,152 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 06:40:53,153 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1596794; bufvoid = 104857600 2017-02-19 06:40:53,154 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25892496(103569984); length = 321901/6553600 2017-02-19 06:40:54,087 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 06:40:54,105 INFO org.apache.hadoop.mapred.Task: Task:attempt_local818542887_0001_m_000002_0 is done. And is in the process of committing 2017-02-19 06:40:54,117 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 06:40:54,123 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local818542887_0001_m_000002_0' done. 2017-02-19 06:40:54,124 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local818542887_0001_m_000002_0 2017-02-19 06:40:54,125 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete. 2017-02-19 06:40:54,137 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks 2017-02-19 06:40:54,137 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local818542887_0001_r_000000_0 2017-02-19 06:40:54,215 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 06:40:54,217 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 06:40:54,261 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@44d88759 2017-02-19 06:40:54,384 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2017-02-19 06:40:54,411 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local818542887_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2017-02-19 06:40:54,413 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 06:40:54,675 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local818542887_0001_m_000001_0 decomp: 694987 len: 694991 to MEMORY 2017-02-19 06:40:54,697 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 694987 bytes from map-output for attempt_local818542887_0001_m_000001_0 2017-02-19 06:40:54,725 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 694987, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->694987 2017-02-19 06:40:54,747 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local818542887_0001_m_000002_0 decomp: 468708 len: 468712 to MEMORY 2017-02-19 06:40:54,758 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 468708 bytes from map-output for attempt_local818542887_0001_m_000002_0 2017-02-19 06:40:54,767 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 468708, inMemoryMapOutputs.size() -> 2, commitMemory -> 694987, usedMemory ->1163695 2017-02-19 06:40:54,771 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local818542887_0001_m_000000_0 decomp: 1489328 len: 1489332 to MEMORY 2017-02-19 06:40:54,821 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1489328 bytes from map-output for attempt_local818542887_0001_m_000000_0 2017-02-19 06:40:54,822 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1489328, inMemoryMapOutputs.size() -> 3, commitMemory -> 1163695, usedMemory ->2653023 2017-02-19 06:40:54,827 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning 2017-02-19 06:40:54,829 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 06:40:54,830 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs 2017-02-19 06:40:54,871 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments 2017-02-19 06:40:54,892 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 2653012 bytes 2017-02-19 06:40:56,115 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 2653023 bytes to disk to satisfy reduce memory limit 2017-02-19 06:40:56,118 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 2653023 bytes from disk 2017-02-19 06:40:56,119 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2017-02-19 06:40:56,119 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2017-02-19 06:40:56,130 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2653016 bytes 2017-02-19 06:40:56,133 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 06:40:56,190 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2017-02-19 06:41:00,234 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:41:00,433 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 72% 2017-02-19 06:41:03,247 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:41:03,441 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 96% 2017-02-19 06:41:03,469 INFO org.apache.hadoop.mapred.Task: Task:attempt_local818542887_0001_r_000000_0 is done. And is in the process of committing 2017-02-19 06:41:03,477 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:41:03,485 INFO org.apache.hadoop.mapred.Task: Task attempt_local818542887_0001_r_000000_0 is allowed to commit now 2017-02-19 06:41:03,494 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local818542887_0001_r_000000_0' to file:/home/cloudera/workspace/bpa/Assign1/output_Q4/_temporary/0/task_local818542887_0001_r_000000 2017-02-19 06:41:03,497 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 06:41:03,500 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local818542887_0001_r_000000_0' done. 2017-02-19 06:41:03,503 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local818542887_0001_r_000000_0 2017-02-19 06:41:03,505 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete. 2017-02-19 06:41:03,585 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local818542887_0001 java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpUriRequest at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:573) Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpUriRequest at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more 2017-02-19 06:41:04,442 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 100% 2017-02-19 06:41:04,443 INFO org.apache.hadoop.mapreduce.Job: Job job_local818542887_0001 failed with state FAILED due to: NA 2017-02-19 06:41:04,548 INFO org.apache.hadoop.mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=95043275 FILE: Number of bytes written=15246149 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=507535 Map output records=507535 Map output bytes=9364993 Map output materialized bytes=2653035 Input split bytes=351 Combine input records=507535 Combine output records=110746 Reduce input groups=87026 Reduce shuffle bytes=2653035 Reduce input records=110746 Reduce output records=87026 Spilled Records=221492 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=1047 Total committed heap usage (bytes)=773603328 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=26057874 File Output Format Counters Bytes Written=2507837 2017-02-19 07:13:25,679 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 07:13:28,663 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 07:13:28,692 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 07:13:30,345 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2017-02-19 07:13:30,476 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3 2017-02-19 07:13:30,875 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3 2017-02-19 07:13:32,391 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local1212592229_0001 2017-02-19 07:13:34,151 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/ 2017-02-19 07:13:34,153 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local1212592229_0001 2017-02-19 07:13:34,164 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null 2017-02-19 07:13:34,233 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 07:13:34,241 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2017-02-19 07:13:34,662 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks 2017-02-19 07:13:34,664 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1212592229_0001_m_000000_0 2017-02-19 07:13:34,910 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 07:13:35,028 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 07:13:35,056 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935 2017-02-19 07:13:35,156 INFO org.apache.hadoop.mapreduce.Job: Job job_local1212592229_0001 running in uber mode : false 2017-02-19 07:13:35,158 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0% 2017-02-19 07:13:36,392 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 07:13:36,392 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 07:13:36,392 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 07:13:36,393 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 07:13:36,393 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 07:13:36,430 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 07:13:36,497 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 07:13:41,014 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:13:44,021 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:13:44,192 INFO org.apache.hadoop.mapreduce.Job: map 1% reduce 0% 2017-02-19 07:13:47,024 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:13:50,032 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:13:50,238 INFO org.apache.hadoop.mapreduce.Job: map 4% reduce 0% 2017-02-19 07:13:53,036 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:13:53,243 INFO org.apache.hadoop.mapreduce.Job: map 6% reduce 0% 2017-02-19 07:13:56,049 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:13:56,247 INFO org.apache.hadoop.mapreduce.Job: map 8% reduce 0% 2017-02-19 07:13:59,052 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:13:59,254 INFO org.apache.hadoop.mapreduce.Job: map 10% reduce 0% 2017-02-19 07:14:02,056 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:02,259 INFO org.apache.hadoop.mapreduce.Job: map 12% reduce 0% 2017-02-19 07:14:05,058 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:05,268 INFO org.apache.hadoop.mapreduce.Job: map 13% reduce 0% 2017-02-19 07:14:08,060 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:08,278 INFO org.apache.hadoop.mapreduce.Job: map 16% reduce 0% 2017-02-19 07:14:11,071 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:11,284 INFO org.apache.hadoop.mapreduce.Job: map 18% reduce 0% 2017-02-19 07:14:14,075 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:14,290 INFO org.apache.hadoop.mapreduce.Job: map 20% reduce 0% 2017-02-19 07:14:17,078 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:17,299 INFO org.apache.hadoop.mapreduce.Job: map 22% reduce 0% 2017-02-19 07:14:17,814 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:17,817 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 07:14:17,823 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 07:14:17,824 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5624480; bufvoid = 104857600 2017-02-19 07:14:17,824 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25005312(100021248); length = 1209085/6553600 2017-02-19 07:14:20,082 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 07:14:23,088 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 07:14:26,090 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 07:14:26,291 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 07:14:26,314 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1212592229_0001_m_000000_0 is done. And is in the process of committing 2017-02-19 07:14:26,325 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 07:14:26,332 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1212592229_0001_m_000000_0' done. 2017-02-19 07:14:26,334 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1212592229_0001_m_000000_0 2017-02-19 07:14:26,337 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1212592229_0001_m_000001_0 2017-02-19 07:14:26,355 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 07:14:26,356 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 07:14:26,363 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889 2017-02-19 07:14:26,783 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 07:14:26,800 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 07:14:26,801 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 07:14:26,801 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 07:14:26,801 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 07:14:26,811 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 07:14:26,820 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 07:14:27,322 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 07:14:32,367 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:33,331 INFO org.apache.hadoop.mapreduce.Job: map 42% reduce 0% 2017-02-19 07:14:35,370 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:36,338 INFO org.apache.hadoop.mapreduce.Job: map 47% reduce 0% 2017-02-19 07:14:38,371 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:39,347 INFO org.apache.hadoop.mapreduce.Job: map 52% reduce 0% 2017-02-19 07:14:40,357 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:40,363 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 07:14:40,363 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 07:14:40,363 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2143719; bufvoid = 104857600 2017-02-19 07:14:40,364 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25715252(102861008); length = 499145/6553600 2017-02-19 07:14:41,373 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 07:14:41,992 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 07:14:42,015 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1212592229_0001_m_000001_0 is done. And is in the process of committing 2017-02-19 07:14:42,024 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 07:14:42,030 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1212592229_0001_m_000001_0' done. 2017-02-19 07:14:42,032 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1212592229_0001_m_000001_0 2017-02-19 07:14:42,032 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1212592229_0001_m_000002_0 2017-02-19 07:14:42,039 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 07:14:42,040 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 07:14:42,055 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050 2017-02-19 07:14:42,366 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 07:14:42,461 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 07:14:42,486 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 07:14:42,487 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 07:14:42,487 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 07:14:42,487 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 07:14:42,492 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 07:14:42,493 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 07:14:48,047 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:48,377 INFO org.apache.hadoop.mapreduce.Job: map 81% reduce 0% 2017-02-19 07:14:51,049 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:51,381 INFO org.apache.hadoop.mapreduce.Job: map 88% reduce 0% 2017-02-19 07:14:51,477 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:14:51,479 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 07:14:51,480 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 07:14:51,480 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1596794; bufvoid = 104857600 2017-02-19 07:14:51,481 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25892496(103569984); length = 321901/6553600 2017-02-19 07:14:52,419 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 07:14:52,428 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1212592229_0001_m_000002_0 is done. And is in the process of committing 2017-02-19 07:14:52,440 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 07:14:52,446 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1212592229_0001_m_000002_0' done. 2017-02-19 07:14:52,450 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1212592229_0001_m_000002_0 2017-02-19 07:14:52,452 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete. 2017-02-19 07:14:52,466 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks 2017-02-19 07:14:52,467 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1212592229_0001_r_000000_0 2017-02-19 07:14:52,542 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 07:14:52,557 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 07:14:52,584 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4127f9f0 2017-02-19 07:14:52,704 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2017-02-19 07:14:52,741 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local1212592229_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2017-02-19 07:14:52,979 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1212592229_0001_m_000002_0 decomp: 450109 len: 450113 to MEMORY 2017-02-19 07:14:53,004 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 450109 bytes from map-output for attempt_local1212592229_0001_m_000002_0 2017-02-19 07:14:53,015 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 450109, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->450109 2017-02-19 07:14:53,053 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1212592229_0001_m_000001_0 decomp: 663948 len: 663952 to MEMORY 2017-02-19 07:14:53,059 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 663948 bytes from map-output for attempt_local1212592229_0001_m_000001_0 2017-02-19 07:14:53,073 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 663948, inMemoryMapOutputs.size() -> 2, commitMemory -> 450109, usedMemory ->1114057 2017-02-19 07:14:53,079 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1212592229_0001_m_000000_0 decomp: 1428220 len: 1428224 to MEMORY 2017-02-19 07:14:53,107 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1428220 bytes from map-output for attempt_local1212592229_0001_m_000000_0 2017-02-19 07:14:53,122 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1428220, inMemoryMapOutputs.size() -> 3, commitMemory -> 1114057, usedMemory ->2542277 2017-02-19 07:14:53,123 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning 2017-02-19 07:14:53,124 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 07:14:53,124 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs 2017-02-19 07:14:53,167 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments 2017-02-19 07:14:53,183 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 2542266 bytes 2017-02-19 07:14:53,386 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 07:14:54,371 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 2542277 bytes to disk to satisfy reduce memory limit 2017-02-19 07:14:54,375 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 2542277 bytes from disk 2017-02-19 07:14:54,382 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2017-02-19 07:14:54,387 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2017-02-19 07:14:54,394 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2542270 bytes 2017-02-19 07:14:54,398 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 07:14:54,500 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2017-02-19 07:14:58,600 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 07:14:59,397 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 72% 2017-02-19 07:15:01,601 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 07:15:02,406 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 78% 2017-02-19 07:15:03,504 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1212592229_0001_r_000000_0 is done. And is in the process of committing 2017-02-19 07:15:03,510 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 07:15:03,510 INFO org.apache.hadoop.mapred.Task: Task attempt_local1212592229_0001_r_000000_0 is allowed to commit now 2017-02-19 07:15:03,524 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local1212592229_0001_r_000000_0' to file:/home/cloudera/workspace/bpa/Assign1/output_Q4/_temporary/0/task_local1212592229_0001_r_000000 2017-02-19 07:15:03,529 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 07:15:03,532 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1212592229_0001_r_000000_0' done. 2017-02-19 07:15:03,533 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1212592229_0001_r_000000_0 2017-02-19 07:15:03,536 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete. 2017-02-19 07:15:03,660 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local1212592229_0001 java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpUriRequest at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:573) Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpUriRequest at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more 2017-02-19 07:15:04,412 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 100% 2017-02-19 07:15:04,413 INFO org.apache.hadoop.mapreduce.Job: Job job_local1212592229_0001 failed with state FAILED due to: NA 2017-02-19 07:15:04,568 INFO org.apache.hadoop.mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=94821783 FILE: Number of bytes written=14543312 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=507535 Map output records=507535 Map output bytes=9364993 Map output materialized bytes=2542289 Input split bytes=351 Combine input records=507535 Combine output records=110746 Reduce input groups=87026 Reduce shuffle bytes=2542289 Reduce input records=110746 Reduce output records=87026 Spilled Records=221492 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=994 Total committed heap usage (bytes)=773603328 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=26057874 File Output Format Counters Bytes Written=2284613 2017-02-19 07:45:24,231 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-02-19 07:45:27,338 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2017-02-19 07:45:27,371 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 2017-02-19 07:45:29,491 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2017-02-19 07:45:29,618 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 3 2017-02-19 07:45:30,133 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:3 2017-02-19 07:45:31,716 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local1717752541_0001 2017-02-19 07:45:33,420 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/ 2017-02-19 07:45:33,421 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local1717752541_0001 2017-02-19 07:45:33,437 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null 2017-02-19 07:45:33,501 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 07:45:33,513 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2017-02-19 07:45:33,927 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks 2017-02-19 07:45:33,928 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1717752541_0001_m_000000_0 2017-02-19 07:45:34,219 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 07:45:34,362 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 07:45:34,366 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg3200.txt:0+16013935 2017-02-19 07:45:34,427 INFO org.apache.hadoop.mapreduce.Job: Job job_local1717752541_0001 running in uber mode : false 2017-02-19 07:45:34,430 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0% 2017-02-19 07:45:35,890 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 07:45:35,891 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 07:45:35,891 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 07:45:35,891 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 07:45:35,891 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 07:45:35,968 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 07:45:37,206 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 07:45:40,350 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:45:43,355 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:45:43,463 INFO org.apache.hadoop.mapreduce.Job: map 1% reduce 0% 2017-02-19 07:45:46,360 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:45:46,490 INFO org.apache.hadoop.mapreduce.Job: map 2% reduce 0% 2017-02-19 07:45:49,363 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:45:49,499 INFO org.apache.hadoop.mapreduce.Job: map 4% reduce 0% 2017-02-19 07:45:52,366 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:45:52,510 INFO org.apache.hadoop.mapreduce.Job: map 6% reduce 0% 2017-02-19 07:45:55,369 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:45:55,522 INFO org.apache.hadoop.mapreduce.Job: map 8% reduce 0% 2017-02-19 07:45:58,374 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:45:58,531 INFO org.apache.hadoop.mapreduce.Job: map 10% reduce 0% 2017-02-19 07:46:01,378 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:01,546 INFO org.apache.hadoop.mapreduce.Job: map 12% reduce 0% 2017-02-19 07:46:04,382 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:04,558 INFO org.apache.hadoop.mapreduce.Job: map 13% reduce 0% 2017-02-19 07:46:07,387 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:07,562 INFO org.apache.hadoop.mapreduce.Job: map 16% reduce 0% 2017-02-19 07:46:10,389 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:10,569 INFO org.apache.hadoop.mapreduce.Job: map 18% reduce 0% 2017-02-19 07:46:13,394 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:13,575 INFO org.apache.hadoop.mapreduce.Job: map 20% reduce 0% 2017-02-19 07:46:16,399 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:16,581 INFO org.apache.hadoop.mapreduce.Job: map 22% reduce 0% 2017-02-19 07:46:17,197 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:17,202 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 07:46:17,204 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 07:46:17,205 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5624480; bufvoid = 104857600 2017-02-19 07:46:17,209 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25005312(100021248); length = 1209085/6553600 2017-02-19 07:46:19,400 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 07:46:22,402 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 07:46:25,405 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 07:46:25,888 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 07:46:25,912 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1717752541_0001_m_000000_0 is done. And is in the process of committing 2017-02-19 07:46:25,924 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 07:46:25,928 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1717752541_0001_m_000000_0' done. 2017-02-19 07:46:25,929 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1717752541_0001_m_000000_0 2017-02-19 07:46:25,930 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1717752541_0001_m_000001_0 2017-02-19 07:46:25,949 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 07:46:25,950 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 07:46:25,952 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg100.txt:0+5589889 2017-02-19 07:46:26,357 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 07:46:26,375 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 07:46:26,376 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 07:46:26,376 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 07:46:26,376 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 07:46:26,381 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 07:46:26,429 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 07:46:26,633 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 07:46:31,955 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:32,648 INFO org.apache.hadoop.mapreduce.Job: map 42% reduce 0% 2017-02-19 07:46:34,968 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:35,653 INFO org.apache.hadoop.mapreduce.Job: map 47% reduce 0% 2017-02-19 07:46:37,977 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:38,657 INFO org.apache.hadoop.mapreduce.Job: map 52% reduce 0% 2017-02-19 07:46:39,867 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:39,869 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 07:46:39,869 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 07:46:39,869 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 2143719; bufvoid = 104857600 2017-02-19 07:46:39,869 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25715252(102861008); length = 499145/6553600 2017-02-19 07:46:40,978 INFO org.apache.hadoop.mapred.LocalJobRunner: map > sort 2017-02-19 07:46:41,619 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 07:46:41,633 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1717752541_0001_m_000001_0 is done. And is in the process of committing 2017-02-19 07:46:41,645 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 07:46:41,648 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1717752541_0001_m_000001_0' done. 2017-02-19 07:46:41,649 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1717752541_0001_m_000001_0 2017-02-19 07:46:41,650 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1717752541_0001_m_000002_0 2017-02-19 07:46:41,658 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 07:46:41,658 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 07:46:41,666 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 07:46:41,683 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/bpa/Assign1/pg31100.txt:0+4454050 2017-02-19 07:46:42,089 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2017-02-19 07:46:42,094 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2017-02-19 07:46:42,094 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2017-02-19 07:46:42,094 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2017-02-19 07:46:42,094 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2017-02-19 07:46:42,099 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2017-02-19 07:46:42,141 INFO org.apache.hadoop.mapreduce.lib.input.LineRecordReader: Found UTF-8 BOM and skipped it 2017-02-19 07:46:47,678 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:47,684 INFO org.apache.hadoop.mapreduce.Job: map 81% reduce 0% 2017-02-19 07:46:50,681 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:50,695 INFO org.apache.hadoop.mapreduce.Job: map 88% reduce 0% 2017-02-19 07:46:51,012 INFO org.apache.hadoop.mapred.LocalJobRunner: map > map 2017-02-19 07:46:51,014 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2017-02-19 07:46:51,015 INFO org.apache.hadoop.mapred.MapTask: Spilling map output 2017-02-19 07:46:51,015 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 1596794; bufvoid = 104857600 2017-02-19 07:46:51,020 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25892496(103569984); length = 321901/6553600 2017-02-19 07:46:51,903 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2017-02-19 07:46:51,940 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1717752541_0001_m_000002_0 is done. And is in the process of committing 2017-02-19 07:46:51,946 INFO org.apache.hadoop.mapred.LocalJobRunner: map 2017-02-19 07:46:51,954 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1717752541_0001_m_000002_0' done. 2017-02-19 07:46:51,955 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1717752541_0001_m_000002_0 2017-02-19 07:46:51,956 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete. 2017-02-19 07:46:51,997 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks 2017-02-19 07:46:51,999 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local1717752541_0001_r_000000_0 2017-02-19 07:46:52,071 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-02-19 07:46:52,081 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2017-02-19 07:46:52,111 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@278a1cfb 2017-02-19 07:46:52,263 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2017-02-19 07:46:52,294 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local1717752541_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2017-02-19 07:46:52,564 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1717752541_0001_m_000001_0 decomp: 694987 len: 694991 to MEMORY 2017-02-19 07:46:52,584 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 694987 bytes from map-output for attempt_local1717752541_0001_m_000001_0 2017-02-19 07:46:52,601 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 694987, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->694987 2017-02-19 07:46:52,638 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1717752541_0001_m_000002_0 decomp: 468708 len: 468712 to MEMORY 2017-02-19 07:46:52,652 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 468708 bytes from map-output for attempt_local1717752541_0001_m_000002_0 2017-02-19 07:46:52,655 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 468708, inMemoryMapOutputs.size() -> 2, commitMemory -> 694987, usedMemory ->1163695 2017-02-19 07:46:52,662 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1717752541_0001_m_000000_0 decomp: 1489328 len: 1489332 to MEMORY 2017-02-19 07:46:52,674 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1489328 bytes from map-output for attempt_local1717752541_0001_m_000000_0 2017-02-19 07:46:52,697 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0% 2017-02-19 07:46:52,706 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1489328, inMemoryMapOutputs.size() -> 3, commitMemory -> 1163695, usedMemory ->2653023 2017-02-19 07:46:52,706 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning 2017-02-19 07:46:52,708 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 07:46:52,709 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs 2017-02-19 07:46:52,771 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments 2017-02-19 07:46:52,774 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 2653012 bytes 2017-02-19 07:46:54,022 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 3 segments, 2653023 bytes to disk to satisfy reduce memory limit 2017-02-19 07:46:54,029 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 2653023 bytes from disk 2017-02-19 07:46:54,068 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2017-02-19 07:46:54,068 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2017-02-19 07:46:54,069 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2653016 bytes 2017-02-19 07:46:54,069 INFO org.apache.hadoop.mapred.LocalJobRunner: 3 / 3 copied. 2017-02-19 07:46:54,238 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2017-02-19 07:46:58,091 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 07:46:58,736 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 72% 2017-02-19 07:47:01,096 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 07:47:01,744 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 81% 2017-02-19 07:47:02,469 INFO org.apache.hadoop.mapred.Task: Task:attempt_local1717752541_0001_r_000000_0 is done. And is in the process of committing 2017-02-19 07:47:02,474 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 07:47:02,474 INFO org.apache.hadoop.mapred.Task: Task attempt_local1717752541_0001_r_000000_0 is allowed to commit now 2017-02-19 07:47:02,486 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local1717752541_0001_r_000000_0' to file:/home/cloudera/workspace/bpa/Assign1/output_Q4/_temporary/0/task_local1717752541_0001_r_000000 2017-02-19 07:47:02,489 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce 2017-02-19 07:47:02,493 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local1717752541_0001_r_000000_0' done. 2017-02-19 07:47:02,495 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local1717752541_0001_r_000000_0 2017-02-19 07:47:02,496 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete. 2017-02-19 07:47:02,617 WARN org.apache.hadoop.mapred.LocalJobRunner: job_local1717752541_0001 java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpUriRequest at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:573) Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpUriRequest at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more 2017-02-19 07:47:02,747 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 100% 2017-02-19 07:47:02,747 INFO org.apache.hadoop.mapreduce.Job: Job job_local1717752541_0001 failed with state FAILED due to: NA 2017-02-19 07:47:02,935 INFO org.apache.hadoop.mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=95043275 FILE: Number of bytes written=15252029 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=507535 Map output records=507535 Map output bytes=9364993 Map output materialized bytes=2653035 Input split bytes=351 Combine input records=507535 Combine output records=110746 Reduce input groups=87026 Reduce shuffle bytes=2653035 Reduce input records=110746 Reduce output records=87026 Spilled Records=221492 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=954 Total committed heap usage (bytes)=773603328 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=26057874 File Output Format Counters Bytes Written=2507837