Skip to content
Snippets Groups Projects
Commit 60f1f8c1 authored by Wen Yao Jin's avatar Wen Yao Jin
Browse files

go

parent 7cbfe4da
Branches
No related tags found
No related merge requests found
...@@ -1549,3 +1549,119 @@ Caused by: java.lang.NullPointerException ...@@ -1549,3 +1549,119 @@ Caused by: java.lang.NullPointerException
Bytes Written=5365335 Bytes Written=5365335
similarity.WordSort$DocLineCounter similarity.WordSort$DocLineCounter
NUM=124787 NUM=124787
2017-03-10 16:04:16,924 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-03-10 16:04:17,595 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2017-03-10 16:04:17,605 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2017-03-10 16:04:17,661 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/home/cloudera/workspace/WordCount/output already exists
2017-03-10 16:04:38,662 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-03-10 16:04:39,236 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2017-03-10 16:04:39,243 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2017-03-10 16:04:39,836 WARN org.apache.hadoop.mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2017-03-10 16:04:39,846 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 1
2017-03-10 16:04:39,896 INFO org.apache.hadoop.mapreduce.JobSubmitter: number of splits:1
2017-03-10 16:04:40,232 INFO org.apache.hadoop.mapreduce.JobSubmitter: Submitting tokens for job: job_local2107526736_0001
2017-03-10 16:04:40,744 INFO org.apache.hadoop.mapreduce.Job: The url to track the job: http://localhost:8080/
2017-03-10 16:04:40,745 INFO org.apache.hadoop.mapreduce.Job: Running job: job_local2107526736_0001
2017-03-10 16:04:40,753 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter set in config null
2017-03-10 16:04:40,762 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-03-10 16:04:40,772 INFO org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2017-03-10 16:04:40,922 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for map tasks
2017-03-10 16:04:40,925 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local2107526736_0001_m_000000_0
2017-03-10 16:04:40,998 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-03-10 16:04:41,021 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2017-03-10 16:04:41,024 INFO org.apache.hadoop.mapred.MapTask: Processing split: file:/home/cloudera/workspace/WordCount/input/pg100.txt:0+5589886
2017-03-10 16:04:41,117 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2017-03-10 16:04:41,117 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100
2017-03-10 16:04:41,117 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2017-03-10 16:04:41,117 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600
2017-03-10 16:04:41,117 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2017-03-10 16:04:41,121 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-03-10 16:04:41,747 INFO org.apache.hadoop.mapreduce.Job: Job job_local2107526736_0001 running in uber mode : false
2017-03-10 16:04:41,749 INFO org.apache.hadoop.mapreduce.Job: map 0% reduce 0%
2017-03-10 16:04:44,514 INFO org.apache.hadoop.mapred.LocalJobRunner:
2017-03-10 16:04:44,515 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2017-03-10 16:04:44,515 INFO org.apache.hadoop.mapred.MapTask: Spilling map output
2017-03-10 16:04:44,515 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 5541084; bufvoid = 104857600
2017-03-10 16:04:44,515 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 25753980(103015920); length = 460417/6553600
2017-03-10 16:04:44,850 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
2017-03-10 16:04:45,645 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2017-03-10 16:04:45,648 INFO org.apache.hadoop.mapred.Task: Task:attempt_local2107526736_0001_m_000000_0 is done. And is in the process of committing
2017-03-10 16:04:45,653 INFO org.apache.hadoop.mapred.LocalJobRunner: map
2017-03-10 16:04:45,654 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local2107526736_0001_m_000000_0' done.
2017-03-10 16:04:45,654 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local2107526736_0001_m_000000_0
2017-03-10 16:04:45,654 INFO org.apache.hadoop.mapred.LocalJobRunner: map task executor complete.
2017-03-10 16:04:45,658 INFO org.apache.hadoop.mapred.LocalJobRunner: Waiting for reduce tasks
2017-03-10 16:04:45,659 INFO org.apache.hadoop.mapred.LocalJobRunner: Starting task: attempt_local2107526736_0001_r_000000_0
2017-03-10 16:04:45,664 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-03-10 16:04:45,664 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2017-03-10 16:04:45,666 INFO org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@a9b03af
2017-03-10 16:04:45,680 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=679778688, maxSingleShuffleLimit=169944672, mergeThreshold=448653952, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2017-03-10 16:04:45,687 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_local2107526736_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2017-03-10 16:04:45,730 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.deflate]
2017-03-10 16:04:45,730 INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local2107526736_0001_m_000000_0 decomp: 5771296 len: 2404026 to MEMORY
2017-03-10 16:04:45,758 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 0%
2017-03-10 16:04:45,813 INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 5771296 bytes from map-output for attempt_local2107526736_0001_m_000000_0
2017-03-10 16:04:45,813 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 5771296, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->5771296
2017-03-10 16:04:45,817 INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning
2017-03-10 16:04:45,818 INFO org.apache.hadoop.mapred.LocalJobRunner: 1 / 1 copied.
2017-03-10 16:04:45,818 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2017-03-10 16:04:45,825 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments
2017-03-10 16:04:45,826 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 5771286 bytes
2017-03-10 16:04:46,671 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 1 segments, 5771296 bytes to disk to satisfy reduce memory limit
2017-03-10 16:04:46,672 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 2404034 bytes from disk
2017-03-10 16:04:46,672 INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2017-03-10 16:04:46,672 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments
2017-03-10 16:04:46,673 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 5771286 bytes
2017-03-10 16:04:46,674 INFO org.apache.hadoop.mapred.LocalJobRunner: 1 / 1 copied.
2017-03-10 16:04:46,680 INFO org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2017-03-10 16:04:47,525 INFO org.apache.hadoop.mapred.Task: Task:attempt_local2107526736_0001_r_000000_0 is done. And is in the process of committing
2017-03-10 16:04:47,532 INFO org.apache.hadoop.mapred.LocalJobRunner: 1 / 1 copied.
2017-03-10 16:04:47,532 INFO org.apache.hadoop.mapred.Task: Task attempt_local2107526736_0001_r_000000_0 is allowed to commit now
2017-03-10 16:04:47,533 INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_local2107526736_0001_r_000000_0' to file:/home/cloudera/workspace/WordCount/output/_temporary/0/task_local2107526736_0001_r_000000
2017-03-10 16:04:47,533 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce > reduce
2017-03-10 16:04:47,534 INFO org.apache.hadoop.mapred.Task: Task 'attempt_local2107526736_0001_r_000000_0' done.
2017-03-10 16:04:47,534 INFO org.apache.hadoop.mapred.LocalJobRunner: Finishing task: attempt_local2107526736_0001_r_000000_0
2017-03-10 16:04:47,534 INFO org.apache.hadoop.mapred.LocalJobRunner: reduce task executor complete.
2017-03-10 16:04:47,766 INFO org.apache.hadoop.mapreduce.Job: map 100% reduce 100%
2017-03-10 16:04:47,767 INFO org.apache.hadoop.mapreduce.Job: Job job_local2107526736_0001 completed successfully
2017-03-10 16:04:47,785 INFO org.apache.hadoop.mapreduce.Job: Counters: 34
File System Counters
FILE: Number of bytes read=16712056
FILE: Number of bytes written=13108535
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=124787
Map output records=115105
Map output bytes=5541084
Map output materialized bytes=2404026
Input split bytes=120
Combine input records=0
Combine output records=0
Reduce input groups=115105
Reduce shuffle bytes=2404026
Reduce input records=115105
Reduce output records=115105
Spilled Records=230210
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=142
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=331227136
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=5589886
File Output Format Counters
Bytes Written=5365335
similarity.WordSort$DocLineCounter
NUM=124787
This diff is collapsed.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment