hive--map数量过小导致hive运行缓慢_hive在map阶段特别的慢-程序员宅基地

技术标签： 错误集锦  hive  

map数量过小导致hive运行缓慢

  • 网页显示
    在这里插入图片描述
  • 日志
    • Not uberizing job_1607937945101_247633 because: not enabled; too much RAM;
    • nput size for job job_1607937945101_246670 = 38627311. Number of splits = 1
    • Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0这条一直出现。。。。。。。
  • 思路
    • 数据量过大,map数过少
    • 调整 参数
      • set mapred.max.split.size=25000000;
      • set mapred.min.split.size.per.node=25000000;
      • set mapred.min.split.size.per.rack=25000000;
    • 再次查看日志 Number of splits = 4，速度大大加快
    • 如果不行（则应该是压缩不可切分造成）
  • 完整日志可以参考下面

Logged in as: yarn
Application
About
Jobs
Tools
Log Type: syslog

Log Upload Time: 星期一 三月 01 18:18:41 +0800 2021

Log Length: 169813

2021-03-01 16:57:51,172 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1607937945101_246670_000001
2021-03-01 16:57:51,410 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2021-03-01 16:57:51,410 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@33bc72d1)
2021-03-01 16:57:51,604 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter
2021-03-01 16:57:51,606 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter
2021-03-01 16:57:52,081 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-03-01 16:57:52,215 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2021-03-01 16:57:52,216 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2021-03-01 16:57:52,216 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2021-03-01 16:57:52,217 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2021-03-01 16:57:52,217 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2021-03-01 16:57:52,221 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2021-03-01 16:57:52,221 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2021-03-01 16:57:52,222 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2021-03-01 16:57:52,257 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:52,277 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:52,293 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:52,306 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2021-03-01 16:57:52,340 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2021-03-01 16:57:52,536 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2021-03-01 16:57:52,591 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2021-03-01 16:57:52,591 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2021-03-01 16:57:52,603 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1607937945101_246670 to jobTokenSecretManager
2021-03-01 16:57:52,709 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1607937945101_246670 because: not enabled; too much RAM;
2021-03-01 16:57:52,729 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1607937945101_246670 = 38627311. Number of splits = 1
2021-03-01 16:57:52,729 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1607937945101_246670 = 0
2021-03-01 16:57:52,729 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from NEW to INITED
2021-03-01 16:57:52,730 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1607937945101_246670.
2021-03-01 16:57:52,753 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
2021-03-01 16:57:52,762 INFO [Socket Reader #1 for port 36866] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36866
2021-03-01 16:57:52,793 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2021-03-01 16:57:52,826 INFO [IPC Server listener on 36866] org.apache.hadoop.ipc.Server: IPC Server listener on 36866: starting
2021-03-01 16:57:52,826 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2021-03-01 16:57:52,828 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at cdhuatdn62-209.com/10.9.62.209:36866
2021-03-01 16:57:52,883 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2021-03-01 16:57:52,889 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2021-03-01 16:57:52,893 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2021-03-01 16:57:52,902 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2021-03-01 16:57:52,932 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2021-03-01 16:57:52,932 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2021-03-01 16:57:52,934 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2021-03-01 16:57:52,934 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2021-03-01 16:57:52,941 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 46085
2021-03-01 16:57:52,941 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2021-03-01 16:57:52,964 INFO [main] org.mortbay.log: Extract jar:file:/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/jars/hadoop-yarn-common-2.6.0-cdh5.13.3.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_46085_mapreduce____.frd8sy/webapp
2021-03-01 16:57:53,196 INFO [main] org.mortbay.log: Started [email protected]:46085
2021-03-01 16:57:53,196 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 46085
2021-03-01 16:57:53,435 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2021-03-01 16:57:53,439 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1607937945101_246670
2021-03-01 16:57:53,440 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
2021-03-01 16:57:53,441 INFO [Socket Reader #1 for port 43229] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 43229
2021-03-01 16:57:53,481 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2021-03-01 16:57:53,481 INFO [IPC Server listener on 43229] org.apache.hadoop.ipc.Server: IPC Server listener on 43229: starting
2021-03-01 16:57:53,501 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2021-03-01 16:57:53,502 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2021-03-01 16:57:53,502 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2021-03-01 16:57:53,564 INFO [main] org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider: Failing over to rm709
2021-03-01 16:57:53,600 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:131072, vCores:32>
2021-03-01 16:57:53,600 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.users.w_hanzhiying
2021-03-01 16:57:53,603 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2021-03-01 16:57:53,603 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
2021-03-01 16:57:53,612 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from INITED to SETUP
2021-03-01 16:57:53,615 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2021-03-01 16:57:53,617 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from SETUP to RUNNING
2021-03-01 16:57:53,752 INFO [Socket Reader #1 for port 36866] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for hive (auth:SIMPLE) for protocol=interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB
2021-03-01 16:57:53,772 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1607937945101_246670, File: hdfs://lhService01:8020/user/hive/.staging/job_1607937945101_246670/job_1607937945101_246670_1.jhist
2021-03-01 16:57:53,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1607937945101_246670_m_000000 Task Transitioned from NEW to SCHEDULED
2021-03-01 16:57:53,792 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2021-03-01 16:57:53,793 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:2048, vCores:1>
2021-03-01 16:57:53,904 INFO [IPC Server handler 0 on 36866] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for MAP   job_1607937945101_246670. Report-size will be 1
2021-03-01 16:57:53,964 INFO [IPC Server handler 0 on 36866] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for REDUCE   job_1607937945101_246670. Report-size will be 0
2021-03-01 16:57:53,983 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:54,603 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2021-03-01 16:57:54,632 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1607937945101_246670: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:179712, vCores:199> knownNMs=7
2021-03-01 16:57:55,642 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
2021-03-01 16:57:55,671 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e64_1607937945101_246670_01_000002 to attempt_1607937945101_246670_m_000000_0
2021-03-01 16:57:55,673 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2021-03-01 16:57:55,743 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://lhService01/user/hive/.staging/job_1607937945101_246670/job.jar
2021-03-01 16:57:55,746 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hive/.staging/job_1607937945101_246670/job.xml
2021-03-01 16:57:55,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container
2021-03-01 16:57:55,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1
2021-03-01 16:57:55,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
2021-03-01 16:57:55,784 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx1639m
2021-03-01 16:57:55,786 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2021-03-01 16:57:55,791 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e64_1607937945101_246670_01_000002 taskAttempt attempt_1607937945101_246670_m_000000_0
2021-03-01 16:57:55,794 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1607937945101_246670_m_000000_0
2021-03-01 16:57:55,855 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1607937945101_246670_m_000000_0 : 13562
2021-03-01 16:57:55,858 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1607937945101_246670_m_000000_0] using containerId: [container_e64_1607937945101_246670_01_000002 on NM: [cdhuatdn62-209.com:8041]
2021-03-01 16:57:55,864 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2021-03-01 16:57:55,864 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1607937945101_246670_m_000000 Task Transitioned from SCHEDULED to RUNNING
2021-03-01 16:57:56,675 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1607937945101_246670: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:177664, vCores:194> knownNMs=7
2021-03-01 16:57:57,308 INFO [Socket Reader #1 for port 43229] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1607937945101_246670 (auth:SIMPLE)
2021-03-01 16:57:57,317 INFO [Socket Reader #1 for port 43229] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for job_1607937945101_246670 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
2021-03-01 16:57:57,327 INFO [IPC Server handler 1 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1607937945101_246670_m_70368744177666 asked for a task
2021-03-01 16:57:57,327 INFO [IPC Server handler 1 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1607937945101_246670_m_70368744177666 given task: attempt_1607937945101_246670_m_000000_0
2021-03-01 16:58:10,060 INFO [IPC Server handler 6 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0
2021-03-01 16:58:16,148 INFO [IPC Server handler 12 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0


此处省略

org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0
2021-03-01 18:18:33,300 INFO [IPC Server handler 15 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 1.0
2021-03-01 18:18:33,305 INFO [IPC Server handler 26 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1607937945101_246670_m_000000_0
2021-03-01 18:18:33,309 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from RUNNING to SUCCESS_FINISHING_CONTAINER
2021-03-01 18:18:33,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1607937945101_246670_m_000000_0
2021-03-01 18:18:33,348 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1607937945101_246670_m_000000 Task Transitioned from RUNNING to SUCCEEDED
2021-03-01 18:18:33,351 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
2021-03-01 18:18:33,352 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from RUNNING to COMMITTING
2021-03-01 18:18:33,362 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_COMMIT
2021-03-01 18:18:33,405 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent 
2021-03-01 18:18:33,406 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from COMMITTING to SUCCEEDED
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2021-03-01 18:18:33,426 INFO [Thread-298] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
2021-03-01 18:18:33,794 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2021-03-01 18:18:34,090 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://lhService01:8020/user/hive/.staging/job_1607937945101_246670/job_1607937945101_246670_1.jhist to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist_tmp
2021-03-01 18:18:34,215 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist_tmp
2021-03-01 18:18:34,222 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://lhService01:8020/user/hive/.staging/job_1607937945101_246670/job_1607937945101_246670_1_conf.xml to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml_tmp
2021-03-01 18:18:34,307 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml_tmp
2021-03-01 18:18:34,317 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670.summary_tmp to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670.summary
2021-03-01 18:18:34,319 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml_tmp to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml
2021-03-01 18:18:34,321 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist_tmp to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist
2021-03-01 18:18:34,321 INFO [Thread-298] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2021-03-01 18:18:34,322 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1607937945101_246670_m_000000_0
2021-03-01 18:18:34,359 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from SUCCESS_FINISHING_CONTAINER to SUCCEEDED
2021-03-01 18:18:34,364 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to 
2021-03-01 18:18:34,365 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://cdhuatmaster62-201.com:19888/jobhistory/job/job_1607937945101_246670
2021-03-01 18:18:34,374 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
2021-03-01 18:18:35,375 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2021-03-01 18:18:35,376 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://lhService01 /user/hive/.staging/job_1607937945101_246670
2021-03-01 18:18:35,388 INFO [Thread-298] org.apache.hadoop.ipc.Server: Stopping server on 43229
2021-03-01 18:18:35,392 INFO [IPC Server listener on 43229] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 43229
2021-03-01 18:18:35,392 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2021-03-01 18:18:35,394 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
2021-03-01 18:18:35,396 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
2021-03-01 18:18:40,394 INFO [Thread-298] org.apache.hadoop.ipc.Server: Stopping server on 36866
2021-03-01 18:18:40,394 INFO [IPC Server listener on 36866] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 36866
2021-03-01 18:18:40,394 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2021-03-01 18:18:40,396 INFO [Thread-298] org.mortbay.log: Stopped [email protected]:0
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_46893497/article/details/113846400

智能推荐

while循环&CPU占用率高问题深入分析与解决方案_main函数使用while(1)循环cpu占用99-程序员宅基地

文章浏览阅读3.8k次，点赞9次，收藏28次。直接上一个工作中碰到的问题，另外一个系统开启多线程调用我这边的接口，然后我这边会开启多线程批量查询第三方接口并且返回给调用方。使用的是两三年前别人遗留下来的方法，放到线上后发现确实是可以正常取到结果，但是一旦调用，CPU占用就直接100%(部署环境是win server服务器)。因此查看了下相关的老代码并使用JProfiler查看发现是在某个while循环的时候有问题。具体项目代码就不贴了，类似于下面这段代码。​​​​​​while(flag) {//your code;}这里的flag._main函数使用while(1)循环cpu占用99

【无标题】jetbrains idea shift f6不生效_idea shift +f6快捷键不生效-程序员宅基地

文章浏览阅读347次。idea shift f6 快捷键无效_idea shift +f6快捷键不生效

node.js学习笔记之Node中的核心模块_node模块中有很多核心模块,以下不属于核心模块,使用时需下载的是-程序员宅基地

文章浏览阅读135次。Ecmacript 中没有DOM 和 BOM核心模块Node为JavaScript提供了很多服务器级别，这些API绝大多数都被包装到了一个具名和核心模块中了，例如文件操作的 fs 核心模块 ，http服务构建的http 模块 path 路径操作模块 os 操作系统信息模块// 用来获取机器信息的var os = require('os')// 用来操作路径的var path = require('path')// 获取当前机器的 CPU 信息console.log(os.cpus._node模块中有很多核心模块,以下不属于核心模块,使用时需下载的是

数学建模【SPSS 下载-安装、方差分析与回归分析的SPSS实现(软件概述、方差分析、回归分析)】_化工数学模型数据回归软件-程序员宅基地

文章浏览阅读10w+次，点赞435次，收藏3.4k次。SPSS 22 下载安装过程7.6 方差分析与回归分析的SPSS实现7.6.1 SPSS软件概述1 SPSS版本与安装2 SPSS界面3 SPSS特点4 SPSS数据7.6.2 SPSS与方差分析1 单因素方差分析2 双因素方差分析7.6.3 SPSS与回归分析SPSS回归分析过程牙膏价格问题的回归分析_化工数学模型数据回归软件

利用hutool实现邮件发送功能_hutool发送邮件-程序员宅基地

文章浏览阅读7.5k次。如何利用hutool工具包实现邮件发送功能呢？1、首先引入hutool依赖<dependency> <groupId>cn.hutool</groupId> <artifactId>hutool-all</artifactId> <version>5.7.19</version></dependency>2、编写邮件发送工具类package com.pc.c..._hutool发送邮件

docker安装elasticsearch，elasticsearch-head，kibana，ik分词器_docker安装kibana连接elasticsearch并且elasticsearch有密码-程序员宅基地

文章浏览阅读867次，点赞2次，收藏2次。docker安装elasticsearch，elasticsearch-head，kibana，ik分词器安装方式基本有两种，一种是pull的方式，一种是Dockerfile的方式，由于pull的方式pull下来后还需配置许多东西且不便于复用，个人比较喜欢使用Dockerfile的方式所有docker支持的镜像基本都在https://hub.docker.com/docker的官网上能找到合..._docker安装kibana连接elasticsearch并且elasticsearch有密码

随便推点

Python 攻克移动开发失败！_beeware-程序员宅基地

文章浏览阅读1.3w次，点赞57次，收藏92次。整理 | 郑丽媛出品 | CSDN（ID：CSDNnews）近年来，随着机器学习的兴起，有一门编程语言逐渐变得火热——Python。得益于其针对机器学习提供了大量开源框架和第三方模块，内置..._beeware

Swift4.0_Timer 的基本使用_swift timer 暂停-程序员宅基地

文章浏览阅读7.9k次。//// ViewController.swift// Day_10_Timer//// Created by dongqiangfei on 2018/10/15.// Copyright 2018年 飞飞. All rights reserved.//import UIKitclass ViewController: UIViewController { ..._swift timer 暂停

元素三大等待-程序员宅基地

文章浏览阅读986次，点赞2次，收藏2次。1.硬性等待让当前线程暂停执行，应用场景：代码执行速度太快了，但是UI元素没有立马加载出来，造成两者不同步，这时候就可以让代码等待一下，再去执行找元素的动作线程休眠，强制等待 Thread.sleep(long mills)package com.example.demo;import org.junit.jupiter.api.Test;import org.openqa.selenium.By;import org.openqa.selenium.firefox.Firefox.._元素三大等待

Java软件工程师职位分析_java岗位分析-程序员宅基地

文章浏览阅读3k次，点赞4次，收藏14次。Java软件工程师职位分析_java岗位分析

Java：Unreachable code的解决方法_java unreachable code-程序员宅基地

文章浏览阅读2k次。Java：Unreachable code的解决方法_java unreachable code

标签data-*自定义属性值和根据data属性值查找对应标签_如何根据data-*属性获取对应的标签对象-程序员宅基地

文章浏览阅读1w次。1、html中设置标签data-*的值 标题 11111 222222、点击获取当前标签的data-url的值$('dd').on('click', function() { var urlVal = $(this).data('ur_如何根据data-*属性获取对应的标签对象

推荐文章

热门文章

相关标签