博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
java代码读取yarn聚合目录日志
阅读量:4320 次
发布时间:2019-06-06

本文共 2808 字,大约阅读时间需要 9 分钟。

可以直接使用org.apache.hadoop.yarn.client.cli.LogsCLI(yarn logs -applicationId)中的main方法逻辑,如

public static void main(String[] args)

throws Exception
{
Configuration conf = new YarnConfiguration();

conf.addResource(new Path("/etc/hadoop/conf.cloudera.yarn/core-site.xml"));

conf.addResource(new Path("/etc/hadoop/conf.cloudera.yarn/yarn-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf.cloudera.yarn/hdfs-site.xml"));

LogsCLI logDumper = new LogsCLI();
logDumper.setConf(conf);
int exitCode = logDumper.run(args);
System.exit(exitCode);
}

也可以仿照他的逻辑自己实现如下:

Configuration conf = new YarnConfiguration();

conf.addResource(new Path("/etc/hadoop/conf.cloudera.yarn/core-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf.cloudera.yarn/yarn-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf.cloudera.yarn/hdfs-site.xml"));
String appIdStr="application_1529377575914_8380";
PrintStream out=new PrintStream(appIdStr);
ApplicationId appId = null;
appId = ConverterUtils.toApplicationId(appIdStr);
Path remoteRootLogDir = new Path(conf.get("yarn.nodemanager.remote-app-log-dir", "/tmp/logs"));

String user = UserGroupInformation.getCurrentUser().getShortUserName();;

String logDirSuffix = LogAggregationUtils.getRemoteNodeLogDirSuffix(conf);
Path remoteAppLogDir = LogAggregationUtils.getRemoteAppLogDir(remoteRootLogDir, appId, user, logDirSuffix);
RemoteIterator<FileStatus> nodeFiles;
try
{
Path qualifiedLogDir = FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
nodeFiles = FileContext.getFileContext(qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
}
catch (FileNotFoundException fnf)
{
logDirNotExist(remoteAppLogDir.toString());
return -1;
}
boolean foundAnyLogs = false;
while (nodeFiles.hasNext())
{
FileStatus thisNodeFile = (FileStatus)nodeFiles.next();
if (!thisNodeFile.getPath().getName().endsWith(".tmp"))
{
AggregatedLogFormat.LogReader reader = new AggregatedLogFormat.LogReader(conf, thisNodeFile.getPath());
try
{
//System.out.println(thisNodeFile.getPath().getName());
AggregatedLogFormat.LogKey key = new AggregatedLogFormat.LogKey();
DataInputStream valueStream = reader.next(key);
for (;;)
{
if (valueStream != null)
{
String containerString = "\n\nContainer: " + key + " on " + thisNodeFile.getPath().getName();
out.println(containerString);
out.println(StringUtils.repeat("=", containerString.length()));
try
{
for (;;)
{
AggregatedLogFormat.LogReader.readAContainerLogsForALogType(valueStream, out, thisNodeFile.getModificationTime());
foundAnyLogs = true;
}
}
catch (EOFException eof)
{
key = new AggregatedLogFormat.LogKey();
valueStream = reader.next(key);
}
}else{
break;
}
}
}
finally
{
reader.close();
}
}
}
if (!foundAnyLogs)
{
emptyLogDir(remoteAppLogDir.toString());
return -1;
}
return 0;
}

转载于:https://www.cnblogs.com/lyy-blog/p/9262935.html

你可能感兴趣的文章
chromium浏览器开发系列第三篇:chromium源码目录结构
查看>>
java开发操作系统内核:由实模式进入保护模式之32位寻址
查看>>
第五讲:单例模式
查看>>
Python编程语言的起源
查看>>
Azure ARMTemplate模板,VM扩展命令
查看>>
在腾讯云上创建您的SQL Cluster(4)
查看>>
linux ping命令
查看>>
Activiti源码浅析:Activiti的活动授权机制
查看>>
数位dp整理
查看>>
UNIX基础知识
查看>>
bzoj 1179: [Apio2009]Atm
查看>>
利用LDA进行文本聚类(hadoop, mahout)
查看>>
第三周作业
查看>>
js添加删除行
查看>>
浏览器性能测试网址
查看>>
[MTK FP]用Python把图片资源image.rar中为.pbm后缀的文件更改为.bmp后缀的方法
查看>>
实验二
查看>>
[LeetCode]203. Remove Linked List Elements 解题小结
查看>>
测试一下
查看>>
vue base64
查看>>