【原创】大数据基础之Ambari（1）简介、安装、使用

官方：http://ambari.apache.org/

The Apache Ambari project is aimed at making Hadoop management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use Hadoop management web UI backed by its RESTful APIs.

ambari通过web UI使得Hadoop管理更加简单，覆盖安装、管理和监控等方面；

Ambari enables System Administrators to:

  • Provision a Hadoop Cluster

    • 　　Ambari provides a step-by-step wizard for installing Hadoop services across any number of hosts.
    • 　　Ambari handles configuration of Hadoop services for the cluster.
  • Manage a Hadoop Cluster
    • 　　Ambari provides central management for starting, stopping, and reconfiguring Hadoop services across the entire cluster.
  • Monitor a Hadoop Cluster
    • 　　Ambari provides a dashboard for monitoring health and status of the Hadoop cluster.
    • 　　Ambari leverages Ambari Metrics System for metrics collection.
    • 　　Ambari leverages Ambari Alert Framework for system alerting and will notify you when your attention is needed (e.g., a node goes down, remaining disk space is low, etc).

功能包括：安装向导、集群配置、各个组件的启动停止和配置更新、监控页面、指标收集、报警等；

Ambari enables Application Developers and System Integrators to:

  • Easily integrate Hadoop provisioning, management, and monitoring capabilities to their own applications with the Ambari REST APIs.

一 安装

官方安装指南：https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.7.3

1 下载编译打包

# wget http://www.apache.org/dist/ambari/ambari-2.7.3/apache-ambari-2.7.3-src.tar.gz
# tar xfvz apache-ambari-2.7.3-src.tar.gz
# cd apache-ambari-2.7.3-src
# mvn versions:set -DnewVersion=2.7.3.0.0

# pushd ambari-metrics
# mvn versions:set -DnewVersion=2.7.3.0.0
# popd

# mvn -B clean install rpm:rpm -DnewVersion=2.7.3.0.0 -DbuildNumber=4295bb16c439cbc8fb0e7362f19768dde1477868 -DskipTests -Dpython.ver="python >= 2.6"

最后一步要build很久（里边的坑不少，还与网络情况有关，要反复执行多遍，后续执行建议将clean去掉，节省时间，估计至少要1天），并且可能出现多种报错，详见附录；

成功build之后是这样的：

[INFO] Ambari Main 2.7.3.0.0 .............................. SUCCESS [ 0.738 s]
[INFO] Apache Ambari Project POM 2.7.3.0.0 ................ SUCCESS [ 0.115 s]
[INFO] Ambari Web 2.7.3.0.0 ............................... SUCCESS [ 44.403 s]
[INFO] Ambari Views 2.7.3.0.0 ............................. SUCCESS [ 0.359 s]
[INFO] Ambari Admin View 2.7.3.0.0 ........................ SUCCESS [ 5.570 s]
[INFO] ambari-utility 1.0.0.0-SNAPSHOT .................... SUCCESS [ 0.541 s]
[INFO] ambari-metrics 2.7.3.0.0 ........................... SUCCESS [ 0.381 s]
[INFO] Ambari Metrics Common 2.7.3.0.0 .................... SUCCESS [ 4.635 s]
[INFO] Ambari Metrics Hadoop Sink 2.7.3.0.0 ............... SUCCESS [ 2.082 s]
[INFO] Ambari Metrics Flume Sink 2.7.3.0.0 ................ SUCCESS [ 1.214 s]
[INFO] Ambari Metrics Kafka Sink 2.7.3.0.0 ................ SUCCESS [ 1.307 s]
[INFO] Ambari Metrics Storm Sink 2.7.3.0.0 ................ SUCCESS [ 1.775 s]
[INFO] Ambari Metrics Storm Sink (Legacy) 2.7.3.0.0 ....... SUCCESS [ 1.546 s]
[INFO] Ambari Metrics Collector 2.7.3.0.0 ................. SUCCESS [02:03 min]
[INFO] Ambari Metrics Monitor 2.7.3.0.0 ................... SUCCESS [ 0.612 s]
[INFO] Ambari Metrics Grafana 2.1.0.0.0 ................... SUCCESS [ 2.052 s]
[INFO] Ambari Metrics Host Aggregator 2.7.3.0.0 ........... SUCCESS [ 44.355 s]
[INFO] Ambari Metrics Assembly 2.7.3.0.0 .................. SUCCESS [01:18 min]
[INFO] Ambari Service Advisor 1.0.0.0-SNAPSHOT ............ SUCCESS [ 0.175 s]
[INFO] Ambari Server 2.7.3.0.0 ............................ SUCCESS [11:39 min]
[INFO] Ambari Functional Tests 2.7.3.0.0 .................. SUCCESS [ 0.818 s]
[INFO] Ambari Agent 2.7.3.0.0 ............................. SUCCESS [04:35 min]
[INFO] ambari-logsearch 2.7.3.0.0 ......................... SUCCESS [ 1.587 s]
[INFO] Ambari Logsearch Appender 2.7.3.0.0 ................ SUCCESS [ 4.570 s]
[INFO] Ambari Logsearch Config Api 2.7.3.0.0 .............. SUCCESS [ 0.196 s]
[INFO] Ambari Logsearch Config JSON 2.7.3.0.0 ............. SUCCESS [ 0.209 s]
[INFO] Ambari Logsearch Config Solr 2.7.3.0.0 ............. SUCCESS [ 20.562 s]
[INFO] Ambari Logsearch Config Zookeeper 2.7.3.0.0 ........ SUCCESS [ 1.872 s]
[INFO] Ambari Logsearch Config Local 2.7.3.0.0 ............ SUCCESS [ 0.182 s]
[INFO] Ambari Logsearch Log Feeder Plugin Api 2.7.3.0.0 ... SUCCESS [02:45 min]
[INFO] Ambari Logsearch Log Feeder Container Registry 2.7.3.0.0 SUCCESS [ 22.523 s]
[INFO] Ambari Logsearch Log Feeder 2.7.3.0.0 .............. SUCCESS [01:45 min]
[INFO] Ambari Logsearch Web 2.7.3.0.0 ..................... SUCCESS [02:24 min]
[INFO] Ambari Logsearch Server 2.7.3.0.0 .................. SUCCESS [03:55 min]
[INFO] Ambari Logsearch Assembly 2.7.3.0.0 ................ SUCCESS [ 2.389 s]
[INFO] Ambari Logsearch Integration Test 2.7.3.0.0 ........ SUCCESS [01:20 min]
[INFO] ambari-infra 2.7.3.0.0 ............................. SUCCESS [ 4.214 s]
[INFO] Ambari Infra Solr Client 2.7.3.0.0 ................. SUCCESS [ 18.131 s]
[INFO] Ambari Infra Solr Plugin 2.7.3.0.0 ................. SUCCESS [05:11 min]
[INFO] Ambari Infra Manager 2.7.3.0.0 ..................... SUCCESS [02:14 min]
[INFO] Ambari Infra Assembly 2.7.3.0.0 .................... SUCCESS [ 7.399 s]
[INFO] Ambari Infra Manager Integration Tests 2.7.3.0.0 ... SUCCESS [ 13.159 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

生成的rpm在这里

# ls ambari-server/target/rpm/ambari-server/RPMS/x86_64/
ambari-server-2.7.3.0-0.x86_64.rpm

ls ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/
ambari-agent-2.7.3.0-0.x86_64.rpm

2 分发

将build完成的ambari-server-2.7.3.0-0.x86_64.rpm和ambari-agent-2.7.3.0-0.x86_64.rpm目录拷贝到集群对应的服务器上，比如ansible 详见 https://www.cnblogs.com/barneywill/p/10263278.html

3 启动

3.1 server

安装

yum install ambari-server*.rpm

Run the setup command to configure your Ambari Server, Database, JDK, LDAP, and other options:

ambari-server setup

具体过程为：

# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is ‘disabled‘
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /data/jdk1.8.0_191/
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): $mysql_host
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Re-enter password:
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.3.0.0.jar

Ambari repo file doesn‘t contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server ‘setup‘ completed successfully.

需要手工初始化数据库：/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql，导入之后表结构如下，导入详见：https://www.cnblogs.com/barneywill/p/10271024.html

mysql> show tables;
+-------------------------------+
| Tables_in_ambari |
+-------------------------------+
| ClusterHostMapping |
| QRTZ_BLOB_TRIGGERS |
| QRTZ_CALENDARS |
| QRTZ_CRON_TRIGGERS |
| QRTZ_FIRED_TRIGGERS |
| QRTZ_JOB_DETAILS |
| QRTZ_LOCKS |
| QRTZ_PAUSED_TRIGGER_GRPS |
| QRTZ_SCHEDULER_STATE |
| QRTZ_SIMPLE_TRIGGERS |
| QRTZ_SIMPROP_TRIGGERS |
| QRTZ_TRIGGERS |
| adminpermission |
| adminprincipal |
| adminprincipaltype |
| adminprivilege |
| adminresource |
| adminresourcetype |
| alert_current |
| alert_definition |
| alert_group |
| alert_group_target |
| alert_grouping |
| alert_history |
| alert_notice |
| alert_target |
| alert_target_states |
| ambari_configuration |
| ambari_operation_history |
| ambari_sequences |
| artifact |
| blueprint |
| blueprint_configuration |
| blueprint_setting |
| clusterconfig |
| clusters |
| clusterservices |
| clusterstate |
| confgroupclusterconfigmapping |
| configgroup |
| configgrouphostmapping |
| execution_command |
| extension |
| extensionlink |
| groups |
| host_role_command |
| host_version |
| hostcomponentdesiredstate |
| hostcomponentstate |
| hostconfigmapping |
| hostgroup |
| hostgroup_component |
| hostgroup_configuration |
| hosts |
| hoststate |
| kerberos_descriptor |
| kerberos_keytab |
| kerberos_keytab_principal |
| kerberos_principal |
| key_value_store |
| kkp_mapping_service |
| members |
| metainfo |
| permission_roleauthorization |
| remoteambaricluster |
| remoteambariclusterservice |
| repo_applicable_services |
| repo_definition |
| repo_os |
| repo_tags |
| repo_version |
| request |
| requestoperationlevel |
| requestresourcefilter |
| requestschedule |
| requestschedulebatchrequest |
| role_success_criteria |
| roleauthorization |
| servicecomponent_version |
| servicecomponentdesiredstate |
| serviceconfig |
| serviceconfighosts |
| serviceconfigmapping |
| servicedesiredstate |
| setting |
| stack |
| stage |
| topology_host_info |
| topology_host_request |
| topology_host_task |
| topology_hostgroup |
| topology_logical_request |
| topology_logical_task |
| topology_request |
| upgrade |
| upgrade_group |
| upgrade_history |
| upgrade_item |
| user_authentication |
| users |
| viewentity |
| viewinstance |
| viewinstancedata |
| viewinstanceproperty |
| viewmain |
| viewparameter |
| viewresource |
| viewurl |
| widget |
| widget_layout |
| widget_layout_user_widget |
+-------------------------------+
111 rows in set (0.00 sec)

后续想修改数据库可以看这里

/etc/ambari-server/conf/ambari.properties

启动

ambari-server start

输出

Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start................
Server started listening on 8080

3.2 agent

安装

yum install ambari-agent*.rpm

配置

/etc/ambari-agent/conf/ambari-agent.ini

将其中的hostname修改为master的host或ip

[server]
hostname=localhost

启动

ambari-agent start

输出

Verifying Python version compatibility...
Using python /usr/bin/python
Checking for previously running Ambari Agent...
Checking ambari-common dir...
Starting ambari-agent
Verifying ambari-agent process status...
Ambari Agent successfully started
Agent PID at: /run/ambari-agent/ambari-agent.pid
Agent out at: /var/log/ambari-agent/ambari-agent.out
Agent log at: /var/log/ambari-agent/ambari-agent.log

4 打开Web

http://<ambari-server-host>:8080

Log in with username admin and password admin and follow on-screen instructions.

当你看到以下页面的时候，就说明你成功了

附：build过程错误

1）MojoExecutionException

[ERROR] Failed to execute goal org.codehaus.mojo:rpm-maven-plugin:2.0.1:rpm (default-cli) on project ambari: RPM build execution returned: ‘127‘ executing ‘/bin/sh -c cd /data/apache-ambari-2.7.3-src/target/rpm/ambari/SPECS && rpmbuild -bb --buildroot /data/apache-ambari-2.7.3-src/target/rpm/ambari/buildroot --define ‘_topdir /data/apache-ambari-2.7.3-src/target/rpm/ambari‘ --target noarch-redhat-linux ambari.spec‘ -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:rpm-maven-plugin:2.0.1:rpm (default-cli) on project ambari: RPM build execution returned: ‘127‘ executing ‘/bin/sh -c cd /data/apache-ambari-2.7.3-src/target/rpm/ambari/SPECS && rpmbuild -bb --buildroot /data/apache-ambari-2.7.3-src/target/rpm/ambari/buildroot --define ‘_topdir /data/apache-ambari-2.7.3-src/target/rpm/ambari‘ --target noarch-redhat-linux ambari.spec‘

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)

at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)

at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)

at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)

at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)

at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)

at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)

at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke (Method.java:566)

at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)

at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)

at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)

at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

Caused by: org.apache.maven.plugin.MojoExecutionException: RPM build execution returned: ‘127‘ executing ‘/bin/sh -c cd /data/apache-ambari-2.7.3-src/target/rpm/ambari/SPECS && rpmbuild -bb --buildroot /data/apache-ambari-2.7.3-src/target/rpm/ambari/buildroot --define ‘_topdir /data/apache-ambari-2.7.3-src/target/rpm/ambari‘ --target noarch-redhat-linux ambari.spec‘

at org.codehaus.mojo.rpm.RPMHelper.buildPackage (RPMHelper.java:150)

at org.codehaus.mojo.rpm.AbstractRPMMojo.execute (AbstractRPMMojo.java:736)

at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)

at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)

at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)

at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)

at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)

at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)

at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)

at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke (Method.java:566)

at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)

at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)

at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)

at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

原因是缺少rpm-build

yum install -y rpm-build

2）超时

[INFO] Downloading https://github.com/yarnpkg/yarn/releases/download/v0.23.2/yarn-v0.23.2.tar.gz to /root/.m2/repository/com/github/eirslett/yarn/0.23.2/yarn-0.23.2./yarn-v0.23.2.tar.gz

[get] Getting: http://dev.hortonworks.com.s3.amazonaws.com/HDP/centos7/3.x/BUILDS/3.0.0.0-1634/tars/hbase/hbase-2.0.0.3.0.0.0-1634-bin.tar.gz
[get] To: /data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-timelineservice/target/embedded/hbase.tar.gz

有好几个jar下载过程可能很慢或者报超时错误和连接被重置，这时要么换个环境手工下载后放到maven local repository，要么直接重试；

3）编译错误

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project ambari-metrics-common: Compilation failure: Compilation failure: 
[ERROR] /data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/TimelineMetrics.java:[24,33] package javax.xml.bind.annotation does not exist

这是因为你使用了jdk11，因为java.xml.bind (JAXB) 在11中 REMOVED，改用11之前的jdk版本即可；

Java 8 - OK
Java 9 - DEPRECATED
Java 10 - DEPRECATED
Java 11 - REMOVED

4）RatCheckException

[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project ambari: Too many files with unapproved license: 1 See RAT report in: /data/apache-ambari-2.7.3-src/target/rat.txt -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project ambari: Too many files with unapproved license: 1 See RAT report in: /data/apache-ambari-2.7.3-src/target/rat.txt

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)

at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)

at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)

at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)

at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)

at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)

at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)

at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke (Method.java:498)

at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)

at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)

at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)

at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

Caused by: org.apache.rat.mp.RatCheckException: Too many files with unapproved license: 1 See RAT report in: /data/apache-ambari-2.7.3-src/target/rat.txt

at org.apache.rat.mp.RatCheckMojo.check (RatCheckMojo.java:183)

at org.apache.rat.mp.RatCheckMojo.execute (RatCheckMojo.java:156)

at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)

at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)

at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)

at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)

at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)

at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)

at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)

at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke (Method.java:498)

at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)

at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)

at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)

at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

mvn命令中增加

-Drat.skip=true

5）MojoExecutionException

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (psutils-compile) on project ambari-metrics-host-monitoring: An Ant BuildException has occured: exec returned: 1

[ERROR] around Ant part ...<exec failonerror="true" dir="/data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/src/main/python/psutil" executable="/data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/../../ambari-common/src/main/unix/ambari-python-wrap">... @ 4:275 in /data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/target/antrun/build-psutils-compile.xml

[ERROR] -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (psutils-compile) on project ambari-metrics-host-monitoring: An Ant BuildException has occured: exec returned: 1

around Ant part ...<exec failonerror="true" dir="/data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/src/main/python/psutil" executable="/data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/../../ambari-common/src/main/unix/ambari-python-wrap">... @ 4:275 in /data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/target/antrun/build-psutils-compile.xml

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)

at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)

at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)

at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)

at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)

at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)

at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)

at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke (Method.java:498)

at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)

at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)

at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)

at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 1

around Ant part ...<exec failonerror="true" dir="/data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/src/main/python/psutil" executable="/data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/../../ambari-common/src/main/unix/ambari-python-wrap">... @ 4:275 in /data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/target/antrun/build-psutils-compile.xml

at org.apache.maven.plugin.antrun.AntRunMojo.execute (AntRunMojo.java:355)

at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)

at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)

at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)

at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)

at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)

at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)

at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)

at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke (Method.java:498)

at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)

at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)

at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)

at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

Caused by: org.apache.tools.ant.BuildException: exec returned: 1

at org.apache.tools.ant.taskdefs.ExecTask.runExecute (ExecTask.java:646)

at org.apache.tools.ant.taskdefs.ExecTask.runExec (ExecTask.java:672)

at org.apache.tools.ant.taskdefs.ExecTask.execute (ExecTask.java:498)

at org.apache.tools.ant.UnknownElement.execute (UnknownElement.java:291)

at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke (Method.java:498)

at org.apache.tools.ant.dispatch.DispatchUtils.execute (DispatchUtils.java:106)

at org.apache.tools.ant.Task.perform (Task.java:348)

at org.apache.tools.ant.Target.execute (Target.java:390)

at org.apache.tools.ant.Target.performTasks (Target.java:411)

at org.apache.tools.ant.Project.executeSortedTargets (Project.java:1399)

at org.apache.tools.ant.Project.executeTarget (Project.java:1368)

at org.apache.maven.plugin.antrun.AntRunMojo.execute (AntRunMojo.java:327)

at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)

at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)

at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)

at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)

at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)

at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)

at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)

at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)

at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke (Method.java:498)

at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)

at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)

at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)

at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

这个看不出直接原因，错误是因为执行python返回1，具体的错误没报出来，只能手工重现问题；

首先看报错的脚本：

# cat /data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/target/antrun/build-psutils-compile.xml

<?xml version="1.0" encoding="UTF-8" ?>

<project name="maven-antrun-" default="psutils-compile"  >

<target name="psutils-compile">

<exec failonerror="true" dir="/data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/src/main/python/psutil" executable="/data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/../../ambari-common/src/main/unix/ambari-python-wrap">

<arg value="setup.py"/>

<arg value="build"/>

<arg value="--build-platlib"/>

<arg value="/data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/target/psutil_build"/>

</exec>

</target>

</project>

然后手工重现：

# cd /data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/src/main/python/psutil

# /data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/../../ambari-common/src/main/unix/ambari-python-wrap setup.py build --build-platlib /data/apache-ambari-2.7.3-src/ambari-metrics/ambari-metrics-host-monitoring/target/psutil_build
running build
running build_py
running build_ext
building ‘_psutil_linux‘ extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
unable to execute gcc: No such file or directory
error: command ‘gcc‘ failed with exit status 1

终于看到直接原因了，缺少gcc（你的原因可能不同，手工执行可以看到）

yum install gcc

原文地址：https://www.cnblogs.com/barneywill/p/10264135.html

时间： 2024-11-05 18:34:45

【原创】大数据基础之Ambari（1）简介、安装、使用的相关文章

【原创】大数据基础之Ambari（2）通过Ambari部署ElasticSearch

ambari的hdp中原生不支持elasticsearch安装,下面介绍如何通过mpack方式使ambari支持elasticsearch安装 安装过程 1 下载 Mpack include version 6.3.2 of ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat # wget https://community.hortonworks.com/storage/attachments/87416-elasticsearc

【原创】大数据基础之Ambari（3）通过Ambari部署Airflow

ambari2.7.3(hdp3.1) 安装 airflow1.10 ambari的hdp中原生不支持airflow安装,下面介绍如何通过mpack方式使ambari支持airflow安装: 1 下载 # wget https://github.com/miho120/ambari-airflow-mpack/raw/master/airflow-service-mpack.tar.gz 2 安装 # ambari-server install-mpack --mpack airflow-ser

区块链这些技术与h5房卡斗牛平台出售，大数据基础软件干货不容错过

在IT产业发展中,包括CPU.操作系统h5房卡斗牛平台出售 官网:h5.super-mans.com 企娥:2012035031 vx和tel:17061863513 h5房卡斗牛平台出售在内的基础软硬件地位独特,不但让美国赢得了产业发展的先机,成就了产业巨头,而且因为技术.标准和生态形成的壁垒,主宰了整个产业的发展.错失这几十年的发展机遇,对于企业和国家都是痛心的. 当大数据迎面而来,并有望成就一个巨大的应用和产业机会时,企业和国家都虎视眈眈,不想错再失这一难得的机遇.与传统的IT产业一样,大

大数据基础教程：创建RDD的二种方式

大数据基础教程:创建RDD的二种方式 1.从集合中创建RDD val conf = new SparkConf().setAppName("Test").setMaster("local")      val sc = new SparkContext(conf)      //这两个方法都有第二参数是一个默认值2  分片数量(partition的数量)      //scala集合通过makeRDD创建RDD,底层实现也是parallelize      val 

大数据平台CDH5.14.2 的安装配置

大数据平台CDH5.14.2 的安装配置 标签(空格分隔): 大数据平台构建 一:系统环境初始化 二:安装CDH5.14.2 平台 三:分配主机与分配角色 一: 系统环境初始化 1.1: 系统环境介绍 系统: CentOS7.5X64 cat /etc/hosts --- 172.17.100.11 node-01.flyfish 172.17.100.12 node-02.flyfish 172.17.100.13 node-03.flyfish 172.17.100.14 node-04.f

【原创】大数据基础之Impala（1）简介、安装、使用

impala2.12 官方:http://impala.apache.org/ 一 简介 Apache Impala is the open source, native analytic database for Apache Hadoop. Impala is shipped by Cloudera, MapR, Oracle, and Amazon. impala是hadoop上的开源分析性数据库: Do BI-style Queries on Hadoop Impala provides

大数据基础架构详解

简介:本文是对大数据领域的基础论文的阅读总结,相关论文包括GFS,MapReduce.BigTable.Chubby.SMAQ. 大数据出现的原因: 大多数的技术突破来源于实际的产品需要,大数据最初诞生于谷歌的搜索引擎中.随着web2.0时代的发展,互联网上数据量呈献爆炸式的增长,为了满足信息搜索的需要,对大规模数据的存储提出了非常强劲的需要.基于成本的考虑,通过提升硬件来解决大批量数据的搜索越来越不切实际,于是谷歌提出了一种基于软件的可靠文件存储体系GFS,使用普通的PC机来并行支撑大规模的存

大数据基础和hadoop

一.大数据的特点 大数据是什么?其实很简单,大数据其实就是海量资料巨量资料,这些巨量资料来源于世界各地随时产生的数据,在大数据时代,任何微小的数据都可能产生不可思议的价值.大数据有4个特点,为别为:Volume(大量).Variety(多样).Velocity(高速).Value(价值),一般我们称之为4V. 所谓4V,具体指如下4点: 1．大量.大数据的特征首先就体现为“大”,从先Map3时代,一个小小的MB级别的Map3就可以满足很多人的需求,然而随着时间的推移,存储单位从过去的GB到TB,

“大数据“基础知识普及

大数据,官方定义是指那些数据量特别大.数据类别特别复杂的数据集,这种数据集无法用传统的数据库进行存储,管理和处理.大数据的主要特点为数据量大(Volume),数据类别复杂(Variety),数据处理速度快(Velocity)和数据真实性高(Veracity),合起来被称为4V. 大数据中的数据量非常巨大,达到了PB级别.而且这庞大的数据之中,不仅仅包括结构化数据(如数字.符号等数据),还包括非结构化数据(如文本.图像.声音.视频等数据).这使得大数据的存储,管理和处理很难利用传统的关系型数据库去