cdh5 hadoop redhat 本地仓库配置

cdh5 hadoop redhat 本地仓库配置

cdh5 在网站上的站点位置:

http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/

在RHEL6上配置指向这个repo非常简单,只要把:

http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/cloudera-cdh5.repo

下载存储到本地即可:

/etc/yum.repos.d/cloudera-cdh5.repo

但是如果是离线情况下,网络连接不可用,就需要把整个资源镜像到本地,然后在cloudera-cdh5.repo中配置。我写了一个脚本,用于下载整个站点。虽然用wget一个命令可以搞定,为了练练shell脚本,我还是写了一个。基本思路就是分析网页,找到资源链接,存储到本地目录。脚本中:PATH_MUST_BE_EXSITED必须指向已经存在的本地目录。不废话,上代码:

#!/bin/bash
#
# @file
#   cdh5_rhel6-downloads.sh
#
# @date
#   2014-12-18
#
# @author
#   cheungmine
#
# @version
#   0.0.1pre
#
# downloads all from CDH_URL_PREFIX:
#   http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/
#
################################################################################
# specify where you want to save downloaded packages here:
#
PATH_MUST_BE_EXSITED="../libs/cdh"

# get real path from relative path
function real_path() {
    \cd "$1"
    /bin/pwd
}

# server dist resources:
#
CDH_URL_PREFIX="http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh"

CDH_GPGKEY=$CDH_URL_PREFIX"/RPM-GPG-KEY-cloudera"
CDH_REPO=$CDH_URL_PREFIX"/cloudera-cdh5.repo"

CDH5_REPODATA=$CDH_URL_PREFIX"/5/repodata/"
CDH5_RPMS_NOARCH=$CDH_URL_PREFIX"/5/RPMS/noarch/"
CDH5_RPMS_X86_64=$CDH_URL_PREFIX"/5/RPMS/x86_64/"

# source packages not used:
CDH5_SRPMS=$CDH_URL_PREFIX"/5/SRPMS/"

# get local absolute path for storing the downloaded:
CDH5_LOCALPATH=$(real_path $PATH_MUST_BE_EXSITED)
echo "**** downloaded packages will be stored in folder: "$CDH5_LOCALPATH

# first we get index pages:
#
repodata_html=$CDH5_LOCALPATH"/.repodata.index.html"
x86_64_html=$CDH5_LOCALPATH"/.x86_64.index.html"
noarch_html=$CDH5_LOCALPATH"/.noarch.index.html"

wget -c $CDH5_REPODATA -P $CDH5_LOCALPATH -O $repodata_html
wget -c $CDH5_RPMS_NOARCH -P $CDH5_LOCALPATH -O $noarch_html
wget -c $CDH5_RPMS_X86_64 -P $CDH5_LOCALPATH -O $x86_64_html

wget -c $CDH_GPGKEY -P $CDH5_LOCALPATH
wget -c $CDH_REPO -P $CDH5_LOCALPATH

# download repodata
# CDH5_REPODATA
repodata_dir=$CDH5_LOCALPATH"/5/repodata"
mkdir -p $repodata_dir
echo -e "process file: ‘$repodata_html‘"
while read line
do
    # start with: <td><a href="
    a=`echo $line | sed -n ‘/<td><a href="/p‘`

    if [ -n "$a" ]; then
        b=`echo $a | sed -n ‘/Parent Directory/p‘`

        # do including: Parent Directory
        if [ -z "$b" ]; then
            # end with: </a></td>
            b=`echo $a | sed -n ‘/<\/a><\/td>/p‘`

            if [ -n "$b" ]; then
                a=`echo $a | sed -e ‘s/.*<td><a href="//;s/">.*//‘`
                url=$CDH5_REPODATA$a

                echo -e "download: $url"

                wget -c $url -P $repodata_dir -O $repodata_dir/$a
            fi
        fi
    fi
done < $repodata_html

# download noarch
# CDH5_RPMS_NOARCH
noarch_dir=$CDH5_LOCALPATH"/5/RPMS/noarch"
mkdir -p $noarch_dir
echo -e "process file: ‘$noarch_html‘"
while read line
do
    # start with: <td><a href="
    a=`echo $line | sed -n ‘/<td><a href="/p‘`

    if [ -n "$a" ]; then
        b=`echo $a | sed -n ‘/Parent Directory/p‘`

        # do including: Parent Directory
        if [ -z "$b" ]; then
            # end with: </a></td>
            b=`echo $a | sed -n ‘/<\/a><\/td>/p‘`

            if [ -n "$b" ]; then
                a=`echo $a | sed -e ‘s/.*<td><a href="//;s/">.*//‘`
                url=$CDH5_RPMS_NOARCH$a

                echo -e "download: $url"

                wget -c $url -P $noarch_dir -O $noarch_dir/$a
            fi
        fi
    fi
done < $noarch_html

# download x86_64
# CDH5_RPMS_X86_64
x86_64_dir=$CDH5_LOCALPATH"/5/RPMS/x86_64"
mkdir -p $x86_64_dir
echo -e "process file: ‘$x86_64_html‘"
while read line
do
    # start with: <td><a href="
    a=`echo $line | sed -n ‘/<td><a href="/p‘`

    if [ -n "$a" ]; then
        b=`echo $a | sed -n ‘/Parent Directory/p‘`

        # do including: Parent Directory
        if [ -z "$b" ]; then
            # end with: </a></td>
            b=`echo $a | sed -n ‘/<\/a><\/td>/p‘`

            if [ -n "$b" ]; then
                a=`echo $a | sed -e ‘s/.*<td><a href="//;s/">.*//‘`
                url=$CDH5_RPMS_X86_64$a

                echo -e "download: $url"

                wget -c $url -P $x86_64_dir -O $x86_64_dir/$a
            fi
        fi
    fi
done < $x86_64_html

# TODO: do we need to check all packages?
# remove index pages:
rm -f $repodata_html $x86_64_html $noarch_html

echo "download all packages successfully."

上面脚本可以重复多次运行,不会重复下载。PATH_MUST_BE_EXSITED里面保存了cdh5全部内容。最后把PATH_MUST_BE_EXSITED的全部内容上传到本地ftp服务器上,确保可以访问:

ftp://your-server-ip/pub/libs/cdh/

然后在需要访问的RHEL6机器上,增加一个repo文件,我的是:

# /etc/yum.repos.d/cdh5.repo

[cloudera-cdh5]
# Packages for Cloudera‘s Distribution for Hadoop, Version 5, on RedHat	or CentOS 6 x86_64
name     = Cloudera‘s Distribution for Hadoop, Version 5
baseurl  = ftp://your-server-ip/pub/libs/cdh/5/
gpgkey   = ftp://your-server-ip/pub/libs/cdh/RPM-GPG-KEY-cloudera
gpgcheck = 1

好了,cdh的本地仓库建好了。只要能访问到这个ftp的服务器中存在这个 /etc/yum.repos.d/cdh5.repo 文件,就可以安装cdh hadoop的软件包。比如安装个zookeeper服务端:

#Installing the ZooKeeper Base Package
  $ yum install zookeeper

# Installing the ZooKeeper Server Package
  $ yum install zookeeper-server

# start zookeeper-server
  $ service zookeeper-server init --myid=1
Using myid of 1

OK!

时间: 2024-10-22 00:38:37

cdh5 hadoop redhat 本地仓库配置的相关文章

Maven本地仓库配置

本地仓库是远程仓库的一个缓冲和子集,当你构建Maven项目的时候,首先会从本地仓库查找资源,如果没有,那么Maven会从远程仓库下载到你本地仓库.这样在你下次使用的时候就不需要从远程下载了.如果你所需要的jar包版本在本地仓库没有,而且也不存在于远程仓库,Maven在构建的时候会报错,这种情况可能是有些jar包的新版本没有在Maven仓库中及时更新. Maven缺省的本地仓库路径为${user.home}/.m2/repository. 可以通过修改${user.home}/.m2/settin

Maven本地仓库配置及修改Maven的本地仓库路径

安装Maven后我们会在用户目录下发现.m2 文件夹.默认情况下,该文件夹下放置了Maven本地仓库.m2/repository.所有的Maven构件(artifact)都被存储到该仓库中,以方便重用.但是windows用户的操作系统都安装在C盘,把Maven仓库放到C盘是很危险的,为此我们需要修改Maven的本地仓库路径. 修改方式如下:     1.手动创建本地仓库的存放路径,例如:D:\Java\m2\repository:     2.在windows的环境变量中增加了新的用户变量M2_

Maven 教程(3)— Maven仓库介绍与本地仓库配置

原文地址:https://blog.csdn.net/liupeifeng3514/article/details/79537837 1.Maven本地仓库/远程仓库的基本介绍 本地仓库是指存在于我们本机的仓库,在我们加入依赖时候,首先会跑到我们的本地仓库去找,如果找不到则会跑到远程仓库中去找.对于依赖的包大家可以从这个地址进行搜索:http://mvnrepository.com/. 远程仓库是指其他服务器上的仓库,包括全球中央仓库,公司内部的私服,又或者其他公司提供的公共库.后面会专门聊聊私

maven本地仓库配置到指定目录和中央仓库镜像的相关配置

一.本地仓库的配置 1. 在maven的解压目录,找到settings.xml文件(如:D:\Program Files\apache-maven-3.6.3\conf\settings.xml) 找到如下的位置添加以下内容: <!-- 将本地仓库(jar包目录)配置到指定路径(*可以不进行配置,默认在C盘)--> <localRepository>D:\myFiles\apache-maven-3.6.3\repository</localRepository> 二.

淘淘商城 本地仓库配置和仓库jar包下载

SVN服务器的搭建请查看该文:<Win7 x64 svn 服务器搭建> 1:仓库包存放位置: 2:setting.xml 文件配置信息 1 <?xml version="1.0" encoding="UTF-8"?> 2 3 <!-- 4 Licensed to the Apache Software Foundation (ASF) under one 5 or more contributor license agreements.

配置docker本地仓库遇到的一些问题

在执行一下命令的时候遇到一些问题,记录如下: 000 # pip install docker-registry Searching for M2Crypto==0.22.3 Reading https://pypi.python.org/simple/M2Crypto/ Best match: M2Crypto 0.22.3 Downloading https://pypi.python.org/packages/source/M/M2Crypto/M2Crypto-0.22.3.tar.gz

gradle本地、远程仓库配置--转

https://blog.csdn.net/x_iya/article/details/75040806 本地仓库配置配置环境变量GRADLE_USER_HOME,并指向你的一个本地目录,用来保存Gradle下载的依赖包. 远程仓库配置一般Gradle.maven从中央仓库mavenCentral() http://repo1.maven.org/maven2/下载依赖包,但是在国内下载速度巨慢,我们只能使用国内的镜像. 所以每个Gradle构建的项目中,我们可以在build.gradle做如下

hadoop hive2.0 mysql本地仓库安装 错误解决

参考资料: hive安装手册. Hadoop2.7实战v1.0之Hive-2.0.0+MySQL远程模式安装  http://m.blog.itpub.net/30089851/viewspace-2082805/ 安装环境 Ubuntu 12.04 server java 1.7.0_95 hadoop 2.6.4 步骤: 1.安装mysql 直接使用命令安装: 更新源 sudo apt-get update 安装 sudo apt-get install mysql-server mysql

ambari 搭建hadoop大数据平台系列3-搭建本地仓库

1.官方参考 https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/setting_up_a_local_repository_with_no_internet_access.html 2.本地实例 10.1.11.32 localRepository-11-32   下载路径:记得下载后md5校验,下载时间很长. HDP: http://public-repo-1.hort