实时计算<1>环境准备

硬件准备

机器名 IP 硬件配置 操作系统 软件配置
nimbus 192.168.55.173 4cpu 2.93GHz 4G内存 Linux x86_64 redis2.8、 storm0.9.1 nimbus/UI/DRPC、zookeeper3.4.6
supervisor1 192.168.55.174 4cpu 2.93GHz 4G内存 Linux x86_64 storm0.9.1 supervisor、zookeeper3.4.6
supervisor2 192.168.55.175 4cpu 2.93GHz 4G内存 Linux x86_64 storm0.9.1 supervisor、zookeeper3.4.6

软件准备

三台虚拟机,一台作为Storm的nimbus和redis数据库,另外两台做Storm的supervisor。

设置hosts文件

    vi /ets/hosts

如下

    127.0.0.1 nimbus
    192.168.55.173  nimbus
    192.168.55.174 supervisor1
    192.168.55.175 supervisor2

基本软件安装

三台机器必备安装软件,jdk、storm、zookeeper

Jdk1.7.0_15安装

  1. 下载JDK
     $ wget http://download.oracle.com/otn-pub/java/jdk/8u31-b13/jdk-7u15-linux-x64.tar.gz
    
  2. 解压JDK

     tar zvxf jdk-7u15-linux-x64.tar.gz
    
  3. 设置环境变量

     $ set PATH=/your_jdk_unzip_dir/jdk1.7.0_15/bin:$PATH
     $ set JAVA_HOME=/your_jdk_unzip_dir/jdk1.7.0_15
    
  4. 验证Jdk版本

     $ jdk -version
    

Zookeeper3.4.6安装

  1. 下载zookeeper

     $ wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
    
  2. 解压zookeeper

     $ tar zvxf zookeeper-3.4.6.tar.gz
    
  3. 修改配置文件

         $ vi /your_zookeeper_unzip_dir/conf/zoo.cfg
    
         # The number of milliseconds of each tick
         tickTime=2000
         # The number of ticks that the initial
         # synchronization phase can take
         initLimit=10
         # The number of ticks that can pass between
         # sending a request and getting an acknowledgement
         syncLimit=5
         # the directory where the snapshot is stored.
         dataDir=/your_zookeeper_unzip_dir/dataDir/zookeeper
         # the port at which the clients will connect
         clientPort=2181            
    
         server.1=192.168.55.173:2888:3888
         server.2=192.168.55.174:2888:3888
         server.3=192.168.55.175:2888:3888
    

    接下来在dataDir所指定的目录下创建一个文件名为myid的文件,文件中的内容只有一行,为本主机对应的id值,也就是上图中server.id中的id。例如:在服务器1中的myid的内容应该写入1。

  4. 启动

      $ ./zkServer.sh start
    

Storm0.9.0.1安装

  1. 下载storm

      $ wget https://dl.dropboxusercontent.com/s/tqdpoif32gufapo/storm-0.9.0.1.tar.gz
    
  2. 解压storm

      $ tar zvxf storm-0.9.0.1.tar.gz
    
  3. 修改配置文件

      $ vi /your_storm_unzip_dir/conf/storm.yaml 
    
      nimbus.host: "nimbus"
      storm.zookeeper.servers:
             - "nimbus"
             - "supervisor1"
             - "supervisor2"
      storm.local.dir: "/your_storm_unzip_dir/storm-local"
      storm.messaging.transport: "backtype.storm.messaging.netty.Context"
      storm.messaging.netty.server_worker_threads: 1
      storm.messaging.netty.client_worker_threads: 1
      storm.messaging.netty.buffer_size: 5242880
      storm.messaging.netty.max_retries: 100
      storm.messaging.netty.max_wait_ms: 1000
      storm.messaging.netty.min_wait_ms: 100
      drpc.queue.size: 1024
      drpc.worker.threads: 512
      drpc.request.timeout.secs: 30
      drpc.servers:
          - "nimbus"
    
  4. 启动

    nimbus

     $ ./storm nimbus
    

    supervisor

     $ ./storm supervisor
    

    drpc

     $ ./storm drpc
    

    storm ui

     $ ./storm ui
    

Redis2.8.18安装

  1. 下载安装

     $ wget http://download.redis.io/releases/redis-2.8.18.tar.gz
     $ tar xzf redis-2.8.18.tar.gz
     $ cd redis-2.8.18
     $ make
    
  2. 启动

     $ src/redis-server