Configure Hadoop and Start cluster services using Ansible Playbook.

About Hadoop

About Ansible

  • rhel8_arth — 192.168.29.202 — machine where Ansible is installed
  • rhel-exp-2 — 192.168.29.225 — Name-node Machine
  • rhel8 hadoop slave — 192.168.29.195 — Data-node Machine

Configuring Control Node.

  • Setting the inventory file so that ansible can connect with the managed nodes.
Setting inventory file
  • Now since inventory file is configured. Ansible can ping to managed nodes by the command -
anisble <ip/name of managed nodes> -m ping.
Checking the connectivity

Namenode Configuration

  1. core-site.xml
  2. hdfs-site.xml
  3. hadoop.yml
  4. nam_var.yml
  • core-site.xml
core-site.xml
  • hdfs-site.xml
hdfs-site.xml
  • nam_var.yml file
hadoop_path: "/root/hadoop-1.2.1-1.x86_64.rpm"
jdk_path: "/root/jdk-8u171-linux-x64.rpm"
hadoop_software: "hadoop-1.2.1-1.x86_64.rpm"
jdk_software: "jdk-8u171-linux-x64.rpm"
  • Playbook for Namenode
- hosts: namenode
vars_files:
- name_var.yml
tasks:
- name: "copying jdk files on namenode"
copy:
dest: "/root"
src: "{{ jdk_path }}"
- name: "copying hadoop files on namenode"
copy:
dest: "/root"
src: "{{ hadoop_path }}"
- name: "installing jdk"
command: rpm -i "{{ jdk_software }}"
ignore_errors: yes


- name: "installing hadoop"
command: rpm -i hadoop-1.2.1-1.x86_64.rpm --force
ignore_errors: yes
- name: configuring core-site.xml file
template:
dest: "/etc/hadoop/core-site.xml"
src: "core-site.xml"
- name: configuring hdfs-site.xml file
template:
dest: "/etc/hadoop/hdfs-site.xml"
src: "hdfs-site.xml"
- name: formatting master
command: hadoop namenode -format
- name: starting hadoop service
command: hadoop-daemon.sh start namenode
ansible-playbook <name_of_YML_file>
Running playbook to configure Namenode
Running playbook to configure Namenode

Datanode Configuration

  • core-site.xml
core-site.xml
  • hdfs-site.xml
hdfs-site.xml
  • nam_var.yml file
hadoop_path: "/root/hadoop-1.2.1-1.x86_64.rpm"
jdk_path: "/root/jdk-8u171-linux-x64.rpm"
hadoop_software: "hadoop-1.2.1-1.x86_64.rpm"
jdk_software: "jdk-8u171-linux-x64.rpm"
  • Playbook for Datanode
- hosts: datanode
vars_files:
- name_var.yml
tasks:
- name: "copying jdk files on datanode"
copy:
dest: "/root"
src: "{{ jdk_path }}"
- name: "copying hadoop files on datanode"
copy:
dest: "/root"
src: "{{ hadoop_path }}"
- name: "installing jdk"
command: rpm -i "{{ jdk_software }}"
ignore_errors: yes


- name: "installing hadoop"
command: rpm -i hadoop-1.2.1-1.x86_64.rpm --force
ignore_errors: yes
- name: configuring core-site.xml file
template:
dest: "/etc/hadoop/core-site.xml"
src: "core-site.xml"
- name: configuring hdfs-site.xml file
template:
dest: "/etc/hadoop/hdfs-site.xml"
src: "hdfs-site.xml"


- name: starting hadoop service on datanode
command: hadoop-daemon.sh start datanode
ansible-playbook <name_of_yml_file>
Running playbook to configure Datanode
Running playbook to configure Datanode
hadoop dfsadmin -report

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store