Apache Hadoop Installation

This guide is a collection or a summary on how to install and use a footprint of Apache Hadoop. I tried to follow an old version 2.7.1 guide that I created few years ago and adjusted this to use the latest version. Apache Hadoop 3.5.0 is used below; check the Apache releases page before future installations. These instructions target Linux (Ubuntu/Debian) for development or testing. Production clusters need Kerberos, network controls, encryption, monitoring, backups, and an upgrade plan. Do not expose HDFS or YARN ports to the internet. Native single-node installation Prerequisites sudo apt-get update sudo apt-get install -y openjdk-17-jdk openssh-client openssh-server pdsh curl tar java -version Hadoop requires Java and SSH; pdsh is recommended by the current Apache single-node documentation. Find JAVA_HOME if needed: readlink -f "$(command -v java)" | sed 's:/bin/java::' Download and install Pin the version for repeatable installs and verify Apache's SHA-512 checksum: export HADOOP_VERSION=3.5.0 cd /tmp curl -fLO "https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz" curl -fLO "https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz.sha512" sha512sum -c "hadoop-${HADOOP_VERSION}.tar.gz.sha512" sudo tar -xzf "hadoop-${HADOOP_VERSION}.tar.gz" -C /opt sudo ln -sfn "/opt/hadoop-${HADOOP_VERSION}" /opt/hadoop sudo chown -R "$USER":"$USER" "/opt/hadoop-${HADOOP_VERSION}" Add this to ~/.bashrc, adjusting JAVA_HOME if necessary: export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 export HADOOP_HOME=/opt/hadoop export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop" export HADOOP_HDFS_HOME="$HADOOP_HOME" export HADOOP_YARN_HOME="$HADOOP_HOME" export HADOOP_MAPRED_HOME="$HADOOP_HOME" export PATH="$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin" Then load and verify it: source ~/.bashrc sed -i "s|^# export JAVA_HOME=.*|export JAVA_HOME=${JAVA_HOME}|" "$HADOOP_HOME/etc/hadoop/hadoop-env.sh" had