Hadoop 121 Post Clone Startup

Hadoop 1.2.1 Post-Clone Startup and Verification Skill

This skill assists students with the remaining steps after the first Hadoop VM has been cloned and /etc/hosts has been updated on both VMs.

This skill is for Apache Hadoop 1.2.1 in a two-VM instructional cluster.

The expected cluster layout is:

groupN-1 = NameNode + JobTracker + DataNode + TaskTracker
groupN-2 = DataNode + TaskTracker

The first VM has a dual role and should run both master and worker daemons. The second VM should run only worker daemons.

This skill assumes the earlier pre-clone configuration skill has already configured Hadoop files using Hadoop 1.x properties such as:

fs.default.name
mapred.job.tracker
dfs.replication

Do not use Hadoop 2.x or Hadoop 3.x YARN commands or properties.

Scope of this skill

Use this skill only after all of the following are expected to be true:

1. The first VM has already been cloned to create the second VM.
2. The first VM hostname is groupN-1.
3. The second VM hostname is groupN-2.
4. /etc/hosts has been updated on both VMs.
5. Hadoop 1.2.1 is installed at /usr/local/hadoop-1.2.1.
6. Hadoop configuration files were already prepared by the pre-clone configuration skill.

This skill can help with:

1. Opening required Hadoop firewall ports on both VMs.
2. Checking hostname resolution between both VMs.
3. Checking passwordless SSH from groupN-1 to both groupN-1 and groupN-2.
4. Checking whether the NameNode appears to have already been formatted.
5. Formatting the NameNode only if needed and only on groupN-1.
6. Starting Hadoop HDFS and MapReduce daemons only if they are not already running.
7. Verifying daemon processes on both VMs using jps.

Execution safety policy

The assistant may inspect the system using safe read-only commands.

The assistant must not run commands that modify the system unless the student explicitly asks it to run them.

Read-only checks are allowed when useful:

hostname
cat /etc/hosts
getent hosts groupN-1
getent hosts groupN-2
ssh -o BatchMode=yes -o ConnectTimeout=5 groupN-1 hostname
ssh -o BatchMode=yes -o ConnectTimeout=5 groupN-2 hostname
jps
sudo firewall-cmd --zone=public --query-port=50010/tcp
sudo firewall-cmd --zone=public --query-port=50030/tcp
sudo firewall-cmd --zone=public --query-port=50060/tcp
sudo firewall-cmd --zone=public --query-port=50070/tcp
sudo firewall-cmd --zone=public --query-port=50075/tcp
sudo firewall-cmd --zone=public --query-port=50090/tcp
sudo firewall-cmd --zone=public --query-port=54310/tcp
sudo firewall-cmd --zone=public --query-port=54311/tcp

Potentially modifying commands require explicit student permission:

sudo firewall-cmd --zone=public --add-port=PORT/tcp
hadoop namenode -format
start-dfs.sh
start-mapred.sh

The assistant must never format the NameNode automatically just because formatting is the next step.

The assistant must first check whether formatting appears to have already happened.

The assistant must never start Hadoop daemons before verifying that hostname resolution and passwordless SSH work.

The assistant must never repeatedly start daemons that are already running.

Hadoop directory policy

Use HADOOP_PREFIX, not HADOOP_HOME.

The Hadoop installation directory is:

export HADOOP_PREFIX=/usr/local/hadoop-1.2.1

The Hadoop executable directory is:

$HADOOP_PREFIX/bin

When generating commands, use:

export HADOOP_PREFIX=/usr/local/hadoop-1.2.1
export PATH=$HADOOP_PREFIX/bin:$PATH

Do not use YARN commands.

Do not use:

start-yarn.sh
stop-yarn.sh
yarn
mapred --daemon
hdfs --daemon

For Hadoop 1.2.1, use:

$HADOOP_PREFIX/bin/hadoop namenode -format
$HADOOP_PREFIX/bin/start-dfs.sh
$HADOOP_PREFIX/bin/start-mapred.sh

Hostname detection and role policy

First detect the current hostname:

hostname

The current host should match one of these patterns:

groupN-1
groupN-2

If the current host is groupN-1, treat it as the master VM.

If the current host is groupN-2, treat it as the worker VM.

If the hostname does not match either pattern, warn the student and do not run Hadoop startup or formatting commands.

Infer the peer host as follows:

current host groupN-1 -> peer host groupN-2
current host groupN-2 -> peer host groupN-1

Formatting and cluster startup must be run only from groupN-1.

If the student is on groupN-2 and asks to format or start Hadoop, explain that these commands should be run from groupN-1.

Required pre-start checks

Before formatting or starting Hadoop, verify all of the following.

1. Verify hostname resolution

Use getent hosts:

getent hosts groupN-1
getent hosts groupN-2

Replace groupN with the detected group number.

If either command fails, tell the student to fix /etc/hosts on both VMs before proceeding.

Do not start Hadoop if either hostname does not resolve.

2. Verify passwordless SSH

From groupN-1, verify SSH to both hosts:

ssh -o BatchMode=yes -o ConnectTimeout=5 groupN-1 hostname
ssh -o BatchMode=yes -o ConnectTimeout=5 groupN-2 hostname

Expected output:

groupN-1
groupN-2

If either SSH command fails, do not format or start Hadoop.

Tell the student that passwordless SSH must work from groupN-1 to both VMs.

Do not ask the student to type a password into SSH for Hadoop startup verification.

3. Verify Hadoop configuration files exist

Check:

export HADOOP_PREFIX=/usr/local/hadoop-1.2.1
ls -l $HADOOP_PREFIX/conf/core-site.xml
ls -l $HADOOP_PREFIX/conf/hdfs-site.xml
ls -l $HADOOP_PREFIX/conf/mapred-site.xml
ls -l $HADOOP_PREFIX/conf/slaves
ls -l $HADOOP_PREFIX/conf/masters

Check that slaves contains both hosts:

cat $HADOOP_PREFIX/conf/slaves

Expected:

groupN-1
groupN-2

If the files are missing or inconsistent, tell the student to rerun or review the pre-clone Hadoop configuration skill before startup.

Firewall policy

The port numbers used by Hadoop processes are blocked by Chameleon Cloud’s strict firewall policy.

The following ports must be open on both VMs:

50010/tcp
50030/tcp
50060/tcp
50070/tcp
50075/tcp
50090/tcp
54310/tcp
54311/tcp

Before adding a port, always check whether it is already open.

Use this safe idempotent firewall logic on each VM:

for port in 50010 50030 50060 50070 50075 50090 54310 54311; do
    if sudo firewall-cmd --zone=public --query-port=${port}/tcp >/dev/null 2>&1; then
        echo "Port ${port}/tcp is already open."
    else
        sudo firewall-cmd --zone=public --add-port=${port}/tcp
        echo "Opened port ${port}/tcp."
    fi
done

This uses runtime firewall rules. Do not add --permanent unless the student or instructor explicitly asks for permanent firewall rules.

If the student asks to open the firewall on both VMs from groupN-1, first verify passwordless SSH to groupN-2, then use:

for host in groupN-1 groupN-2; do
    echo "Checking firewall ports on ${host}"
    ssh -o BatchMode=yes -o ConnectTimeout=5 ${host} '
        for port in 50010 50030 50060 50070 50075 50090 54310 54311; do
            if sudo firewall-cmd --zone=public --query-port=${port}/tcp >/dev/null 2>&1; then
                echo "Port ${port}/tcp is already open."
            else
                sudo firewall-cmd --zone=public --add-port=${port}/tcp
                echo "Opened port ${port}/tcp."
            fi
        done
    '
done

Replace groupN-1 and groupN-2 with the detected hostnames.

If sudo requires a password or the SSH remote command fails, report the failure and provide the local command for the student to run manually on each VM.

NameNode formatting policy

Formatting the NameNode is destructive for a fresh HDFS metadata directory and should only happen once during initial cluster setup.

Format only on groupN-1.

Before formatting, check whether the NameNode appears to have already been formatted.

Use:

export HADOOP_PREFIX=/usr/local/hadoop-1.2.1
NAMENODE_DIR="/app/hadoop/tmp/dfs/name"

if [ -d "$NAMENODE_DIR/current" ]; then
    echo "NameNode metadata already exists at $NAMENODE_DIR/current. Do not format again unless your instructor explicitly tells you to reset HDFS."
else
    echo "NameNode metadata directory was not found. This appears to be an unformatted fresh cluster."
fi

If the metadata directory exists, do not format.

If the metadata directory does not exist and all pre-start checks pass, the assistant may run the following only when the student explicitly asks it to format:

export HADOOP_PREFIX=/usr/local/hadoop-1.2.1
$HADOOP_PREFIX/bin/hadoop namenode -format

After formatting, verify that NameNode metadata now exists:

test -d /app/hadoop/tmp/dfs/name/current && echo "NameNode format appears complete."

If the format command reports an error, stop and explain the error. Do not proceed to daemon startup.

Daemon startup policy

Start daemons only from groupN-1.

Before starting, check current daemons with:

jps

Expected daemons on groupN-1 after successful startup:

NameNode
JobTracker
DataNode
TaskTracker

Expected daemons on groupN-2 after successful startup:

DataNode
TaskTracker

If all expected daemons are already running, do not run startup scripts again.

If only some daemons are running, report which ones are missing. Then start only the relevant subsystem if appropriate.

For HDFS:

$HADOOP_PREFIX/bin/start-dfs.sh

For MapReduce v1:

$HADOOP_PREFIX/bin/start-mapred.sh

Do not use start-all.sh unless the student explicitly asks for it. Prefer start-dfs.sh and start-mapred.sh because they are clearer for teaching.

Do not run start-dfs.sh if both NameNode and all expected DataNode processes are already running.

Do not run start-mapred.sh if JobTracker and all expected TaskTracker processes are already running.

Safe daemon verification logic

From groupN-1, use this verification script:

for host in groupN-1 groupN-2; do
    echo "===== ${host} ====="
    ssh -o BatchMode=yes -o ConnectTimeout=5 ${host} jps
    echo
 done

Expected output on groupN-1 should include:

NameNode
JobTracker
DataNode
TaskTracker

Expected output on groupN-2 should include:

DataNode
TaskTracker

Additional Java-related processes may also appear and are not necessarily errors.

If a required process is missing, explain which one is missing and suggest the likely area to check:

Missing NameNode: check NameNode formatting and core-site.xml.
Missing DataNode: check slaves file, SSH, firewall, and HDFS logs.
Missing JobTracker: check mapred-site.xml and start-mapred.sh output.
Missing TaskTracker: check slaves file, SSH, firewall, and MapReduce logs.

Recommended full post-clone workflow

When a student asks what to do next after cloning, guide them through this sequence:

1. Confirm they are on groupN-1.
2. Verify /etc/hosts resolution for groupN-1 and groupN-2.
3. Verify passwordless SSH from groupN-1 to both VMs.
4. Check firewall ports on both VMs.
5. Open missing firewall ports only if the student asks the assistant to run the commands.
6. Check whether the NameNode has already been formatted.
7. Format the NameNode only if it has not been formatted and the student explicitly asks.
8. Check current daemon status using jps.
9. Start missing Hadoop daemons only if the student explicitly asks.
10. Verify final daemon status on both VMs.

One-shot guided command block for students

If the student asks for commands to run manually, provide this block from groupN-1 after replacing groupN with the correct group number:

export HADOOP_PREFIX=/usr/local/hadoop-1.2.1
export PATH=$HADOOP_PREFIX/bin:$PATH

# 1. Verify hostname resolution
getent hosts groupN-1
getent hosts groupN-2

# 2. Verify passwordless SSH
ssh -o BatchMode=yes -o ConnectTimeout=5 groupN-1 hostname
ssh -o BatchMode=yes -o ConnectTimeout=5 groupN-2 hostname

# 3. Check/open firewall ports on both VMs
for host in groupN-1 groupN-2; do
    echo "Checking firewall ports on ${host}"
    ssh -o BatchMode=yes -o ConnectTimeout=5 ${host} '
        for port in 50010 50030 50060 50070 50075 50090 54310 54311; do
            if sudo firewall-cmd --zone=public --query-port=${port}/tcp >/dev/null 2>&1; then
                echo "Port ${port}/tcp is already open."
            else
                sudo firewall-cmd --zone=public --add-port=${port}/tcp
                echo "Opened port ${port}/tcp."
            fi
        done
    '
done

# 4. Check whether NameNode appears already formatted
if [ -d /app/hadoop/tmp/dfs/name/current ]; then
    echo "NameNode metadata already exists. Skipping format."
else
    echo "NameNode metadata not found. Format is needed for a fresh cluster."
    echo "Run this only once if your instructor says this is a fresh cluster:"
    echo "$HADOOP_PREFIX/bin/hadoop namenode -format"
fi

# 5. Check local daemons
jps

Do not include the actual format command as an automatically executed line in a one-shot command block unless the student explicitly asks to execute formatting.

One-shot execution policy if the student asks the assistant to run everything

If the student says something like “run the remaining steps for me,” the assistant must still perform checks and avoid duplicate or unsafe actions.

Use this decision order:

1. Detect hostname.
2. Stop if not on groupN-1.
3. Verify groupN-1 and groupN-2 resolve.
4. Verify passwordless SSH to both VMs.
5. Check and open missing firewall ports on both VMs.
6. Check NameNode formatting status.
7. If NameNode metadata exists, skip formatting.
8. If NameNode metadata does not exist, ask for explicit confirmation before formatting unless the student’s latest message explicitly included permission to format.
9. Check daemon status on both VMs.
10. Start HDFS only if HDFS daemons are missing and NameNode has been formatted.
11. Start MapReduce only if MapReduce daemons are missing.
12. Verify final daemon status on both VMs.

If the latest student message explicitly says to format if needed, then formatting may proceed after the safety checks pass.

Command execution script for assistant use when explicitly authorized

When the student explicitly asks the assistant to run the post-clone setup, use a cautious script like this from groupN-1.

Before running it, replace groupN-1 and groupN-2 with the detected hostnames.

#!/usr/bin/env bash
set -u

export HADOOP_PREFIX=/usr/local/hadoop-1.2.1
export PATH="$HADOOP_PREFIX/bin:$PATH"

MASTER="groupN-1"
WORKER="groupN-2"
HOSTS="$MASTER $WORKER"
PORTS="50010 50030 50060 50070 50075 50090 54310 54311"

current_host="$(hostname)"
if [ "$current_host" != "$MASTER" ]; then
    echo "ERROR: This startup script must be run on $MASTER, but current hostname is $current_host."
    exit 1
fi

for host in $HOSTS; do
    if getent hosts "$host" >/dev/null 2>&1; then
        echo "Hostname resolves: $host"
    else
        echo "ERROR: Hostname does not resolve: $host"
        exit 1
    fi
done

for host in $HOSTS; do
    if ssh -o BatchMode=yes -o ConnectTimeout=5 "$host" hostname >/tmp/ssh-check-${host}.out 2>/tmp/ssh-check-${host}.err; then
        echo "Passwordless SSH works for $host: $(cat /tmp/ssh-check-${host}.out)"
    else
        echo "ERROR: Passwordless SSH failed for $host."
        cat /tmp/ssh-check-${host}.err
        exit 1
    fi
done

for host in $HOSTS; do
    echo "===== Checking firewall on $host ====="
    ssh -o BatchMode=yes -o ConnectTimeout=5 "$host" "
        for port in $PORTS; do
            if sudo firewall-cmd --zone=public --query-port=\${port}/tcp >/dev/null 2>&1; then
                echo \"Port \${port}/tcp is already open.\"
            else
                sudo firewall-cmd --zone=public --add-port=\${port}/tcp
                echo \"Opened port \${port}/tcp.\"
            fi
        done
    "
done

if [ -d /app/hadoop/tmp/dfs/name/current ]; then
    echo "NameNode metadata already exists. Skipping format."
else
    echo "NameNode metadata not found. Formatting NameNode because explicit permission was given."
    "$HADOOP_PREFIX/bin/hadoop" namenode -format
fi

master_jps="$(jps)"
worker_jps="$(ssh -o BatchMode=yes -o ConnectTimeout=5 "$WORKER" jps)"

if echo "$master_jps" | grep -q NameNode && echo "$master_jps" | grep -q DataNode && echo "$worker_jps" | grep -q DataNode; then
    echo "HDFS daemons appear to already be running. Skipping start-dfs.sh."
else
    echo "Starting HDFS daemons."
    "$HADOOP_PREFIX/bin/start-dfs.sh"
fi

sleep 3
master_jps="$(jps)"
worker_jps="$(ssh -o BatchMode=yes -o ConnectTimeout=5 "$WORKER" jps)"

if echo "$master_jps" | grep -q JobTracker && echo "$master_jps" | grep -q TaskTracker && echo "$worker_jps" | grep -q TaskTracker; then
    echo "MapReduce daemons appear to already be running. Skipping start-mapred.sh."
else
    echo "Starting MapReduce daemons."
    "$HADOOP_PREFIX/bin/start-mapred.sh"
fi

sleep 3

echo "===== Final daemon status on $MASTER ====="
jps

echo "===== Final daemon status on $WORKER ====="
ssh -o BatchMode=yes -o ConnectTimeout=5 "$WORKER" jps

This script is intentionally idempotent for firewall opening and cautious about Hadoop startup.

Final success criteria

The final verification should show the following.

On groupN-1:

NameNode
JobTracker
DataNode
TaskTracker

On groupN-2:

DataNode
TaskTracker

If these are present, tell the student that the Hadoop 1.2.1 two-VM cluster appears to be running.

If any daemon is missing, report the missing daemon and suggest checking logs under:

/usr/local/hadoop-1.2.1/logs

Important warnings to preserve

Always preserve these warnings:

Do not format the NameNode more than once unless your instructor explicitly tells you to reset HDFS.

Run formatting and startup commands from groupN-1, not groupN-2.

For Hadoop 1.2.1, use MapReduce v1 commands. Do not use YARN commands.