Download Hadoop

On the authorized_keys file, run:sudo chmod 640 ~/.ssh/authorized_keysFinally, you are ready to test SSH configuration:ssh localhostNotes:If you didn’t set a passphrase, you should be logged in automatically.If you set a passphrase, you’ll be prompted to enter it.Step 3: Download the latest stable releaseTo download Apache Hadoop, visit the Apache Hadoop download page. Find the latest stable release (e.g., 3.3.4) and copy the download link.Also, you can download the release using wget command:wget extract the downloaded file:tar -xvzf hadoop-3.3.4.tar.gzTo move the extracted directory, run:sudo mv hadoop-3.3.4 /usr/local/hadoopUse the command below to create a directory for logs:sudo mkdir /usr/local/hadoop/logsNow, you need to change ownership of the Hadoop directory. So, use:sudo chown -R hadoop:hadoop /usr/local/hadoopStep 4: Configure Hadoop Environment VariablesEdit the .bashrc file using the command below:sudo nano ~/.bashrcAdd environment variables to the end of the file by running the following command:export HADOOP_HOME=/usr/local/hadoopexport HADOOP_INSTALL=$HADOOP_HOMEexport HADOOP_MAPRED_HOME=$HADOOP_HOMEexport HADOOP_COMMON_HOME=$HADOOP_HOMEexport HADOOP_HDFS_HOME=$HADOOP_HOMEexport YARN_HOME=$HADOOP_HOMEexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/binexport HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"To save changes and source the .bashrc file, type:source ~/.bashrcWhen you are finished, you are ready for Ubuntu Hadoop setup.Step 5: Configure Hadoop Environment VariablesFirst, edit the hadoop-env.sh file by running the command below:sudo nano $HADOOP_HOME/etc/hadoop/hadoop-env.shNow, you must add the path to Java. If you haven’t already added the JAVA_HOME variable in your .bashrc file, include it here:export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64export HADOOP_CLASSPATH+=" $HADOOP_HOME/lib/*.jar"Save changes and exit when you are done.Then, change your current working directory to /usr/local/hadoop/lib:cd /usr/local/hadoop/libThe below command lets you download the javax activation file:sudo wget you are finished, you can check the Hadoop version:hadoop versionIf you have passed the steps correctly, you can now configure Hadoop Core Site. To edit the core-site.xml file, run:sudo nano $HADOOP_HOME/etc/hadoop/core-site.xmlAdd the default filesystem URI: fs.default.name hdfs://0.0.0.0:9000 The default file system URI Save changes and exit.Use the following command to create directories for NameNode and DataNode:sudo mkdir -p /home/hadoop/hdfs/{namenode,datanode}Then, change ownership of the directories:sudo chown -R hadoop:hadoop /home/hadoop/hdfsTo change the ownership of the created directory to the hadoop user:sudo chown -R hadoop:hadoop /home/hadoop/hdfsTo edit the hdfs-site.xml file, first run:sudo nano $HADOOP_HOME/etc/hadoop/hdfs-site.xmlThen, paste the following line to set the replication factor: dfs.replication 1 Save changes and exit.At this point, you can configure MapReduce. Run the command below to edit the mapred-site.xml file:sudo nano $HADOOP_HOME/etc/hadoop/mapred-site.xmlTo set the MapReduce framework, paste the following line: mapreduce.framework.name yarn Save changes and exit.To configure YARN, run the command below and edit the yarn-site.xml file:sudo nano $HADOOP_HOME/etc/hadoop/yarn-site.xmlPaste the following to enable the MapReduce shuffle service: yarn.nodemanager.aux-services mapreduce_shuffle Save changes and exit.Format the NameNode by

Hadoop is a distributed computing framework for processing and storing massive datasets. It runs on Ubuntu and offers scalable data storage and parallel processing capabilities.Installing Hadoop enables you to efficiently handle big data challenges and extract valuable insights from your data.To Install Hadoop on Ubuntu, the below steps are required:Install Java.Create a User.Download Hadoop.Configure Environment.Configure Hadoop.Start Hadoop.Access Web Interface.Prerequisites to Install Hadoop on UbuntuComplete Steps to Install Hadoop on UbuntuStep 1: Install Java Development Kit (JDK)Step 2: Create a dedicated user for Hadoop & Configure SSHStep 3: Download the latest stable releaseStep 4: Configure Hadoop Environment VariablesStep 5: Configure Hadoop Environment VariablesStep 6: Start the Hadoop ClusterStep 7: Open the web interfaceWhat is Hadoop and Why Install it on Linux Ubuntu?What are the best Features and Advantages of Hadoop on Ubuntu?What to do after Installing Hadoop on Ubuntu?How to Monitor the Performance of the Hadoop Cluster?Why Hadoop Services are Not starting on Ubuntu?How to Troubleshoot issues with HDFS?Why My MapReduce jobs are failing?ConclusionPrerequisites to Install Hadoop on UbuntuBefore installing Hadoop on Ubuntu, make sure your system is meeting below specifications:A Linux VPS running Ubuntu.A non-root user with sudo privileges.Access to Terminal/Command line.Complete Steps to Install Hadoop on UbuntuOnce you provided the above required options for Hadoop installation Ubuntu including buying Linux VPS, you are ready to follow the steps of this guide.In the end, you will be able to leverage its capabilities to efficiently manage and analyze large datasets.Step 1: Install Java Development Kit (JDK)Since Hadoop requires Java to run, use the following command to install the default JDK and JRE:sudo apt install default-jdk default-jre -yThen, run the command below to Verify the installation by checking the Java version:java -versionOutput:java version "11.0.16" 2021-08-09 LTSOpenJDK 64-Bit Server VM (build 11.0.16+8-Ubuntu-0ubuntu0.22.04.1)As you see, if Java is installed, you’ll see the version information.Step 2: Create a dedicated user for Hadoop & Configure SSHTo create a new user, run the command below and create the Hadoop user:sudo adduser hadoopTo add the user to the sudo group, type:sudo usermod -aG sudo hadoopRun the command below to switch to the Hadoop user:sudo su - hadoopTo install OpenSSH server and client, run:sudo apt install openssh-server openssh-client -yThen, generate SSH keys by running the following command:ssh-keygen -t rsaNotes:Press Enter to save the key to the default location.You can optionally set a passphrase for added security.Now, you can add the public key to authorized_keys:cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keysTo set permissions

Database Service Node to Run the Examples with Oracle SQL Connector for HDFSYou must configure the co-managed Database service node in order to run the examples, as shown below. See Oracle Big Data Connectors User's Guide, section Installing and Configuring a Hadoop Client on the Oracle Database System for more details. Generate Oracle SQL Connector for HDFS zip file on the cluster node and copy to the database node. Example:cd /opt/oraclezip -r /tmp/orahdfs-.zip orahdfs-/*Unzip the Oracle SQL Connector for HDFS zip file on the database node. Example:mkdir -p /u01/misc_products/bdcunzip orahdfs-.zip -d /u01/misc_products/bdcInstall the Hadoop client on the database node in the /u01/misc_products/ directory.Connect as the sysdba user for the PDB and verify that both OSCH_BIN_PATH and OSCH_DEF_DIR database directories exist and point to valid operating system directories. For example, create or replace directory OSCH_BIN_PATH as '/u01/misc_products/bdc/orahdfs-/bin'; grant read,execute on directory OSCH_BIN_PATH to OHSH_EXAMPLES; where OHSH_EXAMPLES is the user created in Step 2: Create the OHSH_EXAMPLES User, above.create or replace directory OSCH_DEF_DIR as '/u01/misc_products/bdc/xtab_dirs'; grant read,write on directory OSCH_DEF_DIR to OHSH_EXAMPLES; Note: create the xtab_dirs operating system directory if it doesn't exist. Change to your OSCH (Oracle SQL Connector for HDFS) installation directory, and edit the configuration file hdfs_stream. For example,sudo su -l oracle cd /u01/misc_products/bdc/orahdfs- vi bin/hdfs_streamCheck that the following variables are configured correctly. Read the instructions included in the hdfs_stream file for more details.#Include Hadoop client bin directory to the PATH variable export PATH=/u01/misc_products/hadoop-/bin:/usr/bin:/bin export JAVA_HOME=/usr/java/jdk #See explanation below export HADOOP_CONF_DIR=/u01/misc_products/hadoop-conf#Activate the Kerberos configuration for secure clustersexport HADOOP_CLIENT_OPTS="-Djava.security.krb5.conf=/u01/misc_products/krb5.conf"Configure the Hadoop configuration directory (HADOOP_CONF_DIR). If it's not already configured, use Apache Ambari to download the Hadoop Client configuration archive file, as follows:Login to Apache Ambari. the HDFS service, and select the action Download Client Configuration. Extract the files under the HADOOP_CONF_DIR (/u01/misc_products/hadoop-conf) directory. Ensure that the hostnames and ports configured in HADOOP_CONF_DIR/core-site.xml are accessible from your co-managed Database service node (see the steps below). For example, fs.defaultFS hdfs://bdsmyhostmn0.bmbdcsxxx.bmbdcs.myvcn.com:8020 In this example host bdsmyhostmn0.bmbdcsxxx.bmbdcs.myvcn.com and port 8020 must be accessible from your co-managed Database service node. For secure clusters:Copy the Kerberos configuration file from the cluster node to the database node. Example:cp krb5.conf /u01/misc_products/Copy the Kerberos keytab file from the cluster node to the database node. Example:cp /u01/misc_products/Run the following commands to verify that HDFS access is working. #Change to the Hadoop client bin directory cd /u01/misc_products/hadoop-/bin #--config points to your HADOOP_CONF_DIR directory. ./hadoop --config /u01/misc_products/hadoop-conf fs -ls This command should list the HDFS contents. If you get a timeout or "no route to host" or "unknown host" errors, you will need to update your /etc/hosts file and verify your Big Data Service Console network configuration, as follows:Sign into the Cloud Console, click Big Data, then Clusters, then your_cluster>, then Cluster Details. Under the List of cluster nodes section, get the fully qualified name of all your cluster nodes and all the IP addresses .Edit your co-managed Database service configuration file /etc/hosts, for example: #BDS hostnames xxx.xxx.xxx.xxx bdsmynodemn0.bmbdcsad1.bmbdcs.oraclevcn.com bdsmynodemn0 xxx.xxx.xxx.xxx bdsmynodewn0.bmbdcsad1.bmbdcs.oraclevcn.com bdsmynodewn0 xxx.xxx.xxx.xxx bdsmynodewn2.bmbdcsad1.bmbdcs.oraclevcn.com bdsmynodewn2 xxx.xxx.xxx.xxx bdsmynodewn1.bmbdcsad1.bmbdcs.oraclevcn.com bdsmynodewn1 xxx.xxx.xxx.xxx bdsmynodeun0.bmbdcsad1.bmbdcs.oraclevcn.com bdsmynodeun0