Variations of Hadoop ls Shell Command
$ hadoop fs –ls /
Returns all the available files and subdirectories present under the root directory.
$ hadoop fs –ls –R /user/cloudera
Returns all the available files and recursively lists all the subdirectories under /user/Cloudera
4) mkdir- Used to create a new directory in HDFS at a given location.
Example of HDFS mkdir Command -
$ hadoop fs –mkdir /user/cloudera/sample
The above command will create a new directory named dezyre1 under the location /user/cloudera
Note : Cloudera and other
hadoop distribution vendors provide /user/
directory with read/write permission to all users but other directories are available as read-only.Thus, to create a folder in the root directory, users require superuser permission as shown below -
$ sudo –u hdfs hadoop fs –mkdir /test
This command will create a new directory named dezyre under the / (root directory).
5) copyFromLocal
Copy a file from local filesytem to HDFS location.
For the following examples, we will use Sample.txt file available in the /home/Cloudera location.
xample - $ hadoop fs –copyFromLocal Sample1.txt /user/cloudera/test
Copy/Upload Sample1.txt available in /home/cloudera (local default) to /user/cloudera/dezyre1 (hdfs path)
6) put –
This hadoop command uploads a single file or multiple source files from local file system to hadoop distributed file system (HDFS).
Ex - $ hadoop fs –put Sample2.txt /user/cloudera/test
Copy/Upload Sample2.txt available in /home/cloudera (local default) to /user/cloudera/dezyre1 (hdfs path)
7) moveFromLocal
This hadoop command functions similar to the put command but the source file will be deleted after copying.
Example - $ hadoop fs –moveFromLocal Sample3.txt /user/cloudera/dezyre1
Move Sample3.txt available in /home/cloudera (local default) to /user/cloudera/dezyre1 (hdfs path). Source file will be deleted after moving.
8) du
Displays the disk usage for all the files available under a given directory.
Example - $ hadoop fs –du /user/cloudera/test
9) df
Displas disk usage of current hadoop distributed file system.
Example - $ hadoop fs –df
10) Expunge
This HDFS command empties the trash by deleting all the files and directories.
Example - $ hadoop fs –expunge
11) Cat
This is similar to the cat command in Unix and displays the contents of a file.
Example - $ hadoop fs –cat /user/cloudera/test/Sample1.txt
12) cp
Copy files from one HDFS location to another HDFS location.
Example – $ hadoop fs –cp /user/cloudera/test/war_and_peace /user/cloudera/test/
13) mv
Move files from one HDFS location to another HDFS location.
Example – $ hadoop fs –mv /user/cloudera/test/Sample1.txt /user/cloudera/tes/
14) rm
Removes the file or directory from the mentioned HDFS location.
Example – $ hadoop fs –rm -r /user/cloudera/test
Example – $ hadoop fs –rm -r /user/cloudera/test
Deletes or removes the directory and its content from HDFS location in a recursive manner.
Example – $ hadoop fs –rm /user/cloudera/test
Delete or remove the files from HDFS location.
15) tail
This hadoop command will show the last kilobyte of the file to stdout.
Example – $ hadoop fs -tail /user/cloudera/test/war_and_peace
16) copyToLocal
Copies the files to the local filesystem . This is similar to hadoop fs -get command but in this case the destination location msut be a local file reference
Example - $ hadoop fs –copyFromLocal /user/cloudera/test/Sample1.txt /home/cloudera/hdfs_bkp/
Copy/Download Sample1.txt available in /user/cloudera/test (hdfs path) to /home/cloudera/hdfs_bkp/ (local path)
17) get
Downloads or Copies the files to the local filesystem.
Example - $ hadoop fs –get /user/cloudera/test/Sample2.txt /home/cloudera/hdfs_bkp/
Copy/Download Sample2.txt available in /user/cloudera/test (hdfs path) to /home/cloudera/hdfs_bkp/ (local path)
18) touchz
Used to create an emplty file at the specified location.
Example - $ hadoop fs –touchz /user/cloudera/test/Sample4.txt
It will create a new empty file Sample4.txt in /user/cloudera/test/ (hdfs path)
19) setrep
This hadoop fs command is used to set the replication for a specific file.
Example - $ hadoop fs –setrep –w 1 /user/cloudera/test/Sample1.txt
It will set the replication factor of Sample1.txt to 1
20) chgrp
This hadoop command is basically used to change the group name.
Example - $ sudo –u hdfs hadoop fs –chgrp –R cloudera /test
It will change the /test directory group membership from supergroup to cloudera (To perform this operation superuser permission is required)
21) chown
This command lets you change both the owner and group name simulataneously.
Example - $ sudo –u hdfs hadoop fs –chown –R cloudera /test
It will change the /dezyre directory ownership from hdfs user to cloudera user (To perform this operation superuser is permission required)
22) hadoop chmod
Used to change the permissions of a given file/dir.
Example - $ hadoop fs –chmod /test
It will change the /dezyre directory permission to 700 (drwx------).
