Skip to main content

Sqoop example to pull data from mysql



Example1:
sqoop import --connect jdbc:mysql:<localhost>/<databasename> --username=<username> --password-file <password file location> -table <tablename> -m 1 --compression-codec=snappy --as-avrodatafile --warehouse-dir=<hdfs location for creating datafiles>

Example 2:
sqoop import --connect jdbc:mysql:<localhost>/<databasename> --username=<username> --password=<password> -table <tablename> -m 1 --compression-codec=snappy --as-avrodatafile --warehouse-dir=<hdfs location for creating datafiles>

We can use this flag to overwrite the data pulled
--hive-overwrite flag
Labels:

Comments

Post a Comment

Popular posts from this blog

UNIX : How to ignore lines with certain names

Sometimes we need to ignore multiple lines with certain words and get the list out of the file. usually it will be a log file to read . The below grep command can be used to ignore multiple words present in a text file. Lets say the file contain $ cat list.txt apple orange apple banana papaya Now we need to ignore line with orange , banana and papaya . So we can use the below grep command. $ cat list.txt | grep -Ev "orange|banana|papaya" apple apple It will ignore lines with the words in -v part of grep.
Post a Comment
Read more

UNIX : unpack and pack using tar

Tarball or tar is a utility in linux to unpack or pack the files. Lets say we have a file name hello.tar.gz which is packed and compressed. To unpack the file use the below command > tar xvzf hello.tar.gz This is unpack the compressed tarball file to the directory hello in current location. To pack a list of files to a compressed tarball file , we can follow the below steps. Let say we have the following list of files in Hello directory. > cd Hello > ls helloworld.py README.txt Move back to parent directory and give the below command. > cd .. > tar zcvf hello.tar.gz Hello This will create the compressed tar file in the current location
Post a Comment
Read more