What is Sqoop?

What is Sqoop?
As we all know Relational databases are the main data sources for Big Data, and Hadoop is a framework which we use to analyze big data. So Sqoop is a tool which imports the data from Relational databases to Hadoop HDFS and also exports the data from Hadoop HDFS to Relational databases.
Relational databases can be MySQL, PostgreSQL ,Oracle and Redshift or any other RDBMS.
Sqoop uses MapReduce to import and export the data, which provides parallel operation as well as fault tolerance.
Sqoop is an open source software product of the Apache Software Foundation.

Prerequisites
Before we start with Sqoop following prerequisite knowledge is required to run Sqoop jobs:
·         Basic knowledge of linux operating system with commands
·         Concepts of Relational database management systems
·         Concepts of Hadoop and HDFS, with basic commands


Starting with Sqoop :-
Let’s start with very basic and important command which will tell you about all the available commands of Sqoop

$ sqoop help
Available commands:
  codegen                            Generate code to interact with database records
  create-hive-table            Import a table definition into Hive
  eval                                   Evaluate a SQL statement and display the results
  export                               Export an HDFS directory to a database table
  help                                   List available commands
  import                               Import a table from a database to HDFS
  import-all-tables             Import tables from a database to HDFS
  import-mainframe          Import datasets from a mainframe server to HDFS
  job                                     Work with saved jobs
  list-databases            List available databases on a server
  list-tables                    List available tables in a database
  merge                          Merge results of incremental imports
  metastore                  Run a standalone Sqoop metastore
  version                       Display version information


In next blog will see how to use all above Sqoop commands one by one.

Comments

Post a Comment

Popular posts from this blog

1. What is Big Data ?

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone. According to IBM, 80% of data captured today is unstructured, from sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals, to name a few. All of this unstructured data is Big Data. When dealing with larger data sets, organizations face difficulties in being able to create, manipulate, and manage big data. Big data is particularly a problem in business analytics because standard tools and procedures are not designed to search and anal
Read more

How to install Cloudera QuickStart VM on VMware - Part1?

Start using Hadoop with Cloudera's QuickStart VMs. The QuickStart VMs contain a single-node Apache Hadoop cluster, complete with example data, queries, scripts. These virtual machines make it easy to get started with CDH (Cloudera’s 100% open source Hadoop platform that includes Impala, Search, Spark, and more) and Cloudera Manager. They come complete with everything you need to learn Hadoop.The VMs run CentOS 6.4 and are available for VMware, VirtualBox, and KVM.   Requirements:- a) System Requirement:- T hese are a 64-bit VMs. They requires a 64-bit host OS and a virtualization product that can support a 64-bit guest OS.   To use a VMware VM, you must use a player compatible with WorkStation 8.x or higher: Player 4.x or higher, ESXi 5.x or higher, or Fusion 4.x or higher. Older versions of WorkStation can be used to create a new VM using the same virtual disk (VMDK file), but some features in VMware Tools won't be availa
Read more

Different flavours of Hadoop?

There are so many flavors of Hadoop available as Cloudera, Hortonworks etc. The nice thing is that you can download and try a free version. The base Apache distribution is good when you are just learning and getting started.   The real advantage of the bundled distributions such as Cloudera and Hortonworks is that you don't have to think about version x of Hive will work with Hadoop version y and version z of Hbase. They also come with better tools for deployment and management in an operational environment for example Cloudera comes with Cloudera Manager which provide you the web GUI for everything. There are three modes to start a Hadoop cluster. 1.       Local (Standalone) Mode 2.       Pseudo-Distributed Mode 3.       Fully-Distributed Mode In next post will start with Cloudera Quickstart VM and will see how to install in Virtual Machine. 
Read more