2. What is Hadoop ?


What is Hadoop?
Apache Hadoop is an open-source software framework written in Java for distributed storage and processing of large data sets across clusters of commodity hardware using simple programming models. Hadoop is designed to scale up from single servers to thousands of machines, each offering local computation and storage. 
Apache Hadoop has two core parts one is Hadoop Distributed File System (HDFS- For - Storage) and second is MapReduce (For - Processing).
Hadoop splits files into large blocks and distributes them amongst the nodes in the cluster. To process the data, Hadoop MapReduce transfers packaged code for nodes to process in parallel, based on the data each node needs to process.
Hadoop is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
The current Apache Hadoop ecosystem consists of the Hadoop kernel, MapReduce, the Hadoop distributed file system (HDFS) and a number of related projects such as Apache Hive, HBase and Zookeeper.

 Hadoop was inspired by Google's MapReduce, a software framework in which an application is broken down into numerous small parts. Any of these parts (also called fragments or blocks) can be run on any node in the cluster. Doug Cutting, Hadoop's creator, named the framework after his child's stuffed toy elephant.

The Hadoop framework is used by major players including Google, Yahooand IBM, largely for applications involving search engines and advertising. 

The project includes these modules:

        Hadoop Distributed File System (HDFS): A distributed file system.
     Hadoop : A framework for job scheduling and cluster resource management.
      MapReduce: A  parallel processing system for large data sets.

Other Hadoop-related projects at Apache include:

     Avro: A data serialization system.
     Cassandra: A scalable multi-master database with no single points of failure.
     Chukwa: A data collection system for managing large distributed systems.
     HBASE: A scalable, distributed database that supports structured data storage for large tables.
     Hive: A data warehouse infrastructure that provides data summarization and ad hoc querying.
     Mahout: A Scalable machine learning and data mining library.
     Sqoop: A tools which import the data from Relational Databases to HDFS or Hive and also 
                    export data from HDFS or Hive to Relational databases like MySQL and Oracle.
       Pig: A high-level data-flow language and execution framework for parallel computation.
     Spark: A fast and general compute engine for Hadoop data. Spark provides a simple and 
                  expressive programming model that supports a wide range of applications, including ETL,
                  machine learning, stream processing, and graph computation.
     ZooKeeper: A high-performance coordination service for distributed applications.



Comments

Post a Comment

Popular posts from this blog

1. What is Big Data ?

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone. According to IBM, 80% of data captured today is unstructured, from sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals, to name a few. All of this unstructured data is Big Data. When dealing with larger data sets, organizations face difficulties in being able to create, manipulate, and manage big data. Big data is particularly a problem in business analytics because standard tools and procedures are not designed to search and anal
Read more

How to install Cloudera QuickStart VM on VMware - Part1?

Start using Hadoop with Cloudera's QuickStart VMs. The QuickStart VMs contain a single-node Apache Hadoop cluster, complete with example data, queries, scripts. These virtual machines make it easy to get started with CDH (Cloudera’s 100% open source Hadoop platform that includes Impala, Search, Spark, and more) and Cloudera Manager. They come complete with everything you need to learn Hadoop.The VMs run CentOS 6.4 and are available for VMware, VirtualBox, and KVM.   Requirements:- a) System Requirement:- T hese are a 64-bit VMs. They requires a 64-bit host OS and a virtualization product that can support a 64-bit guest OS.   To use a VMware VM, you must use a player compatible with WorkStation 8.x or higher: Player 4.x or higher, ESXi 5.x or higher, or Fusion 4.x or higher. Older versions of WorkStation can be used to create a new VM using the same virtual disk (VMDK file), but some features in VMware Tools won't be availa
Read more

Different flavours of Hadoop?

There are so many flavors of Hadoop available as Cloudera, Hortonworks etc. The nice thing is that you can download and try a free version. The base Apache distribution is good when you are just learning and getting started.   The real advantage of the bundled distributions such as Cloudera and Hortonworks is that you don't have to think about version x of Hive will work with Hadoop version y and version z of Hbase. They also come with better tools for deployment and management in an operational environment for example Cloudera comes with Cloudera Manager which provide you the web GUI for everything. There are three modes to start a Hadoop cluster. 1.       Local (Standalone) Mode 2.       Pseudo-Distributed Mode 3.       Fully-Distributed Mode In next post will start with Cloudera Quickstart VM and will see how to install in Virtual Machine. 
Read more