Data Storage & Analysis in Hadoop

If you have more data that means you can comfortably beat better algorithms that work with fewer data as we give more accurate results. Big Data Big Data Big Data is here. What's the problem? The bad news is that we are struggling to store and analyze that.

The problem is simple: although the storage capacities of hard drives have increased

massively over the years, access speeds—the rate at which data can be read from drives have around five not kept up. One typical drive from 1990 could store 1,370 MB of data and had a

transfer speed of 4.4 MB/s so you could read all the data from a full around five

minutes. Over 20 years later, 1-terabyte drives are the norm, but the transfer speed

is around 100 MB/s, so it takes more than two and a half hours to read all the data off the disk.

The obvious way to reduce the time is to read from multiple disks at once. Imagine if we had

100 drives, each holding one hundredth of the data. Working in parallel, we could read

the data in under two minutes.

Using only one hundredth of a disk may seem wasteful. But we can store 100 data sets,

each of which is 1 terabyte, and provide shared access to them. We can imagine that the

users of such a system would be happy to share access in return for shorter analysis

times, and statistically, that their analysis jobs would be likely to be spread over time,

so they wouldn’t interfere with each other too much. There’s more to being able to

read and write data in parallel to or from multiple disks, though.

The first problem to solve is hardware failure: as soon as you start using many pieces of

hardware, the chance that one will fail is fairly high. A common way of avoiding data

loss is through replication: redundant copies of the data are kept by the system so that

in the event of failure, there is another copy available. This is how RAID works, for

instance, although Hadoop’s filesystem, the Hadoop Distributed Filesystem (HDFS),

takes a slightly different approach, as you shall see later. 

The second problem is that most analysis tasks need to be able to combine the data in someway 

, and data read from one disk may need to be combined with data from any of

the other 99 disks. Various distributed systems allow data to be combined from mul‐

tiple sources, but doing this correctly is notoriously challenging. MapReduce provides a

 programming model that abstracts the problem from disk reads and writes, trans‐

forming it into a computation over sets of keys and values. We look at the details of this

model in later chapters, but the important point for the present discussion is that there

are two parts to the computation—the map and the reduce—and it’s the interface be‐

tween the two where the “mixing” occurs. Like HDFS, MapReduce has built inreliability.

In a nutshell, this is what Hadoop provides: a reliable, scalable platform for storage and

analysis. What’s more, because it runs on commodity hardware and is open source,

Hadoop is affordable.

Brief History of Hadoop

 Hadoop is concocted by Doug Cutting while he and his friend Mike Cafarella were under development of  Nutch(open source web search engine). Before Hadoop invention, Google released a paper on Google File System(GFS) in 2003 and a paper on Map Reduce in 2004. These papers are best utilized by Doug and helped him in hand pioneering in their own Nutch Distributed File System(NDFS) and Map Reduce.Later on in 2006 he and Mike logged out of Nutch and framed the independent subproject of Lucene(Name of the entire project where all these subprojects like Nutch etc., stays in) called HADOOP. Hadoop name doesn't mean anything. It's just name of a toy(yellow baby elephant) that Doug's son used to play. Hadoop is maintained by Apache Software Foundation.

At around the same time, Doug joined Yahoo!, which provided a dedicated team and the resources to turn Hadoop into a system that ran at web scale. This was demonstrated in February 2008 when yahoo! announced that it's production search index was being generated by a 10,000-core Hadoop cluster.
Currently in 2016, still Yahoo! holds the biggest Hadoop cluster with 43,000 nodes in it.
Hadoop is currently playing a major role as general purpose storage and analysis platform for big data in many other companies besides Yahoo!, such as Microsoft, Last.FM, Facebook, and the New York Times.
Commercial Hadoop support is available from large, established enterprise vendors, including EMC, IBM, Microsoft, Oracle, as well as from specialist Hadoop companies such as Cloudera, Hortonworks, and MapR.
Doug Cutting is currently working in Cloudera.

Time Line of how things are progressed:

• 2004—Initial versions of what is now Hadoop Distributed Filesystem and MapReduce implemented by Doug Cutting and Mike Cafarella.
• December 2005—Nutch ported to the new framework. Hadoop runs reliably on 20 nodes.
• January 2006—Doug Cutting joins Yahoo!.
• February 2006—Apache Hadoop project officially started to support the standalone development of MapReduce and HDFS.
• February 2006—Adoption of Hadoop by Yahoo! Grid team.
• April 2006—Sort benchmark (10 GB/node) run on 188 nodes in 47.9 hours.
• May 2006—Yahoo! set up a Hadoop research cluster—300 nodes.
• May 2006—Sort benchmark run on 500 nodes in 42 hours (better hardware than April benchmark).
• October 2006—Research cluster reaches 600 nodes.
• December 2006—Sort benchmark run on 20 nodes in 1.8 hours, 100 nodes in 3.3 hours, 500 nodes in 5.2 hours, 900 nodes in 7.8 hours.
• January 2007—Research cluster reaches 900 nodes.
• April 2007—Research clusters—2 clusters of 1000 nodes.
• April 2008—Won the 1 terabyte sort benchmark in 209 seconds on 900 nodes.
• October 2008—Loading 10 terabytes of data per day on to research clusters.
• March 2009—17 clusters with a total of 24,000 nodes.
• April 2009—Won the minute sort by sorting 500 GB in 59 seconds (on 1,400 nodes) and the 100 terabyte sort in 173 minutes (on 3,400 nodes).

For more info on the history of Hadoop. click here.

Hadoop Solution for Big Data

The Hadoop platform was designed to solve problems where you have a lot of data — perhaps a mixture of unstructured and structured data — and it doesn’t fit nicely into tables. It’s for situations where you want to run analytics that is deep and computationally extensive, like clustering and targeting. 

Hadoop applies to a bunch of markets. In finance, if you want to do accurate portfolio evaluation and risk analysis, you can build sophisticated models that are hard to jam into a database engine. But Hadoop can handle it. In online retail, if you want to deliver better search answers to your customers so they’re more likely to buy the thing you show them, that sort of problem is well addressed by the platform Google built.

Generally, in vast companies, if you want to maintain a data center, it needs a very expensive hardware machines and network system in order to be less failure effective. What if you have a small company(basically a start-up) you cannot go for an extortionate expensive hardware. Because the cost of Data machines itself implies your entire start-up economy. Since we are dealing with the big data thing, we need high space machines and we want data storage to be installed in very less valuation. What's the solution? Simple. Use Hadoop! in less expensive machines instead of going for more priced ones. 

Hadoop is designed to run on a large number of machines that don’t share any memory or disks. That means you can buy a whole bunch of commodity(less cost) servers, slap them in a rack, and run the Hadoop software on each one. When you want to load all of your organization’s data into Hadoop, what the software does is the bust that data into pieces that it then spreads across your different servers. There’s no one place where you go to talk to all of your data; Hadoop keeps track of where the data resides. And because there are multiple copy stores, data stored on a server that goes offline or dies can be automatically replicated from a known good copy. We store data in multiple systems because as we are using commodity hardware, we should be in a safe mode in order to protect our data.

In a centralized database system, you’ve got one big disk connected to four or eight or 16 big processors. But that is as much horsepower as you can bring to bear. In a Hadoop cluster(distributed system), every one of those servers has two or four or eight CPUs. You can run your indexing job by sending your code to each of the dozens of servers in your cluster, and each server operates on its own little piece of the data. Results are then delivered back to you in a unified whole. That’s MapReduce: you map the operation out to all of those servers and then you reduce the results back into a single result set. Architecturally, the reason you’re able to deal with lots of data is because Hadoop spreads it out.

In a more detailed way, We, in general, write the code, bring the data into the code by performing some file operations, execute them and send the result back. But in Hadoop architecture, we write the code, and we send the code to the place where data is stored in distributed systems. This is the main advantage in Hadoop systems. How? How it is done..I'll post in further posts. Please stay tuned.

Is only Hadoop available in the market to deal with the big data?

The answer to this question is NO. There are many alternative frameworks that are available in the current world to maintain the big data. Some of them are Cloudera, Horton Works, MapR, Spark, Octopy, Sphere etc., We cannot compare each and every vendor since each got some special feature which is not present in the another one. 

Advantages of Hadoop:

1. Scalable

• Hadoop is a highly scalable storage platform because it can stores and distribute very large data sets across hundreds of inexpensive servers that operate in parallel. Unlike traditional relational database systems (RDBMS) that can’t scale to process large amounts of data, Hadoop enables businesses to run applications on thousands of nodes involving many thousands of terabytes of data.

2. Cost effective

• Hadoop also offers a cost effective storage solution for businesses’ exploding data sets. The problem with traditional relational database management systems is that it is extremely cost prohibitive to scale to such a degree in order to process such massive volumes of data. In an effort to reduce costs, many companies in the past would have had to down-sample data and classify it based on certain assumptions as to which data was the most valuable. The raw data would be deleted, as it would be too cost-prohibitive to keep. While this approach may have worked in the short term, this meant that when business priorities changed, the complete raw data set was not available, as it was too expensive to store.

3. Flexible

• Hadoop enables businesses to easily access new data sources and tap into different types of data (both structured and unstructured) to generate value from that data. This means businesses can use Hadoop to derive valuable business insights from data sources such as social media, email conversations.  Hadoop can be used for a wide variety of purposes, such as log processing, recommendation systems, data warehousing, market campaign analysis and fraud detection.

4. Fast

• Hadoop’s unique storage method is based on a distributed file system that basically ‘maps’ data wherever it is located in a cluster. The tools for data processing are often on the same servers where the data is located, resulting in much faster data processing. If you’re dealing with large volumes of unstructured data, Hadoop is able to efficiently process terabytes of data in just minutes, and petabytes in hours.

5. Resilient to failure

• A key advantage of using Hadoop is its fault tolerance. When data is sent to an individual node, that data is also replicated to other nodes in the cluster, which means that in the event of failure, there is another copy available for use.

Disadvantages of Hadoop:

As the backbone of so many implementations, Hadoop is almost synonymous with big data.

1. Security Concerns

Just managing complex applications such as Hadoop can be challenging. A simple example can be seen in the Hadoop security model, which is disabled by default due to sheer complexity. If whoever managing the platform lacks of know how to enable it, your data could be at huge risk. Hadoop is also missing encryption at the storage and network levels, which is a major selling point for government agencies and others that prefer to keep their data under wraps.

2. Vulnerable By Nature

Speaking of security, the very makeup of Hadoop makes running it a risky proposition. The framework is written almost entirely in Java, one of the most widely used yet controversial programming languages in existence. Java has been heavily exploited by cyber criminals and as a result, implicated in numerous security breaches.

3. Not Fit for Small Data

While big data is not exclusively made for big businesses, not all big data platforms are suited for small data needs. Unfortunately, Hadoop happens to be one of them. Due to its high capacity design, the Hadoop Distributed File System lacks the ability to efficiently support the random reading of small files. As a result, it is not recommended for organizations with small quantities of data.

4. Potential Stability Issues

Like all open source software, Hadoop has had its fair share of stability issues. To avoid these issues, organizations are strongly recommended to make sure they are running the latest stable version, or run it under a third-party vendor equipped to handle such problems.

5. General Limitations

The article introduces Apache Flume, MillWheel, and Google’s own Cloud Dataflow as possible solutions. What each of these platforms has in common is the ability to improve the efficiency and reliability of data collection, aggregation, and integration. The main point the article stresses is that companies could be missing out on big benefits by using Hadoop alone.

Stay tuned for more posts.Thank You for reading and spending your time here.

Big Data Challenges

Challenges tangled in Big Data:

Generally, many organizations generate the enormous amount of data each day. How can they handle the data for future plans? What is the use of that data? How can they use that data in decision making? and there are many challenges arise while dealing with the precipitated data.

Here are some of the challenges that generally many companies fight with:

1. Using data to generate business value is already a reality in many industries. With the advent of the “Internet of things,” improved analytics and increased connectivity through new technology and software bring significant opportunities for companies. First, companies have difficulty identifying the right data and determining how to best use it. Building data-related business cases often mean thinking outside of the box and looking for revenue models that are very different from the traditional business.
2. Second, companies are struggling to find the right talent capable of both working with new technologies and of interpreting the data to find meaningful business insights.
3. Third, data access and connectivity can be an obstacle. A majority of data points are not yet connected today, and companies often do not have the right platforms to aggregate and manage the data across the enterpriseThird, data access, and connectivity can be an obstacle. A majority of data points are not yet connected today, and companies often do not have the right platforms to aggregate and manage the data across the enterprise.
4. Fourth, the technology landscape in the data world is evolving extremely fast. Leveraging data means working with a strong and innovative technology partner that can help create the right IT architecture that can adapt to changes in the landscape in an efficient manner.
5. Fifth, leveraging big data often means working across functions like IT, engineering, finance and procurement and the ownership of data is fragmented across the organization. To address these organizational challenges means finding new ways of collaborating across functions and businesses.
6. Finally, security concerns about data protection are a major obstacle preventing companies from taking full advantage of their data.

For some more challenges that big data faces click this link.

Hence, we need to find some solution to cope with this Huge Enormous Heavy Big Data for analyzing purposes.

Data and Big Data

What is Data?

     Data means nothing but information in raw or unorganized format. It is present everywhere in the world. The amount of data that's been created and stored on a global level is almost inconceivable and it just keeps growing every second. 90% of world's data today has been created in the last 2 years alone. In general, data is of worthless since it is unorganized. But, Information is more potential and valuable and is based on the concepts of data. There are generally 3 types of data: -

• Structured Data
• Unstructured Data
• Semi-Structured Data

Why is data important?

Suppose you want to buy one from below two cars. Which one do you buy?

  

What do you mean you can't make that decision? Are you thinking to yourself you need more information? You probably do if you want to make a good decision.

So, what is it you need? You need data! This is true of business decisions as well - more data generally means less risk in decision making. The better the data, the lower the risk. Making a decision without data increases the risk that the decision will be faulty. More data means more accuracy.

So you know you need lots of data, but what does data look like? You're probably thinking of lots of numbers and maybe some charts and graphs. But data comes in many forms and is often categorized as structured and unstructured. Structured data generally consists of numerical information and is objective. Unstructured data is more subjective and is usually text-heavy. It can't generally be put into a data structure like columns or rows. based on around 80% of business decisions are relied on Unstructured data and only 20% of Structured data. The below two pictures demonstrate the structured data and unstructured data.

 
Semi-structured data is data that has not been organized into a specialized repository, such as a database, but that nevertheless has associated information, such as metadata, that makes it more amenable to processing than raw data.
Examples of unstructured data:  RAW files, Videos, Images, Audio files, Social Media etc.,
Examples of semi-structured data: JSON files, XML files etc.,

Big data

Big data is a term for datasets that are so large or complex that traditional data processing applications are inadequate to deal with them. The data can be of any voluminous amount of Structured data, Unstructured data and Semi-Structured data that has the potential to be mined for information. Actually, any data i.e., beyond the storage and processing capacity of a single centralized or physical machine or server is considered as Big Data.

In other words, Big Data describes the collection of complex and large datasets such that it is difficult to capture, process, store, search and analyze using conventional database systems. Big Data doesn't revolve around how much data you have, but what you do with it and the processing and analysis of it and the insights, products, and services that emerge from the analysis.

You can take data from any source and analyze it to find answers that enable 1) cost reductions, 2) time reductions, 3) new product development and optimized offerings, and 4) smart decision making. When you combine big data with high-powered analytics, you can accomplish business-related tasks like 

• Determining root causes of failures, issues and defects in near-real time.

• Generating coupons at the point of sale based on the customer’s buying habits.

• Recalculating entire risk portfolios in minutes.

• Detecting fraudulent behavior before it affects your organization.

Big Data is characterized of 7Vs but in general, it mainly characterized of 3Vs:

• Volume: The amount of generated and stored data.
• Variety: The type and nature of the data. Can be structured & unstructured data, text data, sensor data, log files etc.,
• Velocity: Is the speed at which the data is generated and the speed at which it is processed, stored and analyzed.

The remaining Vs are

• Variability: Variability is different from variety. The same is true of data if the meaning is constantly changing it can have a huge impact on your data homogenization. For example, A coffee shop may offer 6 different blends of coffee, but if you get the same blend every day and it tastes different every day, that is variability.
• Veracity: Veracity is all about making sure the data is accurate, which requires processes to keep the bad data from accumulating in your systems. 
• Visualization: Visualization is critical in today’s world. Using charts and graphs to visualize large amounts of complex data is much more effective in conveying meaning than spreadsheets and reports chock-full of numbers and formulas.
• Value: Value is the end game. After addressing volume, velocity, variety, variability, veracity, and visualization – which takes a lot of time, effort, and resources – you want to be sure your organization is getting value from the data.

Big data in action:

Banking:

With large amounts of information streaming in from countless sources, banks are faced with finding new and innovative ways to manage big data. While it’s important to understand customers and boost their satisfaction, it’s equally important to minimize risk and fraud while maintaining regulatory compliance. Big data brings big insights, but it also requires financial institutions to stay one step ahead of the game with advanced analytics.

Education:

Educators armed with data-driven insight can make a significant impact on school systems, students, and curriculums. By analyzing big data, they can identify at-risk students, make sure students are making adequate progress, and can implement a better system for evaluation and support of teachers and principals.

Government:

When government agencies are able to harness and apply analytics to their big data, they gain significant ground when it comes to managing utilities, running agencies, dealing with traffic congestion or preventing crime. But while there are many advantages to big data, governments must also address issues of transparency and privacy.

Health Care:

Patient records. Treatment plans. Prescription information. When it comes to health care, everything needs to be done quickly, accurately – and, in some cases, with enough transparency to satisfy stringent industry regulations. When big data is managed effectively, health care providers can uncover hidden insights that improve patient care.

Manufacturing:

Armed with insight that big data can provide, manufacturers can boost quality and output while minimizing waste – processes that are key in today’s highly competitive market. More and more manufacturers are working in an analytics-based culture, which means they can solve problems faster and make more agile business decisions.

Retail:

Customer relationship building is critical to the retail industry – and the best way to manage that is to manage big data. Retailers need to know the best way to market to customers, the most effective way to handle transactions, and the most strategic way to bring back lapsed business. Big data remains at the heart of all those things.