You typically go from problem-to-problem in applied machine learning and you require to get up to speed on a fresh dataset, quickly. 

A traditional and under-utilized strategy that you can leverage to swiftly develop a relationship with a new data problem is Exploratory Data Analysis. 

In this blog article by AICorespot, you will find out about Exploratory Data Analysis (EDA), the strategies and techniques that you can leverage and why you should be performing EDA on your subsequent problem. 

Build a relationship with the Data 

The procedure of traditional statistics is to evaluate hypothesis already held about the problem. 

This is performed by fitting particular models and illustrating particular relationships in the information. It’s an efficient strategy, however, it assumes you already possess hypotheses about the problem, that you already comprehend the data. This is uncommonly the scenario within applied machine learning. 

Prior to modelling the data and evaluating your hypothesis, you require to develop a relationship with the data. You can develop this relationship by spending time summarizing, plotting, and reviewing actual real information from the field. 

This approach of analysis prior to modelling is referred to as Exploratory Data Analysis. 

In using time with the data up-front you can develop an intuition with the data formats, values, and relationships that can assist to explain observations and modelling results later on. 

It is referred to as exploratory data analysis as you are looking into your comprehension of the data, developing an intuition for how the underlying procedure that produced it functions and provoking questions and ideas that you can leverage as the foundation for your modelling. 

The procedure can be leveraged to sanity check the data, to detect outliers and come up with particular techniques for managing them. In spending time with the information, you can identify corruption in the values that might indicate a fault in the data logging procedure. 

Origin of Exploratory Data Analysis 

Exploratory Data Analysis was created by John Tukey at Bell Labs as a way of methodically leveraging the tools of statistics on a problem prior to a hypotheses about the data being formulated. It is an alternative or opposite approach to “confirmatory data analysis” 

The seminal description of the procedure was in Tukey’s 1977 book Exploratory Data Analysis. 

The goal is to comprehend the issue in order to produce testable hypothesis. As such, the results like the graphs and summary statistics are just for you to enhance and complement your comprehension, not to demonstrate a relationship in the data to a general audience. This provides the agile flavour to the procedure. 

The S language was generated in the same laboratory and was leveraged as the tool for EDA. The leveraging of scripts to produce data summaries and perspectives is a natural and intentional fit for the procedure. 

Wikipedia furnishes a nice short listing of the goals of EDA. 

• Suggest hypotheses about the reasons behind unobserved phenomena 
• Evaluate assumptions on which statistical inference will be based 
• Assist the selection of relevant statistical tools and strategies 
• Furnish a basis for subsequent data collection through surveys or experiments 

Strategies for Exploratory Data Analysis 

Exploratory data analysis is typically carried out with a representative sample of the data. You do not require to leverage all information available nor big data infrastructure. 

Spend time with the raw data. 

Beginning with eyeballing tables of numbers is a smart move. Skimming through tables can swiftly highlight the form of every data attribute, obvious perversions and big outliers in the values and begin to suggest candidate relationships to explore amongst attributes. Take notes. 

Simple univariate and multivariate strategies that provide you a perspective on the data can be leveraged. 

For instance, five strategies that we would consider as must have are: 

• Five number summaries (mean/median, min, max, q1, q3) 
• Histogram graphs 
• Line Charts 
• Box and whisker plots 
• Pairwise scatterplots (scatterplots matrices) 

On top of summaries, also observe transforms of the data and re-scalings of the data. Flush out faConcscinating structures that you can detail. 

Ensure to jot down a ton of notes. 

Ask lots of queries of the data, for instance: 

• What values do you observe? 
• What distributions do you observe? 
• What relationships do you observe? 
• What relationships do you think might provide advantages to the forecasting problem? 
• What ideas about the domain does the data spark? 

Concentrate on understanding 

You are not developing a report, you are attempting to comprehend the problem. 

The outcomes are ultimately throw-away, and all that you ought to be left with is a greater comprehension and intuition for the data and a long listing of hypotheses to look into during modelling. 

The code does not require to be beautiful – but they are required to be precise. Leverage reproducible scripts and standard packages. 

You are not required to delve into sophisticated statistical strategies or plots. Keep it simple and spend time with the data. 

A query interface like SQL can assist you simulate a lot of what-if situations very swiftly with a sampling of your data. 

The models will just be as good as the questions and comprehension you have of the data and the problem. 

Resources 

The book Doing Data Science: Straight Talk from the Frontline has a brief section on EDA and furnishes an amazing reading list for additional data: 

• Exploratory data analysis 
• The Visual display of Quantitative Information (very recommended) 
• The elements of graphing data 
• Statistical graphics for visualizing multivariate data 

Try out exploratory data analysis on your present or subsequent project. 

If you already perform it, attempt some strategies you have not leveraged prior or attempt to be systematic, even sketch out a checklist of stuff to look at to cover your bases as a first pass on the data.