Chapter Summary: “First Graphs and Conclusions”

Getting to Know the Job

Data in the CSV format can be separated by commas, semicolons, tab characters, or other symbols. Even if decimals are written with commas they can give us problems.

The parameters of the read_csv() function indicate which symbols you’re going to use in order to break up decimals and columns. Column separators are given by the sep parameter, while decimal separators are given by the decimal parameter:

file = pd.read_csv('file.csv',  sep=';' , decimal=',')

Using pivot tables for calculating the mean

When you were preprocessing the data, you used pivot_table(), a method for building pivot tables. You previously set the value of aggfunc to sum, summing up all the elements in the column. However, if you don’t indicate an aggfunc parameter, then the pivot_table() method will by default calculate the arithmetic mean of the values listed in the values parameter.

Essential data verification

There are always surprises in store for you when you’re working with data:

—Incomplete or incorrect data was accidentally uploaded or because the task was misinterpreted.

—There were mistakes in algorithms used to calculate the value you needed.

—The data's in the wrong format.

—Some important facts were missed.

It could pretty much be anything, and as an analyst, you’re solely responsible for how accurate the data is. So you should always be mindful of whether or not it's trustworthy. Start by performing a basic check, something like answering questions about the data and then working on your own or together with other coworkers to see whether the answers match up to what you assume the truth to be.

That basic check might reveal a problem within the data. Then again, it might also tell you that everything’s okay, as for now.

Histograms

Histograms are graphs that show how frequently values appear in a particular data set. They combine numerical values inside a range, calculating how many times values appear within each specified interval. The way histograms are structured is something similar to the value_counts() method you're already familiar with, the one we used to calculate the number of unique values in a list. But value_counts() **is only great for calculating frequencies in lists of categorical variables since it just groups purely identical values.

In pandas, we make histograms using a special hist() method, which is applied to a list or DataFrame column. The hist() method scans through sets of numbers in order to locate the minimum and maximum values, then divides that range into sections, or so-called bins. It then calculates how many values fall into each bin, depicting them in a graph. The bins parameter determines how many sections a range of data will be divided into, with a default being bins=10. The histogram then defaults to displaying all the values, from low to high. You can also manually set the scale by using the range parameter: range=(min_value, max_value).

In order to create graphs, including histograms, import the matplotlib library and the show() **method will display the graphs.