You often come across duplicate entries when you’re analyzing data. If you don’t locate them in time, your analysis is in danger of ending up with a flawed conclusion.
There are two ways to look for duplicate data.
Method 1. You’re already familiar with the duplicated() method. When we combine it with sum(), it returns the number of duplicates. If you execute the duplicated() method without summation, all rows will be displayed on the screen. We’ll get a True value where there’s a duplicate, a False value where there isn’t.
Method 2. Call the value_counts() method*,* which returns unique values and their frequency. We apply it to Series objects. The method gets us a list of value–frequency pairs sorted in descending order, which means the duplicate entries we’re looking for are at the beginning.
Duplicates in string data demand particular attention given that the case plays a role: from Python’s point of view, a capital 'A' and a lowercase 'a' are different symbols, though they both have the same value of A.
To catch duplicate entries like those, we change all the characters in the string to lowercase by calling the lower() method**.**
Over in pandas, we change characters to lowercase with a method that follows a similar syntax: str.lower().
When you’re categorizing strings, checking to make sure particular substrings are included in their strings doesn’t always give you a correct result. One solid option for handling that is stemming.
Stemming is the process of finding the stem of a given word.
There is a Python library called NLTK that is used for stemming. It contains stemmers, or objects with the rules for identifying stems. The stem() method is used to find the stem of each word in a string.
from nltk.stem import SnowballStemmer
russian_stemmer = SnowballStemmer('russian')
russian_stemmer.stem(text)
Stemming isn’t the only algorithm used to find words written in different forms. Lemmatization is a more advanced process that returns words back to their dictionary form (lemma).
Here’s how lemmas are formed in English:
—For nouns, it’s the singular.
—For adjectives, it’s the corresponding positive form.