For cleansing the data after you’ve given a general overview, you can get down to the details of your dataset. You’ll need to come up with a standard way to find and handle different kinds of data. In this blog, we will see how data diagnosing can be done for a particular category of data set.
If you don’t plan well, you might end up cleaning only some of the data points, which would give you a skewed set of data.
Here, we’ll talk about how to fix some of the most common problems with dirty data:
- Duplicate data
- Invalid data
- Missing values
- Outliers
De-duplication Diagnosing
De-duplication means finding and getting rid of any copies of data that are the same, so that your dataset only has unique cases or participants. In this type of data diagnosing you see for the same type of data and get rid of them.
If you don’t get rid of duplicates from your dataset, they will change your results. Some people’s answers will be given more weight than others’.
Invalid data
Using data standardization, you can find data in different formats and change it to a single format.
Standardization is different from data validation in that it can be used on data that has already been collected. This means making codes to turn your dirty data into formats that are consistent and valid.
Data standardization is helpful if you don’t have any rules about how to enter data or if the formats of your data are different.
Methods for matching strings.
You can use strict or fuzzy string-matching methods to find exact or close matches between your data and valid values. This will make your data more consistent.
A string is a group of characters in a row. You compare your data strings to the expected valid values and then get rid of or change the strings that don’t match.
Strict string matching means that strings that don’t match the valid values exactly aren’t valid.
Fuzzy string matching means that strings that are close to or almost the same as valid values are found and fixed.
After matching, you can change your text data into numbers so that the format of all the values is the same.
Most of the time, fuzzy string-matching is better than strict string-matching because more data are kept.
Missing data.
In any set of data, there are usually some pieces that are missing. In your spreadsheet, these cells are empty.
Missing data can be caused by random or planned events. This is what all you do in data diagnosing.
Randomly missing data can be caused by mistakes when entering data, not paying attention, or misreading measurements.
Measurements or questions that are hard to understand, poorly made, or not right for the job can cause missing data that is not random.
dealing with data that isn’t there.
Usually, you can do the following to deal with missing data:
- Taking the information as it is.
- Taking the case out of the analysis.
- Making up the missing information.
- Most of the time, random missing data are left alone, but missing data that isn’t random may need to be removed or replaced.
With deletion, you take people out of your analysis who are missing data. But your sample may get smaller than you planned, which means you may lose statistical power. This is how diagnosing can help you.
You can also use imputation to fill in a missing value with another value that is based on a good guess. For a more complete set of data, you use other data to fill in the missing value.
Imputation should be used with care because there is a chance of bias or wrong results.
Outliers
These are values that are very different from the rest of the data in a set. Outliers can be true values or errors.
True outliers should always be kept because they just show that your sample has natural differences. For example, people who train for the 100-meter Olympic sprint are much faster than most other people. Their sprint times stand out from the rest.
Outliers can also happen because of measurement errors, mistakes in entering data, or sampling that doesn’t represent the whole. For example, if you misread the timer, you might record a very slow sprint time.
Finding the odd ones.
Outliers are always at the extreme ends of a set of data that has many variables.
There are several ways to find outliers:
- Sorting your values from lowest to highest and making sure the minimum and maximum values are correct.
- Use a boxplot to see your data and look for “outliers.”
- Using statistical methods to find values that are very high or very low.
Taking care of outliers.
Once you’ve found outliers, you’ll decide what to do with them in your dataset. You can either keep them or get rid of them.
In general, you should try to accept outliers as much as possible unless it’s clear that they are errors or bad data.
It’s important to write down what you did to get rid of each outlier and why, so that other researchers can follow your steps.