To clean up dirty data, you need to use different methods for each dataset, but you need to do this in a systematic way. You will want to keep as much of your data as possible while also making sure you have a clean dataset at the end.
Cleaning up data is hard because mistakes are hard to find after you collect all the data. You won’t always be able to tell if a data point shows the real value of something precisely and accurately.
In practise, you may instead focus on finding and fixing data points that don’t fit with the rest of your dataset in more obvious ways. These numbers could be missing values, outliers, formatted wrong, or not important.
You can choose from a few ways to clean data based on what works best. You want to end up with a set of data that is valid, consistent, unique, uniform, and as full as possible.
Workflow to clean data.
Most of the time, you start cleaning up your data by looking at it in a big picture way. You look at problems and figure out what’s wrong in a systematic way. Then, you change each thing based on standard procedures. This could be how your work flow looks:
Use techniques for data validation to stop people from entering bad data.
Check your dataset for mistakes or things that don’t make sense.
Diagnose your data entries.
Make codes that translate your data into valid values.
Change your data or get rid of it based on standard procedures.
Not every dataset will need to do all of these steps. You can carefully use data cleaning techniques where they are needed, and make sure that your processes are well-documented for transparency.
By writing down your workflow, you make sure that other people can look at it and follow your steps.
Data validation.
Data validation is the process of putting limits on data to make sure it is correct and consistent. It is usually used before you even start collecting data, when making questionnaires or other measuring tools that need to be entered by hand.
Using different data validation constraints can help you clean up your data as little as possible.
Data-type constraints: Only certain types of values, like numbers or text, can be accepted.