What is Data Poisoning?

Data poisoning attacks aim to damage the target model during training or retraining, which often occurs during the lifespan of a machine learning model.

FREMONT, CA: While machine learning systems may be vulnerable to typical security risks at the hardware, application, and network levels, they may also be susceptible to domain-specific attacks that are currently unnoticed. One of the most significant risks they face is data poisoning. By introducing erroneous data into the training set, data poisoning attacks jeopardize the integrity of machine learning models. Let's understand the concept behind data poisoning and what's data poisoning attack?

Data poisoning attack

The quality and quantity of data used to train a machine learning model significantly impact its performance. A considerable quantity of training data is frequently necessary to prepare an appropriate machine learning model. In order to get adequate training data, practitioners may turn to potentially untrustworthy sources. The decrease in data quality, especially if the data didn't get systematically checked to verify the correctness of its labels, opens the door to data poisoning attacks, in which intentionally incorrectly labeled data gets inserted into a model's training set to compromise the model's accuracy.

A data poisoning attack tries to change a training set such that the model trained with it makes erroneous predictions. Data poisoning attacks aim to damage the target model during training or retraining, which often occurs during the lifespan of a machine learning model. Data poisoning attacks have a long-term impact since they weaken the model's integrity and lead it to produce persistent mistakes while generating predictions at runtime. After a model has got poisoned, recovering from the attack later is a difficult task.

Data poisoning attacks may get carried out against nearly any machine learning model that uses third-party data for training. The sensitivity of machine learning models to data poisoning attacks varies depending on a variety of factors. The capacity of the target model is one key characteristic that influences this vulnerability: simple models with limited capacity are more prone to denial-of-service poisoning attacks, whereas sophisticated models are more vulnerable to backdoor poisoning assaults. It means that there is no one-size-fits-all solution for protecting models against data poisoning attacks by design. To effectively minimize data poisoning assaults, an extra line of protection must get provided throughout the model's training phase.

Model poisoning attacks, in addition to data poisoning attacks, may be carried out in distributed training settings such as federated learning setups.