The risk to one’s privacy should not substantially increase as a result of participating in a statistical database.

The Privacy Problems in Training Data

In recent years, numerous incidents have highlighted the vulnerability of private and sensitive personal information in training data. Model inversion attacks have demonstrated the capability to infer private information about individuals within training datasets. For example, adversaries can extract large amounts of (potentially private) information from production LLMs. Consequently, companies need to be very cautious regarding Personally Identifiable Information (PII) included in training or fine-tuning datasets. These adversarial attacks necessitate treating open-source models or models with black-box access as if they were data-in-storage to mitigate privacy risks effectively.

What is Differential Privacy?

Differential privacy is a technique designed to safeguard the privacy of individuals within a dataset. Introduced in the 2006 paper “Differential Privacy” by Cynthia Dwork (Microsoft Research), it operates by injecting noise into the data or statistical computations. This approach ensures that the overall utility of the data is maintained while limiting the amount of sensitive information that can be inferred about any individual.

Differential privacy works by adding carefully calibrated noise to the outputs of queries made to a database. The noise is typically drawn from a Laplace distribution and is scaled according to a parameter known as the privacy budget (often denoted by ϵ​). The privacy budget quantifies the level of privacy guaranteed: a smaller ϵ provides stronger privacy but may reduce the accuracy of the query results. This ensures that the presence or absence of any single individual's data in the dataset has a minimal impact on the output, thus protecting individual privacy.

Methods for Applying Differential Privacy in ML

1. Differential-private Synthetic Data

Differential privacy can also play a critical role in generating synthetic data. This method involves creating artificial datasets that mimic the statistical properties of the original data without compromising individual privacy. By incorporating differential privacy in the synthetic data generation process, organisations can ensure that the synthetic data is useful for training models while safeguarding sensitive information.

2. Differential Privacy in Stochastic Gradient Descent (DP-SGD)

Differential privacy can be incorporated directly into the model training process, particularly through stochastic gradient descent (SGD). This techniques involves adding noise to the gradients during the training process, based on a specified privacy budget, to ensure that the model does not memorise specific data points. Popular machine learning frameworks, such as TensorFlow and PyTorch, offer modules like TensorFlow Privacy and Opacus that support DP-SGD. However, this method can impact training time and model performance, making it a trade-off between privacy and efficiency.

Conclusion

Differential privacy provides a robust framework for enhancing data privacy in machine learning. By injecting noise into data or statistical processes, differential privacy ensures that sensitive information remains protected while preserving the utility of the data. Implementing differential privacy in training data, generating synthetic data, and integrating it into the training process through methods like differentially-private SGD are crucial strategies for mitigating privacy risks.

—-

Photo by Google DeepMind from Pexels.