Towards Standardized Evaluation in Differentially Private Image Classification: A Critical Approach

Abstract

This thesis critically examines differentially private machine learning (DPML) in image classification, addressing recent critiques about the effectiveness of current techniques and the validity of existing benchmarks. We focus on three key questions: the accuracy of current benchmarks in measuring DPML progress, the impact of public pre-training datasets on DPML model performance, and strategies for unifying future research efforts.

Our study introduces standardized benchmark datasets and evaluation settings using two medical image datasets as private data sources. We assess various DPML methods across different scenarios, including those with no public data, training from scratch, and fine-tuning approaches.

The main contributions include: proposing standardized benchmark datasets and evaluation settings, conducting a validation study of previous DPML techniques, and introducing a moderated public leaderboard to track progress in DPML. This research aims to provide a comprehensive assessment of DPML in image classification, offering insights into existing methods and suggesting future research directions in machine learning and privacy.