Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

3-2025

Abstract

In this paper, we present generalization bounds for the unsupervised risk in the Deep Contrastive Representation Learning framework, which employs deep neural networks as representation functions. We approach this problem from two angles. On the one hand, we derive a parameter-counting bound that scales with the overall size of the neural networks. On the other hand, we provide a norm-based bound that scales with the norms of neural networks’ weight matrices. Ignoring logarithmic factors, the bounds are independent of k, the size of the tuples provided for contrastive learning. To the best of our knowledge, this property is only shared by one other work, which employed a different proof strategy and suffers from very strong exponential dependence on the depth of the network which is due to a use of the peeling technique. Our results circumvent this by leveraging powerful results on covering numbers with respect to uniform norms over samples. In addition, we utilize loss augmentation techniques to further reduce the dependency on matrix norms and the implicit dependence on network depth. In fact, our techniques allow us to produce many bounds for the contrastive learning setting with similar architectural dependencies as in the study of the sample complexity of ordinary loss functions, thereby bridging the gap between the learning theories of contrastive learning and DNNs.

Discipline

Artificial Intelligence and Robotics

Research Areas

Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

Proceedings of the 39th AAAI conference on Artificial Intelligence, Philadelphia, Pennyslvania, 2025 February 25 - March 4

Volume

39

First Page

17186

Last Page

17194

Identifier

10.1609/aaai.v39i16.33889

Publisher

AAAI Press

City or Country

United States

Additional URL

https://doi.org/10.1609/aaai.v39i16.33889

Download

Share

COinS