Jae Myung Kim

I am a PhD student at the University of Tübingen and a member of ELLIS & IMPRS-IS programs advised by Zeynep Akata (TU Munich, Helmholtz Munich) and Cordelia Schmid (Google, Inria). Previously, I completed my B.S. and M.S. degrees at the Seoul National University.

I am broadly interested in efficient data-centric approaches. At the moment I'm working on the following questions:

1. Synthetic data as training data: Exploring how generative models can be leveraged as a robust data source for pertaining or downstream tasks.
2. Weak alignment: Multi-modal data are normally weakly aligned. How can we have better representations with weak data alignment?
3. Zero-shot and few-shot learning: How can we better leverage foundation models with access to minimum data?

Email  /  Google Scholar  /  LinkedIn

Selected papers

DataDream: Few-shot Guided Dataset Generation

We generate a synthetic dataset guided by few-shot real samples, which more faithfully represents the real data distribution of the targeted classification task.

Improving Intervention Efficacy via Concept Realignment in Concept Bottleneck Models

We learn concept relations to realign concept assignments post-intervention in CBMs. This effectively reduces the number of necessary interventions to reach a target performance.

Waffling around for Performance: Visual Classification with Random Words and Broad Concepts

We achieve comparable zero-shot CLIP performance without access to external models by using random characters and random word descriptors.

Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval

We find that image-text retrieval models commonly learn to memorize spurious correlations in the training data. We introduce a metric that measures a model’s robustness to spurious correlations in the training data, and de-bias those models by finetuning them with the controlled synthetic dataset.

Bridging the Gap between Model Explanations in Partially Annotated Multi-label Classification

We observe that the explanation of two models, trained with full and partial labels each, highlights similar regions but with different scaling. We then propose to boost the attribution scores of the model trained with partial labels to make its explanation resemble that of the model trained with full labels.

Large Loss Matters in Weakly Supervised Multi-Label Classification

We frame the partially labeled setting as a noisy multi-label classification task, and observe the memorization effect. We then propose to reject or correct high-loss samples, preventing the memorization of noise.

Keep CALM and Improve Visual Feature Attribution

Class Activation Mapping (CAM) is widely used for visual feature attribution, but its reliance on ad-hoc calibration steps outside the training graph limits its interpretability. We address this issue by introducing a latent variable for cue location, an explanation by itself, in the training graph.

Design and source code from Leonid Keselman's website
Keywords color-coding inspired from Seong Joon Oh's website