Excited to present my work at CoLLAs 2025 @CoLLAs_Conf! In our paper arxiv.org/abs/2501.12254, we tackle the challenge of self-supervised learning from scratch with continuous, unlabeled egocentric video streams, where we propose to use temporal segmentation and a two-tier memory.

CDS Asst. Prof. Mengye Ren (@mengyer), Courant PhD students @alexandernwang and Christopher Hoang, and @ylecun introduce PooDLe: a self-supervised learning method enhancing AI vision in real-world videos by improving small object detection. nyudatascience.medium.com/le…

Check out our latest paper on representation learning from naturalistic videos →

🚀 Why Lifelong Learning Matters 🚀 Modern ML systems struggle in non-stationary environments, while humans adapt seamlessly. How do we bridge this gap? 📖 Read our latest blog on the vision behind #CoLLAs2025 and the future of lifelong learning research: 🔗 lifelong-ml.cc/blogs/1 #MachineLearning #ContinualLearning #AI #LifelongLearning

New research by CDS MS student Amelia (Hui) Dai, PhD student Ryan Teehan (@rteehas), and Asst. Prof. Mengye Ren (@mengyer) shows that models’ accuracy on current events drops 20% over time—even when given the source articles. Presented at #NeurIPS2024. nyudatascience.medium.com/la…

Just finished my first in-person NeurIPS journey. It’s great to meet many friends, old ones and new ones. Happy to see that my work is well-received in the poster session!

Thrilled to be back at NYU CDS and continuing my research journey! The MSDS program provides incredible research opportunities that shaped my path. If you’re passionate about data science, this is the place to be!

For more info, check out the links below! arXiv preprint: arxiv.org/abs/2411.08324 Project webpage and dataset download: agenticlearning.ai/daily-ora… Work done by @ameliadai_, @rteehas, @mengyer

There are still many interesting open questions: Would there be a limit to in-context learning? Do we need continuous pretraining to keep the model up-to-date?

Same even when feeding the models with the gold article (a.k.a. the actual news article for QA).

Can RAG save us from declining? Only partially. Retrieving relevant news articles can sometimes help, but it does not stop the declining trend.

We find a smooth temporal trend—LLMs' performance degrades over time, with a sharp decline beyond its "knowledge cutoff" period.

Will LLMs ever get out-dated? Can LLMs predict the future? Today, we release Daily Oracle, a daily news QA benchmark testing LLM’s temporal generalization and forecasting capability. 🧵

Humans and animals learn visual knowledge through continuous streams of experiences. How do we perform unsupervised continual learning (UCL) in the wild? Yipeng's latest paper reveals three essential components for UCL success in real-world scenarios: Plasticity, Stability, and Cross-Task Consolidation. Our Osiris algorithm is currently the state-of-the-art on UCL benchmarks. Excitingly, on new structured environment benchmarks, Osiris surpasses even IID training. Surprising? Perhaps, but an optimal continual learner should exploit temporal structures instead of falling behind in random sequences. Thrilled to share that our paper will be presented at #CoLLAs2024! arxiv.org/abs/2404.19132

🔍 New LLM Research 🔍 Conventional wisdom says that deep neural networks suffer from catastrophic forgetting as we train them on a sequence of data points with distribution shifts. But conventions are meant to be challenged! In our recent paper led by @YanlaiYang, we discovered a curious behavior in overparameterized networks, especially LLMs—as we train the network on a cyclic sequence of documents, it starts to anticipate the next document and reverses the forgetting trend! ⤴️ ▶️ After 3-4 cycles, the network reverses over 90% of the forgetting right before seeing the original document again. ▶️ The amount of anticipation emerges with the size of the network. LLMs <= 160M show no anticipation. ▶️ We showed that you can reproduce such an effect in a toy network! Check out more details in our arXiv preprint on anticipatory recovery: Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training. 🚀 arxiv.org/abs/2403.09613 🚀 #LLM #AI #Research

Wondering about how to train deep neural networks without backprop? Check out our ICLR 2023 paper: arxiv.org/abs/2210.03310 Forward gradient computes gradient information from forward pass. But it is slow and noisy — it computes the directional gradient along a random weight perturbation. Gradient variance explodes with deeper and wider networks. Our key insight is to utilize local activity perturbation and we introduce a new architecture with many local losses throughout the network so that each loss is associated with a small number of weights. Both supervised and self-supervised (contrastive) learning works. Much better than a lot of backprop-free methods on large scale problems. Joint work with @geoffreyhinton @skornblith @lrjconan Our ICLR poster will be presented on Tuesday May 2, in Kigali, Rwanda!

Introducing LifelongMemory, an LLM-based personalized AI for egocentric video natural language query (NLQ). This amazing work is led by Ying Wang @yingwww_

🚨 New Research Alert! People have found safety training of LLMs can be easily undone through finetuning. How can we ensure safety in customized LLM finetuning while making finetuning still useful? Check out our latest work led by Jiachen Zhao! @jcz12856876 🔍 Our study reveals: 1️⃣ Traditional sequential safety finetuning? It leads to forgetting of unsafe data but also useful downstream data. 2️⃣ Surprising find: LLMs are much more likely to forget unsafe examples than other downstream examples after safety finetuning. The selective forgetting behavior only emerges in large scale models. 3️⃣ Enter ForgetFilter! Inspired by the selective forgetting behavior, we can filter out both explicit and implicit unsafe content based on forgetting, ensuring LLMs' safety without compromising task performance. 4️⃣ Bonus results: Interleaving between safe/unsafe data, LLMs can recall "forgotten" unsafe knowledge info despite safety finetuning. Long-term safety requires data filtering. 🔗 arxiv.org/abs/2312.12736 #AI #MachineLearning #DataSafety #LLMs #ForgetFilter #LargeLanguageModels