Schedule

• Course Introduction
• Logistics

• Jurafsky & Martin Chapter 1

• Text classification
• Generative Models vs Discriminative Models

• Jurafsky & Martin Chapter 4 & 5

• Text classification (cont.)
• Evaluation
• N-Gram Language Models

• Jurafsky & Martin Chapter 4
• Jurafsky & Martin Chapter 3

• The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (Dror et al., 2018) [link](https://aclanthology.org/P18-1128.pdf)

• Word embeddings and vector semantics

• Jurafsky & Martin Chapter 6

HW1 out 1/29

• Word embeddings and vector semantics (cont.)

• Jurafsky & Martin Chapter 6

• Distributed Representation of Words and Phrases and their Compositionality (Mikolov et al., 2013) [link]
• Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013) [link]
• Word2vec Explained- deriving Mikolov et al.'s negative-sampling word-embedding method (Goldberg and Levy, 2014) [link]

• Information theory
• Sequence labeling

• Eisenstein Chapter 7

• Sequential classification
• Conditional Random Fields

• Eisenstein 7.1-7.3
• Micheal Collins' notes on CRFs [link]

• Lafferty et al. (2001) Conditional Random Fields- Probabilistic Models for Segmenting and Labeling Sequence Data [link]
• Sutton and McCallum (2010) An Introduction to Conditional Random Fields [link]

• Basics of Neural Networks and Language Model Training

• The Matrix Calculus You Need For Deep Learning (Terence Parr and Jeremy Howard) [link]
• Little book of deep learning (François Fleuret) - Ch 3

2/11 HW 1 due

• Autograd
• Building blocks of Neural Networks
• Convolutional layers
• Network layers and optimizers

• Little book of deep learning (François Fleuret) - Ch 4

Project teams due

• Building blocks of Neural Networks for NLP
• Taks specific neural network architectures
• RNNs

• Goldberg Chapter 9

2/16 HW 2 out

• RNNs (contd.)
• Machine translation

• Understanding LSTM Networks (Christopher Olah) [link]
• Eisenstein, Chapter 18

• Neural Machine Translation and Sequence-to-sequence Models- A Tutorial (Graham Neubig) [link]

• Machine translation (contd.)
• Attention
• Transformers

• Statistical Machine Translation (Koehn) [link]
• Neural Machine Translation and Sequence-to-sequence Models- A Tutorial (Graham Neubig) [link]
• Learning to Align and Translate with Attention (Bahdanau et al., 2015) [link]
• Luong et al. (2015) Effective Approaches to Attention-based Neural Machine Translation [link]
• Attention is All You Need (Vaswani et al., 2017) [link]
• Illustrated Transformer [link]

• Transformers (cont'd.)
• Language modeling with Transformers

• Illustrated Transformer [link]
• Attention is All You Need (Vaswani et al., 2017) [link]
• The Annotated Transformer (Harvard NLP) [link]
• GPT-2 (Radford et al., 2019) [link]

• Pre-training and transfer learning
• Objective functions for pre-training
• Model architectures
• ELMO, BERT, GPT, T5

• The Illustrated BERT, ELMo, and co. (Jay Alammar) [link]
• BERT- Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018) [link]
• GPT-2 (Radford et al., 2019) [link]

3/1 HW 2 due

• Transfer learning (contd.)
• Encoder-decoder pretrained models
• Architecture and pretraining objectives

• T5- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2020) [link]
• BART- Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (Lewis et al., 2019) [link]
• What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (Wang et al, 2022) [link]

3/8 Project proposals due;
3/10 HW 3 out

• Decoding and generation
• Large language models and impact of scale
• In-context learning and prompting

• The Curious Case of Neural Text Degeneration (Holtzman et al., 2019) [link]
• How to generate text- using different decoding methods for language generation with Transformers [link]
• Scaling Laws for Neural Language Models (Kaplan et al., 2020) [link]
• Training Compute-Optimal Large Language Models (Hoffmann et al., 2022) [link]
• GPT3 paper - Language Models are Few-Shot Learners (Brown et al., 2020) [link]

• Retreival-Augmented Language Models

• In-context learning and prompting (cont'd)
• Improving instruction following and few-shot learning

• Few-Shot Learning with Language Models (Brown et al., 2020) [link]
• Finetuned Language Models Are Zero-Shot Learners (Wei et al., 2022) [link]
• Multitask Prompted Training Enables Zero-Shot Task Generalization (Sanh et al., 2021) [link]
• Scaling Instruction-Finetuned Language Models (Chung et al., 2022) [link]
• Are Emergent Abilities of Large Language Models a Mirage? (Sha et al., 2023) [link]
• Emergent Abilities of Large Language Models (Wei et al., 2022) [link]

4/2 HW3 due;

• Reinforcement learning from Human Feedback
• Alignment

• Training language models to follow instructions with human feedback (Ouyang et al., 2022) [link]
• Fine-Tuning Language Models from Human Preferences (Ziegler et al., 2019) [link]
• Direct Preference Optimization- Your Language Model is Secretly a Reward Model (Rafailov et al., 2023) [link]
• RLAIF- Scaling Reinforcement Learning from Human Feedback with AI Feedback (Lee et al., 2023) [link]

HW 4 out 4/7

• The Quest to build an (O)pen (L)anguage (Mo)del

• OLMo- Accelerating the Science of Language Models (Groeneveld et al., 2024) [link]
• Dolma- an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research [link]

• AI safety, security, and ethics
• Privacy and societal implications

• Self-Alignment of Large Language Models

• Self-Alignment with Instruction Backtranslation (Li et al., 2023) [link]
• Self-Rewarding Language Models (Yuan et al., 2024) [link]

• Efficiency in large language models
• Efficient inference and quantization
• Efficient transformers

• Project presentations

• Project presentations

Last meeting of the class;
4/28 HW 4 due
5/8 Final project due