Introduction

​​Foundation models are a recent class of AI models that are large-scale in terms of number of parameters and are trained on broad data (generally using self-supervision at scale). These models have demonstrated exceptional capabilities in natural language processing, computer vision, and other tasks. Examples of Foundation Models are GPT-4, ChatGPT, GPT-3, Dall-E, Stable Diffusion, etc. In this course, we discuss building blocks of foundation models, including transformers, self-supervised learning, transfer learning, learning from human feedback, power of scale, large language models, in-context learning, chain-of-thought prompting, parameter-efficient fine-tuning, vision transformers, diffusion models, generative modeling, safety, ethical and societal considerations, their impact, etc. While the course primarily focuses on advances on large language models, we also cover foundation models in computer vision, as well as multi-modal foundation models.

Learning Resources

Textbook

There is no required textbook and we will be mostly reading publicly available research papers. The papers will be mostly from major conferences in the field including ACL, EMNLP, NAACL, ICLR, CVPR, NeurIPS, etc.

Optional book chapter: Bommasani, Rishi et al. “On the Opportunities and Risks of Foundation Models.” Stanford, ArXiv abs/2108.07258 (2021)

Anonymous feedback

If you wish to share comments, questions, or feedback anonymously please use this form: Anonymous Form.
I will check this regularly and respond to questions/comments in the next lecture.

Communication

We use Canvas and email for main announcements. For questions about the course, discussions about material, and faciliatating discussions for projects between students, we will mainly use Slack

Grading

Final grades will be comprised of:

  • 30%: Assignments, which includes both written and coding problem sets
  • 20%: Midterm, in person, closed book
  • 10%: Participation and quizzes
  • 40%: Final projects, including project proposal (5%), progress report (5%), final report and presentation (20%), code and reproducibility checklist (10%)

AI Assistant policies

Using assistance from AIs such as ChatGPT to complete your homeworks, quizzes, projects, and exam is not allowed except for the following circumstances: 1- The assignment explicitly askes for it 2- AI Assistant is used to improve writing. If you take advantage of any sort of AI assistance for an assignment, you will be required to submit the specific prompts you used.

Late submissions

You can still submit your assignment after the deadlines for up to 5 days. You will, however, receive partial credit for late submissions. Every late day will result in 10% deduction in full credit for that assignment

Note: Late days can only be used on the assignments, and not on the final project proposal or final report.

Grading for graduate students

Grading components for graduate students will be the same as undergraduate students. The only difference is the following:

For class projects we expect graduate students to work on a research problem (The project should propose either a novel research, a novel investigation of existing methods, an extension of prior work for a specific purpose, or a new application.). Graduate student projects are also expected to have a more thorough literature review component in their final project report.

Class project (40%)

Students must complete a final research project on a topic of their choice related to the class. The students can optionally team up with other students but the team size is limited to 2 students. (In rare cases and depending on the scope of the proposed project, a group of size 3 may be also allowed.) 

  • 5%: proposal
    • Students should submit a 1-2 page proposal for their project. The proposal should state and motivate the problem, and position the proposed project within related work. The project proposal should also include a brief description of the approach as well as the experimental plan (e.g., baselines, datasets, etc) to validate the effectiveness of the approach. Here are some ideas on types of projects.:
    • For undergraduate students the project could be reimplementation of an exsiting method, a new user-facing application that uses foundation models for a new problem, a comprehensive survey into a subtopic of interest, deeper investigation of a paper and providing further insights by conducting additional experiments, or novel reseach.
    • For graduate students the project should include a component of novelty. E.g., it could propose a novel research, a novel investigation of existing methods, an extension of prior work for a specific purpose, or a new application.
  • 5%: project progress report
    • 1 page document due by week 10-11 (around 2 weeks after mid-term). It should describe the project goal and related work, initial results, and the plan continuing the project. 
  • 10%: code and reproducibility checklist
    • Your project code should be clean, readable, with clear running instructions, and the results should be fully reproducible. We will provide a reproducibility checklist that should be returned.
  • 20%: final project report
    • 4-6 (no more than 6 pages) page conference format report (e.g., NeurIPS) detailing the project motivation, related work, proposed approach, results, and discussion. You can think of this as a conference paper. Negative results will not be penalized, but should be accompanied with detailed analysis of why the proposed methods didn’t work and provide some additional insights into the problem. 
    • References and appendix won’t count towards the page limit
    • A 3 minute pre-recorded video presentation of your project.

Integrity

Academic integrity requires that students at Yale acknowledge all of the sources that inform their coursework. Most commonly, this means (a) citing the sources of any text or data that you include in papers and projects, and (b) only collaborating with other students or using AI composition software in ways that are explicitly endorsed by the assignment. Yale’s dedication to academic integrity flows from our two primary commitments: supporting research and educating students to contribute to ongoing scholarship. A safe and ethical climate for research demands that previous authors and artists receive credit for their work. And learning requires that you do your own work. Conventions for acknowledging sources vary across disciplines, and instructors should instruct you in the forms they expect; they should also delineate which forms of collaboration among students are permitted. But ultimately it is the student’s responsibility to act with integrity, and the burden is on you to ask questions if anything about course policies is unclear.