EP-390 Syllabus

C
Charles•

Generative AI for Music, Code, and Image

Course Description

AI Hype is everywhere! Major tech companies are betting on AI and offering annual salaries in the hundreds of millions to AI researchers.

AI Scorn is everywhere too. Detractors point to the environmental costs, ethical implications, and technical limitations of generative AI.

So which is it? Over-hyped purveyor of AI Slop? Or impending economic revolution posing a real threat to artists, musicians, and anyone not poised to benefit from the success of major tech companies? Can it be both?

Generative models can be very helpful for writing code, especially for beginners who want to learn fast. Can they also be useful for other artistic practices? Will they be useful in the future? Learn to see through the hype and make your own informed opinions about the utility, trajectory, and ethics of modern generative AI.

Topic List

Students will start with code generation and learn to write better code faster using AI tools. We will use the efficacy of coding tools as a benchmark to evaluate the value and trajectory of generative AI for music and image modalities.

Each week will introduce a new topic, and work on an introductory lab exercise. Each exercise is an opportunity to expand on further in the mid-term and final projects.

  • Hands on: Tools for agentic code generation
  • Hands on: Symbolic Music Generation
  • Research Review: Can models “reason”?
  • Ethics: TESCREAL
  • Ethics: AI and the Environment
  • Mid-term project!
  • Theory: Transformers
  • Theory and Practice: Music Generation
  • Theory: Diffusion
  • Theory and practice: Model internals and Comfy-UI
  • Theory and practice: Image Gen – On-device and in the cloud
  • Hands on: Video Games
  • Final project!

AI is changing fast! The exact make-up of this list may change subtly to keep up with important developments in the field.

Course Activities

In this class, you will:

  • Get hands-on practice with modern tools for generating media
  • Study how the models work under the hood
  • Contrast the strengths and weaknesses of different modalities: Image, Music, and Code to understand where these technologies are headed and what they are likely to mean for artists in the future.
  • Explore research and criticism of Generative AI
  • Build a portfolio of content and reflections that expose AI strengths and weaknesses
  • Develop an informed critical take on the utility and ethics of generative AI

Learning Outcomes

Upon completion of this course, students will be able to:

  • Demonstrate proficiency with contemporary generative AI tools for creating music, code, and images
  • Analyze and explain the fundamental technical principles underlying transformer and diffusion models
  • Critically evaluate generative AI research papers and distinguish between hype and substantive claims
  • Assess the environmental, ethical, and economic implications of generative AI deployment
  • Identify and articulate pitfalls in Generative AI
  • Develop well-reasoned perspectives on AI's role in creative industries

Prerequisites

  • We will be writing code, using the terminal, and submitting assignments on GitHub
  • The class requires a B or better in LMSC-261 to enroll
  • We will be using both Python and JavaScript – Comfort with at least one of these is required

Preamble

Some contents of the curriculum may change in small ways based on our progress throughout the semester and based on new and ongoing developments in the field. The course policies, expectations, and grading metrics will not change.

Expectations

All students are expected to

  • Attend fourteen lectures
  • Work on weekly assignments which will be visible in the class website
  • Present 1-2 research works
  • Design and implement mid-term and final projects
  • Participate in the mid-term and final project presentations
  • Experiment with unfamiliar technologies

The last bullet above is important! Each time you expose yourself to a new instrument or technology there is some discomfort as you learn your way around it and become proficient. The good news is that the more you do this, the easier and more fun it gets – and the FASTER you start to learn other related instruments and technologies.

We will be using Python, JavaScript, and TypeScript. You do not have to be good at them. You do have to be willing to dive in and try. The goal is to learn to get through the discomfort so that you can move on to the fun parts quickly.

Grade Determination

The grade is determined as follows:

Description Grade
Research Work 20%
Lab exercises 30%
Mid-term Project 25%
Final Project 25%

Assignments submitted after the cutoff will not be accepted.

Bias towards Open Models

Currently, the best generative models are "closed weight," meaning that in order to access them you need to pay to use a model by sending a request to the model provider's web-service.

Sometimes proprietary closed-source generative models are promoted for free.

In this class we will prioritize free and open-weight models when possible and appropriate.

We will avoid proprietary services like Suno that focus on web-based UIs.

← Back to Home