NeuralAcademy
CoursesLearning PathsAI AgentsPromptsAI ToolsWorkflowsResourcesPlaygroundCommunityLiveBlogLeaderboardCategoriesInstructorsPricingAboutContact
Sign inGet started
LLM Foundations: A Deep Dive
Module 2
Lesson 1 of 4
02.01 · Pretraining

Objectives and data mixtures

AI-generated
Lesson content · AI-crafted

My notes

Sign in to jot down private notes as you learn.

5/24
Course outline
Module 1
From tokens to transformers
4
  • 01.01Tokenization deep dive
  • 01.02Embeddings and positional encoding
  • 01.03Attention from scratch
  • 01.04The transformer block
Module 2
Pretraining
4
  • 02.01Objectives and data mixtures
  • 02.02Scaling laws
  • 02.03Compute and hardware
  • 02.04Reading the training loss
Module 3
Alignment
4
  • 03.01Supervised fine-tuning
  • 03.02RLHF and DPO
  • 03.03Constitutional methods
  • 03.04Failure modes
Module 4
Inference
4
  • 04.01Sampling strategies
  • 04.02KV cache and batching
  • 04.03Speculative decoding
  • 04.04Quantization
Module 5
Adapting models
4
  • 05.01LoRA and QLoRA
  • 05.02Continued pretraining
  • 05.03Distillation
  • 05.04Choosing an approach
Module 6
The frontier
4
  • 06.01Long context
  • 06.02Mixture of experts
  • 06.03Multimodal models
  • 06.04Where the field is going
NeuralAcademy

The training academy for AI engineers. Ship real AI products.

Learn

  • All courses
  • AI Agents
  • LLMs
  • RAG

Academy

  • Instructors
  • Pricing
  • FAQ
  • Contact

Legal

  • Terms
  • Privacy
  • Cookies

© 2026 NeuralAcademy. All rights reserved.

Built for the AI-native engineer.

Edit with