Workload planning desk / Reading notes
Build a workload memory budget
Estimate the working set before choosing an accelerator.
List the components
Include parameters, activations, optimizer state where relevant, inputs and temporary buffers. The stored dataset size is not the same as peak device memory.
Measure a representative case
Profile a realistic input shape and execution mode. Keep headroom for allocator behaviour and temporary peaks instead of sizing exactly to an average.
Scale deliberately
Change batch size or partitioning one step at a time. A workload that fits on several devices may still require explicit distribution and communication support.