Practice
Learn
← Quit
0 / 0 correct
LLMs & AI
Hard
1 of 40
What does continuous (in-flight) batching do in servers like vLLM or TGI?
A
Sends each request to a separate GPU
B
Adds and removes sequences from the running batch at every decode step, instead of waiting for a whole batch to finish
C
Merges prompts into one long prompt
D
Batches requests once per minute