Practice
Learn
← Quit
0 / 0 correct
LLMs & AI
Hard
1 of 10
What does continuous (in-flight) batching do in servers like vLLM or TGI?
A
Sends each request to a separate GPU
B
Batches requests once per minute
C
Merges prompts into one long prompt
D
Adds and removes sequences from the running batch at every decode step, instead of waiting for a whole batch to finish