Config Explorer should consider `--dtype` and `--kv-cache-dtype`

[jgchn]

Component

Configuration Explorer

Desired use case or feature

In vLLM, there are various choices for controlling the precision of the model. To start off, consider the three arguments: --dtype, --kv-cache-dtype, and --quantization.

--dtype: https://docs.vllm.ai/en/latest/configuration/engine_args.html#-dtype

• Controls the precision used for loading model weights and activations
• auto: will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models
• Default is determined from torch_dtype in config.json

--quantization: https://docs.vllm.ai/en/latest/configuration/engine_args.html#-quantization-q

• This probably takes precedence over --dtype.
• Options include bitesandbytes, fp8, quark. Looks like it is model- and hardware- specific.

--kv-cache-dtype: KV cache can be quantized to reduce memory footprint.

• Options include auto (unquantized, defaults to the model data type, fp8, fp8_e4m3, and fp8_e5m2`
• Given this example, it seems that kv cache dtype is independent of the weight dtype. One can use kv cache dtype fo fp8 while the tensor weights are unquantized

Proposed solution

The Capacity Planner should accept quantization for model weights and activation. Currently, the calculation for model weights use the dtype exposed by safetensor metadata, which might be different for some parameters. However, when loading into vllm, the weights use the same precision (it should be what --dtype is). Furthermore, allow users to configure --kv-cache-dtype. Finally, update the vllm serve command.

Alternatives

No response

Additional context or screenshots

No response

[jgchn]

I also noticed that for some quantized models, there is a quantization_config in config.json. That seems to be what vLLM use when loading the model for determining the default quant method.

After some empirical testing aka actually starting vLLM I observed the following for two models

• Qwen/Qwen3-8B
  • There is no quantization_config
  • Current Capcaity planner estimates model memory = 15.26GB
• Qwen/Qwen3-8B-FP8
  • The quant method in config.json is fp8
  • • Current Capcaity planner estimates model memory = 8.79GB

Model	--dtype	--quantization	Memory size (GB)
Qwen/Qwen3-8B	NA	NA	15.27
	bfloat16	NA	15.27
	bfloat16	fp8	8.80
	NA	fp8	8.80
Qwen/Qwen3-8B-FP8	NA	NA	8.80
	bfloat16	NA	8.80
	bfloat16	fp8	8.80
	NA	fp8	8.80

Conclusions

• If quant_method exists in config.json, that is the default quant method applied, regardless if the user explicitly sets it or not.
  • A caveat is that I noticed for RedHatAI/Qwen3-8B-FP8-dynamic, the quant_method is compressed-tensors. One will need to look at the config group to determine the number of bits used
• For unquantized models, it is possible to use quantization even if config.json doesn't supply it
• For quantized models, one can leave out --dtype and --quantization and vLLM will load it in quant mode by default

Something that still troubles me is that it seems like vLLM does NOT use the same precision for each parameter. For Qwen/Qwen3-8B-FP8, there are two sets of parameters with different dtypes:

Data type	Size in bytes	Number of parameters	Memory in GB (params x bytes)
BF16	2	1245391872	2.319723129272461
F8_E4M3	1	6945767424	6.46875

Our estimation is 8.79GB which is close to the actual vLLM run. This shows that despite some params are BF16, vLLM does not cast it to F8, so the original assumption that vLLM will use the same precision for every parameter given --dtype or -q is incorrect.

Would appreciate some help or pointers to clarify this.

[achandrasekar]

Agree on not relying on --dtype and using quantization config where available. For example, Deepseek-R1 is FP8 quantized and the model weights are loaded in FP8 by default, but the default dtype is BF16 - https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/blob/main/config.json.

A caveat is that I noticed for RedHatAI/Qwen3-8B-FP8-dynamic, the quant_method is compressed-tensors. One will need to look at the config group to determine the number of bits used

This is ok I think because the num_bits under weights tells us how the model weights are quantized. We can make looking at num_bits the default for compressed-tensors?

For unquantized models, it is possible to use quantization even if config.json doesn't supply it

We don't have to worry as much about this I think since in our case, we are trying to provide recommendations based on the model information available. Any runtime quantization config is something the users of the tool should account for themselves.

Something that still troubles me is that it seems like vLLM does NOT use the same precision for each parameter. For Qwen/Qwen3-8B-FP8, there are two sets of parameters with different dtypes:

This is interesting. Where are these numbers from? vLLM logs? Are some of the model weights loaded in different precision than others?

[jgchn]

Thanks Ashok @achandrasekar for the feedback. Sounds like we should make the data type configurable in capacity planning in case users wants to load in a quantized manner for unquantized models?

Something that still troubles me is that it seems like vLLM does NOT use the same precision for each parameter. For Qwen/Qwen3-8B-FP8, there are two sets of parameters with different dtypes:

This is interesting. Where are these numbers from? vLLM logs? Are some of the model weights loaded in different precision than others?

I obtained these numbers from the HFAPI ModelInfo, also available via Capacity Planner.

from config_explorer.capcaity_planner import get_model_info_from_hf
model = "Qwen/Qwen3-8B-FP8"
info = get_model_info_from_hf(model)
print(info.safetensors)

SafeTensorsInfo(parameters={'BF16': 1245391872, 'F8_E4M3': 6945767424}, total=8191159296)

The capacity planner estimates model to take 8.79GB, and vLLM log say that it is 8.80GB, which is pretty close. Our formula today takes into account the dtype for each dtype-parameter pair (so 1245391872 = 2 bytes and 6945767424 = 1 byte). This means vLLM actually loads the parameters differently, which is different from my assumption when I created this post that if --dtype is set, vLLM would load the parameters using the same dtype.