If there is a specified torch_dtype, it will be used instead.