Skip to content
This repository was archived by the owner on Oct 9, 2024. It is now read-only.

Add configs to run int4 inference #37

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add configs to run int4 inference #37

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
572e644
Add configs to run int4 inference
Nov 18, 2022
132d99d
fix quantization-bit config & turn off ds_sample
Nov 18, 2022
99cd7c9
change the quantization config format to work with the new style at D…
Nov 19, 2022
32779e8
Update bloom-ds-inference.py
awan-10 Nov 21, 2022
b472e48
Merge pull request #1 from awan-10/patch-1
RezaYazdaniAminabadi Nov 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions bloom-inference-scripts/bloom-ds-inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
parser = ArgumentParser()

parser.add_argument("--name", required=True, type=str, help="model_name")
parser.add_argument("--dtype", type=str, help="float16 or int8", choices=["int8", "float16"], default="float16")
parser.add_argument("--dtype", type=str, help="float16 or int8 or int4", choices=["int8", "float16", "int4"], default="float16")
parser.add_argument("--local_rank", required=False, type=int, help="used by dist launchers")
parser.add_argument("--batch_size", default=1, type=int, help="batch size")
parser.add_argument("--benchmark", action="store_true", help="additionally run benchmark")
Expand Down Expand Up @@ -100,7 +100,7 @@ def get_checkpoint_files(model_name_or_path):


model_name = args.name
infer_dtype = args.dtype
infer_dtype = args.dtype if args.dtype != 'int4' else 'int8'
Copy link
Contributor

@stas00 stas00 Nov 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make for a more user-friendly API to

  1. keep the dtype intact
  2. drop quantization_bits
  3. let deepspeed.init_inference derive the number of bits from dtype?

not only the currently suggested override is confusing, I fail to see what purpose serves carrying the same information in dtype and and quantization_bits twice

Copy link
Contributor

@stas00 stas00 Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, wait, torch.init4 still doesn't exist, does it?

let's find the feature request.

Copy link
Contributor

@stas00 stas00 Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still not implemented pytorch/pytorch#74627

so that's why you had to do the odd workarounds, right?

Copy link
Collaborator

@mayank31398 mayank31398 Nov 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we can drop it once its implemented @stas00 ?
For now, this might be the best way to do it.

Copy link
Contributor

@stas00 stas00 Nov 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see #37 (comment)

Copy link
Contributor

@stas00 stas00 Nov 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's pointless to wait, since they won't have int3 and int12

Copy link

@awan-10 awan-10 Nov 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make for a more user-friendly API to

  1. keep the dtype intact
  2. drop quantization_bits
  3. let deepspeed.init_inference derive the number of bits from dtype?

not only the currently suggested override is confusing, I fail to see what purpose serves carrying the same information in dtype and and quantization_bits twice

@stas00 and @RezaYazdaniAminabadi - just clarifying that we have introduced a new DeepSpeedInferenceConfig that can be passed to init_inference. We are keeping it backwards compatible but if we are okay to make changes to this file, I would advocate for writing a config dictionary for DeepSpeed and pass that to init_inference instead of the various kwargs. Please see here for an example: https://gist.github.com/awan-10/6e3d5c756be3a876522e860c6bbf702d#file-bloom-ds-inference-py-L173

Also, see the docs for the new config: https://deepspeed.readthedocs.io/en/latest/inference-init.html

Copy link
Contributor

@stas00 stas00 Nov 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That definitely works.

@awan-10, may I suggest you make the inference config accept dict_or_path just like zero does? it might be for some users easier to write out a separate file.

Copy link

@awan-10 awan-10 Nov 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stas00 - thanks for the suggestion. Created an issue so we can track it: deepspeedai/DeepSpeed#2532. Mike and I will work on it.

Copy link
Contributor

@stas00 stas00 Nov 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much, @awan-10


tp_presharded_mode = True if model_name in tp_presharded_models else False

Expand Down Expand Up @@ -191,6 +191,7 @@ def write_checkponts_json():
mp_size=world_size,
base_dir=repo_root,
dtype=getattr(torch, infer_dtype),
quantization_bits=8 if args.dtype == 'int8' else 4,
Copy link
Contributor

@stas00 stas00 Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens with --dtype float16?

probably best to set this in kwargs only if quantization dtype is provided

Copy link
Author

@RezaYazdaniAminabadi RezaYazdaniAminabadi Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The quabtization-bit should not be used when running in half-precision. But, I agree we can do it in the kwargs and only for qunatized inference mode.

Copy link
Contributor

@stas00 stas00 Nov 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these demos are already used by many users so let's make those nice and clean configuration-wise, so it's clear to the reader when what bits should be enabled.

checkpoint=checkpoints_json,
**kwargs,
)
Expand Down Expand Up @@ -227,7 +228,7 @@ def write_checkponts_json():
# dynamically extend to support larger bs by repetition
input_sentences *= math.ceil(args.batch_size / len(input_sentences))

generate_kwargs = dict(max_new_tokens=num_tokens, do_sample=False)
generate_kwargs = dict(max_new_tokens=num_tokens, do_sample=True)
Copy link
Contributor

@stas00 stas00 Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is already a very different type of change.

If int4 requires do_sample=True, then again, let's change it only if it's --dtype int4

Copy link
Author

@RezaYazdaniAminabadi RezaYazdaniAminabadi Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I will double check the do_sample=False again to see if the text generated makes sense. If not, I set it to true for int4

Copy link
Author

@RezaYazdaniAminabadi RezaYazdaniAminabadi Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just checked with do_sample=False and I see the text is produced in the same way as for FP16 and INT8:

in=DeepSpeed is a machine learning framework
out=DeepSpeed is a machine learning framework for deep learning. It is a Python library, and it is also a framework. It is a framework, and it is a library. It is a framework, and it is a library. It is a framework, and it is a library. It is a framework, and it is a library. It is a framework, and it is a library. It is a framework, and it is a library. It is a framework, and it is a library. It is a framework, and

So, I am gonna turn it off for now.



print_rank0(f"Generate args {generate_kwargs}")
Expand Down