piǍ $dZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl Z ddl mZddlmZej"dZ dd Zdd ZddZddZddZdZid d d iddgdfdZdZdZ d dZid d d ddddddddgf dZy)!zWeightOnly for onnxrt adaptor.N) numpy_helper)np_dtype_to_tensor_dtype) ONNXModel)simple_progress_barneural_compressorc  ||zdz} tj|jd| fd} |jdd|d|z} |jd| g} g} i}d}|d k(r1|d d d d d f|d d dd d fd zz}|d d d | f| d d d d f<n!|dk(r|} ntj d |d tj | d|| f} tj |d|f}|jtjk(s|jtjk(sJtjj|jddzt|j|j|jd}| j|j | j||~|dk(r|j#d}n|d k(rtj$|jddzd zdd}tj&|jd|z|zj d}|d d d }|dd d }||d zdz||j)z||d z<||d zdz||j)d zz||d z<nt+d |d tj ||ddf}tjj|jddzd |j|jd}| j|j | j||d|d<|d|d<||d<||d<|dkDr||d<tjj| d | j| jd}| j|tjj,|f| |j.|j r|j dzt1|zn dt1|zdd|}|| fS)aBBuild MatMulNBits node. Args: node: original matmul node weight_shape: original weight shape num_bits (int): num_bits group_size (int): how many elements share one scale/zp k_blocks (int): block number q_weight (array): quantized weight scale (array): scale zero_point (array): zero point accuracy_level (int): accuracy level. Support 0 (unset), 1(fp32), 2(fp16), 3(bf16), or 4(int8). Returns: matmul_weight_only_node: MatMulNBits node new_inits: initializers of the new node ruint8dtyper_QG MatMulNBitsNz8MatMulNBits does not have kernel support for num_bits = ._scaleTname data_typedimsvalsraw_zpKNbits block_sizeaccuracy_levelz com.microsoft)inputsoutputsrdomain)npzerosshapeinputloggererrorreshaper float32float16onnxhelper make_tensorrtobytesappendrastypefullarangeravel ValueError make_nodeoutputstr)node weight_shapenum_bits group_sizek_blocksq_weightscale zero_pointr$ blob_sizepacked q_weight_name input_names new_initskwargsop_typeq_weight_pairs scale_tensor packed_zpidxeven_idxodd_idx zp_tensorq_weight_tensormatmul_weight_only_nodes x/opt/services/ai/voice_agent/venv/lib/python3.12/site-packages/onnxruntime/quantization/neural_compressor/weight_only.pymake_matmul_weight_only_noderW,s8X%*I XXx~~a()4G DFJJqMb Aj^$DDM::a=-0KI FG1}!!SqS&)HQ1W,=,BB%a)m4q!t Q OPXzYZ[\ ZZXy 9 :F JJur8n -E ;;"** $ rzz(AA A;;** ZZ]X %*5;;7 [[ ]]_  +L|(() \" q="))'2I ]!1!1!!4q!8Q >7SI))J,,Q/8;hFGOOPRSC3Q3xH!$Q$iG(1(a-(@4(G:V^K_KeKeKg'gIh!m $'0A'>'E*U\J]JcJcJeijJj&kIgl #WX`Waabcd dJJy<?B*?@ KK++A&!)//PYPaPaPcim,  9>>*#q/F3Kq/F3KF6N%F<#1 kk--  \\ ^^   .O_%"kk33 15TYY H -s8}@T    #I --r asymc .tj|d|f}|dk(s|dk(r d|zdz }d}n'|dk(r"|dk7r d|dz zdz nd}|dk7r d|dz z nd}tj|dd |z}tj|dd |z} |dk(rtjtj |tj | } tj | j} | dkD} | | d zjtjz z | | <|d k(rtj| jn)tj | jd d|dz zz} ntj | j} tj| |z || k7jjDcgc]}t|z z c}| || k7<|d k(r3tj| j|z | z jnitjdtj tj| j|z | z jjd } tj"|| j$ }tj&|| |tj(|| |tj||tj*|||| | fScc}w)a Quantize tensor per group. Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. scheme (str, optional): quantization scheme. Defaults to "asym". dtype (str, optional): data type. Defaults to "int". ratio (float, optional): percentile of clip. Defaults to 1.0. Returns: output: quantized weight scale: scale zero_point: zero point rrZuintrrrsymTaxiskeepdimsg@intr r out)r(r.minmaxmaximumabsonesr*r6float64r)arrayflattentolistfloatroundminimum empty_liker divideaddclip)datar@rAschemer ratiomaxqminqrminrmax max_rangerDmaskrEirCs rV quant_tensorr~s ::dR, -D 5F?({Q 5*2a-qX\"Q&Q)1Qx!|$%B 66$Q . 6D 66$Q . 6D JJrvvd|RVVD\:  #1} ,44RZZ@D4KPd %*e^BHHU[[ !SZ9[_`empqeq_r9s  # hh04t TT\/J/R/R/T/[/[/] ^!U1X % ^ ddl ~hhu{{#d*e 3 : : <Arzz$"((5;;2G$2NRW1W0^0^0`abiijqr  }}T5HIIdEx(FF8ZX.HHX8$GGHdDh/ UJ && _sLctj|d|fjtj}d|zdz }d}tj|dzdd}tj ||z }tj |tj|}tj|dd}tj|dd} tj|dd} tj||zdd} tj| j|j} || k7} ||z | | || z z | | <d| z }tjtj| ||z z||}||z|z|z }tj||dzzdd}d}d }d}t|D]}tj| j|j}tj |||zz|z|z gj|jd}|| k7} || | || z z || <tjtj|||z z||}||z}tj|dd}tj||zdd}tj||zdd}tj"| |z|dz}| |z| |zz |z }|| z||zz |z }||z|z|z }tj||dzzdd} tj | }!tj |}"tj$|!|"kd}#||#d d f||#d d f<| |#||#<||#||#<||#||#<tj| |z jd|jd }$|jtj&}tj(||j}%tj*|||% tj |%|$|% tj|%|% tj|%|||% |%||$fS) aQuantize tensor per group based on k quant. Ref: https://github.com/ggml-org/llama.cpp/blob/64eda5deb9859e87a020e56bab5d2f9ca956f1de/ggml/src/ggml-quants.c Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 32. Returns: output: quantized weight scale: scale zero_point: zero point rrrrTr^r 皙?Nr rb)r(r.r6r/sumsqrtrrrgrdrerhr*r rsrnrangerjsubtractwhererirprq)&rtr@rArwrxsum_x2av_xweightsryrzsum_wsum_xiscaler|rD quant_datadiffbest_madnsteprdeltarrminis_ iscale_newfactorquant_data_newmul_weights_quant_data_newsum_lsum_l2sum_xlD this_scalethis_minmadmad_1 best_mad_1idx_to_replacerErCs& rVquant_tensor_k_quant_cpurs ::dR, - 4 4RZZ @D h;?D D VVD!G!d 3F 776J& 'DffT266$<(G 66$Q .D 66$Q .D FF7T 2E FF7T>D 9E WWTZZtzz 2F 4?FFtzzRSTUt|!T$Z$t*%<= 4*t *D!EtTR%,~%="1DI2^C!VZ[2T9DQ KKq 1fnuu}49 UNUV^3q8N*X5<ffWtQw&Q> XXh' %*"45a8(6~q7H(I >1$%#&~#6 *> :n'7^184D5E/002At<CCGLJ LL $E}}T5HIIdEx(FF8ZX.HHX8$GGHdDh/ UJ &&rXc& ddl}ddl}|jjr|j |}|j d|fj |j}d|zdz }d}|j|dzdd}|j||z }|j||j|} |j|dd} |j|dd} |j| dd} |j| |zdd} |j| j|j }| | k7}||z | || |z z ||<d|z }|j#|j%||| z z||}||z| z|z }|j| |dzzdd}d }d }d}t'|D]}|j| j|j }|j)|||zz|z|z gj |j d}| | k7}|| || |z z ||<|j#|j%||| z z||}| |z}|j|dd}|j||zdd}|j||zdd}|j+| |z|dz}| |z| |zz |z } || z||zz |z }!| |z|!z|z }|j| |dzzdd}"|j)|"}#|j)|}$|j-|#|$kd}%||%ddf||%ddf<|"|%||%<| |%||%<|!|%| |%<|j#| |z j%d|j d }&|j |j.}|j1||j }'|j3|||' |j|'|&|' |j%|'|' |j#|'|||' |'j5|j5|&j5fSt6j9d t;|||S#t<$r%t6j?dt;|||cYSwxYw)aQuantize tensor per group based on k quant. Ref: https://github.com/ggml-org/llama.cpp/blob/64eda5deb9859e87a020e56bab5d2f9ca956f1de/ggml/src/ggml-quants.c Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. Returns: output: quantized weight scale: scale zero_point: zero point rNrrrTr^r rrr rbzqTry to use k-quant quantization on CUDA. However, CUDA is not available.Fall back to k-quant quantization on CPU.zNow we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.) cupytorchcuda is_availableasarrayr.r6r/rrrrrgrdrerhr*r rsrnrrjrrrirprqgetr,warningr ImportErrorinfo)(rtr@rAcprrwrxrrrryrzrrrr|rDrrrrrrrrrrrrrrrrrrrrrrErCs( rVquant_tensor_k_quant_cudarsID :: " " $::d#D<<Z 0188DDh;?DDVVD!G!dV;F776J./DffT266$<0G66$Q66D66$Q66DFF7TF:EFF7T>DFAEWWTZZtzzW:F4#E"FGNNtzzZ[\]t|#)T$Z$t*-D#E 4 !#*t 2L)MtUY!Z-4~-E*9DQ :^ KRS^bc :T ATXYKKq9#fnuu}<A "UNUV^;q@!N2X=DffWtQw.QfF XXh/ !#%**~q?P0Q >1,-+.~+>((2>(Bn%'/'?^$1 @4D5E/!8!8!:AtDKKGTJLL,E}}T}=H IIdExI 0 FF8ZXF 6 HHX8H , GGHdDhG 7<<>599; 0@@ @ NN< ,D(JG G D  D (h CC DsP=Q"!Q""+RRcz|j}t||||||\}}} tj||| z z|S)aQuant dequant tensor per group. Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. scheme (str, optional): quantization scheme. Defaults to "asym". dtype (str, optional): data type. Defaults to "int". ratio (float, optional): percentile of clip. Defaults to 1.0. Returns: output: quant-dequant weight )r*r~r(r.) rtr@rArur rv org_shapeweightrDzps rV qdq_tensorrysC I$T8ZPUVFE2 ::ev{+Y 77rXc|dk(r|S|j}||z}||dz }|dkDrtj|d|fdfd}|S)aPad tensor rowi so that it can be is divisible by group_size. Args: weight (array): weight group_size (int): how many elements share one scale/zp k_blocks (int): the number of block Returns: weight: paded weight rr)rrconstant)r*r(pad)rrArB org_w_shape padded_rowspad_lens rV pad_tensorrsYR ,,KZ'KKN*G{!Wv 6 C MrXCPUExecutionProviderk_quantc tt|}|j)tjj |jnd} g} g} t |j D cgc]} | jdvs| c} } d}|j D]n}|jdvr|dz }t| ||jdvs2|j|jdQ|j|jidk7sq|j|jd}tj|| j}t |j dk7r|j"}|j|vr6||jd }||jd }||jd }|j }|d k7r|n|d}|ddz |zdz}|j%|jd}t'|||}|d k(xs|dk(}|r|dk(rt)|j*||\}}}n;t-|j*|||d|j|jdd\}}}t/||||||j1d|j1||dk(s|dk(r|nd| \}}|j3|| j5|| j5|n t7|j*|||d|j|jdd}t9j:||dd f}t9j<|}|d|dddfj1|}t>j@jC|jdd|d|ztE||j |jGd}|jI||j|jd<|dk(s^|jK|q|jM| |jO| |jQ|Scc} w)aQuant the model with round to nearst method. Args: model (ModelProto or ONNXModel): onnx model weight_config (dict): quantization config For example, weight_config = { 'fc2': { 'bits': 4, 'group_size': 32, 'scheme': 'sym', 'algorithm': 'RTN' } } num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". ratios (dict, optional): percentile of clip. Defaults to {}. accuracy_level (int): accuracy level. Support 0 (unset),1(fp32), 2(fp16), 3(bf16), or 4(int8). providers (list): providers to use Returns: model: fake quantized ONNXModel NMatMulrrfp32)base_dirrr"rArurrr rr\r rZ r>r?r@rArBrCrDrEr$rarrTr))r model_pathospathdirnamelennodesrLrget_initializerr+rrrto_arraycopyr*r get_initializer_share_numrrTr~rWr6add_initializersr5rr(r. transposer1r2r3rr4add_initializerremove_initializer add_nodes remove_nodestopological_sort)model weight_configr@rAruratiosr$ providers algorithmr new_nodesrr} total_numcurr_idr> weight_tensorrr rrBinit_share_numsatisfy_MatMulNBits_conditionrCrDr q_matmul_noderJrTs rV rtn_quantizersH e E494D4D4Prwwu//0VXHIL I1j1HQIJIG E8 <<: % qLG  7 3 LLJ &%%djjm4@!!$))R0F:!11$**Q-@M!**=8LQQSF6<< A%LLEyyM)(3F;*4995lC &tyy1(; ,,K'1R'7[^J#A*z9A=H"<tAw<3|#44 1#l2C1DIcRVWXRYl^\ 4 d1gt $ MM$QUVWQXQ^Q^Q`a~tXx'9 :ab c Q , MM$SPTUVPWyaf=gh cs hi j MM$s.find_params~sLL ^^FNN$41=Fhhv||A'zz"&&a0#6zz"&&a0#6 U?::bffTlD1D(Cvvc{!#YJS qyTQY'S S $ U?775;;'4!84q8D88TEEM*D 77FLLO,-e >%A,Q#GX\cdIef fIDAJ  "R%( "Q;r"uax "#q& RYYtBCBJ/66 ;7>**T" gqjM 1aggA HrXct|}|j)tjj |jnd}t |||| \}}~t j|jjj}|j|Dcgc]}|jc}g}|jD]}|jdvs|j|jidk7s2|j|jijdddk(sb|j!|j"dt%t'|}|j)||j*r1t-j.|j|jdzd d d |j*s0t1j2|jj5|| n$t1j2|jdz|| }t7|D]\}}t9t;||d zg}g}|j<|D].}|jdvs|j|jidk7s3|j|jijdddk(sc|j?|j"d tAjB|j?|jE|jj"d |j }t;|jFdk7r|j!||j!|jE|j1t;|dk(rt|Dcgc]3}tIjJ|jFd|jFdf5}}d}|D]}|jM|g|d}|jFd}tIjN|d|jFdf}|Dcgc] }||||zz z}}||z }tIjPd|z |z}|Dcgc]%}|tIjR|jT|z'}}tW|||d D]%\}}} |j|vr6||jd}||jd}||jd}|dk7r|n|jFd}|jX}!t[|| |||||| | |  }"|j?|j"d }#|j]|j"d }$|dk(}%|%r|jF}&|&d|zd z |z}'t_|"||'}"ta|"jT|||d\}"}(})tc||&|||'|"jed|(je|!|dk(r|)nd|  \}*}+|jg|+|ji||jk|*nt,jljo|j"d d|d|ztq|!|"jF|"je|!jsd },|ju|,|,j|j"d <|$d k(s|jw|#(|j||jjjjy||j{|j*rCddl>m?}-|-|jtjj|jd|Scc}wcc}wcc}wcc}w)aQuant the model with GPTQ method. Args: model (ModelProto or ONNXModel): onnx model dataloader (object): dataloader for calibration. weight_config (dict): quantization config For example, weight_config = { 'fc2': { 'bits': 4, 'group_size': 32, 'scheme': 'sym', 'algorithm': 'GPTQ' } } num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". n_samples (int, optional): calibration sample number. percdamp (float, optional): percent of the average Hessian diagonal to use for dampening. blocksize (int, optional): blocksize to quantize weight. actorder (bool, optional): whether rearrange Hessian matrix considering the diag's value. mse (bool, optional): whether get scale and zero point with mse error. perchannel (bool, optional): whether quantize weight per-channel. accuracy_level (int): accuracy level. Support 0 (unset), 1(fp32), 2(fp16), 3(bf16), or 4(int8). providers (list): providers to use Returns: model: fake quantized ONNXModel NrrrrGPTQrrTFrrrrrrr"rAru)r@rArur8r9r:r)r+rr\r rZrrrr)load_external_data_for_model)Arrrrrrrr3rgraphr<remove_tensors_from_outputsrrrLrr5r+listsetadd_tensors_to_outputsrr1rrrrrrrinput_name_to_nodesrrrget_noder*r(r)runr.rr5rrr rMrrr~rWr6r remove_nodeadd_noder2r3rr4rr MergeFromronnx.external_data_helperrPsplit).rr rr@rArur r9r8r:r)r+r$rrr%r  org_outputr} output_namesr>r rP input_name node_listrrHsnsamplesrtrrr7r rCrrrrrBrDrrrJrTrPs. rV gptq_quantizerds^ e E494D4D4Prwwu//0VXHy*iHJFBu{{00778J %%z&B!qvv&BCL / LLJ &!!$))R0F:!!$))R044[&IVS    1 . /L)*L   .   KK    ."&$(#  ## U[[::!%%dii488fMQWW))$**Q-8D%..))%..*C*I*I!*LMx$&v||$)v&   !:; < w<1  :A BQbhh AGGAJ/0 B B 9D++zlD1!4C))A,C**S2syy}"56C=?@!x8c>23@B@ OH''!h,'#-C578!biis++8B8 9GR 6 = 8   yyM)(3F;*4995lC &tyy1(;'1R'7V\\!_JLLE!%#!!% H"11$**Q-@M"<rms0%   0!%   . /p.f3'lF'RXDv8&4   %&tn6x    I ^     %&|rX