o m9:jM#@s(UdZddlZddlmZddlmZddlmZddlZddl m Z edZ edZ iZ eejjefed <ehd Zd ed ed ee e fdeee e fgee e fffddZ dCdddddejdejdejdejdBdedededejfddZedde dCdddddejdejdejdejdBdedededejfddZd edBd!edefd"d#Zd$ejd%ededejfd&d'Zd(ejd)ejd*ed+ed edBd,edejfd-d.Zd*ed+eddfd/d0Zd(ejd)ejd*ed+ed edBdejf d1d2Z   dDdddddd3dd4d(ejd)ejd5ejd6ejdBd7ejdBd8ejdBd9ed:ed;ed,ed edBdd?Z"ed@de"   dDdddddd3dd4d(ejd)ejd5ejd6ejdBd7ejdBd8ejdBd9ed:ed;ed,ed edBd op_type opset_version fake_implreturncs.dtttfdtttfffdd }|S)zDDecorator to register an ONNX operator with a custom implementation.funcrcsPd}tjjdd|dd|}|ttttjj|<||S)Nopsetzonnx::.) mutates_args)torchlibrary custom_oprgetattropsonnx register_fake)roverloadtorch_oprr rr[/home/nk/hobo-godmode/plappi-mvp/.venv/lib/python3.10/site-packages/torch/onnx/ops/_impl.py decorator's  z_onnx_op..decorator)rrr)r rrr!rrr _onnx_op"s* r"F) interleaved num_headsrotary_embedding_dimx cos_cache sin_cache position_idsr#r$r%cCs|S)zFFake implementation for RotaryEmbedding-23 for torch.compile purposes.)clone)r&r'r(r)r#r$r%rrr _rotary_embedding_23_fake_impl5s r+RotaryEmbeddingcs|jt}dddurWtdkfddtjdkfddtjdkfd dtdkoNdkfd dntd kodd kfd d|d krwt|d}n$|d krt|dkfddd}||} || g} t|| }tt|jd kdd|jd } |dkr| }|ddddddd|f} |dddddd|df} |ddurntjdkojdkfddtjdkojdkfddtjdkfddtjdkfddtdtd|rk| dddddddddf} | dddddddddf}n tj| ddd\} }| |}| |}|rt|d}t|d}tj ||fdd}t|| j} n tj ||fdd} tj | | fdd}|d krt|St|dS)z_RotaryEmbedding-23 https://onnx.ai/onnx/operators/onnx__RotaryEmbedding.html#rotaryembedding-23rNcs djS)Nz6position_ids must be 2D when provided. Received shape shaper)r)rr Zs z%rotary_embedding_23..csddjdS)Nz6position_ids first dim (batch) must match x.shape[0] ( ). Received rr0r) batch_sizer)rr r2^r csddjdS)Nz;position_ids second dim (sequence) must match x.shape[-2] (r3r r0r)r)sequence_lengthrr r2br5cdjdjS)NzWcos_cache/sin_cache must be 2D when position_ids is provided. Received cos_cache shape , sin_cache shape r0rr'r(rr r2f cr7)Nz[cos_cache/sin_cache must be 3D when position_ids is not provided. Received cos_cache shape r8r0rr9rr r2lr:)rr/r r;cs dS)NzKnum_heads must be provided for 3D inputs. Received input tensor with shape rr) input_shaperr r2ys cSdS)Nzx should be a 4D tensor by nowrrrrr r2csdjdddS)Nzcos has shape  but expected (batch=, seq=, ...)r0r)r4cosr6rr r2csdjdddS)Nzsin has shape r@rArBr0r)r4r6sinrr r2rDcsdjdddS)NzLast dimension of cos cache (rF') should match rotary_embedding_dim/2 ().r0r)rCrotary_embedding_dim_halfrr r2csdjdddS)NzLast dimension of sin cache (rFrGrHr0r)rIrErr r2rJdim) r1lenr_checkrLpermutereshape unsqueezechunkcat)r&r'r(r)r#r$r% input_rank hidden_size head_size new_shapex_rotate x_not_rotatex1x2realimagx_rotate_concatoutputr) r4rCr'r=r)rIr6rEr(r rotary_embedding_23Cs                  "$     r`scalerVcCs|dur|Sdt|S)z/Get the scale factor for attention computation.Ng?)mathsqrt)rarVrrr _get_scale_factorsrdtensorr4cCs:|jd|jd}}||}|||||ddS)z1Reshape 3D tensor to 4D for multi-head attention.r r/)r1view transpose contiguous)rer4r$r6rUrVrrr _reshape_3d_to_4ds riQKcurrent_q_num_headscurrent_kv_num_headsqk_matmul_output_modec Cs2|dkr t|||||Stt||ddS)z1Get QK output tensor based on the specified mode.rr.rF)_compute_qk_output_for_mode_0r zeros_likematmulrg)rjrkrlrmrarnrrr _get_qk_output_for_aten_spdas   rrcs"tdkfdddS)z-Validate Group Query Attention configuration.rcsdddS)Nz q_num_heads (z%) must be divisible by kv_num_heads (z ) for GQArrrmrlrr r2sz-_validate_gqa_configuration..N)rrN)rlrmrrsr _validate_gqa_configurations  rtc Cs`|}||kr||}|j|dd}t||jd}t|}||} ||} t| | ddS)zDHelper function to compute QK output for qk_matmul_output_mode == 0.r rKr;r.rF)repeat_interleaverdr1rbrcrrqrg) rjrkrlrmraK_for_qk repeat_factor scale_factor sqrt_scaleQ_scaledK_scaledrrr ros  ro) is_causal kv_num_heads q_num_headsrnrasoftcapsoftmax_precisionV attn_maskpast_key past_valuer}r~rrrcCsZ|jd} t|jdkrE|jd}|j}|dur,| ||jd|jd|jd|f}n| ||jd|jd|f}|}| |||df}n:|jd}|j}|duri|jd|jd|jd|jd|jdf}n|j}|}|jd|jd|jd|df}tj||j|jd}tj||j|jd}tj||j|jd}tj||j|jd}||||fS)z@Fake implementation for Attention-23 for torch.compile purposes.rr;r Nr/dtypedevice)r1rMremptyrr)rjrkrrrrr}r~rrnrarrr4q_sequence_length output_shapepresent_key_shapepresent_value_shapeqk_output_shaper_ present_key present_value qk_outputrrr _attention_23_fake_implsR      r Attentionc' Csd\} }}t|j}|jd}t|jdkr;t|dko|dkdd|jd}t|||}t|||}t|||}tt|jdkoQt|jdkoQt|jdkdd|j|}t| |} |d urmtj||g|d n|}|d ur~tj||g|d n|}||}}|j| }|j| }|j|}|j|}| d ko| dko| d uo|d up|jtj k}t |||rtj j j ||||d || t ||kd }t||||| | }n||kr||}|j|| d }|j|| d }tj|||j|jd }|rt|d uddttj||tj |jd }||td}|d ur4|jtj kr0||td}n||}t| |jd}t|} || }!|| }"t|!|"dd}#|#}|#|}$| dkr`|$}| d krn| t|$| }$| dkru|$}| d ur| tvr|$j}%|$tj| }$tj|$dd }&|&|%}&ntj|$dd }&ntj|$dd }&| dkr|&}t|&|}|dkr|dd !||d}||||fS)zMAttention-23 https://onnx.ai/onnx/operators/onnx__Attention.html#attention-23)r r/r;rr;cSr>)Nz;q_num_heads and kv_num_heads must be provided for 3D inputsrrrrr r2r?zattention_23..r r<cSr>)Nz'Q, K, and V should be 4D tensors by nowrrrrr r2r?NrKr|)r dropout_pr}ra enable_gqarcSr>)Nz'Cannot use both is_causal and attn_maskrrrrr r2r?z-infr.rFr/)"rMr1rrNrirdrSr*rboolrtnn functionalscaled_dot_product_attentionrrruzerosrtrilones masked_fillfloatrbrcrqrgtanh-_ATTENTION_23_ALLOWED_INTERMEDIATE_PRECISIONStorONNX_DTYPE_TO_TORCH_DTYPEsoftmaxrhrf)'rjrkrrrrr}r~rrnrarr num_head_dim sequence_dimhead_diminput_shape_lenr4r q_head_sizerrrlrmkv_sequence_length can_use_sdpar_rrw attn_bias causal_maskrxryrzr{qk_matmul_output qk_with_biasoriginal_dtype qk_softmaxrrr attention_23cs       (                        r)N)NNN)$__doc__rbcollections.abcrtypingrtyping_extensionsrrtorch.onnx.opsrrrrdict_ops OpOverload__annotations__ frozensetrstrintr"Tensorrr+r`rrdrirrrtrotuplerrrrrr s                     T