o r::j@s>ddlmZddlZddlmZd d d ZGd d d eZdS)) annotationsN)OpRunx np.ndarray num_headsintreturncCsZ|j\}}}||dkrtd|d|d|jd||}|||||ddddS) z$Reshape (B, T, H*D) -> (B, H, T, D).rz Last dim z not divisible by num_heads z for shape .)shape ValueErrorreshape transpose)rrbthddrm/home/nk/hobo-godmode/plappi-mvp/.venv/lib/python3.10/site-packages/onnx/reference/ops/op_linear_attention.py_unpack_3d_to_4d s  rc@s&eZdZ        dddZdS)LinearAttentionNc + Cs| durd} | dvrtd| d| dus|durtd| dks,|dks,| |dkr7td| d|d | d v} | d v} | rM|durMtd | d | s[|dur[td | d| ri|duritd | d| sw|durwtd | dd|fd|fd|ffD]\}}|jdkrt|d|jdq|j}|j\}}}|jd| }|jd|}| |}t|| tj}t||tj}t||tj}|dur|jdkrtd|jd|jd}||kr||||d dddd}n|||krt||}ntd|d|d||d |tj}|durZ|jdkr/td|jd|jd}||dfvrFtd|d |d!||||d dddd}|tj}|dur|j||||fkr~td"|jd#|d$|d$|d$|d |j}|tj }n|}tj ||||ftjd%}| dus| d&krd't |} nt | } tj || ||ftjd%}!t|D]}"|dddd|"ddf}#|dddd|"ddf}$|dddd|"ddf}%| r|dddd|"ddf}&|t|&d(}| r td)||$}'|dddd|"ddf|%|'}%||$d*dddf|%d*dddf}|dkr<|}(ntj||dd+}(| td,|#|(|!dddd|"ddf<q|! dddd||| |})|)|})||}*|)|*fS)-N gated_delta)lineargateddeltarzUnsupported update_rule 'z>'. Expected one of: 'linear', 'gated', 'delta', 'gated_delta'.z5q_num_heads and kv_num_heads are required attributes.rz q_num_heads (z/) must be a positive multiple of kv_num_heads (z).)rr)rrz update_rule 'z' requires decay input.z' forbids decay input.z' requires beta input.z' forbids beta input.querykeyvaluer z' must be rank 3 (B, T, H*D), got shape r z decay must be rank 3, got shape r r zdecay last dim z must equal kv_num_heads (z) or kv_num_heads*d_k (zbeta must be rank 3, got shape zbeta last dim z must be kv_num_heads (z) or 1.zpast_state shape z does not match (z, )dtypegg?).Nz bhdm,bhd->bhm.)axisz bhd,bhdm->bhm)rndimr r!rastypenpfloat32rrcopyzerossqrtfloatrangeexpeinsumrepeat)+selfrrr past_statedecaybeta chunk_size kv_num_heads q_num_headsscale update_rulegatingdelta_correctionnamearr out_dtyperr_d_kd_v group_sizeq4k4v4 decay_lastdecay4 beta_lastbeta4state_in_dtypestate scale_valoutputsiq_tk_tv_tg_t retrievedstate_for_readoutput present_staterrr_runs                   $( "  zLinearAttention._run)NNNNNNNN)__name__ __module__ __qualname__rUrrrrrsr)rrrrrr) __future__rnumpyr%onnx.reference.op_runrrrrrrrs