o m9:jX$@sdZddlZddlmZddlmZmZddlZee Z gdZ de e dBde e fdd Zed d d e defd dZGdddeZejjdid       dFdejdejdejdejdejdBde de dededBde e dBdedejdBdejdBd e dBdeejejejffd!d"Zej       dFdejdejdejdejdejdBde de dededBde e dBdedejdBdejdBd e dBdeejejejffd#d$Zddd%ddddd&dejdejdejdejdejdBde de d'edBdedBdee e fdedejdBdejdBd e dBdejeejejfBfd(d)Zejjd*d+hd       dFd+ejdejdejdejdejdejdBde de dededBde e dBdedejdBdejdBd e dBdejf d,d-Zej       dFd+ejdejdejdejdejdejdBde de dededBde e dBdedejdBdejdBd e dBdejf d.d/Zddd%ddddd&d+ejdejdejdejdejdejdBde de d'edBdedBdee e fdedejdBdejdBd e dBdejeejejfBf d0d1Zd2ed3eed4fd5eddfd6d7Zejjd8id  dGd9ejdejdejdejd+ejd:ejdejdejde de ded;ejdedBde e dBdeejejejffdd?Z d2ed9ejd@ejdAejdeejdBd4ff dBdCZ!ej"e!edDej#$ej%j&j'ddEl(m)Z)m*Z*m+Z+m,Z,e*e,ej%j-j<e+e,ej%j-j<e)e,ej%j-j<dS)Hz Variable-length attention implementation using Flash Attention. This module provides a high-level Python interface for variable-length attention that calls into the optimized Flash Attention kernels. N) lru_cache)Any NamedTuple) varlen_attnvarlen_attn_out AuxRequest window_sizereturncCs2|durddg}t|dkrtdt||S)Nz$window_size must have length 2, got )len ValueError)rr`/home/nk/hobo-godmode/plappi-mvp/.venv/lib/python3.10/site-packages/torch/nn/attention/varlen.py_normalize_window_sizes  r)maxsize device_indexcCsdS)z;Cache device capability check to avoid repeated CUDA calls.Fr)rrrr_should_use_cudnnsrc@seZdZUdZdZeed<dS)rz Request which auxiliary outputs to compute from varlen_attn. Each field is a boolean indicating whether that auxiliary output should be computed. FlseN)__name__ __module__ __qualname____doc__rbool__annotations__rrrrr#s rztorch_attn::_varlen_attn) mutates_argsFquerykeyvaluecu_seq_qcu_seq_kmax_qmax_k is_causalscale enable_gqa seqused_k block_table num_splitscCs*t| } |jo t|jj}|ratd| rtd| dur"td| ddks.| ddkr2td| dus:| dur>td tj j j |||d||||d d |d |d }|d|d|d}}}n%tdtj j j |||||||d |d || d| d| | | d\}}}}}tj dtj|jd}|||fS)z Private custom op for variable-length attention. This is the internal implementation. Users should use the public varlen_attn function instead. #Using cuDNN backend for varlen_attnz,GQA is not supported with the cuDNN backend.Nz3num_splits is not supported with the cuDNN backend.rr TcuDNN backend does not support window attention. Please use Flash Attention backend.zBseqused_k/block_table is not yet supported with the cuDNN backend.TFr%-Using Flash Attention backend for varlen_attn)return_debug_maskr%window_size_leftwindow_size_rightr'r(r)r dtypedevice)ris_cudarr7indexloginfo RuntimeErrortorchopsaten_cudnn_attention_forward_flash_attention_forwardzerosuint64)rrrr r!r"r#r$r%rr&r'r(r) use_cudnnresultoutput softmax_lse rng_state_ rng_state_rrr _varlen_attn-sn    rKcCst| } t|}|d}|d}tj||ftj|jd}tjjrCtj }|tj j j krC|dd}tj|||ftj|jd}tjdtj |jd}|||fS)z Fake implementation for meta tensor computation and tracing. Based on the 3D varlen path from meta__flash_attention_forward: - query shape: (total, num_heads, head_dim) - logsumexp shape: (num_heads, total_q) rr+r5r4)rr= empty_likesizeemptyfloatr7versionhip_C_get_rocm_fa_preferred_backend_ROCmFABackendAOTritonrC)rrrr r!r"r#r$r%rr&r'r(r)rFtotal_q num_heads logsumexp preferred batch_sizerHrrr_varlen_attn_fakes      r[)r r ) return_auxr%rr&r'r(r)r\cCs|d}| dur|dn|d}| s$||kr$td|d|d| r7||dkr7td|d|d | d k}tjj|||||||||t| | | | | \}}}|dur_|jr_||fS|S) aCompute variable-length attention using Flash Attention. This function is similar to scaled_dot_product_attention but optimized for variable-length sequences using cumulative sequence position tensors. Args: query (Tensor): Query tensor; shape :math:`(T_q, H_q, D)` key (Tensor): Key tensor; shape :math:`(T_k, H_{kv}, D)`, or :math:`(\text{total\_pages}, \text{page\_size}, H_{kv}, D)` when ``block_table`` is provided. value (Tensor): Value tensor; shape :math:`(T_k, H_{kv}, D)`, or :math:`(\text{total\_pages}, \text{page\_size}, H_{kv}, D)` when ``block_table`` is provided. cu_seq_q (Tensor): Cumulative sequence positions for queries; shape :math:`(N+1,)` cu_seq_k (Tensor): Cumulative sequence positions for keys/values; shape :math:`(N+1,)` max_q (int): Maximum query sequence length in the batch. max_k (int): Maximum key/value sequence length in the batch. return_aux (Optional[AuxRequest]): If not None and ``return_aux.lse`` is True, also returns the logsumexp tensor. scale (float, optional): Scaling factor for attention scores window_size (tuple[int, int], optional): Window size for sliding window attention as (left, right). Use (-1, -1) for full attention (default), (-1, 0) for causal attention, or (W, 0) for causal attention with sliding window of size W. enable_gqa (bool): If set to True, enables Grouped Query Attention (GQA) and allows key/value to have fewer heads than query. Each KV head is shared by a group of :math:`H_q / H_{kv}` query heads, so :math:`H_q` must be divisible by :math:`H_{kv}`. Default is False. seqused_k (Tensor, optional): Number of valid KV tokens per batch element; shape :math:`(N,)`. When set, only the first ``seqused_k[i]`` tokens in the key/value sequence for batch element *i* participate in attention. Useful for KV-cache decoding where the cache slot is larger than the actual sequence. Inference-only (not supported in backward). block_table (Tensor, optional): Block table for paged KV cache; shape :math:`(N, \text{max\_pages\_per\_seq})`, dtype ``int32``. Requires ``seqused_k``. Inference-only (not supported in backward). When ``block_table`` is provided, ``key`` and ``value`` are a "pool" of pages of tokens of KV data and the pages belong to any sequence/order. The ``block_table`` is what maps each sequence's logical chunks back to physical pages in this pool. ``seqused_k[i]`` tells the kernel how many tokens in sequence *i* are actually valid, since the last page is typically only partially filled. num_splits (int, optional): Number of splits for split-KV. Set to ``1`` to disable split-KV which enables batch invariance. Split-KV parallelizes the key/value sequence dimension across multiple thread blocks and combines partial results. The split decision depends on ``max_k`` (the longest sequence in the batch), so different batch compositions can change the reduction order and produce different floating-point results for the same sequence. When this is disabled, bitwise identical outputs are guaranteed for a given sequence regardless of what other sequences are in the batch, at the cost of lower GPU utilization when there are few queries. When ``None`` (default), the kernel chooses automatically. Returns: output (Tensor): Output tensor from attention computation; shape :math:`(T_q, H_q, D)`. If ``return_aux`` is not None and ``return_aux.lse`` is True: lse (Tensor): Log-sum-exp of attention scores; shape :math:`(T_q, H_q)`. Shape legend: - :math:`N`: Batch size - :math:`T_q`: Total number of query tokens in the batch (sum of all query sequence lengths) - :math:`T_k`: Total number of key/value tokens in the batch (sum of all key/value sequence lengths) - :math:`H_q`: Number of query attention heads - :math:`H_{kv}`: Number of key/value attention heads (equal to :math:`H_q` unless GQA is enabled) - :math:`D`: Head dimension Example:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> batch_size, max_seq_len, embed_dim, num_heads = 2, 512, 1024, 16 >>> head_dim = embed_dim // num_heads >>> seq_lengths = [] >>> for _ in range(batch_size): ... length = torch.randint(1, max_seq_len // 64 + 1, (1,)).item() * 64 ... seq_lengths.append(min(length, max_seq_len)) >>> seq_lengths = torch.tensor(seq_lengths, device="cuda") >>> total_tokens = seq_lengths.sum().item() >>> >>> # Create packed query, key, value tensors >>> query = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> key = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> value = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> >>> # Build cumulative sequence tensor >>> cu_seq = torch.zeros(batch_size + 1, device="cuda", dtype=torch.int32) >>> cu_seq[1:] = seq_lengths.cumsum(0) >>> max_len = seq_lengths.max().item() >>> >>> # Call varlen_attn >>> output = varlen_attn( ... query, key, value, cu_seq, cu_seq, max_len, max_len ... ) r+Nr GExpect query and key/value to have the same number of heads but got Hq= and Hkv=&. Try setting enable_gqa=True for GQA.rMExpect number of query heads to be a multiple of kv heads for GQA but got Hq=.r r)rMr r=r> torch_attnrKlistr)rrrr r!r"r#r\r%rr&r'r(r) num_heads_q num_heads_kr$outrrIrrrrsL u  rztorch_attn::_varlen_attn_outrgcCslt| } |jo t|jj}|rtdtdtj j j ||||||||d|d| | d| d| | |d}|S)z Private custom op for variable-length attention with pre-allocated output. Same as _varlen_attn but writes the attention output into the provided out tensor. z+cuDNN backend does not support out variant.z1Using Flash Attention backend for varlen_attn_outr-Frr+)r%r2r3r'r(r)) rr8rr7r9r<r:r;r=r>r?+_flash_attention_forward_no_dropout_inplace)rgrrrr r!r"r#r$r%rr&r'r(r)rDrGrrr_varlen_attn_outRs2 ricCsx|d}|d}tj||ftj|jd}tjjr:tj}|tjj j kr:|dd}tj|||ftj|jd}|S)F Fake implementation for meta tensor computation and tracing. rr+r5) rMr=rNrOr7rPrQrRrSrTrU)rgrrrr r!r"r#r$r%rr&r'r(r)rVrWrXrYrZrrr_varlen_attn_out_fakes   rkcCs|d}| dur|dn|d}| s$||kr$td|d|d| r7||dkr7td|d|d | d k}tjj|||||||||| t| | | | |}|dur]|jr]||fS|S) zCompute variable-length attention using Flash Attention with a pre-allocated output tensor. Same as :func:`varlen_attn` but writes the attention output into the provided ``out`` tensor instead of allocating a new one. r+Nr r]r^r_rr`rarb)rMr r=r>rcrirdr)rgrrrr r!r"r#r\r%rr&r'r(r)rerfr$rrrrrsN  rctxinputs.rFcCs|\}}}}}}} } } } } }}}|\}}}|durtd|dur%td|||||||||||_| |_| |_| |_| |_dS)Nz)seqused_k is an inference-only parameter.z+block_table is an inference-only parameter.)r<save_for_backwardr"r#r$r%r)rlrmrFrrrr r!r"r#r$r%rr&r'r(r)rgrrHrrr_setup_contexts6  roz!torch_attn::_varlen_attn_backwardgrad_outrrHcCst| } tjd|jd}|jot|jj}|rFtd| ddks(| ddkr,t dtj j j |||||||||| d| | || d\}}}n$td tj j j |||||||||| d| | || | d| dd \}}}|||fS) Nr)r7r*r r+r,r-r.r0)r%r2r3)rr=rNr7r8rr9r:r;r<r>r?_cudnn_attention_backward_flash_attention_backward)rprrrrgrr r!r"r#r$rHr%runusedrDdqdkdvrrr_varlen_attn_backward s^   rwcCs0t| } t|}t|}t|}|||fS)rj)rr=rL)rprrrrgrr r!r"r#r$rHr%r grad_querygrad_key grad_valuerrr_varlen_attn_backward_fakeQs     r{grad_lsegrad_rngcCsz|j\}}}}}} } } |j} |j} |j}|j}|j}tjj ||||| | ||| | || ||\}}}d}|||gd|RS)N )N) saved_tensorsr"r#r$r%rr=r>rcrw)rlrpr|r}rrrr r!rgrrHr"r#r$r%rrtrurv num_paramsrrr _backwardns0 r) setup_context)_varlen_attn_backward_flop_varlen_attn_forward_flop_varlen_attn_out_flop flop_registry)FNNFNNN)NN).rlogging functoolsrtypingrrr= getLoggerrr:__all__rdintrrrrlibrary custom_opTensorrOtuplerK register_faker[rrirkrrorwr{rregister_autograd_dynamodisallow_in_graphr>r?rhtorch.utils.flop_counterrrrrrcrrrrs           Y       :             5       /     "=!       D        !