o m9:jA9 @sUdZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z m Z dd lmZe r7dd lmZddlZdd lmZd gZdad ed<eGdddZedoddZ dpdqdd ZedrddZdsdd Z !dtdud*d+Z  dvdwd7d8Zdxd_sz,_fa4_common_support_error..zinputs must be CUDA tensorscSsh|]}|jqSr)rrJrrr asz,_fa4_common_support_error..r inputs must share devicez'query dtype must be float16 or bfloat16z dtype must be float32zdense query must be 4Dzragged query must be 3DzCUDA not available) z+FA4 requires compute capability 9.0 or 10.0) alllendtyper float16bfloat16float32dimr! is_availabler%r)rArCrErGnametensorrrr_fa4_common_support_errorYs&   r^keyvalue dropout_pfloatreturn_debug_maskbool alibi_slopes seqused_k block_table num_splits int | Nonec Cs|dkrdS|r dS|durdS|dur!|jtjkrdS|js!dSt|j} |dur4| dkr4d| d S| durF| d krF| dkrFd | d St||||f|} | dur[| d krYd S| SdS)Ndropout_p must be 0zreturn_debug_mask must be Falsezalibi_slopes not supportedzseqused_k must be int32zseqused_k must be CUDArSz+paged KV (block_table) not supported on SM 0r z-SplitKV (num_splits > 1) not supported on SM rOz(query, key, value must be on same device)rVr int32rIr%rr^) rAr_r`rarcrerfrErgrhr#errorrrr_fa4_forward_support_errorss4     rograd_outout logsumexpc Cs>|dkrdSt|||||||f|d|ffd}|dur|SdS)Nrjrkrr)rG)r^) rprAr_r`rqrrrarErnrrr_fa4_backward_support_errors rsvalcCs|dkrdS|S)z"need to convert -1 to None for FA4Nr)rtrrr_aten_to_fa4_window_sizesrvTs Unpack[Ts]tuple[Unpack[Ts]]cGstdd|DS)Ncss|] }|ddVqdS)r N) transposerJrrrrMsz#_transpose_dense..)tuple)rCrrr_transpose_densesr}cu_seq_qcu_seq_kmax_qmax_kscale float | None is_causalwindow_size_leftwindow_size_right!tuple[torch.Tensor, torch.Tensor]cCsztdurtdtt}||t| t| d||||| dur!| nd| |p&d| d }|j|||fi|\} }| |fS)NFA4 not registeredTr ) softmax_scalecausalrr return_lse cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_krf page_tablerhrq)rr0r)rv contiguousr+)rAr_r`r~rrrrrrrrfrqrgrhr1kwargslserrr_fa4_run_forwards& rF deterministic/tuple[torch.Tensor, torch.Tensor, torch.Tensor]c CsXtdurtdtt} | j|||||||| t| t| ||| d \}}}|||fS)Nr)rrrrrrr)rr0r)r,rrv)rprAr_r`rqrrr~rrrrrrr1dqdkdvrrr_fa4_run_backwards& rT rrrrfrerqrgcompute_auxiliaryrh cum_seq_krc Cst||||| || ||| }|durtd|t|||||||| || | | |||\}}|rMtjdtj|jd}tjdtj|jd}tjd|j|jd}nd}d}d}|||||fS)Nz)FA4 flash_attention forward unsupported: )rz)rVrrr) ror0rr zerosuint64remptyrV)rAr_r`rErrrrarrcrrrrfrerqrgrrhrnr rng_state philox_offset debug_maskrrrr; sN r;)rrrrfrergrhc Cs<t||||||||| | | | | ||||d|d\}}}}}|S)NFr)r;)rqrAr_r`rErrrrarrcrrrrfrergrhr$rrrrr<Js,r<)rrrrunusedcCsdt||||||| |}|durtd|t}t|||||||||| ||| \}}}|||fS)Nz*FA4 flash_attention backward unsupported: )rsr0r $are_deterministic_algorithms_enabledr)rprAr_r`rqrrrErrrrarrrrrrrnrrrrrrrr=ws:  r=rjrcCst|||||ddd}|durtd|t|||\}} } t|} | dd} |d} | d}t|| | dd| |||||| d \}}}}}|d}|d}| |dd|||||f S)NzFA4 SDPA forward unsupported: r rz)rrq)ror0r}r empty_liker{sizer;)rAr_r`rarrcrrnqkvout_bhsdout_bshd max_q_flash max_k_flashr$rrrrrrrrrr>sV        r> philox_seedrcCst||||||| d}|durtd|t|||||\}}}}}|d}|d} t||||||dd|| | | | | |d\}}}t|||\}}}|||fS)NzFA4 SDPA backward unsupported: rzr)rsr0r}rr=)rprAr_r`rqrrrErrrrarrrrrnrrrogorrrrrrr?sD     r?FA4) register_fn)rrrr)r&)r'r(rr)r'r(rr )rr )r) rArBrCrDrErFrGrHrr)NN)rArBr_rBr`rBrarbrcrdrerFrfrFrErFrgrFrhrirr)rprBrArBr_rBr`rBrqrBrrrBrarbrErFrr)rtrirri)rCrxrry)NNN) rArBr_rBr`rBr~rFrrFrrirrirrrrdrrirrirfrFrqrFrgrFrhrirr)F)rprBrArBr_rBr`rBrqrBrrrBr~rFrrFrrrrdrrirrirrdrr)&rArBr_rBr`rBrErFrrFrrrrrarbrrdrcrdrrrrirrirfrFrerFrqrFrgrFrrdrhri)$rqrBrArBr_rBr`rBrErFrrFrrrrrarbrrdrcrdrrrrirrirfrFrerFrgrFrhri)"rprBrArBr_rBr`rBrqrBrrrBrErFrrFrrrrrarbrrdrrBrrBrrrrirri)rjFF)rArBr_rBr`rBrarbrrdrcrdrr)rprBrArBr_rBr`rBrqrBrrrBrErFrrFrrrrrarbrrdrrBrrBrr)+__doc__ __future__rr- dataclassesr functoolsrtypingrrtyping_extensionsrrr typesr r torch.libraryr __all__rrrr%r r)r*r^rorsrvrwr}rrr;r<r=r>r?register_flash_attention_implrrrrs            # (   5 0M=7M6