o m9:j @sddlZddlmZddlmZddlmZddlZddlm Z m Z ddl m Z m Z mZmZmZmZmZmZmZmZmZmZmZgdZedd ZGd d d ejZd ejfd ejdededefddZ GdddeZ!GdddeZ"d a#ddZ$d a%ddZ&dS)N) namedtuple)Callable)Any))sparse_semi_structured_from_dense_cutlass'sparse_semi_structured_to_dense_cutlass) fallback_dispatchersemi_sparse_addmmsemi_sparse_clonesemi_sparse_detachsemi_sparse_indicessemi_sparse_linearsemi_sparse_mmsemi_sparse_scaled_mm semi_sparse_tsemi_sparse_tosemi_sparse_to_copysemi_sparse_valuessemi_sparse_view)SparseSemiStructuredTensor!SparseSemiStructuredTensorCUTLASS$SparseSemiStructuredTensorCUSPARSELTto_sparse_semi_structured_SEMI_STRUCTURED_SPARSE_CONFIGz=sparse_min_rows sparse_min_cols dense_min_rows dense_min_colsc@seZdZUdZdZeed<eej e fed<dZ e ed<dZ e ed<dZe ed<eed <eeefed <ejd Bed <ejd Bed <ejd Bed<ejd Bed<ejd Bed<e ed<eed<gdZe   d1dejd ejd Bd ejd Bdejd Bdejd Bdejd Bde dede fddZdefddZdeeeeeje ee fffddZedeeje ee fdejfddZejjZedefd d!Z ed2d3d"d#Z!ed$ejdd fd%d&Z"d'd(Z#eefd$ejd)eddfd*d+Z$d d,d-ejd.ejd Bdejfd/d0Z%d S)4ra This class implements semi-structured sparsity as a Tensor subclass. Semi-structured sparsity describes a sparsity pattern where n in every 2n elements are sparse, depending on the datatype. It is also referred to as 2:4 sparsity or fine-grained structured sparsity. There are two backends available for semi_structred sparsity, either cuSPARSELt or CUTLASS. This class is meant to serve as a base class for both implementations. SparseSemiStructuredCUTLASS and SparseSemiStructuredCUSPARSELT both inherit from this class and define three backend-specific items. Note that as such, this class cannot be instantiated directly. -`_DTYPE_SHAPE_CONSTRAINTS` - A dictionary holding backend specific dense/sparse min shape constraints - `def from_dense()` - backend specific compression routines - `def _mm()` - backend specific mm op (either torch._cslt_sparse_mm or torch._sparse_semi_structured_(mm|addmm)) r_DEFAULT_ALG_ID_DTYPE_SHAPE_CONSTRAINTSF_FORCE_CUTLASS_FUSE_TRANSPOSE_PROTOTYPE_WARNING_SHOWNBACKENDSPARSE_DISPATCHNpackedmetapacked_tmeta_tcompressed_swizzled_bitmaskfuse_transpose_cusparseltalg_id_cusparselt)r r!r"r#r$shape requires_gradc Cs|jstjdtddd|_|tj||dur|} n |dur&|} ntdtj j ||| j | j | j | d} || _|| _|| _|| _|| _|| _|| _| S)a0 Create a new instance of the tensor subclass from the compressed sparse representation. We have the option to create the subclass with the compressed representations of both X and X', for training. For inference, we only need a single representation (either X or X'), while the corresponding other set will be None. Depending on the backend selected, certain fields will be set to None. (CUSPARSELT vs CUTLASS) Args: shape: The shape of the original dense tensor packed: The compressed representation of the original dense tensor meta: The metadata of the original dense tensor, if it is stored separately packed_t: The compressed representation of the transposed original dense tensor meta_t: The metadata of the transposed original dense tensor, if it is stored separately compressed_swizzled_bitmask: The masks used by the CUTLASS backend to determine which threads should participate in the computation. Used for pointwise ops. fuse_transpose_cusparselt: When running with cuSPARSELt, we have the option to fuse a transposition with a matmul, which is useful in the case of 2:4 sparse training. alg_id_cusparselt: The algorithm id to use when using cuSPARSELT, will have effect on performance Returns: torch.Tensor: A torch.Tensor wrapper subclass. Raises: ValueError: If all of the tensor arguments are None. zThe PyTorch API of SparseSemiStructuredTensor is in prototype stage and will change in the near future. Please open a Github issue for features requests and see our documentation on the torch.sparse module for further information about the project. stacklevelTNz3At least one of packed or packed_t must be provided)devicedtypelayoutr()rwarningswarn UserWarning_load_dispatch_tabletorch_dynamoallow_in_graph ValueErrorTensor_make_wrapper_subclassr,r-r.r r!r"r#r$r%r&) clsr'r r!r"r#r$r%r&r(previous_tensortensorr<c/home/nk/hobo-godmode/plappi-mvp/.venv/lib/python3.10/site-packages/torch/sparse/semi_structured.py__new__Os<'   z"SparseSemiStructuredTensor.__new__returncCs(t|ds td|jjd|jdS)Nr'ztensor has no shape attributez(shape=))hasattrAssertionError __class____name__r'selfr<r<r=__repr__s z#SparseSemiStructuredTensor.__repr__cs4ttfddj}jjjjf}||fS)Ncst|duSN)getattr)xrEr<r=sz?SparseSemiStructuredTensor.__tensor_flatten__..)listfilter __slots__r'r%r&r()rF inner_tensors tensor_metar<rEr=__tensor_flatten__sz-SparseSemiStructuredTensor.__tensor_flatten__rPc CsN|\}}}}|||dd|dd|dd|dd|dd|||d S)Nr r!r"r#r$ r'r r!r"r#r$r%r&r()get) r9rOrP outer_size outer_strider'r%r&r(r<r<r=__tensor_unflatten__s     z/SparseSemiStructuredTensor.__tensor_unflatten__cCs:|j|jvrt|jd|jd|j|j||||S)NzI only supports a specific set of operations, can't perform requested op (r@)_overloadpacketrNotImplementedErrorrD)r9functypesargskwargsr<r<r=__torch_dispatch__s z-SparseSemiStructuredTensor.__torch_dispatch__cCst|dddurbtjjjttjjjttjjjt tjjj t tjjj t tjjj ttjjjttjjjttjjjttjjjttjjjttjjjttjjjttjjjttjjjti|_ |durd|j !|dSdSdS)zT Loads the op overload sparse dispatch table for the current class. rN)"rIr3opsatenvaluesrindicesr is_same_sizerdetach_detachr trviewrmmr matmuladdmmrlinearr _to_copyr _scaled_mmrcloner torrupdate)r9custom_dispatch_tabler<r<r=r2s*               z/SparseSemiStructuredTensor._load_dispatch_tableoriginal_tensorcCs|js td|jd|dkrtd|d|s$td|j|jvr6td|jd|d |j\}}|j|jj}|j|jj }||ksY||sY||ksY||rhtd |jd |d |d dS)z_ Assert that the given tensor is valid for semi-structured sparse compression. zError original_tensor.device= z= is not supported! Only CUDA tensors are currently supported.r)zError original_tensor.dim = z; is not supported! Only 2d tensors are currently supported.zXError original_tensor is not contiguous!Only contiguous tensors are currently supported.zError original_tensor.dtype z is not a supported dtype for !zError original_tensor.shape zS is not supported! Both dimensions must be larger or equal than and a multiple of (z, r@N) is_cuda RuntimeErrorr,dim is_contiguousr-rr'sparse_min_rowssparse_min_cols)r9rqmnmin_rowsmin_colsr<r<r= _validate_device_dim_dtype_shapes8      z;SparseSemiStructuredTensor._validate_device_dim_dtype_shapecCs&|jd}t|tj||j|jdS)Nr-r,)r'r3rgeyer-r,)rFcolr<r<r=to_denses z#SparseSemiStructuredTensor.to_densealg_idcCtrHrXr9rqrr<r<r= from_dense#sz%SparseSemiStructuredTensor.from_dense)biasBrcKrrHr)rFrrr\r<r<r=_mm+szSparseSemiStructuredTensor._mm)FrFrH)r?N)&rD __module__ __qualname____doc__rint__annotations__dictr3r-rrboolrrstrrr7rN staticmethodSizer>rGtuplerLrQ classmethodrV_C_disabled_torch_function_impl__torch_function__rr]r2r}rrrr<r<r<r=r*s        T  *  rFrq transposedrr?cCs8|r tjdtddtjrtjjntjj}|j ||dS)a This function converts a dense tensor into a sparse semi-structured tensor. It will return a SparseSemiStructuredTensor, a subclass of torch.Tensor. This function will check to ensure the dense tensor has the right dtype, size, dims, and device. We currently only support semi-structured sparse tensors for 2d CUDA tensors. Additionally, your tensor must be a positive multiple of the minimum sparse block size, given in `_DTYPE_TO_SHAPE_CONSTRAINTS` for each dtype (float32, float16, bfloat16, int8). Args: original_tensor (Tensor): the dense tensor to convert transposed (bool, optional): deprecated arg to be removed in another release. Do not use. alg_id (int, optional): the algorithm id to use for cuSPARSELt matmul. Defaults to 0. Can be obtained via ``torch._cslt_sparse_mm_search``. Returns: SparseSemiStructuredTensor: A sparse semi-structured tensor created from the given original_tensor Raises: None Example: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> A = torch.Tensor([0, 0, 1, 1]).tile((128, 32)).half().cuda() tensor([[0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], ..., [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.]], device='cuda:0', dtype=torch.float16) >>> A_sparse = to_sparse_semi_structured(A) SparseSemiStructuredTensor(shape=torch.Size([128, 128])) >>> A_sparse.values() tensor([[1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], ..., [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.]], device='cuda:0', dtype=torch.float16), >>> A_sparse.indices() tensor([[-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], ..., [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370]], device='cuda:0', dtype=torch.int16)) zSetting transpose from `to_sparse_semi_structured` is deprecated and will be removed in a future release. `SparseSemiStructuredTensor` only support contiguous input tensors.r)r*)r) r/r0 FutureWarningrrr3sparserrr)rqrrSPARSE_SUBCLASSr<r<r=r5s4 rc seZdZdZdZejeddddejeddddej eddddej eddddiZ e e jfd ejd ed dfd d ZfddZe dd ejd dfddZddddejdejdBded ejfddZZS)ra This class implements semi-structured sparsity for the CUTLASS backend. In this implementation, the specified elements and metadata are stored separately, in packed and meta respectively. When _FORCE_CUTLASS is set, or when cuSPARSELt is not available, this subclass calls into _sparse_semi_structured_(mm|addmm) and sparse_semi_structured_from_dense for conversion to the compressed format. cutlass @rqrr?c Cs0||t|\}}||j||ddd|jdS)Nr r!r"r#r$r()r}rr'r()r9rqrsparse_tensor_cutlassmeta_tensor_cutlassr<r<r=rs z,SparseSemiStructuredTensorCUTLASS.from_densecs@|jdus |jdurtd|jjdkrt|j|jStS)Nz meta and packed must not be Noner))r!r rBndimrsuperrrErCr<r=rs z*SparseSemiStructuredTensorCUTLASS.to_denserc Cs2tj||dd\}}}}}||j|||||ddS)a~ This function takes in a unpruned dense tensor and runs a (branchless) static sort across a 4x4 tile. It greedily picks the largest values in the tile, upholding the 2:4 sparsity constraint across both rows and columns. The algorithm used to prune the matrix is implemented in `_sparse_semi_structured_tile`. Then it creates the packed and meta tensors for the compressed sparse representation of the pruned dense tensor. It also calculates the packed_t and meta_t tensors for the compressed sparse representation of the transposed pruned dense tensor. Since we cannot transpose the compressed representations, we store both for the fw/bw pass respectively. Finally, this function also computes a compressed swizzled bitmask that encodes the sparsity pattern This can be used in the backward pass to mask the gradients. [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to CUTLASS semi-structured -> packed [1 2 6 2] [0 0 6 2] -> metadata -> pack to transposed CUTLASS -> packed_t semi-structured representation -> metadata_t -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same five outputs from the dense tensor can be found below: ``` from torch.sparse import SparseSemiStructuredTensorCUTLASS from torch.sparse._semi_structured_conversions import ( _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask, ) pruned = _sparse_semi_structured_tile(dense) packed_cutlass, meta_cutlass = sparse_semi_structured_from_dense_cutlass(pruned) packed_t_cutlass, meta_t_cutlass = sparse_semi_structured_from_dense_cutlass( pruned.t().contiguous() ) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUTLASS( dense.shape, packed_cutlass, meta_cutlass, packed_t_cutlass, meta_t_cutlass, bitmask, ) ``` T algorithm use_cutlassFr)r3_sparse_semi_structured_tiler'r9rqrr r!r"r#r$r<r<r=prune_dense_static_sorts$=z9SparseSemiStructuredTensorCUTLASS.prune_dense_static_sortNFrshould_transpose_denserrrc Kst|tr td|jj}|jdks|jdkrtd|d|jdus)|jdur1td|dt |j |j }t j j||j|j||jd|j|j|S)NZ`SparseSemiStructuredTensor @ SparseSemiStructuredTensor` is not supported by the hardwarer)`)` matmul: Broadcasting is not implemented$` matmul: operation is not supportedr) isinstancerr6rCrDrrXr r!_ensure_cutlass_mm_registeredrr-r3r^semi_structured cutlass_mmr'dense_min_rowsdense_min_cols)rFrrrr\cls_name constraintsr<r<r=rs2    z%SparseSemiStructuredTensorCUTLASS._mmr)rDrrrrr3int8rfloat16bfloat16float32rrrrr7rrrrrr __classcell__r<r<rr=r|sH    Orc @seZdZdZdZejeddddejeddddej eddddej eddddiZ e e jfdejdeddfd d Ze ddejdd fd dZddddejdejdBdedejfddZdS)ra The cuSPARSELt backend expects the specified elements and the metadata to be stored in a single tensor: packed = [ specified elements of original tensor | metadata ] For an original tensor of size (m, k) we expect the first m * k // 2 elements to be the kept elements The rest of the tensor is metadata. Since there is only one tensor, we only use the packed and packed_t attributes respectively. cuSPARSELt also supports transposition fusion, which is necessary for performant 2:4 sparse training, as well as specifying alg_id, a config that affects the performance of the matmul depending on matmul sizes. cusparseltrrrrqrr?c Cs0||||jt|ddddtj||jd S)NrR)r}r'r3_cslt_compressrrr(rr<r<r=r6s z/SparseSemiStructuredTensorCUSPARSELT.from_denserrc CsVtj||dd\}}}}}||jdd}||jdd}||j|||||ddS)a= This function does the same thing as described in SparseSemiStructuredCUTLASS, but uses the cuSPARSELt metadata layout and sparse matmul. The only functional difference is that cuSPARSELt stores `metadata` and `packed` together into a single tensor. [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to cuSPARSELT semi-structured -> packed [1 2 6 2] [0 0 6 2] -> pack to transposed cuSPARSELt -> packed_t semi-structured representation -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same three outputs from the dense tensor can be found below: ``` from torch.sparse import SparseSemiStructuredTensorCUSPARSELT from torch.sparse._semi_structured_conversions import ( _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask, ) pruned = _sparse_semi_structured_tile(dense) packed_cusparselt = torch._cslt_compress(pruned) packed_t_cusparselt = torch._cslt_compress(pruned.t().contiguous()) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUSPARSELT( dense.shape, packed_cutlass, None, packed_t_cutlass, None, bitmask ) ``` Frrr~r)r3rrfr'rr<r<r=rJs(-z.cutlass_mmtranspose_densec Ss.|r|jdn|jd}tj|||j|jdSNrrrr'r3emptyr-r,) rr r!rrr{r|rrr<r<r=_cutlass_mm_fakes z7_ensure_cutlass_mm_registered.._cutlass_mm_fake)_cutlass_mm_registered torch.libraryrr3r7rr register_fake)rrrr<r<r=rsX    rcCstrdSdaddlm}|ddd dd tjd tjd tjdBd td tdtdtdtdtdtjfdd}|jd tjd tjd tjdBd td tdtdtdtdtdtjfdd}dS)z,Lazily register the cusparselt_mm custom op.NTrrzsemi_structured::cusparselt_mmr<rFrr rrr{r|fuse_transposerrr?c Ss|j\} } | |} | |} | dkp| dk} |}| r'tjj|d| d| f}|r-|n|}tj|||||d}|r?|}| rS|rE| n| }|dd|jtj dS| S)Nr)rtranspose_resultrrr) r'r3rrrre_cslt_sparse_mmrrmrr)rr rrr{r|rrrryrzrrrrrrrr<r<r=rs.   z7_ensure_cusparselt_mm_registered..cusparselt_mmc Ss.|r|jdn|jd} tj|| |j|jdSrr) rr rrr{r|rrrrr<r<r=_cusparselt_mm_fake6s z=_ensure_cusparselt_mm_registered.._cusparselt_mm_fake)F)_cusparselt_mm_registeredrrr3r7rrr)rrrr<r<r=r sb     #  r)'r/ collectionsrcollections.abcrtypingrr3)torch.sparse._semi_structured_conversionsrr!torch.sparse._semi_structured_opsrrr r r r r rrrrrr__all__rr7rrrrrrrrrrrr<r<r<r=sD   < G'!A