o m9:jh@sdZddlZddlZddlmZddlmZmZddlm Z ddl Z ddl m Z gdZ edZ e d Zed Zed Z dQd e d edede jdBde f ddZ dQd e dedede jdBde f ddZ dQd e deded edede jdBde fddZd e dede fddZd e de fddZ dQdedeeBdBdefd d!Z " # dRd e d edede jdBde f d$d%Z " # dRd e dedede jdBde f d&d'Z " # ( ) dSd e deded edede jdBde fd*d+Zd e dede fd,d-Zd e de fd.d/Zd e de fd0d1Zd e de fd2d3Z dTd e d5ede fd6d7Z!d e de"eeffd8d9Z# # dUd e d:ede jdBde fd;d<Z$ # dUd e d:ede jdBde fd=d>Z%d e d?edefd@dAZ&  B C dVd e d ed?edede jdBde f dDdEZ'  B C dVd e d ed?edede jdBde f dFdGZ( 4 dWd e d:ede jdBde fdHdIZ) J dXd e dKedede jdBde f dLdMZ*dNeee fdeee ffdOdPZ+e+eZ,e+eZ-e+eZ.e+e Z/e+e!Z0e+e$Z1e+e%Z2e+e'Z3e+e(Z4e+e)Z5e+e*Z6dS)YzHThis file contains utilities for initializing neural network parameters.N)Callable)LiteralTypeVar) ParamSpec)Tensor)calculate_gainuniform_normal_ trunc_normal_ constant_ones_zeros_eye_dirac_xavier_uniform_xavier_normal_kaiming_uniform_kaiming_normal_ orthogonal_sparse_uniformnormalconstanteyediracxavier_uniform xavier_normalkaiming_uniformkaiming_normal orthogonalsparse_R_P) linearconv1dconv2dconv3dconv_transpose1dconv_transpose2dconv_transpose3dsigmoidtanhrelu leaky_reluselu)fan_infan_outtensorab generatorreturncC<t|j|||dWdS1swYdSNr4)torchno_gradrr1r2r3r4r<T/home/nk/hobo-godmode/plappi-mvp/.venv/lib/python3.10/site-packages/torch/nn/init.py_no_grad_uniform_Es $r>meanstdcCr6r7)r9r:r r1r?r@r4r<r<r=_no_grad_normal_Ls $rBc Cs<|jr|Sdtdtfdd}||d|ks||d|kr%tjdddt||||||||}|dkr|j|d d }|j|d d } |j|||d } | |k| | kB} | sfnt | t | j|||d | } qY|| ur| | nt |t||} d | ||d} t |}t |}|j|||d | ||||dd | |j|d |}| sn?|} |j|||d t ||| } |||dd | t ||j|d ||}| snq| | |WdS1swYdS)Nxr5cSsdt|tddS)N?@)matherfsqrt)rCr<r<r=norm_cdfbsz(_no_grad_trunc_normal_..norm_cdfzjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect. stacklevelg333333?cpu)devicer8Tg)is_metafloatwarningswarnr9r: new_tensoritemr anywhere empty_likecopy_maxminrsub_div_pow_mul_log_gt)r1r?r@r2r3r4rIplohiresultmaskmodelog_peak candidates accept_bufpendingr<r<r=_no_grad_trunc_normal_Vsj        ""  &rkvalcCs6t ||WdS1swYdSN)r9r:fill_r1rlr<r<r=_no_grad_fill_s $rpcCs4t |WdS1swYdSrm)r9r:zero_r1r<r<r=_no_grad_zero_s $rs nonlinearityparamcCsgd}||vs |dkrdS|dkrdS|dkrtdS|dkrM|d ur(d }nt|ts2t|ts7t|tr:|}ntd |d tdd|d S|dkrT dStd|)aReturn the recommended gain value for the given nonlinearity function. The values are as follows: ================= ==================================================== nonlinearity gain ================= ==================================================== Linear / Identity :math:`1` Conv{1,2,3}D :math:`1` Sigmoid :math:`1` Tanh :math:`\frac{5}{3}` ReLU :math:`\sqrt{2}` Leaky Relu :math:`\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}` SELU :math:`\frac{3}{4}` ================= ==================================================== .. warning:: In order to implement `Self-Normalizing Neural Networks`_ , you should use ``nonlinearity='linear'`` instead of ``nonlinearity='selu'``. This gives the initial weights a variance of ``1 / N``, which is necessary to induce a stable fixed point in the forward pass. In contrast, the default gain for ``SELU`` sacrifices the normalization effect for more stable gradient flow in rectangular layers. Args: nonlinearity: the non-linear function (`nn.functional` name) param: optional parameter for the non-linear function Examples: >>> gain = nn.init.calculate_gain( ... "leaky_relu", 0.2 ... ) # leaky_relu with negative_slope=0.2 .. _Self-Normalizing Neural Networks: https://papers.nips.cc/paper/2017/hash/5d44ee6f2c3f71b73125876103c8f6c4-Abstract.html )r#r$r%r&r'r(r)r*r+g?r,rEr-N{Gz?znegative_slope z not a valid numberrJr.g?zUnsupported nonlinearity )rFrH isinstanceboolintrP ValueError)rtru linear_fnsnegative_sloper<r<r=rs.&  rrDcC4tj|rtjjt|f||||dSt||||S)aFill the input Tensor with values drawn from the uniform distribution. :math:`\mathcal{U}(a, b)`. Args: tensor: an n-dimensional `torch.Tensor` a: the lower bound of the uniform distribution b: the upper bound of the uniform distribution generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.uniform_(w) r;)r9 overrideshas_torch_function_variadichandle_torch_functionrr>r;r<r<r=r rcCr)aFill the input Tensor with values drawn from the normal distribution. :math:`\mathcal{N}(\text{mean}, \text{std}^2)`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.normal_(w) rA)r9rrrr rBrAr<r<r=r rr rEcCst||||||dS)aFill the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. For reduced-precision types (``torch.float16`` and ``torch.bfloat16``), sampling quality depends on the underlying ``normal_()`` and ``uniform_()`` implementations which operate at higher internal precision to avoid quantization artifacts. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) r8)rk)r1r?r@r2r3r4r<r<r=r -s!r cCs,tj|rtjjt|f||dSt||S)zFill the input Tensor with the value :math:`\text{val}`. Args: tensor: an n-dimensional `torch.Tensor` val: the value to fill the tensor with Examples: >>> w = torch.empty(3, 5) >>> nn.init.constant_(w, 0.3) ro)r9rrrr rpror<r<r=r Qs   r cCs t|dS)zFill the input Tensor with the scalar value `1`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.ones_(w) rD)rprrr<r<r=r cs r cCst|S)zFill the input Tensor with the scalar value `0`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.zeros_(w) )rsrrr<r<r=r ps r cCsX|dkr tdttj|j||jdWd|S1s%wY|S)a=Fill the 2-dimensional input `Tensor` with the identity matrix. Preserves the identity of the inputs in `Linear` layers, where as many inputs are preserved as possible. Args: tensor: a 2-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.eye_(w) rJ,Only tensors with 2 dimensions are supported)out requires_gradN) ndimensionr{r9r:rshaperrrr<r<r=r}s   rrvgroupsc CsF|}|dvr td|}|d|dkrtd|jr!|S|d|}t||d}tg|t|D]U}t|D]N}|dkrXd||||||ddf<qA|dkrsd||||||dd|ddf<qAd||||||dd|dd|ddf<qAq;Wd |S1swY|S) aFFill the {3, 4, 5}-dimensional input `Tensor` with the Dirac delta function. Preserves the identity of the inputs in `Convolutional` layers, where as many input channels are preserved as possible. In case of groups>1, each group of channels preserves identity Args: tensor: a {3, 4, 5}-dimensional `torch.Tensor` groups (int, optional): number of groups in the conv layer (default: 1) Examples: >>> w = torch.empty(3, 16, 5, 5) >>> nn.init.dirac_(w) >>> w = torch.empty(3, 24, 5, 5) >>> nn.init.dirac_(w, 3) )z5Only tensors with 3, 4, or 5 dimensions are supportedrz!dim 0 must be divisible by groupsrvrrJrN) rr{sizerOrZr9r:rqrange)r1r dimensionssizesout_chans_per_grpmin_dimgdr<r<r=rsP    "         rcCsp|}|dkr td|d}|d}d}|dkr,|jddD]}||9}q%||}||}||fS)NrJzNFan in and fan out can not be computed for tensor with fewer than 2 dimensionsrvr)dimr{rr)r1rnum_input_fmapsnum_output_fmapsreceptive_field_sizesr/r0r<r<r=_calculate_fan_in_and_fan_outs    rgaincCsDt|\}}|tdt||}td|}t|| ||S)aFill the input `Tensor` with values using a Xavier uniform distribution. The method is described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from :math:`\mathcal{U}(-a, a)` where .. math:: a = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain("relu")) rE@)rrFrHrPr>)r1rr4r/r0r@r2r<r<r=rs rcCs4t|\}}|tdt||}t|d||S)aFill the input `Tensor` with values using a Xavier normal distribution. The method is described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_normal_(w) rEr~)rrFrHrPrB)r1rr4r/r0r@r<r<r=rs rrfcCsH|}ddg}||vrtd|d|t|\}}|dkr"|S|S)Nr/r0zMode z" not supported, please use one of )lowerr{r)r1rf valid_modesr/r0r<r<r=_calculate_correct_fans  rr/r-c Cstj|rtjjt|f|||||dSd|jvr"tjddd|St||}t ||}|t |}t d|}t |j | ||dWdS1sRwYdS) aFill the input `Tensor` with values using a Kaiming uniform distribution. The method is described in `Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification` - He, K. et al. (2015). The resulting tensor will have values sampled from :math:`\mathcal{U}(-\text{bound}, \text{bound})` where .. math:: \text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan\_mode}}} Also known as He initialization. Args: tensor: an n-dimensional `torch.Tensor` a: the negative slope of the rectifier used after this layer (only used with ``'leaky_relu'``) mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'`` preserves the magnitude of the variance of the weights in the forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the backwards pass. nonlinearity: the non-linear function (`nn.functional` name), recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default). generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.kaiming_uniform_(w, mode="fan_in", nonlinearity="relu") Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.kaiming_uniform_(w.T, ...)``. )r1r2rfrtr4r,Initializing zero-element tensors is a no-oprJrKrr8N)r9rrrrrrQrRrrrFrHr:r) r1r2rfrtr4fanrr@boundr<r<r=r*s( +    $rcCszd|jvrtjddd|St||}t||}|t|}t|j d||dWdS1s6wYdS)aFill the input `Tensor` with values using a Kaiming normal distribution. The method is described in `Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification` - He, K. et al. (2015). The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}} Also known as He initialization. Args: tensor: an n-dimensional `torch.Tensor` a: the negative slope of the rectifier used after this layer (only used with ``'leaky_relu'``) mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'`` preserves the magnitude of the variance of the weights in the forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the backwards pass. nonlinearity: the non-linear function (`nn.functional` name), recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default). generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.kaiming_normal_(w, mode="fan_out", nonlinearity="relu") Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.kaiming_normal_(w.T, ...)``. rrrJrKr8N) rrQrRrrrFrHr9r:r )r1r2rfrtr4rrr@r<r<r=rks +   $rc Cs|dkr td|dks|jr|S|d}||}|||fjdd|d}||kr5|tj |\}}t |d}| } || 9}||krS|t |||||Wd|S1spwY|S)aFill the input `Tensor` with a (semi) orthogonal matrix. Described in `Exact solutions to the nonlinear dynamics of learning in deep linear neural networks` - Saxe, A. et al. (2013). The input tensor must have at least 2 dimensions, and for tensors with more than 2 dimensions the trailing dimensions are flattened. Args: tensor: an n-dimensional `torch.Tensor`, where :math:`n \geq 2` gain: optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_LAPACK) >>> w = torch.empty(3, 5) >>> nn.init.orthogonal_(w) rJz4Only tensors with 2 or more dimensions are supportedrrvr8N)rr{numelrOr new_emptyr t_r9linalgqrdiagsignr:view_asrXr^) r1rr4rowscols flattenedqrrphr<r<r=rs,       rrwsparsityc Cs|dkr td|jr|S|j\}}t||}t)|jd||dt |D]}t |}|d|} d|| |f<q,Wd|S1sKwY|S)aFill the 2D input `Tensor` as a sparse matrix. The non-zero elements will be drawn from the normal distribution :math:`\mathcal{N}(0, 0.01)`, as described in `Deep learning via Hessian-free optimization` - Martens, J. (2010). Args: tensor: an n-dimensional `torch.Tensor` sparsity: The fraction of elements in each column to be set to zero std: the standard deviation of the normal distribution used to generate the non-zero values generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.sparse_(w, sparsity=0.1) rJrrr8N) rr{rOrrFceilr9r:r rrandperm) r1rr@r4rr num_zeroscol_idx row_indices zero_indicesr<r<r=rs"       rmethcsXjdddtjdtjdtffdd }ddd d |_|_|S) Nargskwargsr5cs,tjdddtdd|i|S)Nz `nn.init.z)` is now deprecated in favor of `nn.init.z`.rJrK)rQrR FutureWarning)rrrnew_nameold_namer<r=deprecated_inits z(_make_deprecate..deprecated_initz z_(...) .. warning:: This method is now deprecated in favor of :func:`torch.nn.init.z"`. See :func:`~torch.nn.init.z` for details.)__name__r"rrr!__doc__)rrr<rr=_make_deprecates " rrm)r~rDN)r~rDrrEN)rv)rDN)rr/r-N)rvN)rwN)7rrFrQcollections.abcrtypingrrtyping_extensionsrr9r__all__r!r"_NonlinearityType_FanModerP Generatorr>rBrkrprsrzrrr r r r r rrtuplerrrrrrrrrrrrrrrrrrrr r<r<r<r=s       M  L   $  8 #   C 7 6 "*