o m9:jؠ@sddlmZddlZddlmZddlmZmZddlm Z m Z m Z ddl m ZddlmZdd lmZgd ZGd d d eZGd ddeZGdddeeZGdddeZGdddeeZGdddeZGdddeeZGdddeZGdddeeZGdddeZ dS))AnyN)Tensor) functionalinit) ParameterUninitializedBufferUninitializedParameter) SyncBatchNorm)LazyModuleMixin)Module) BatchNorm1dLazyBatchNorm1d BatchNorm2dLazyBatchNorm2d BatchNorm3dLazyBatchNorm3dr cseZdZUdZdZgdZeed<eed<edBed<e ed<e ed <  dd d dedededBde d e de ddffddZ dddZ dddZ ddZ ddZ  dfdd ZZS) _NormBasez,Common base of _InstanceNorm and _BatchNorm.)track_running_statsmomentumeps num_featuresaffinerrNrrrh㈵>皙?Tbiasrreturnc s:||d} t||_||_||_||_||_|jr>ttj |fi| |_ |r7ttj |fi| |_ n| ddn | dd| dd|jr| dtj|fi| | dtj|fi| ||| dtj d dtjid d | D|n| dd| dd| dd|dS) Ndevicedtyperweight running_mean running_varnum_batches_trackedrr!cSi|] \}}|dkr||qSr!.0kvr(r(a/home/nk/hobo-godmode/plappi-mvp/.venv/lib/python3.10/site-packages/torch/nn/modules/batchnorm.py Qz&_NormBase.__init__..r)super__init__rrrrrrtorchemptyr"rregister_parameterregister_bufferzerosonestensorlongitemsreset_parameters selfrrrrrr r!rfactory_kwargs __class__r(r-r2&sL       z_NormBase.__init__cCs.|jr|j|jd|jdSdS)Nr )rr#zero_r$fill_r%r>r(r(r-reset_running_stats[s   z_NormBase.reset_running_statscCs<||jrt|j|jdurt|jdSdSdSN)rErrones_r"rzeros_rDr(r(r-r<cs  z_NormBase.reset_parameterscCstrF)NotImplementedErrorr>inputr(r(r-_check_input_dimjsz_NormBase._check_input_dimcCs djdi|jd|jduiS)Nz{{num_features}, eps={eps}, momentum={momentum}, affine={affine}, bias={use_bias}, track_running_stats={track_running_stats}use_biasr()format__dict__rrDr(r(r- extra_reprmsz_NormBase.extra_reprc s|dd}|dus|dkr4|jr4|d} | |vr4|jdur*|jjtdkr*|jntjdtjd|| <t|||||||dS)Nversionrr%metarr') getrr%r r3r9r:r1_load_from_state_dict) r> state_dictprefixlocal_metadatastrict missing_keysunexpected_keys error_msgsrQnum_batches_tracked_keyr@r(r-rTus$  z_NormBase._load_from_state_dictrrTTNNrN)__name__ __module__ __qualname____doc___version __constants__int__annotations__floatboolr2rEr<rLrPrT __classcell__r(r(r@r-rsN      5  rcsdeZdZ      ddddedededBd ed ed ed dffd dZded efddZZ S) _BatchNormrrTNrrrrrrrrc s2||d} tj|||||fi| d|idSNrr)r1r2r=r@r(r-r2s  z_BatchNorm.__init__rKc Cs|||jdur d}n|j}|jr1|jr1|jdur1|jd|jdur.dt|j}n|j} |jr8d}n |jduoA|jdu} t ||jrL|jrO|jnd|jrV|jrY|jnd|j |j |||j S)Nr ?T)rLrtrainingrr%add_rgr#r$F batch_normr"rr)r>rKexponential_average_factor bn_trainingr(r(r-forwards:      z_BatchNorm.forwardr]) r_r`rarergrhr2rrtrir(r(r@r-rjs2   rjcs^eZdZUeed<eed<      ddd dfd d Zdfd d Zdd dZZS) _LazyNormBaser"rrrTNrrc s||d}tjd||ddfi|ddi||_||_|jr2tdi||_|r2tdi||_|jr[tdi||_tdi||_ t j d dt j idd| D|_dSdS) NrrFrr!cSr&r'r(r)r(r(r-r.r/z*_LazyNormBase.__init__..r(r0)r1r2rrrr"rrr#r$r3r9r:r;r%) r>rrrrr r!rr?r@r(r-r2s:    z_LazyNormBase.__init__cs(|s|jdkrtdSdSdS)Nr)has_uninitialized_paramsrr1r<rDr@r(r-r<sz_LazyNormBase.reset_parameterscCs|rO|jd|_|jr6t|jtstd|j|jf|j dur6t|j ts.td|j |jf|j rI|j |jf|j |jf| dSdS)Nr z-self.weight must be an UninitializedParameterz+self.bias must be an UninitializedParameter)rvshaperr isinstancer"rAssertionError materializerrr#r$r<rJr(r(r-initialize_parameterss.     z#_LazyNormBase.initialize_parametersr]r^) r_r`rarrfr2r<r{rir(r(r@r-rus   ,ruc@eZdZdZdddZdS)r aApplies Batch Normalization over a 2D or 3D input. Method described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the number of features or channels of the input). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. At train time in the forward pass, the variance is calculated via the biased estimator, equivalent to ``torch.var(input, correction=0)``. However, the value stored in the moving average of the variance is calculated via the unbiased estimator, equivalent to ``torch.var(input, correction=1)``. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, L)` slices, it's common terminology to call this Temporal Batch Normalization. Args: num_features: number of features or channels :math:`C` of the input eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` Shape: - Input: :math:`(N, C)` or :math:`(N, C, L)`, where :math:`N` is the batch size, :math:`C` is the number of features or channels, and :math:`L` is the sequence length - Output: :math:`(N, C)` or :math:`(N, C, L)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm1d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm1d(100, affine=False) >>> input = torch.randn(20, 100) >>> output = m(input) rNcC4|dkr|dkrtd|ddSdSNrzexpected 2D or 3D input (got D input)dim ValueErrorrJr(r(r-rL{zBatchNorm1d._check_input_dimr^r_r`rarbrLr(r(r(r-r 2sHr c@eZdZdZeZdddZdS)raA :class:`torch.nn.BatchNorm1d` module with lazy initialization. Lazy initialization based on the ``num_features`` argument of the :class:`BatchNorm1d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` rNcCr}r~rrJr(r(r-rLrz LazyBatchNorm1d._check_input_dimr^)r_r`rarbr cls_to_becomerLr(r(r(r-rrc@r|)ra3Applies Batch Normalization over a 4D input. 4D is a mini-batch of 2D inputs with additional channel dimension. Method described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. At train time in the forward pass, the standard-deviation is calculated via the biased estimator, equivalent to ``torch.var(input, correction=0)``. However, the value stored in the moving average of the standard-deviation is calculated via the unbiased estimator, equivalent to ``torch.var(input, correction=1)``. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, H, W)` slices, it's common terminology to call this Spatial Batch Normalization. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` Shape: - Input: :math:`(N, C, H, W)` - Output: :math:`(N, C, H, W)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm2d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm2d(100, affine=False) >>> input = torch.randn(20, 100, 35, 45) >>> output = m(input) rNcC$|dkrtd|ddSNzexpected 4D input (got rrrJr(r(r-rL zBatchNorm2d._check_input_dimr^rr(r(r(r-rIrc@r)raA :class:`torch.nn.BatchNorm2d` module with lazy initialization. Lazy initialization is done for the ``num_features`` argument of the :class:`BatchNorm2d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` rNcCrrrrJr(r(r-rLrz LazyBatchNorm2d._check_input_dimr^)r_r`rarbrrrLr(r(r(r-rrrc@r|)rahApplies Batch Normalization over a 5D input. 5D is a mini-batch of 3D inputs with additional channel dimension as described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. At train time in the forward pass, the standard-deviation is calculated via the biased estimator, equivalent to ``torch.var(input, correction=0)``. However, the value stored in the moving average of the standard-deviation is calculated via the unbiased estimator, equivalent to ``torch.var(input, correction=1)``. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, D, H, W)` slices, it's common terminology to call this Volumetric Batch Normalization or Spatio-temporal Batch Normalization. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, D, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` Shape: - Input: :math:`(N, C, D, H, W)` - Output: :math:`(N, C, D, H, W)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm3d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm3d(100, affine=False) >>> input = torch.randn(20, 100, 35, 45, 10) >>> output = m(input) rNcCrNzexpected 5D input (got rrrJr(r(r-rLarzBatchNorm3d._check_input_dimr^rr(r(r(r-rrrc@r)raA :class:`torch.nn.BatchNorm3d` module with lazy initialization. Lazy initialization is done for the ``num_features`` argument of the :class:`BatchNorm3d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` rNcCrrrrJr(r(r-rLrz LazyBatchNorm3d._check_input_dimr^)r_r`rarbrrrLr(r(r(r-rfrrcseZdZdZ       ddddeded edBd ed ed edBd eddffddZdddZ dddZ de de fddZ e dddZZS)r aApplies Batch Normalization over a N-Dimensional input. The N-D input is a mini-batch of [N-2]D inputs with additional channel dimension) as described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over all mini-batches of the same process groups. :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are sampled from :math:`\mathcal{U}(0, 1)` and the elements of :math:`\beta` are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, correction=0)`. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done for each channel in the ``C`` dimension, computing statistics on ``(N, +)`` slices, it's common terminology to call this Volumetric Batch Normalization or Spatio-temporal Batch Normalization. Currently :class:`SyncBatchNorm` only supports :class:`~torch.nn.DistributedDataParallel` (DDP) with single GPU per process. Use :meth:`torch.nn.SyncBatchNorm.convert_sync_batchnorm()` to convert :attr:`BatchNorm*D` layer to :class:`SyncBatchNorm` before wrapping Network with DDP. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, +)` eps: a value added to the denominator for numerical stability. Default: ``1e-5`` momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` process_group: synchronization of stats happen within each process group individually. Default behavior is synchronization across the whole world bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` Shape: - Input: :math:`(N, C, +)` - Output: :math:`(N, C, +)` (same shape as input) .. note:: Synchronization of batchnorm statistics occurs only while training, i.e. synchronization is disabled when ``model.eval()`` is set or if ``self.training`` is otherwise ``False``. Examples:: >>> # xdoctest: +SKIP >>> # With Learnable Parameters >>> m = nn.SyncBatchNorm(100) >>> # creating process group (optional) >>> # ranks is a list of int identifying rank ids. >>> ranks = list(range(8)) >>> r1, r2 = ranks[:4], ranks[4:] >>> # Note: every rank calls into new_group for every >>> # process group created, even if that rank is not >>> # part of the group. >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]] >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1] >>> # Without Learnable Parameters >>> m = nn.BatchNorm3d(100, affine=False, process_group=process_group) >>> input = torch.randn(20, 100, 35, 45, 10) >>> output = m(input) >>> # network is nn.BatchNorm layer >>> sync_bn_network = nn.SyncBatchNorm.convert_sync_batchnorm(network, process_group) >>> # only single gpu per process is currently supported >>> ddp_sync_bn_network = torch.nn.parallel.DistributedDataParallel( >>> sync_bn_network, >>> device_ids=[args.local_rank], >>> output_device=args.local_rank) rrTNrrrrrr process_grouprrc  s8||d} tj|||||fi| d| i||_dSrk)r1r2r) r>rrrrrrr r!rr?r@r(r-r2s  zSyncBatchNorm.__init__cCs$|dkrtd|ddS)Nrz expected at least 2D input (got rrrJr(r(r-rL rzSyncBatchNorm._check_input_dimcCs|ddkr tddS)Nr rz9SyncBatchNorm number of input channels should be non-zero)sizerrJr(r(r-_check_non_zero_input_channelss z,SyncBatchNorm._check_non_zero_input_channelsrKc Cs|||||jdurd}n|j}|jr:|jr:|jdur$td|jd|jdur7d|j}n|j} |jrAd}n |j duoJ|j du} |jrR|jrU|j nd}|jr]|jr`|j nd}|op|jopt j opt j }|r|jjddd t jfvrtd t jt j jj}|jr|j}t j |}|dk}|st||||j|j|||jS|std t||j|j|||j||| S) z( Runs the forward pass. Nrlz$num_batches_tracked must not be Noner rmTcudahpuxpuz;SyncBatchNorm expected input tensor to be on GPU or XPU or zbn_training must be True)rLrrrnrr%ryroitemr#r$r3 distributed is_availableis_initializedr type_C_get_privateuse1_backend_namergroupWORLDrget_world_sizerprqr"rrsync_batch_normapply) r>rKrrrsr#r$ need_syncr world_sizer(r(r-rts          zSyncBatchNorm.forwardc Cs|}t|tjjjjrXtjj|j|j|j |j |j ||j dud}|j r?t |j|_|j |_ Wdn1s:wY|j|_|j|_|j|_|j|_t|drX|j|_|D]\}}|||||q\~|S)aaConverts all :attr:`BatchNorm*D` layers in the model to :class:`torch.nn.SyncBatchNorm` layers. Args: module (nn.Module): module containing one or more :attr:`BatchNorm*D` layers process_group (optional): process group to scope synchronization, default is the whole world Returns: The original :attr:`module` with the converted :class:`torch.nn.SyncBatchNorm` layers. If the original :attr:`module` is a :attr:`BatchNorm*D` layer, a new :class:`torch.nn.SyncBatchNorm` layer object will be returned instead. Example:: >>> # Network with nn.BatchNorm layer >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> module = torch.nn.Sequential( >>> torch.nn.Linear(20, 100), >>> torch.nn.BatchNorm1d(100), >>> ).cuda() >>> # creating process group (optional) >>> # ranks is a list of int identifying rank ids. >>> ranks = list(range(8)) >>> r1, r2 = ranks[:4], ranks[4:] >>> # Note: every rank calls into new_group for every >>> # process group created, even if that rank is not >>> # part of the group. >>> # xdoctest: +SKIP("distributed") >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]] >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1] >>> sync_bn_module = torch.nn.SyncBatchNorm.convert_sync_batchnorm(module, process_group) Nrqconfig)rxr3nnmodules batchnormrjr rrrrrrno_gradr"r#r$r%rnhasattrrnamed_children add_moduleconvert_sync_batchnorm)clsmoduler module_outputnamechildr(r(r-rys8$    z$SyncBatchNorm.convert_sync_batchnorm)rrTTNNNr^rF)r_r`rarbrergrhrr2rLrrrt classmethodrrir(r(r@r-r sBk     cr )typingrr3rtorch.nnrrprtorch.nn.parameterrrr _functionsr rlazyr rr __all__rrjrur rrrrrr(r(r(r-s&      KON$O$O$