o j9:j H @sddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z ddl Z ddlmZddlmZddlmmZddlmmZddlmZmZddlmZddlmZm Z gd Z!d e"d e#e"e"ffd d Z$dee%dej&de'e"e ffddZ(dej&de'e"e fde jj)fddZ*d8de jj)d e jj)fddZ+dej)d ej)fddZ,dej)de-ej&de-ej&de-ej&fd d!Z.ej/ej0ej1ej2ej3ej4ej5e j6e j7e j8ej6ej9ej:g Z;ejej/ej?ej0ej@ej1d"d#iZAde-ej&de'e"ej)ffd$d%ZBde-ej&de'e"ej)fd&e'ej)ej)ffd'd(ZCGd)d*d*ZDd9d-d.ZEd/eDd eFfd0d1ZGGd2d3d3ZHdejIfde jj)d4e'e"e fdBd5e%ejId e jj)fd6d7ZJdS):N) defaultdict)Iterable)Enum)Anycast)ArgumentTarget) ShapeProp)fuse_conv_bn_evalfuse_linear_bn_eval) matches_module_patternreplace_node_modulefuseremove_dropoutextract_subgraphmodules_to_mkldnn reset_modules MklSubgraphgen_mkl_autotuneruse_mkl_length UnionFindoptimize_for_inferencetargetreturncCs*|dd^}}|r|d|fSd|fS)zp Splits a qualname into parent path and last atom. For example, `foo.bar.baz` -> (`foo.bar`, `baz`) .r)rsplit)rparentnamer i/home/nk/hobo-godmode/plappi-mvp/.venv/lib/python3.10/site-packages/torch/fx/experimental/optimization.py _parent_name%sr"patternnodemodulescCst|jdkr dS|jd|f}t||D]2\}}t|tjs"dS|jdkr*dSt|jts3dS|j|vr;dSt ||j|urGdSqdS)NrF call_moduleT) lenargszip isinstancefxNodeoprstrtype)r#r$r%nodes expected_type current_noder r r!r /s     r new_modulecCsLt|jtstdt|jt|j\}}|||j<t||||dS)NExpected str target, got )r*rr.AssertionErrorr/r"setattr)r$r%r3 parent_namerr r r!r Cs  r Fmodelc Cs4tjtjftjtjftjtjftjtjfg}|st |}|r&t |t j j s,t |}n|}t|}t |j}|D]W}|jD]Q}t|||rt|jdjdkrTqA||jdj} ||j} | jseqA|dtjtjtjfvrwt| | } nt| | } t|jd|| ||jd||qAqeZdZdedeedfdeeefdeffdd Z Z S)z&remove_dropout..DropoutRemoverrr(.kwargsrcsHt|j|tjrt|dkrtdt||dSt|||S)Nrz Expected 1 arg for Dropout, got r)r* submodulesr9Dropoutr'r5superr&)selfrr(rU __class__r r!r&|s  z2remove_dropout..DropoutRemover.call_module) __name__ __module__ __qualname__rtuplerrFr.rr& __classcell__r r rZr!DropoutRemover{s  ra)r+rErC Transformer transform)r8rPrar r r!rus  r orig_moduler0inputsoutputscs|t}i|D] }||j}||<q|D]}||fdd}||<q|fdd|D|t||S)z Given lists of nodes from an existing graph that represent a subgraph, returns a submodule that executes that subgraph. cs|SNr )xenvr r!z"extract_subgraph..csg|]}|qSr r ).0outputrir r! z$extract_subgraph..)r+Graph placeholderr node_copyrnlintrD)rdr0rerfrQinputnew_noder$r rir!rs     rcCs t|Srg) th_mkldnnMkldnnBatchNorm)a_r r r!rks rkcCsi}|D]H}|jdkrLt|jtstdt|j||j}t|tvrLtt||tj}t|t j s?tdt|t |||<t |||q|S)z For each node, if it's a module that can be preconverted into MKLDNN, then we do so and create a mapping to allow us to convert from the MKLDNN version of the module to the original. r&r4zExpected nn.Module, got )r-r*rr.r5r/ mkldnn_maprCfloatr9ModulerArBr )r0r% old_modulesr$ cur_moduler3r r r!rs      rr~cCsZ|D](}|jdkr*t|jtstdt|j||j}||vr*t||||qdS)za Maps each module that's been changed with `modules_to_mkldnn` back to its original. r&r4N)r-r*rr.r5r/r )r0r%r~r$rr r r!rs   rc@seZdZdejfddZdS)rfx_graphcCs||_g|_g|_g|_dSrg)rr0 start_nodes end_nodes)rYrr r r!__init__s zMklSubgraph.__init__N)r\r]r^r+rqrr r r r!rsr rcs*dddtdtffdd }|S)aW This generates a heuristic that can be passed into `optimize_for_inference` that determines whether a subgraph should be run in MKL by running it with the example_inputs. Example usage: heuristic = gen_mkl_autotuner(example_inputs, iters=10) fast_model = optimization.optimize_for_inference(model, heuristic) NrHrcs|j}dur|jj|jjtdd|Dtttj dd|j D}t |j ||fdd}|fdd}t jj t|fdd}||kS) NcSsg|]}t|jqSr )rCrandnshapermr$r r r!rosz@gen_mkl_autotuner..use_mkl_heuristic..cSsg|]}|jdqS)r)r(rr r r!roscs<tD]}|qt}tD]}|qt|Srg)rangetime)frzbegin)iterswarmupr r! benchmarks   z?gen_mkl_autotuner..use_mkl_heuristic..benchmarkcsddddDDS)NcSg|]}|qSr )to_densermir r r!ro szRgen_mkl_autotuner..use_mkl_heuristic....cSrr ) to_mkldnnrr r r!ro rpr r  sample_inputs submoduler r!rk sz>gen_mkl_autotuner..use_mkl_heuristic..csSrgr r rr r!rkrl)rr owning_moduler~r propagaterlistr+r,rrr0rrHrFrG)rH input_nodes output_argsrmkl_time no_mkl_timeexample_inputsrPrr~rrr!use_mkl_heuristics&  z,gen_mkl_autotuner..use_mkl_heuristic)rbool)rrrrr rr!rs "rrHcCst|jdkS)z This is a heuristic that can be passed into `optimize_for_inference` that determines whether a subgraph should be run in MKL by checking if there are more than 2 nodes in it )r'r0)rHr r r!rsrc@sFeZdZddZdefddZdedefddZd ed efd d Zd S)rcCsdg||_dg||_dS)Nrrsize)rYnr r r!r%s zUnionFind.__init__vcCs||j|<d|j|<dS)Nrr)rYrr r r!make_set)s zUnionFind.make_setrcCsF|j|}||kr |S|durtd|||j|<tt|j|S)NzParent is None)rr5findrint)rYrparr r r!r-s zUnionFind.findrybcCsf||||}}||kr|S|j||j|kr ||}}||j|<|j||j|7<dSrg)rrr)rYryrr r r!join6s  zUnionFind.joinN)r\r]r^rrrrrr r r r!r$s  r pass_configtracerc sHdddtid}|duri}|||drt|}|dr#t|}|ddur+|St|dts6td d|dvr@td |dd}|}|t |t |j t| }Gd d d t}tjD]}|j} |jd kr||j} t| tvr|j} t| d} | dur| jtjkrtd| jtdkrtdn|jdkr|jtvr|j} n|jtvr|j} | |jkr"| |jkrt dd|j!Dsqk"|t #|j!fdd} Wdn1swYt$t%t j&j'| |_!(|)dd|f} |*| |f| _!Wdn 1swYqkt+tj|}|_,jD]B}|jdkrr|jdkrr|j!d}t|j-}|D]}|jdkrc|jdkrc|*|.|qKt/|j-dkrr.|q1t/j}t0|fddt1jD]{\}}|jdkr|jdkr||_23|q|jdkr|jdkrƈ|j!ddurtd|j!d|_4qfdd|j5D}t/|dkrِqt dd|Drtd t6|}|d|_7|d!dD] }8|d|qqt9fd"d}jD]9}t:|d#r#|;|j7j<|t:|d$r5|;|j2j=<|t:|d%rG|;|j4j><|q|?D](}||st|j=|j>D]}|j!d}|*|.|qZt@|j||qMd}jD]}|jdks|jdkr|d!7}q{tABtCDd&|Et |}|S)'a Performs a set of optimization passes to optimize a model for the purposes of inference. Specifically, the passes that are run are: 1. Conv/BN fusion 2. Dropout removal 3. MKL layout optimizations The third optimization takes a function `use_mkl_heuristic` that's used to determine whether a subgraph should be explicitly run in MKL layout. Note: As FX does not currently handle aliasing, this pass currently assumes nothing aliases. If that isn't true, use at your own risk. T heuristic) conv_bn_fusermkldnn_layout_optimizeNrrrFz+mkldnn_layout_optimize config is not a dictz4Heuristic not found in mkldnn_layout_optimize configc@seZdZdZdZdZdS)z*optimize_for_inference..MklSupportrrN)r\r]r^NOYESUNKNOWNr r r r! MklSupportlsrr&z)this pass is only for torch.float modulescpuz!this pass is only for CPU modules call_functioncss|]}|jdkVqdS)rN)r)rmargr r r! sz)optimize_for_inference..csd|fS)Nr) call_methodrrr r!rksz(optimize_for_inference..rrrrcs0t|dr |jSt|dr|jSdS)Ncolor start_color)hasattrrrrr)ufr r! get_colors    z)optimize_for_inference..get_colorz!Expected color for to_dense inputcs,g|]}t|tjr|dur|qSrg)r*r+r,r)rr r!ros  z*optimize_for_inference..css|]}|duVqdSrgr rr r r!rszFound None in cur_colorsrcstSrg)rr rr r!rkrlrr end_colorzmkldnn conversions: %s)Frupdaterrr*rF RuntimeErrortracerArBr+rDrootrGrrr0rr-rr/mkldnn_supportedrnext parametersdtyperCr|r5devicemkldnn_supported_unknownranyr(inserting_beforemap_argrr_r$rinserting_after create_noderKrr~rIrLr'r enumeraterrrall_input_nodessortedrrrrrappendrrvaluesrlogging getLoggerr\infort)r8rrdefault_pass_configr cur_tracerr%rr$supports_mkldnnrsample_parameter mkldnn_argsdense_xr~prv_noderIuser num_nodescur_idx cur_colors other_color mkldnn_graphsrHprvmkldnn_conversionsresultr )rrrr!r@s                                       r)FF)rr)KrAroperatorr collectionsrcollections.abcrenumrtypingrrrCtorch.fxr+torch.nnr9torch.nn.functional functionalFtorch.utils.mkldnnutilsmkldnnrw torch.fx.noderrtorch.fx.passes.shape_propr torch.nn.utils.fusionr r __all__r.r_r"r/r,rFr r}r rrrrr<r@r=ReLU MaxPool2d AvgPool2dAdaptiveAvgPool2drelu transposesigmoid avg_pool2dadaptive_avg_pool2draddmulr MkldnnConv2d MkldnnLinearr{rrrrrrrTracerrr r r r!s            (   "   1