o q::j=@sddlmZddlZddlZddlZddlZddlZddlmZddl m Z m Z ddl m Zddlmm ZddlmZmZmZmZmZe rOddlmZmZd[ddZehdZeeZdZ dZ!GdddZ"d\ddZ#d]dd Z$d^d#d$Z%    d_d`d+d,Z& -  . /dadbd3d4Z'dcd5d6Z(ddd8d9Z)dedlengthoffsetbasepathchecksumr dc@seZdZdddZdS) ExternalDataInfotensorr rNonec Csdd|_d|_d|_d|_d|_t}d}|jD]1}|jtvr't ||j|j q|d7}t |t krH|jdt }t |jt krC|d7}||q|rvt|}|t |}t|}|dkre|d|d7}tjd|d|jd td d |jdurt|j|_|jdkrtd |jd|j|jdurt|j|_|jdkrtd |jd|jdSdS)Nrz...z and z morez&Ignoring unknown external data key(s) z for tensor z. Allowed keys: ) stacklevelz/External data offset must be non-negative, got z/External data length must be non-negative, got )rrrrrset external_datakey_ALLOWED_EXTERNAL_DATA_KEYSsetattrvaluelen_MAX_UNKNOWN_KEYS_IN_WARNING_MAX_KEY_DISPLAY_LENGTHaddsortedreprwarningswarnname_SORTED_ALLOWED_KEYSr ValueError) selfr! unknown_keysunknown_key_countentry truncatedshownextrakey_listrrr__init__:sb             zExternalDataInfo.__init__N)r!r rr")__name__ __module__ __qualname__r@rrrrr 9sr data_file IO[bytes]infobytesc Cst|j}|jdur&|j|kr td|jd|d|||j|jdurU|jdur3|jnd}||}|j|krOtd|jd|d|d|||jS|S) a Validate offset/length against actual file size and read data. Layer 3 defense-in-depth (CWE-400): prevents memory exhaustion even if the model was crafted via direct protobuf APIs that bypass Python parsing. Returns the raw bytes read from the file. NzExternal data offset (z) exceeds file size (z ) for tensor rzExternal data length (z) exceeds available data (z bytes from offset ) osfstatfilenost_sizerr7seekrread)rDrFr file_size read_start availablerrr#_validate_external_data_file_boundsms4        rQr!r r"cCs\t|}t||j|jd}t|d}t|||j|_WddS1s'wYdS)zLoads data from an external file for tensor. Ideally TensorProto should not hold any raw data but if it does it will be ignored. Arguments: tensor: a TensorProto object. base_dir: directory that contains the external data. TrbN)r rrr5rHfdopenrQraw_data)r!r rFfdrDrrrload_external_data_for_tensors"rVmodelr cCs8t|D]}t|rt||tj|_|jdd=qdS)zLoads external tensors into model Arguments: model: ModelProto to load external data to base_dir: directory that contains external data N)_get_all_tensorsuses_external_datarVr DEFAULT data_locationr()rWr r!rrrload_external_data_for_models   r\r int | Nonerr str | Nonerc Cs|dstd|jd|jdd=tj|_||dur!t|nd|dur*t|nd||dD]\}}|durG|j }||_ t ||_ q2dS)NrTzTensor zH does not have raw_data field. Cannot set external data for this tensor.)rrrrr) HasFieldr7r5r(r EXTERNALr[ritemsr0r)rr,) r!rrrrrkvr;rrrset_external_datas(      rdTFall_tensors_to_one_filesize_thresholdconvert_attributec Cst|}|r t|}|rIttd}|r0tj|r tdtj |r.t d|d|}|D]}| drFt |j|krFt||q2dS|D]!}| drlt |j|krl|j}t|sgtt}t||qKdS)aCall to set all tensors with raw data as external data. This call should precede 'save_model'. 'save_model' saves all the tensors data as external data after calling this function. Arguments: model (ModelProto): Model to be converted. all_tensors_to_one_file (bool): If true, save all tensors to one external file specified by location. If false, save each tensor to a file named with the tensor name. location: specify the external file relative to the model that all tensors to save to. Path is relative to the model path. If not specified, will use the model name. size_threshold: Threshold for size of data. Only when tensor's data is >= the size_threshold it will be converted to external data. To convert every tensor with raw data to external data set size_threshold=0. convert_attribute (bool): If true, convert all tensors to external data If false, convert only non-attribute tensors to external data Raise: ValueError: If location is not a relative path. FileExistsError: If a file already exists in location. z.datazDlocation must be a relative path that is relative to the model path.zExternal data file exists in .rTN)_get_initializer_tensorsrXruuiduuid1rHpathisabsr7existsFileExistsErrorr_sys getsizeofrTrdr5_is_valid_filename) rWrfrrgrhtensors file_namer!tensor_locationrrrconvert_model_to_external_datas<     rwcCs@t|D]}t|r|dstd|jdd=tj|_qdS)zCall to set all tensors which use external data as embedded data. save_model saves all the tensors data as embedded data after calling this function. Arguments: model (ModelProto): Model to be converted. rTraw_data field doesn't exist.N)rXrYr_r7r(r rZr[)rWr!rrr convert_model_from_external_datas   ry base_pathcCst|}|dstdt||j|jd}t|dC}| dd|j durB| }|j |kr<| d|j || |j | }| |j t||j|| |WddS1scwYdS) aWrites tensor data to an external file according to information in the `external_data` field. The function checks the external is a valid name and located in folder `base_path`. Arguments: tensor (TensorProto): Tensor object to be serialized base_path: System path of a folder where tensor data is to be stored Raises: ValueError: If the external file is invalid. rTrxFzr+brr%N)r r_ onnx_checkerValidationErrorrrr5rHrSrLrtellwriterTrd)r!rzrFrUrDrNrrrrsave_external_datas       "ronnx_model_protoIterable[TensorProto]cCstt|t|S)z=Scan an ONNX model for all tensors and return as an iterator.)rrj_get_attribute_tensorsrrrrrX5srX attributerfunc-Callable[[GraphProto], Iterable[TensorProto]]ccsL|jtjkr||jEdH|jtjkr"|jD] }||EdHqdSdS)zICreate an iterator through processing ONNX model attributes with functor.N)typerGRAPHgGRAPHSgraphs)rrgraphrrr_recursive_attribute_processor=s   rrrccs:|jEdH|jD]}|jD] }t|tEdHqq dS)z@Create an iterator of initializer tensors from ONNX model graph.N) initializernoderr#_get_initializer_tensors_from_graph)rrrrrrrHs    rccst|jEdHdS)z:Create an iterator of initializer tensors from ONNX model.N)rrrrrrrjRsrjgraph_or_functionGraphProto | FunctionProtoccsL|jD]}|jD]}|dr|jV|jEdHt|tEdHq qdS)zSCreate an iterator of tensors from node attributes of an ONNX model graph/function.tN)rrr_rrtr!_get_attribute_tensors_from_graph)rrrrrrrWs     rccs0t|jEdH|jD] }t|EdHq dS)zDCreate an iterator of tensors from node attributes of an ONNX model.N)rr functions)rfunctionrrrres  rfilenamecCstd}||}t|S)z8Utility to check whether the provided filename is valid.z^[^<>:;,?"*|/]+$)recompilematchr)rexprrrrrsls  rscCs|do |jtjkS)z?Returns true if the tensor stores data in an external location.r[)r_r[r r`)r!rrrrYss  rY field_keycCs*t|jD] \}}|j|kr|j|=qdS)aRemoves a field from a Tensor's external_data key-value store. Modifies tensor object in place. Arguments: tensor (TensorProto): Tensor object from which value will be removed field_key (string): The key of the field to be removed N) enumerater(r))r!rifieldrrrremove_external_data_field{s  rfilepathcCs8t|D]}t|r|drt|||dq|S)aSerializes data for all the tensors which have data location set to TensorProto.External. Note: This function also strips basepath information from all tensors' external_data fields. Arguments: model (ModelProto): Model object which is the source of tensors to serialize. filepath: System path to the directory which should be treated as base path for external data. Returns: ModelProto: The modified model object. rT)rXrYr_r ClearField)rWrr!rrrwrite_external_data_tensorss   r) r rrrrrrrrr)rDrErFr rrrrG)r!r r rrr")rWr r rrr")NNNN)r!r rrrr]rr]rr^rr^rr")TNreF) rWr rfrrr^rgrrhrrr")rWr rr")r!r rzrrr")rr rr)rrrrrr)rrrr)rrrr)rrrr)r!r rr)r!r rrrr")rWr rrrr )4 __future__rrHrrqrkr3 itertoolsrtypingrr onnx.checkercheckerr|onnx.onnx_cpp2py_export.checkeronnx_cpp2py_exportr onnx.onnx_pbrrrr r collections.abcr r r frozensetr*r1r6r.r/r rQrVr\rdrwryrrXrrrjrrrsrYrrrrrrs\     4 #   :