CANN/ge Concat无任务优化分析

Concat No Task Feature Analysis【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge1. Feature Background1.1 Problem ScenarioIn deep learning models,ConcatD/ConcatV2Doperators concatenate multiple input tensors along a specified dimension into one output tensor. The traditional Concat operator execution flow is:InputA ──┐ InputB ──┼──► Concat Task ──► Output (concatenated result) InputC ──┘TheConcat Taskrequires launching a computation task on Ascend AI Core. It uses DataMove instructions to transport each input data to a continuous memory region of the output address.1.2 Optimization ApproachWhen Concat operator input tensors arenaturally contiguously arrangedin memory, you do not need to execute any data transportation operations. You can directly reuse the first input address as the output address. The Concat No Task feature identifies this scenario during compilation and marks the Concat operator as a virtual operator (Virtual Op), achieving:No hardware Task generation: Skip AI Core task dispatchNo memory transportation: Output directly reuses input memory addressEliminate redundant computation: Avoid meaningless data movement2. User Usage Scenarios2.1 Typical Scenario: AllGather Concat in Distributed TrainingIn distributed training, multiple cards collect their respective data through AllGather and then concatenate into a complete batch:Card0: Data ──► AllGather ──┐ ├──► ConcatD ──► Complete Batch Card1: Data ──► AllGather ──┘AllGather output data is contiguously arranged in memory by card number order. Concat is logically concatenation, but physically does not require data transportation.2.2 Typical Scenario: Result Merging After Block ComputationSplit a large tensor across multiple operators for parallel computation and then merge results:Input ──► Split ──┬──► Relu ──┐ ├──► Relu ──┼──► ConcatD ──► Output └──► Relu ──┘When Split splits along batch dimension and each branch computation does not change memory layout, Concat inputs are naturally contiguous.2.3 Applicable ConditionsConcat No Task optimization requires simultaneously satisfying the following strict conditions:Condition CategorySpecific RequirementOperator typeOnlyConcatDandConcatV2DShape constraintAll dimension values before concat_dim axis must be 1Alignment constraintconcat_dim axis original size must be integer multiple of align_shape corresponding dimension (no padding)Input constraintCannot have Scalar input; all input tensor sizes must be 32-byte alignedSource constraintCannot have multiple inputs from same output anchorPredecessor nodeCannot be DATA, REFDATA, VARIABLE, CONSTANTOP, CONSTANT node typesPredecessor nodeCannot contain subgraphPredecessor attributeCannot have continuous_input, continuous_output, reference attributesSuccessor nodeSuccessor node cannot already be virtual operator (has _no_task attribute)Output constraintInput cannot simultaneously be model output (NetOutput)Memory typeAll input memory types must be consistentLxFusionCannot involve LxFusion (L1/L2/UB address, lxslice operator)Shape modeDoes not support Unknown Shape (dynamic Shape) scenarioGraph modeDoes not support Single Op scenario and memory non-contiguous allocation scenario3. External Interfaces3.1 Compilation Period Attribute MarkingConcat No Task interacts with other system modules through the following Graph attributes:Attribute NameTypeSetting ObjectDescription_no_taskboolConcat operator OpDescMark this operator does not generate hardware Task_nopadding_continuous_inputboolConcat operator OpDescMark input as continuous memory without padding_output_reuse_inputboolConcat operator OpDESCMark output reuses input memory_reuse_input_on_dim_indexint64Concat operator OpDescSpecify reused input memory dimension index (fixed as 0)can_reused_for_concat_optimizeboolPredecessor node output TensorDescMark this output is occupied by Concat optimization, cannot be reused by other Concat3.2 Pass RegistrationConcatNotaskPass registered as O3 optimization level GraphPass: REG_PASS_OPTION(ConcatNotaskPass).LEVELS(OoLevel::kO3);During compilation flow, this Pass runs inOptimizeStage2final stage, after subgraph merging completes and before memory conflict handling:graph_manager.cc: OptimizeAfterMergeSubGraph() ├── ... (early optimization) ├── BufferPoolMemoryPass ├── ParallelGroupPass └── ConcatNotaskPass ← Execute after graph stabilizes3.3 Runtime BehaviorAt runtime (RT1 and RT2), operators marked with_no_taskreceive special handling:RT1 (Davinci):TbeKernelHandle::NeedInitdetects_no_taskattribute and returns false, skipping Kernel initializationRT2 (V2):IsVirtualOpfunction detects_no_taskattribute, skipping Task generation in AICore Node Converter4. Specific Implementation4.1 Overall Architecture┌─────────────────────────────────────────────────────────────┐ │ Compilation Period (Compiler) │ │ │ │ ┌──────────────────┐ ┌──────────────────────────────┐ │ │ │ ConcatNotaskPass │───►│ Attribute Marking │ │ │ │ │ │ _no_task │ │ │ │ Verification chain│ │ _nopadding_continuous_input │ │ │ │ ├─ InputCheck │ │ _output_reuse_input │ │ │ │ ├─ CheckConcatDim│ │ _reuse_input_on_dim_index │ │ │ │ ├─ OutputCheck │ └──────────────────────────────┘ │ │ │ └─ LxFusionCheck│ │ │ └──────────────────┘ │ └─────────────────────┬───────────────────────────────────────┘ │ Attribute transfer ▼ ┌─────────────────────────────────────────────────────────────┐ │ Memory Allocation (Memory Assigner) │ │ │ │ ┌─────────────────────┐ ┌──────────────────────────┐ │ │ │ BlockMemAssigner │ │ GraphMemAssigner │ │ │ │ │ │ │ │ │ │ Detect NOPADDING_ │ │ Calculate continuous_type │ │ │ │ CONTINUOUS_INPUT │ │ kTypeInputNoPadding │ │ │ │ │ │ │ │ │ │ no_assign_memtrue │ │ Calculate nopadding_size │ │ │ │ (no independent │ │ by dim_index │ │ │ │ memory allocation) │ │ │ │ │ └─────────────────────┘ └──────────────────────────┘ │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Task Generation (Task Generator) │ │ │ │ Detect _no_task attribute → Skip this node Task generation │ │ MarkFirstAndLastOps skip notask nodes │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Runtime (Runtime) │ │ │ │ RT1: TbeKernelHandle skip initialization │ │ RT2: AICoreNodeConverter skip conversion │ │ Output address directly reuse first input address │ └─────────────────────────────────────────────────────────────┘4.2 ConcatNotaskPass Core Verification ChainConcatNotaskPass::Runexecutes the following verification chain for each ConcatD/ConcatV2D node in the graph. It sets attributes only after all pass:4.2.1 Scenario ExclusionSingle Op scenario: Single operator mode requires no optimizationMemory non-contiguous allocation: Graph withATTR_NAME_MEMORY_DISCONTIGUOUS_ALLOCATIONset skipsUnknown Shape: Operators with dynamic Shape in input or output skipDynamic Shape graph: Belonging graph marked with dynamic Shape partition skips4.2.2 InputCheck (Input Verification)Check each input anchor sequentially:IsScalarInput: Exclude dimension count 0 Scalar inputsCheckTensorAlign: In multi-input scenario, each tensor size must be 32-byte alignedHasSameSourceAnchor: ThroughGetFirstOutAnchorNotInRefNodetrace RefNode chain, ensure no multiple inputs trace back to same output anchorIsPreNodeTypeValid: ThroughGetFirstNotRefNodefind actual production node, exclude DATA/REFDATA/VARIABLE/CONSTANTOP/CONSTANTIsPreNodeWithSubgraph: Predecessor node cannot contain subgraph instanceIsPreOutAnchorCanReuseForConcatOptimize: Check predecessor output TensorDesccan_reused_for_concat_optimizeattribute, ensure not occupied by other ConcatIsPreOutAnchorValidMultiRef: If predecessor output simultaneously connects to NetOutput, cannot optimizeIsPreNodeAttrValid: Predecessor node cannot have continuous_input, continuous_output, reference, _no_task, _output_reuse_input, _nopadding_continuous_input attributes, nor atomic outputIsSameInputMemType: All input memory types must be consistent (check throughATTR_NAME_OUTPUT_MEM_TYPE_LIST)4.2.3 CheckConcatDim (Concat Dimension Verification)This is the core verification logic, ensuring all dimensions before concat_dim axis are 1:Original format (like NCHW) ──► Runtime format (like NC1HWC0) │ │ │ GetTransferDims() │ │ (call FE interface) │ ▼ ▼ src_to_dst_transfer_dims dst_to_src_transfer_dims {0},{1,4},{2},{3} {0},{1},{2},{3},{1}GetAlignedShape: Calltransformer::TransferShapeUtils::GetAlignedShapeto get aligned shapeGetTransferDims: Calltransformer::TransferShapeUtils::TransferDimsto get original format to runtime format axis mapping relationshipCheckRealConcatDim: Find actual concat_dim axis in runtime format, verify all dimension values before that axis are 1CheckConcatDimAlignment: Verify concat_dim axis original size is integer multiple of align_shape corresponding dimension (ensure no padding)CheckRealConcatDim key logic:Find real_concat_dim in runtime format throughsrc_to_dst_transfer_dims[concat_dim][0]If real_concat_dim is produced by axis merging (multiple source axes indst_to_src_transfer_dims), need additional verification:All axis values before real_concat_dim axis are 1All values before concat_dim in merged axis are 1If real_concat_dim is not produced by axis merging, only need to verify previous axis values are 14.2.4 OutputCheck (Output Verification)Traverse all successor nodes of Concat node:If successor is Reshape and has output node, continue checking Reshape output nodeSuccessor node cannot already have_no_task,_output_reuse_input,_nopadding_continuous_inputattributes4.2.5 LxFusionCheck (LxFusion Verification)IsLxFusionMem: Check input/output memory type is L1/L2/UB (on-chip memory used by LxFusion)IsLxFusionOp: Check operator name contains lxslice4.3 Attribute SettingAfter all verifications pass,SetAttrForConcatNotaskexecutes following operations:// Set Concat operator itself attributes AttrUtils::SetBool(op_desc, ATTR_NAME_NOTASK, true); AttrUtils::SetBool(op_desc, ATTR_NAME_NOPADDING_CONTINUOUS_INPUT, true); AttrUtils::SetBool(op_desc, ATTR_NAME_OUTPUT_REUSE_INPUT, true); AttrUtils::SetInt(op_desc, ATTR_NAME_REUSE_INPUT_ON_DIM_INDEX, 0); // Mark predecessor node output TensorDesc cannot be reused again for each input: AttrUtils::SetBool(output_tensor_desc, can_reused_for_concat_optimize, false);4.4 Memory Allocation Linkage4.4.1 BlockMemAssignerInAnalyzeSymbolMemReuseInfo, when detecting node hasATTR_NAME_NOPADDING_CONTINUOUS_INPUTattribute:if (is_input_continuous) { symbol_mem_reuse_info_[symbol].no_assign_mem_ true; }This means this symbol (memory block) will not be allocated independent memory, but reuse upstream memory address.GetContinuousNodeLifeTimeBeginmethod handles cascaded continuous input scenarios:a b c (first layer) | | | d e f (second layer) |___|___| | g h i (third layer, h is continuous input) |___|___| | j (fourth layer, j is continuous input)g cannot reuse memory with a/b/c, because d/e/f memory will be replaced by g address (cascaded continuous input). Therefore g lifetime start needs to trace back to earliest among d/e/f.4.4.2 GraphMemAssignerIdentify continuous type inGetContinuousType:kTypeInputNoPadding _nopadding_continuous_input _output_reuse_input kTypeOutputNoPadding _nopadding_continuous_output _output_reuse_inputInGetMemorySize, for nopadding type:Get dimension index through_reuse_input_on_dim_indexCalculatenopadding_size(actual data size) andtensor_size(complete size with padding)Memory allocator allocates precise size memory block based on this4.4.3 SetInputOutputOffsetPassFor no_task Concat nodes, if hasATTR_NAME_CONTINUOUS_INPUTor satisfies BufferFusion condition, will callSetOutputOffsetForConcatto set output offset, ensuring output address correctly points to input data starting position.4.5 Task Generation SkipInTaskGenerator::MarkFirstAndLastOps:bool attr_notask false; if (ge::AttrUtils::GetBool(op_desc, ATTR_NAME_NOTASK, attr_notask) attr_notask) { continue; // Skip notask node, not included in continuous node list }This ensures no_task nodes are not considered as part of continuous computation nodes in stream, not affecting first and last operator marking.4.6 Runtime Processing4.6.1 RT1 (Davinci Model)InTbeKernelHandle::NeedInit:bool attr_no_task false; const bool get_attr_no_task_flag AttrUtils::GetBool(op_desc, ATTR_NAME_NOTASK, attr_no_task); if (get_attr_no_task_flag attr_no_task) { GELOGI(Node[name:%s, type:%s] does not generate task, skip initialization., ...); return false; // Skip Kernel initialization }4.6.2 RT2 (V2 Engine)Inaicore_node_converter.ccIsVirtualOp:bool attr_no_task false; (void)ge::AttrUtils::GetBool(op_desc, ge::ATTR_NAME_NOTASK, attr_no_task); return attr_no_task; // Return true means virtual operator, skip conversion5. Coordination with Other Optimizations5.1 Coordination with Split No TaskConcat No Task frequently works with Split No Task, forming split-compute-merge zero-copy pipeline:Input ──► Split(no_task) ──► Computation ──► Concat(no_task) ──► OutputSplit splits along concat_dim reverse direction, output address directly points to different offsets of input address; Concat treats these offset addresses as continuous memory, directly reuses.5.2 Relationship with Memory ReuseConcat No Task nodes receive special handling in memory reuse check:InReuseChecker, nodes with_no_taskattribute are considered buffer_pool typeInmem_reuse_strategy.cc, nopadding continuous input nodes participate in special memory reuse strategy5.3 Relationship with Address RefreshInMemLayoutConflictUtil, Concat No Task scenario needs to consider address refresh:data | identity (insert identity operator to support address refresh) | split(no_task, no_padding_continuous_output) / \ op1 op2When involving user input and requiring address refresh, insert Identity operator between Data and Split.6. Test VerificationUnit test file located attests/ge/ut/ge/graph/passes/concat_notask_pass_unittest.cc, covering following scenarios:Test CaseVerification Contentallgather_connect_to_concatAllGather output connects to Concat, verify attribute setting correctallgather_connect_to_concat_reshapeAllGather → Reshape → Concat chainMultiple RefData testsVerify RefNode chain trace logicDifferent Format testsdim verification under ND, NCHW, NC1HWC0 formats7. Design Considerations7.1 Why Choose Attribute Marking Rather Than Graph RewritingConcat No Task uses attribute marking rather than node deletion approach, reasons are:Preserve debug information: Dump function needs to preserve operator OpDesc and address information (seeInitNoTaskAndDumpNeededNode)Maintain graph structure integrity: Deleting nodes destroys graph topology relationship, affects other Pass executionSupport dynamic scenarios: Attribute marking can flexibly handle in different compilation stages7.2 Why Verification Conditions Are So StrictConcat No Task verification conditions reach more than ten items, because:Memory safety: If input is not continuous but marked as no_task, will read wrong dataAlignment constraint: Ascend hardware has strict requirements on memory alignment, not satisfying 32B alignment may cause hardware exceptionCascading impact: One no_task node affects downstream memory allocation strategy, wrong marking will propagate7.3 Why Execute at Stage2 EndComment clearly states graph stabilized then do ConcatNotaskPass:Before Stage2, graph structure may still change (subgraph merging, operator fusion etc.)Predecessor node attributes may be modified in subsequent PassExecuting at end can ensure judgment based on final stable graph structure【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考