[Spec][MOE][Internal Op] Specification of MOE internal operation

[mitruska]

Details:

• Specification of MOE internal operation
• Internal ops are used mainly for fusion transformations and optimizations,
  they will not appear in the converted model public IR

Describes MOE used in PR:

• [Transformations][MOE] Add MOE internal op and fuse vectorized MatMul experts into MOE #32183

Tickets:

• 171911

[mitruska]

GPU plugin request is to transpose those weights at conversion stage, so the MatMul both transpose_a/b attrs should be False at this point:

Suggested change

	# Experts computation part (GEMM3_SWIGLU)
	x_proj = matmul(reshaped_hidden_states, weight_0, transpose_a=False, transpose_b=True)
	x_proj2 = matmul(reshaped_hidden_states, weight_1, transpose_a=False, transpose_b=True)
	swiglu = swish(x_proj, beta=expert_beta)
	x_proj = x_proj2 * swiglu
	down_proj = matmul(x_proj, weight_2, transpose_a=False, transpose_b=True)
	# Experts computation part (GEMM3_SWIGLU)
	x_proj = matmul(reshaped_hidden_states, weight_0, transpose_a=False, transpose_b=False)
	x_proj2 = matmul(reshaped_hidden_states, weight_1, transpose_a=False, transpose_b=False)
	swiglu = swish(x_proj, beta=expert_beta)
	x_proj = x_proj2 * swiglu
	down_proj = matmul(x_proj, weight_2, transpose_a=False, transpose_b=False)

cc: @yeonbok

[rkazants]

let us not use MoE name because we can use it for external operation and for real MoE operation. Now it is a sort of FusedExperts.

[rkazants]

I propose to add router_topk_output_indices into this logic. It will show how weights are prepared. Now it is not clear how router_topk_output_indices is used in the specified operation.

[rkazants]

Good job! Thank you, Kasia. Left a couple of comments,