On double cosets of groups $GL(n)$ with respect to subgroups of block strictly triangular matrices

We parametrize the space of double cosets of the group $GL(n,\Bbbk)$ with respect to two subgroups $T_-$, $T_+$ of block strictly triangular matrices. In Addendum, we consider the quasi-regular representation of $GL(n,\Bbb{C})$ in $L^2$ on $T_- \ GL(n,\Bbb{C})$, observe that it admits an additional group of symmetries, find the joint spectrum, and observe that it is multiplicity free.

1 The statement 1.1. Double cosets Let G be a group, K, L subgroups. A double coset of G with respect to K, L is a set of the type K · g · L, i.e., the set of all elements of G that can be represented in the form kgl, where g is fixed, k ranges in K, l ranges in L. We denote the set of all double cosets by K \ G/L.
A description of this set is equivalent to a description of orbits of K on the homogeneous space G/L, and to a description of orbits of L on the homogeneous space K \ G. If G is finite, then a description of double cosets is equivalent to a description of intertwining operators between quasi-regular representations of G in ℓ 2 (G/K) and ℓ 2 (G/L) (see, e.g. [6], Sect. 13.1). For Lie groups and locally compact groups picture is more complicate, in any case understanding double cosets seems necessary for understanding analysis on the corresponding homogeneous spaces.
In any case a problem of description of double cosets arises quite often, but not quite often it admits a tame solution.
1.2. The problem. Let k be a field, V be a finite dimensional linear space over k. Denote by GL[V ] be the group of all invertible linear operator in V . We also use the notation GL(n) = GL(n, k) for GL[k n ]. Split V into a direct sum Denote T + [V 1 , . . . , V p ] = T + (α 1 , . . . , α p ) = T + (|α|) the group of all block strictly upper upper triangular matrices of the size (α 1 + · · · + α p ) × (α 1 + · · · + α p ), i.e., matrices of the form      1 α1 * . . . where 1 m denotes the unit matrix of size m. By P + [. . . ] = P + (. . . ) we denote the group of all block triangular matrices, i.e., we allow arbitrary invertible matrices on places of units. Clearly, T + is normal in P + , we denote the corresponding groups of lower triangular matrices.
In this paper we describe double coset spaces 1.3. The statement. Recall some definitions. Let X, Y be linear spaces over k. A linear relation (see, e.g., [10], Sect. 2.5) L : X ⇒ Y is a linear subspace in X ⊕ Y . A graph of a linear map X → Y is a linear relation but not vice versa. For a linear relation we define: If L is a graph of a linear operator A : X → Y , then kernel, image, and rank are the usual kernel, image, and rank. In this case also dom L = X, indef L = 0.
Any linear relation L determines a canonical invertible operator Moreover a linear relation L : X ⇒ Y is determined by subspaces ker L ⊂ dom L in X, subspaces indef L ⊂ im L in Y and the operator Θ(L). Now consider a linear space V ≃ K n and take two decompositions Consider the double cosets (1.2). For each element A ∈ GL[V ] we assign a canonical collection of linear relations The relations χ ij depend only on the double coset containing A; b) The relations χ ij = χ ij (A) satisfy the conditions: and We say that a collection of linear relations ξ ij :   -Let G be a real semisimple group, H, L are symmetric subgroups 3 . In particular, this problem includes the Jordan normal form (more generally, description of conjugacy classes in all semisimple Lie groups 4 Q), reduction of pairs of nondegenerate quadratic (or symplectic) forms, canonical forms of pairs of subspaces in a Euclidean space, etc. A formal reference to a 'general case' is [8].
-We consider H \ G/P , where G is semisimple group, H is a symmetric subgroup, P is a parabolic subgroup (a block triangular subgroup). A formal reference to a 'general case' is [7].
-For p-adic groups the most important case is related to the Iwahori subgroups, see [5].
There is a cloud of minor variations of these series (we can slightly enlarge G, or slightly reduce subgroups).
Next, there are different ways to assign spectral data to several matrices (this also an be regarded as a classification of double cosets): a spectral curve with a bundle, see [13], [3], [4], or a spectral surface with a sheaf, see [2].
On the other hand, for infinite-dimensional groups quite often a double coset space K \ G/K has a structure of a semigroup. There arise questions about spectral data visualizing such multiplications. This also leads to objects of algebraic-geometric nature as spaces of holomorphic maps of Riemann sphere 2 Cf. similar objects in [9]. 3 Recall that a subgroup H ⊂ G is symmetric if it is the set of fixed points of some involution σ : G → G (i.e., σ(g 1 )σ(g 2 ) = σ(g 2 g 1 ), σ(σ(g)) = g. 4 Namely, we set G = Q × Q, H = L = diag Q is the diagonal subgroup.
to Grassmannians (see [10], Sect.X.3) or rational maps of Grassmannians to Grassmannians (see [11]). Our case arose as a byproduct of a construction of the latter type in [12] (proof of Theorem 1.6), it is quite elementary. However, I could not find it in the literature. Apparently, a natural generality here are spaces H \ G/T , where G is a classical (or semisimple group), H is a symmetric subgroup, and T is the maximal unipotent subgroup in a parabolic subgroup.
A possibility to describe double coset space implies a question about harmonic analysis for L 2 on T + (α 1 , . . . , α p ) \ GL(n). Such an analysis is possible, see Addendum to this paper (but it is not directly related to the description of double cosets). Reformulate the definition of χ ij (A) in the following way. We consider the intersection Next, we send Z to the quotient Clearly, the relation . Then there are x i and x 1 , . . . , x i−1 , y j+2 , . . . , y q satisfying the equation. This implies that x i ∈ indef χ ij .
2.3. Verification of (1.4)-(1.5). To be definite, let us prove the statements from the first row (1.4). Chow But the right hand side must be zero and η = 0. The statement im χ qj = V j follows from the surjectivity of A.

The action of GL[W j ]× GL[V i ] on the double cosets space.
Let G be a group, K, L its subgroups, K and L the normalizers of K and L. Then the group Our subgroups contain the usual subgroups of lower and upper triangle matrices respectively. Applying the usual Gauss reduction we observe that any double coset contains a 0-1-matrix 5 , i.e., an element of the symmetric group S(n). After this reduction we can permute basis elements in each V i and in each W j , so the double coset space (2.2) is in one-to-one correspondence with Our matrix has a natural decomposition into pq blocks, it is important only the number of units in each block. We formulate our observation in the following complicate form.
A representative of the double coset is the map sending each V j i to the corresponding W i j coordinate-wise. Less formally, we get a matrix of the form Here p = 4, q = 3. We present a decomposition of a matrix into blocks corresponding to the decompositions ⊕V i and ⊕W j , and refined blocks corresponding to decompositions ⊕ ⊕ V j i and ⊕ ⊕ W i j . Units are put in bold to make them visible among zeros.
Lemma 2.2 For the matrix (2.5) the corresponding linear relations χ ij are the following: The operator We say that such a bi-hinge is standard and denote it by The statement is obvious. Indeed, let us write χ 32 for the matrix (2.5) (the general case differs from considerations below only by longer notation). We apply this matrix to a vector .

The action of GL[V i ] × GL[W j ] on the set of bi-hinges. Clearly, the group GL[V i ] × GL[W j ] acts on the space
therefore it acts on the set of bi-hinges. b and the unipotent group Proof. a) In a fixed V i we have a flag We Similarly, we fix W j , consider the flag We get the desired canonical form. The statement b) also becomes obvious, since the stabilizer must regard the flags in each V i and W j and the maps Θ(·). Remark. Notice that the inverse inclusion of stabilizers follows from the equivariance.

Coincidence of stabilizers. Thus we have a map
⊠ Proof. The stabilizer of a standard hinge is described in Lemma 2.3. It is a product of subgroups (2.14) and (2.15). For the reductive factor (2.14) the statement is clear. The unipotent factor itself is a product, and it is sufficient to prove the statement for any factor in (2.15), say To be definite, we show what is happened for the matrix J given by (2.5) and a matrix  Multiplying (2.5) by this element we get the matrix We put nonzero symbols in bold to make them visible on the field of zeros. Clearly, this matrix can be reduced to the initial form (2.5) by a left multiplication by an element of T − [W 1 , . . . , W q ]. The boxed units allow to delete x, y, z.
Next, consider the action of GL(n, C) × GL(n, C) on GL(n, C) by left and right multiplications, g → h −1 1 gh 2 . This determines the left-right regular representation of GL(n, C) × GL(n, C) in L 2 GL(n, C) . According Gelfand and Naimark, see, e.g., [1], Sect. 14.4.A, this representation decomposes as a direct integral of representations of GL(n, C) × GL(n, C).
Theorem A.1. The decomposition of L 2 (T + (α 1 , . . . , α p ) \ GL(n, C) under the action of the group G = p i=1 GL(α i , C) × GL(n, C) is multiplicity-free and has the form Proof. We have a space homogeneous with respect to the group G. The stabilizer G 0 of the initial point consists of tuples By definition our representation is induced from the trivial representation of the stabilizer G 0 . Consider a lager group G * 0 = GL(α i , C) × P + (α 1 , . . . , α p ).
The second group is the double GL(α j , C) × GL(α j , C), the first group is GL(α j , C) embedded to the double as the diagonal. So the induced representation is the left-right representation of the double. Clearly, it is equivalent to the tensor product of the left-right regular representations of the factors GL(α j , C) × GL(α j , C), We decompose spaces L 2 (GL(α j , C)) according (A.3).