Let m be a positive integer less than n, and let A B be n×n complex matrices. The mth A-exterior numerical radius of B is defined as
where Cm( A) is the mth compound matrix of A and Un is the group of n×n unitary matrices. Marcus and Sandy conjectured that a (necessary and) sufficient condition for to imply that the rank of B is less than m is that A is nonscalar and counter-example to the conjecture was given in [10]. In this paper, using geometric arguments, we show that the conjecture is true in the special case when A is of rank m. Then we prove that in order that for each B of rank m it is necessary and sufficient that rank A≤m and rank (A—λI)≤min{m, n—m) for all complex numbers λ. We also find other equivalent conditions one of which is that, for any pair of linear subspaces X 9 Y of Cn with dim X= m and dim Y =n— m, there exists UεUn such that Cn = AU(X)⊗U(Y). To establish the equivalence of the above conditions, we give a structure theory of the anti-invariance of matrices, which relies on a combinatorial method and the use of gap between subspaces. As by-products, we obtain two interesting results about matchings of a multi-set. The cases when Unis replaced by GL(n, C) or the underlying field is the set of real numbers are also examined carefully. Some open problems are posed at the end.
