A summation formula for the Rankin-Selberg monoid and a nonabelian trace formula

Let F be a number field and let AF be its ring of adeles. Let B be a quaternion algebra over F and let ν: B → F be the reduced norm. Consider the reductive monoid M over F whose points in an F-algebra R are given by (Formula Presented). Motivated by an influential conjecture of Braverman and Kazhdan we prove a summation formula analogous to the Poisson summation formula for certain spaces of functions on the monoid. As an application, we define new zeta integrals for the Rankin-Selberg L-function and prove their basic properties. We also use the formula to prove a nonabelian twisted trace formula, that is, a trace formula whose spectral side is given in terms of automorphic representations of the unit group of M that are isomorphic (up to a twist by a character) to their conjugates under a simple nonabelian Galois group.