1: What are the mathematical model that he is rejecting?
I guess when he argues against figure/ground architectural thinking he could be talking about Cartesian coordinate system--however, Eisenman only mentions Cartesian coordinate system as only a description of some Leibniz' thinking as 'turning its back to Cartesian rationalism' because he suggested 'the smallest element in the labyrinth of the continuous is not the point but the fold.' Then goes on to talk about the fold as articulating a different kind of relationship between vertical and horizontal, or between figure and ground--something as revolutionary as breaking up Cartesian orders of space.

What is it about that continuous transition between ground and figure or vertical and horizontal that is so impossible in Cartesian space?

2: What is he replacing?
I suppose the answer you're looking for is topology since its been what we've been talking about. But in my limited understanding of topology, I get the impression that topology more about the relationship or connectivity of things rather than the elasticity of space as shown in the folded or bent Cartesian coordinate system that Eisenman obviously has interest in.

3: and why?
Searching in the space between figure and ground (what he calls fold) I think he hopes to find a bridge between Architecture and building--(The ideas associated with building and the manifestation of building). As an idea, its not a bad way to approach the problem but I still don't quite follow the how or the execution of these ideas within any processes yields the non-traditional, or non extruded architecture he envisions.

An extruded plan on a folded coordinate system is still topologically an extruded plan.