Let ${\cal M}$ be an o-minimal expansion of a dense linear order $(M,\lt)$.
Let $f:I \longrightarrow M$ be definable, with $I = (a,b)$ an interval in $M$.
Definition
We call $f$ strictly monotone if $f$ is either constant, or strictly increasing, or strictly decreasing.
Our first goal is to prove the
Monotonicity Theorem
There are $n \in {\mathbb N}$ and $a = a_0 \lt a_1 \lt \cdots \lt a_n = b$ such that the restriction of $f$ to the interval $(a_{i-1},a_i)$ is strictly monotone and continuous, for $i=1, \dots, n$.
See Lemma 4 in Pillay and Steinhorn for original statement and van den Dries for a proof. The proof I will give here is due to Peterzil and Starchenko.
The first step in proving the monotonicity theorem is to reduce it to a “local” statement. We say that $f$ is strictly monotone at $x \in I$ if there exist $a’ \in (a,x)$ and $b’ \in (x,b)$ such that the restriction of $f$ to the interval $(a’,b’)$ is strictly monotone.
Exercise
Assume that $f$ is strictly monotone at every $x \in I$. Prove that $f$ is strictly monotone.
Monotonicity Lemma
There is an open subinterval $J \subseteq I$ such that the restriction of $f$ to $J$ is strictly monotone.
We will prove this lemma later; for now, let us see how to derive the Monotonicity Theorem from it. We need one other
Observation
Assume that $f$ is strictly monotone. Then there exists an open subinterval $J \subseteq I$ on which $f$ is continuous.
Proof. We may assume that $f$ is not constant. So by o-minimality, the image $f(I)$ contains an open interval $J’$, and $f^{-1}(J’)$ is an interval $J$ on which $f$ is either an order-preserving or an order-reversing bijection. $\Box$
Proof of the Monotonicity Theorem from the Monotonicity Lemma: Let $A$ be the set of all $x \in I$ such that $f$ is continuous and strictly monotone at $x$. This $A$ is definable and, by the exercise, the restriction of $f$ to any open interval contained in $A$ is continuous and strictly monotone. By the Monotonicity Lemma and the observation, the complement of $A$ in $I$ contains no open interval. So by o-minimality, the complement of $A$ in $I$ is finite. $\Box$
Exercises
- The limits $\lim_{x \to a^+} f(x)$, $\lim_{x \to b^-} f(x)$ and, for $c \in (a,b)$, the limits $\lim_{x \to c^-} f(x)$ and $\lim_{x \to c^+} f(x)$ exist in $M \cup \{-\infty,+\infty\}$.
- If $c,d \in M$ and $g:[c,d] \longrightarrow M$ is definable and continuous, then $g$ attains a maximum and a minimum in $[c,d]$.
- Let $f_1, \dots, f_k:I \into M$ be definable. Show that there are $n \in {\mathbb N}$ and $a = a_0 \lt a_1 \lt \cdots \lt a_n = b$ such that the restriction of $f_j$ to the interval $(a_{i-1},a_i)$ is strictly monotone and continuous, for $i=1, \dots, n$ and $j=1, \dots, k$.
Here is an attempt at the first exercise:
Since $f$ is definable there exists a formula $\phi$ defining the graph of $f$. We note that the set of $x\in I$ for which there is a neighborhood about $x$ where $f$ is constant is definable (with parameters) via the formula $$\exists y\phi(x,y)\wedge\exists s\exists t(s<x<t\wedge\forall z(s<z<t\to f(z)=k)),$$ where $k$ is some constant in $M$. The set of $x$ for which there is a neighborhood about $x$ where $f$ is strictly increasing is also definable is also definable via the formula $$\exists y\phi(x,y)\wedge\exists s\exists t(s<x<t\wedge\forall u\forall v(s<u<v<t\to f(u)$, and $X_<$. We will show that each is open and closed. To show openness, take $x\in X_=$. Then there is a neighborhood $U$ about $x$ on which $f$ is constant. But then for any $y\in U$ there is a neighborhood around $y$ for which $f$ is constant, namely $U$ itself, and so $X_=$ is open. To show closedness let $x$ be a limit point of $X_=$. By the fact that $f$ is strictly monotone at every $y\in I$, we have that there exists an interval $I$ about $x$ on which $f$ is strictly monotone. Since $x$ is a limit point of $X_=$, this interval $I$ contains a point $y\in X_=$. But then, it must be that $f$ is constant on $I$, since $I$ has constant monotonicity and contains at least part of a region, where $f$ is constant. Showing openness and closedness for $X_$ is identical to the above. Since every interval is definably connected, it follows that each of $X_=$, $X_$ must be empty or all of $I$. The result follows.
For the exercises at the end, I think you can claim that $\lim\limits_{x\to a^+}$ is $\text{inf}(f((a_0,a_1)))$, where $(a_0,a_1) $ is the first interval on which $f$ is strictly monotone and continuous (obtained via monotonicity theorem), assuming $f$ is increasing on that interval. Taking an arbitrary neighborhood around $\text{inf}(f((a_0,a_1)))$ we get that the intersection is some sub-interval of $f((a_0,a_1))$. Since $f$ is continuous the pre-image is open and contains a neighborhood which has $a$ as a limit point… I’m not too sure how to formalize this, but it seems to work.
In your formula defining local constancy, if you replace the parameter $k$ by $f(x)$, then you do not need the parameter $k$.
In your definition of $X_\gt$, you mean $f(u) \lt f(v)$?
To show closedness in $I$, you also assume that the limit point $x$ belongs to $I$ (which you seem to without saying it).
Otherwise, your argument for the first exercise is correct.
For the next exercise, wouldn’t the left endpoint of this first interval have $a_0 = a$ as left endpoint? Here is how I would say this: by the Monotonicity Theorem, there exists $a_1 > a$ such that the restriction of $f$ to the interval $(a,a_1)$ is strictly monotone. Now describe $\lim_{t \to a^+} f(t)$ depending on the three possible cases of monotonicity of this restriction of $f$.