Otdfbt

This NASA source states that the mission was terminated after Neil Armstrong used 75% of the RCS propellent to cancel the rotation from the OAMS thruster failure. If the mission had enough RCS left to have a safe re-entry and policy was to not use the RCS until the end of the mission anyways, why did they not attempt to dock again and complete the mission?

As a second question, if they had not undocked from Agena and had used the RCS to recover anyways, would they have still terminated the mission?

I haven't been able to find the Gemini rules online. But we can infer what they stated based on comments in the Gemini VIII post flight report.

MCC-H made the decision for early mission termination. This
decision was based on data which showed RCS propellant remaining in
both rings to be less than half the amount loaded. Also, both rings
of the RCS had been activated and significant propellant had been
used. Mission rules required termina­tion of the mission under
these conditions.

So the situation was:

• Primary attitude control system failed and unusable.


• Backup attitude control systems have been activated early and
  significant propellant has been used.

This is what we would have called in Shuttle a "zero fault tolerant" situation. In other words, if anything else bad had happened to the RCS, the crew would have died. When the mission rules were written, they must have stated that this type of situation called for early mission termination.

The truly interesting thing about this situation is that the flight director followed the mission rules, terminated the mission early, and it ended his career as a flight director.

As Wayne Hale writes:

NASA management found out about the situation after the crew was in
the ocean. According to the legend, Hodge did not take the time to
pick up the phone and call the Program Manager, the Center Director,
or even his boss, the Chief of the Flight Director office. The
situation was stable, and even though waiting around was not
necessarily a good thing, there was no reason that a couple of hours
delay would have significantly increased the crew risk. Upper
management was severely out of sorts with Blue Flight because they
were not called in to review a critical action that really could have
waited, despite what the Flight Rules called for.

Bottom line: John Hodge never served as Flight Director in Mission
Control again.

Which is why we sarcastically referred to the Flight Rules as "Flight Guidelines".

There are only two ways to mess up in Mission Control: following the Flight Rules, or not following the Flight Rules.

Update: Hale goes on to say

But that is not the point of the fable. The moral of the story for all
rookie Flight Directors is ALWAYS INVOLVE YOUR MANAGEMENT. Any time
that a critical action can reasonably be delayed for even a few
minutes GET ON THE PHONE WITH THE BOSS. No matter what the Flight
Rules say. After all, it’s just your career on the line.

Note: the linked Post Flight Mission Report is a terrific reference on this mission. Highly recommended.