Otdfbt

The Rebel fleet at the end of Rogue One (even after sustaining many losses) still seems considerably larger than the fleet in A New Hope.

General Dodonna essentially answers this question during the briefing:

The battle station is heavily shielded and carries a firepower greater
than half the star fleet. It's defenses are designed around a direct
large-scale assault. A small one-man fighter should be able to
penetrate the outer defense.

Knowing that the Death Star is on the way to the Rebel base, and knowing that a direct assault would be a useless waste of resources, the battle plan sticks exclusively to only those resources that have a chance of making any difference. Also - all of the capital ships didn't necessarily make it to Yavin 4:

"The remaining Alliance forces on Yavin 4 were a ragtag
collection....almost no capital ships had survived the battle at
Scarif"

- Sparks: From a Certain Point of View

Though it does appear that some ships did escape Yavin 4 prior to the battle:

"[I] Don't think they're planning to evacuate," Quersey said. "Most of
the equipment left with the fleet. It's all in the hands of the pilots
upstairs."

- Duty Roster: From a Certain Point of View

We don't see the Rebel Fleet in A New Hope, there are no Captial ship or Escorts that feature in the film.

Apart from the Tantive IV that is captured at the start of the film of course.

We only see the forces posted on Yavin, it's possible that a portion of the remaining Rebel forces fighters are with the larger ships and only some on Yavin.

ANH does not show a capital rebel fleet at any point, because it is simply not involved, for good reason.

After analyzing the Death Stars plans, General Dodonna says in the briefing:

The battle station is heavily shielded and carries a firepower greater than half the star fleet. It's defenses are designed around a direct large-scale assault. A small one-man fighter should be able to penetrate the outer defense. [...] An analysis of the plans provided by Princess Leia has demonstrated a weakness in the battle station. [...] You are required to maneuver straight down this trench and skim the surface to this point.

Rebel pilots were skeptical that the shot that was needed to destroy the DS was at all possible ("That's impossible, even for a computer.").

The rebel leaders aren't stupid. At this point they know that attacking the DS with a capital fleet is just suicide. They've witnessed the destructive power at Scarif. That is why they chose to go another route and only send in small fighters, mainly X-Wings and Y-Wings:

In this shot, we do not see any capital ships involved in the attack – only small fighters – as the trench run was only possible with small fighters and capital ships would just have been target practice for the DS.

Furthermore it doesn't look like there was much left of the rebel fleet that participated at Scarif. Leias blockade runner barely escaped the fray at the end of Rogue One with most of the fleet destroyed, which is confirmed in a short story:

The remaining Alliance forces on Yavin 4 were a ragtag collection of starfighters and light cruisers – almost no capital ships had survived the battle at Scarif.
— Sparks: From a Certain Point of View

Thus, the rebels use only the instruments that actually have a chance to make a difference, without using any capital ships that may or might not be present at Yavin in a guaranteed suicide mission.

There is a large time gap between episode 4 and Rogue One. During this time, it is entirely possible that other battles have weakened the rebel fleet.