I play D&D 3.5 with a group of folks. One of the rules they use in combat allows Alice, on Alice's turn, to ask Bob to yield his space to Alice. Bob chooses a space to move to, Alice moves into Bob's space.

I've tried to find a source for this rule and I've been unable to. I've looked in the PHB, the DMG, the Rules Compendium, the SRD, and I'm not finding anything. I have a strong suspicion that this may be a houserule that they've played with so long that it's just part of the game for them. I've never heard of anything like this and would like some input on where this rule might have come from. Does anyone know of anything like this in a splat, is it from another game?

It does not exist in the official rules. It’s impossible to prove a negative, but I am absolutely confident on this. The word “yield” is not used much at all in the core rules, and even then it’s used for three divinations to describe what information the spell yields, the discussion of a variant rule for summon monster spells, and for tracking, describing soft ground as one that “yields to pressure.” Furthermore, the core rules for acting in combat make no mention of an ability to vacate your space outside your turn.

The rules do specify that friendly creatures can pass through each other’s spaces, but they don’t generally allow you to stop on a square held by an ally. And that ally definitely cannot move into a different square as part of their ally’s movement.

Overall, though, I feel fairly positive about this house rule, at least at first glance. It enables defender types to interpose themselves between enemies and squishy allies, when normally it is extremely difficult to do that. Improving the effectiveness of that role would be good for 3.5e, I think. I have not really thought through all the ramifications, much less playtested it, though. As noted in Chris Morris’s comment and Ryan_L’s answer, care should be made to ensure that this process cannot be done infinitely, or else a character could move an infinite distance by yielding their space to each ally in an infinite line of allies. Ben Bardin’s comment that this could also allow two characters to yield back and forth to attack someone who otherwise could only be attacked by one due to a chokepoint is worth considering, too, though I’m not totally convinced that’s a bad thing per se.

Your playgroup probably hasn't implemented the emergent rule that free actions can only be taken on one's turn

So, most playgroups play with this rule where free actions can only be done on your turn. That's not explicitly stated anywhere, but there are some general passages that seem to sort-of imply it and Wizards of the Coast, the publishing company, said in an old blog post I can no longer find but which is referenced here both that free actions should not be allowed off-turn and that the implication of the general introduction to actions that all actions, including free action, happen only during your turn was, in fact, intended.

However, many groups exist that are unaware of this weird once-posted-about-in-a-blog-but-never-errataed pseudo-rule and, especially if the game sticks pretty close to the core material, allowing free actions off-turn is rarely a problem.

One immediately visible consequence of allowing off-turn free actions is off-turn 5' steps. If a character does not otherwise move any actual distance and isn't in difficult terrain, they can take a 5-foot step without provoking an attack of opportunity, pretty much whenever. How this works out in gameplay is pretty much what you are descibing:

"Hey, Fighter Bob, can you move? I wanna cast burning hands"

"Sure, Wizard Joan" (takes 5' step to an open spot after Joan enters their space but before Joan counts as having 'stopped moving')

Note, however, that Joan couldn't subsequently yield their space to another character in the same round, having already moved, nor could Bob, having already taken a 5' step, without a particularly lenient GM.

Also off-turn free-actions can be a problem in that they can be used by certain builds to do crazy stuff that might be described as breaking the game. But 3.5 isn't exactly a perfectly balanced game anyways and if it's working for your group currently I wouldn't worry about it.

There's a problem if you have a lot of characters adjacent in a line. It seems that, with careful attention paid to initiative, you could use this to get one character to move arbitrarily far in one turn.

Consider if you have 26 characters, adjacent and sorted by initiative such that the character at the leftmost end of the line goes last, but everyone to the right of him goes in order from left to right. Character B goes first, takes A's place. A decides to take B's old place. C takes one step left, takes A's place. A takes C's old place, repeat until you're out of characters. A has now moved 25 spaces for free, and still gets his turn.

To fix this, I would just add that any movement you do like this costs movement on your next turn, and you can't go below 0.

Not exactly, but you can take a 5-foot-step when it's not your turn.

"You can move 5 feet in any round when you don’t perform any other kind of movement." -- SRD

Since a 5-foot-step is a nonaction, just like Attacks of Opportunity are nonactions, you should be able to take a 5ft-step outside your turn. As quoted above, 5ft-steps and "other kind[s] of movement" are mutualy exclusive, and 5ft-steps are once per round. A round is composed of one turn each, in order of initiative. Logically, this means that taking a 5-foot-step on a higher initiative⁽¹⁾ than your own prohibits you from making "other […] movement". I suspect the designers of d&d never thought that far.

(I haven't looked in the rules compendium yet, so I don't know if it changes things)

tl; dr:

No, but you can take a 5-foot-step during any round that you don't otherwise move around.

(1): You might be able to take a 5ft-step during an initiative when there's no turn, or somesuch. Ask your resident RAW expert.