Otdfbt

How was a waitress called in the Middle Ages? Handmaid, waitress, maid, ...? And is there a different name for the ones who did this kind of job inside a castle, in contrast to the women doing this kind of job in, for example, a tavern?

From the top of my head, I remember the word wench, which originally meant girl, then a servant, and later also a prostitute, which is likely why it went out of use for waitresses.

Here’s a somewhat confirming article from 1988. (If link is blocked for you, use Wayback Machine copy.) It describes the job of a wench at Medieval Times, a “dinner theater” which still exists (Wikipedia, Official site). So, even if it’s a misconception, the idea that medieval waitresses were called wenches at least exists in modern mass conscience.

[What] was a waitress called in the Middle Ages?

In Europe, they didn't exist as a recognized occupation.

And is there a different name for the ones who did this kind of job inside a castle, in contrast to the women doing this kind of job in, for example, a tavern?

At a castle, the servants in the great hall would simply be servants (pre-Conquest, þrǣl, ancilla, &c.; post-Conquest, servaunt, bond-womman, ancille, natif, &c.) and the traditional breakdown of roles concerned production and maintenance, not presentation. A large enough household would have separate departments for the table linen (napery), drinking vessels (ewery), alcohol (buttery), lighting (chandlery), &c. and various aspects of the food (larder, spicery, saucery, &c.). The guys who carried the food to the table from whatever department were just "bearers" (ber(e)-man and ber-knight). In Middle English, "waiter" meant "watchman".

Pubs don't seem to be attested between the Romans and the late 10th century, when there were enough ealahuses to show up in Æthelred the Unready's legal code. Breweries aren't attested as professional before the Central European monasteries in the 11th century. Inns were few and far between, with most travelers obliged to seek hospitality from the churches & monasteries.

If women were doing the serving instead of the men, especially if they were hired help or servants instead of daughters, their morals would've been highly suspect. Cleaning their depravity up would've fallen on the local lord; some, like the bishops of Winchester, were content to let 'single women' serve men to prevent more unnatural sins like masturbation and to skim the girls' earnings.

In any case, there wasn't a specialist term for "waitress", although you could coin "bearwoman" and "bearmaid". Instead, they would've used one of twenty-odd words for "hey you girl" (e.g., wench, maide, therne, shelcherne, &c.) or "servant woman" (e.g., ancille, slutte, malkin, ber-wif, &c.) or some local slang like the "Winchester geese" who worked in the bishops' "stews".

The question does not specify the language, so... at least to refute the highly accepted answer there were no such establishments in the Middle Ages.

Old Czech word hospoda or hostinicě - German Obdach, Herberge, Wirtshaus - Latin hospitium - inn, pub
šenk, krčma - German Schenke, Wirtshaus - Latin caupona - rather a place to drink than to sleep

for the proffession:

šenk - from German Schenk, Mundschenk, Hofschenk - a cup bearer - , it was an office, the King of Bohemia was the Erzmundschenk, the Cup-bearer of the Emperor of the Holy-roman empire

the personal of normal inns or pubs:

masc. krčmář, fem. krčmářka or

masc. šenkéř, fem. šenkéřka, obviously from German Schenkwirt, not an office but a proffession, it may be the owner, his wife or an employee

An English popular song (printed in the 16th century but no doubt earlier than that) gives one politically incorrect answer:

He that will an alehouse keep
 Must have three things in store;
A chamber and a feather bed
 A chimney and a... hey nonny nonny no...

"Wench" is the wrong 5-letter word beginning with "w" here :)

Scan of a published version in 1611: http://www.pbm.com/~lindahl/ravenscroft/melismata/mel33small.html

You could try "Pot Girl".

From https://www.yourdictionary.com/pot-girl.

"A girl who works serving customers in an inn or public house".