According to answer on exercise 12.10 in TeXbook, a box's height may be negative:

Yes it did, but only because none of his boxes had a negative height or depth.

But on p.77 it is said:

The result of hbox never has negative height or negative depth, but the
width can be negative.

How does one construct a box with negative height?

Why in the following example prevdepth is not set to the depth of the new box?

vboxhrule depth -5pt showtheprevdepth hboxworld

(on p.80 it is said that prevdepth is set to the depth of the new box)

TeXbook p.80 says:

... the reference point of the lowest box is taken as the reference point of the whole, and the glue is set so that the final height has some desired value.

Consider the following input:

tracingonline=1
setbox0=vbox to 20pthboxxvss
showbox0
abox0 b
end

Output is

Notice, that the lowest box is with "x", then comes the glue:

> box0=
vbox(20.0+0.0)x5.2778, glue set 15.69446fil
.hbox(4.30554+0.0)x5.2778
..tenrm x
.glue 0.0 plus 1.0fil minus 1.0fil

But the reference point of the whole is not reference point of "x" (as seen from the image). Is this a contradiction?

Short answer: as far as I know, a box directly produced with the hbox command can't have a negative height, but a box produced with the vbox command can. A horizontal box can have a negative height if it is stored in a box register, say box0, and one executes an assignment of the form ht0=〈dimen〉 with a negative 〈dimen〉.

Example of a vbox with negative height

Let's consider the following code:

setbox0=vbox to -10ptvsshboxa
This is vboxhrulebox0nointerlineskip hboxtest.
bye

If you do showbox0 before using box0, you'll see:

> box0=
vbox(-10.0+0.0)x5.00002, glue set - 14.30554fil
.glue 0.0 plus 1.0fil minus 1.0fil
.hbox(4.30554+0.0)x5.00002
..tenrm a

The box is -10pt height and 0pt deep. Because of this negative height, the hrule lies exactly 10 points below the baseline of the character box a. Indeed, changing the -10pt to 0pt:

setbox0=vbox to 0ptvsshboxa
showbox0
This is vboxhrulebox0nointerlineskip hboxtest.
bye

gives this:

Here is how it works:

• The reference point of the outer vbox, vboxhrulebox0nointerlineskip hboxtest, is the same as that of hboxtest; it gets aligned with that of the following period, i.e.: lies on the baseline of the paragraph's first and only line.




• Because of the nointerlineskip, box0 sits on top of the hboxtest, with no separation (no interline glue).




• The depth of box0 is that of hboxa (hboxa is a box—not a kern or glue—and the last item of box0), i.e., 0pt. According to what I explain below, this implies that the reference point of box0 is at the same vertical position as the bottom of hboxa: its baseline.



• 
  

  Now, let's consider the case where we do setbox0=vbox to -10ptvsshboxa. We mandate box0 to have height -10pt, i.e., from the reference point we just computed, this means we have to move 10 points down to reach the “top” of box0, which in this precise case is not the top in the usual sense, but means “right before the vbox's first element”. This implies that the vss here behaves as a vskip whose value is -10pt minus the height of hboxa. This can be verified with:

  
  
  

  setbox2=hboxa
  % Without e-TeX's dimexpr: dimen0=-ht2 advancedimen0 by -10pt setbox0=vbox to -10ptvskipdimen0 hboxa
  setbox0=vbox to -10ptvskipdimexpr-ht2-10ptrelax hboxa
  〈rest unchanged〉
  

  
  
  

  which gives the same result. Indeed, when continuing our walk from after-last-item to before-first-item of box0, before getting to the vss, we are at the top of the hboxa, so in order to reach the so-called top of the vbox (what corresponds to “before the first item”), we must move down in order to arrive 10pt below the reference point, and the amplitude of this downward move has to comprise all the height of hboxa, plus 10pt. In other words, vbox to -10pt... says that before-first-item of the vbox must be 10pt below the vbox reference point. So, the first thing to do in order to find the vertical position corresponding to the start of a vbox (what I call before-first-item) is to find the reference point; then move up or down depending either on the natural height of the box, or on its prescribed height in the case of a vbox to.

  
  



• The hrule inside the outer vbox comes right before box0. Since no interline glue is added after a rule box in vertical mode, this hrule comes right above the “top” of box0 which we just described, i.e.: 10pt below the baseline of the hboxa.

Negative height, width or depth of constructed boxes

An hbox command can't produce a box with negative height or depth,¹ but it can produce a box with negative width. A vbox command can't produce a box with negative width (TeXbook p. 81),² but it can produce a box with negative height or depth. Example for a negative depth:

setbox2=vboxhrule height 0pt depth -2pt
showbox2

shows:

> box2=
vbox(0.0+-2.0)x0.0
.rule(0.0+-2.0)x*

To aid memory, note the symmetry between these two sentences:

The height and depth of a constructed hbox are determined by the maximum distances by which the interior boxes reach above and below the baseline, respectively.

(TeXbook p. 77) and

The width of a computed vbox is the maximum distance by which an enclosed box extends to the right of the reference point, taking possible shifting into account.

(TeXbook p. 81). Other box dimensions immediately resulting from an hbox or vbox command may be negative.

prevdepth after a rule box in vertical mode

Regarding prevdepth, the TeXbook pp. 79 and 80 says:

TeX's implementation of interline glue involves another primitive quantity called prevdepth, which usually contains the depth of the most recent box on the current vertical list. However, prevdepth is set to the sentinel value −1000 pt at the beginning of a vertical list, or just after a rule box; (...)

Since your example does showtheprevdepth just after a rule box, we are precisely in one of the particular cases where prevdepth is set to -1000 pt.

Reference point of a constructed vbox

Regarding your addition concerning vbox to 20pthboxxvss, this is a new question! The answer is in the next two paragraphs after the one you quoted on p. 80:

However, this description of vboxes glosses over some technicalities that come up when you consider unusual cases (...) Therefore, the actual rules (...) (2) If there’s at least one box, but if the final box is followed by kerning or glue, possibly with intervening penalties or other things, the depth is zero.

Depth zero means that the reference point of the box is precisely at the bottom of the vss (the box doesn't extend below this vss). This reference point is aligned on a horizontal line with the reference points of the a and b character boxes inside the horizontal box that forms the first and only line of your paragraph. Note that this alignment process of reference points does not correspond to the quote you gave, because the latter concerned alignment of boxes inside a vertical list (these alignment processes work in orthogonal directions).

My method to “see“ the reference point of a vbox is to start after the last item, then move up or down according to the depth computed by the rules given in the paragraph that spans over pages 80 and 81 of the TeXbook. If for instance, the computed depth is 5pt, it means that the reference point of the vbox is 5pt up from the bottom of its final element. The following extension of your example may help:

setbox2=vboxhrule height 0pt depth 5pt
setbox0=vbox to 20pthboxxvssbox2
avrule height 20ptbox0 b
bye

Explanations:

• The width of the hrule in box2 isn't explicitly specified, therefore it is determined by the smallest box or alignment that encloses it, i.e., box2 (cf. TeXbook p. 221). Since there is nothing else inside box2, the width of the hrule is 0pt; that is why we don't see it on the screenshot.




• box2 is a vbox whose last element is a box (a rule box), therefore its depth is the depth of that last element (boxmaxdepth being equal to maxdimen by default). The depth of box2 is thus 5pt.




• This implies that the reference point of box2 lies 5pt above the bottom of its last element, therefore it coincides with the top of the hrule inside box2.




• The natural height plus depth of all the material inside box2 is 5pt. Since box2's depth is also 5pt, the natural height of box2 is 0pt.



• 
  

  This natural height of 0pt is box2's height because box2 is built with vbox as opposed to vbox to. So far, we've explained this:

  
  
  

  > box2=
  vbox(0.0+5.0)x0.0
  .rule(0.0+5.0)x*
  

  
  



• For the same reasons as above, the depth of box0 is that of box2, namely 5pt, and the reference point of box0 lies on the same horizontal line as the top of the hrule inside box2, which coincides with the top of box2 (for both possible meanings of the word “top” in this context).




• The vertical extent of box2 exactly covers box0's 5pt of depth. The remaining hboxxvss inside box0, plus the interline glue inserted before box2, therefore exactly cover the mandated 20pt of height of box0 (the precise amount of computed interline glue before box2 doesn't matter, because the vss will adapt to any amount, making it so that the top of the hboxx lies 20pt above the reference point of box0).




• The previous point explains why the vrule height 20pt reaches up precisely as far as the top of the x, and down 5pt below the baseline of the paragraph's first and only line (since the vrule's depth isn't explicitly specified, it is determined by the “smallest box or alignment that encloses it”, namely the horizontal box corresponding to the paragraph's only line, whose deepest element—apart from such rules with free depth—is box0).

Footnotes

1. However, a horizontal box stored in a register can be made to have negative height or depth via assignments such as ht0=〈dimen〉 or dp0=〈dimen〉, as pointed out by David Carlisle.



2. 
   

   But again, setbox0=vboxhboxawd0=-3pt does cause box register 0 to contain a vertical box with negative width:

   
   
   

   > box0=
   vbox(4.30554+0.0)x-3.0
   .hbox(4.30554+0.0)x5.00002
   ..tenrm a
   

   
   

You can have a horizontal box with negative height, as the dimensions of box registers are assignable, but when constructing a box with hbox the maximum of 0pt and the positive heights of the content is used.

tracingonline1

setbox0=hboxa

showbox0

ht0=-20pt

showbox0

setbox2hboxbox0

showbox2

bye

Produces

hbox(4.30554+0.0)x5.00002

Natural height 4.3pt

hbox(-20.0+0.0)x5.00002

assigned height -20pt

hbox(0.0+0.0)x5.00002

Constructed "natural" height 0pt.