printf: fix buffer overflow in floating point decimal formatting

commit f96e47a introduced a new
overflow past the end of the base-1e9 buffer for floating point to
decimal conversion while fixing a different overflow below the start
of the buffer.

this bug has not been present in any release, and has not been
analyzed in depth for security considerations.

the root cause of the bug, incorrect size accounting for the mantissa,
long predates the above commit, but was only exposed once the
excessive offset causing overflow in the other direction was removed.

the number of slots for expanding the mantissa was computed as if each
slot could peel off at least 29 bits. this would be true if the
mantissa were placed and expanded to the left of the radix point, but
we don't do that because it would require repeated fmod and division.
instead, we start the mantissa with 29 bits to the left of the radix
point, so that they can be peeled off by conversion to integer and
subtraction, followed by a multiplication by 1e9 to prepare for the
next iteration. so while the first slot peels 29 bits, advancing to
the next slot adds back somewhere between 20 and 21 bits: the length
of the mantissa of 1e9. this means we need to account for a slot for
every 8 bits of mantissa past the initial 29.

add a comment to that effect and adjust the max_mant_slots formula.

Author: richfelker

src/stdio/vfprintf.c

@@ -182,7 +182,9 @@ static int fmt_fp(FILE *f, long double y, int w, int p, int fl, int t, int ps)
 {
 	int max_mant_dig = (ps==BIGLPRE) ? LDBL_MANT_DIG : DBL_MANT_DIG;
 	int max_exp = (ps==BIGLPRE) ? LDBL_MAX_EXP : DBL_MAX_EXP;
-	int max_mant_slots = (max_mant_dig+28)/29 + 1;
+	/* One slot for 29 bits left of radix point, a slot for every 29-21=8
+	 * bits right of the radix point, and one final zero slot. */
+	int max_mant_slots = 1 + (max_mant_dig-29+7)/8 + 1;
 	int max_exp_slots = (max_exp+max_mant_dig+28+8)/9;
 	int bufsize = max_mant_slots + max_exp_slots;
 	uint32_t big[bufsize];