Great assessment and analysis, GMJoe et. al.
Approaching this from a top-down historical point of view: The Great Library of Alexandria was estimated at 40,000 to 400,000 scrolls. Let's assume a small number of big, heavy, valuable scrolls for simplicity. If we assume 40,000 scrolls at 3 stone per scroll (the weight of a Torah scroll) that's a library worth 40,000,000gp - the library of all libraries! (6,000 talents in the ancient world. Seems reasonable). The library would weigh 120,000 stone.
From my prior thread, assume you need 1 cubic foot per 3 stone of trade goods. That means we need 40,000 cubic feet of storage. Assume that, for each story of a building, each 10' x 10' square allows for 250 cubic feet of storage - that is 7' x 7' of goods piled 5' high. So we would need (40,000/250) 160 10'x10' squares. That suggests a library that measures 125' by 125'.
According to Wikipedia, "the exact layout of the library is not known, but ancient sources describe the Library of Alexandria as comprising a collection of scrolls, Greek columns, a peripatos walk, a room for shared dining, a reading room, meeting rooms, gardens, and lecture halls, creating a model for the modern university campus. The library itself is known to have had an acquisitions department (possibly built near the stacks, or for utility closer to the harbor) and a cataloguing department. A hall contained shelves for the collections of papyrus scrolls known as bibliothekai (βιβλιοθῆκαι). According to popular description, an inscription above the shelves read: The place of the cure of the soul. The library was but one part of the Musaeum of Alexandria, which functioned as a sort of research institute. In addition to the library, the Musaeum included rooms for the study of astronomy, anatomy, and even a zoo containing exotic animals."
While 125' x 125' seems too small for the whole of the Museum and Library, it seems like a reasonable size for the scroll collection - a room that size filled with scrolls would seem the wonder of the world to an ancient scholar. So I think the basic analysis here makes sense.
If you'd like the libraries to be bigger, you could assume that for every rare book there are some number of more common books which serve as commentaries, glosses, and notes on the rare books. That was actually very common in the ancient and medieval world. It's common today as well, I suppose, too, maybe even more common. The LA Public Library has 16,000 rare books in a collection of 6 million so that is 500 common books per rare book.