Changing the cache management policy and enhancing the mapping of logical to real fonts seem like relatively low hanging fruit or am I missing something?
Currently, if the cache size limit (bytes of all glyph forms in the cache) is hit the cache resets totally and subsequent requests will regenerate the needed glyphs.
Upping the FreeType glyph cache limit helps though the fact that two different instances of a LogicalFont (with the same parameters) will create a new real FreeTypeFont, the cache tends to churn.