Hi Arturo,
The GADM world tree is built from a CSV which I generated from the GADM 2.0 dBase files for the whole world. I checked the dBase format and doesn't include the polygon information you find in the .kmz files. I saw they released 2.5 this month (July 2015) but only in ESRI geodatabase format (which I didn't investigated yet, but not the same as ESRI shapefile format) and will release 2.7 in August 2015.
Another way could be just to parse the KML (is just an XML file) using XPath or XMLPullParser.
About locating the point in a polygon the problem of traditional 2D inclusion tests (ray crossing methods) is that require large storage, given polygons of enough size, complexity and amount of edges. This is a common situation with level 1 and 2 polygons. New methods for example classiffy polygon edges into layers, results in faster inclusion test.
Hern��n