Irradiance with Radiosity maps is a hard combo for what you are doing. Particularly with the Physical renderer. You can try thickening the walls and ceiling but you really have to play with the radiosity settings, decrease the texel size (Map Density) and increase the Sampling Subdivisions. It takes a lot but if you turn the radiosity maps off you can see it's primarily the source of the 'leaks'. You can change to Light Mapping or you can try QMC primary and keep Radiosity Maps as secondary then you don't have to increase the radiosity settings nearly as much.
All radiosity is expensive in render time and the quicker the render is the more likely you are to be affected by the shortcuts the programmers have to take to make it work well in most situations but still be fast (relatively speaking). QMC will be the most accurate but take the longest to get smoothness (non-graininess) close to Irradiance, so you have to find what works for the render you are trying. The help as a lot of good info for GI.