Is there a shop in Brussels where one can re-fill drinking water?
There are vast numbers of shops in Bruxelles where you can BUY drinking water in bottles.
Use the cold water tap in your kitchen surely? Why would you go to a shop to get water?