Former lanlord repeatedly broke contract and now wants all our deposit??

Basically as it says, our former landlord repeatedly broke the rental contract by visiting the property unannounced and possibly when we weren't even there although that part can't be proved I know.
He even admits in emails that he made visits but I can guarantee you he can't come up with any documentation or phone records to prove he did.

Can this be used as any sort of leverage at a court hearing?

I realise we should of made complaints about it at the time, even called the police but unfortunately we didn't.