Also there are four or five places in the York Street (opposite the bank side) around the corner. I think the names are Royal, Swiss, Windsor, Kamal etc Negotiate and go for the best rate. I always go to these places. Never had a counterfeit note. Also inside the Sea Street Gold markets. The highest rate is here - but it's a curse to get there.
A long time ago there was a person on a small tall desk & a chair near the Chartered Bank, Fort. A tamil guy in a white shirt and Sarong. But he did business more than a bank would in a day. Nice person, gave the best rates. One day on my way to the Tap (Hotel Taprobane- basement bar) I went to change money and noticed that I have forgotten the wallet. He said "mahattayata keeyakda one. ayith ena dawasaka denna" and he advanced the money without asking another question.
Sadly he was killed in the bomb explosion there.