There are a number of late night bars (note: not nightclubs) in Edinburgh. Whistle Binkies on South Bridge is open until 3am and I've been in one on Cowgate (that I now can't remember the name of, can't think why!) that is open even later.
Whistle Binkies has live music but isn’t really a “club”, you need to be in before 2.
The Banshee Labyrinth is a bit more “gothic” and seems popular with the staff from other pubs that have closed earlier.
Both of those are quite interesting architecturally as they extend into the vaults under South Bridge. The Banshee in particular has lots of nooks and crannies to explore.
Stramash is the place to go to dance. It also has live bands.
B Bar is another possibility.
All have free admission and all close at 3. They are all quite pleasant places to spend an hour or two.