Local boy Jack White opened Third Man, and it's a super-fun browse. The store sells records (of course), turntables, T-shirts and other gear, but the coolest bits are the recording booth (where you can make your own record for $20) and the record pressing plant that you can peek into for free (or tour for $15 on select Saturdays).
Free concerts and other performances take place on the in-store stage; check Third Man's calendar for the schedule.