I've been doing a lot of thinking lately about why I have low self-esteem, and I think a large part of it is that I have severe difficulty talking to people who aren't as nerdy as I am. My level of comfort with talking to someone is directly related to how much of a geek they are, and if someone is into more normal things my confidence plummets and anxiety takes over.
I don't have much trouble talking to the people in the nerd trivia group I attend, but I hardly ever speak to anyone else beyond a little bit of small talk. I know with my friends I can talk about just about any geeky thing and they'll be interested (if they aren't themselves fans of whatever I'm talking about). With other people, though, I feel at a loss for what I could possibly talk to them about. I'll wrack my brain for a possible topic of conversation and not find anything. I usually won't speak to most of my co-workers all day. I did try to make more of an effort with talking to them earlier this year, but I quit after several negative experiences. I'm also very uncomfortable with outing myself as a nerd unless I think someone else may be one, as in my experience normal people often use it as ammunition for mockery.
It doesn't help that I have trouble retaining information beyond the things I personally find interesting. For the most blatant example, I once learned that a woman I found attractive who worked at a local store was into music, especially metal. I have hardly listened to music but spent hours a day over the next few weeks researching it, looking up notable bands in different genres, listening to songs on YouTube or Spotify, using Shazam on every song I heard while out and about to identify it, reading up on artists and songs on Wikipedia while also recording relevant information (genre, band name, particularly famous members, most well-known songs, years active, etc) in a Word document, etc. But then I happened to see her profile on a dating site where she said she wanted kids (I don't myself) and lost all interest both in dating her and in learning about music. Spending so much time and effort learning about it didn't spark an interest in me at all.
This negatively impacted my last couple of relationships as well. The former stayed with me for four months but we both decided to break up because we had nothing in common (although we both tried to look into the others' hobbies but couldn't get into them). The latter was into a few nerdy things, like horror movies, aliens, sci-fi, and the cartoon Adventure Time, so it was easy to talk to her about those things, but beyond that it was hard to feel confident while talking to her. It was enough for me to start developing feelings for her, but she must not have felt the same and ended up breaking up with me after only three weeks.
I think another thing that limits what I have available to talk about is that I wasn't very social growing up, either, and don't have many stories I can tell. I also don't have any brothers, sisters, or close family who can invite me to social events or act as support.
Not sure if this is relevant, but a therapist once said he thought I may have Aspergers as well as an anxiety disorder, although I don't think he made an official diagnosis.