"I'm curious what things hold back Lisp from being more popular."
Inertia. Things like Java and Python were close enough to C (syntax and imperative paradigm) that moving to them was just a minor course change. Lisp feels like a U-turn, though I have argued elsewhere it is not really. Meanwhile Java had Sun behind it, giving PHBs a vital warm fuzzy. Lisp looks like the north wall of the Eiger in winter to PHBs.
The IDE problem just confirms anyone's fears if they even get that far, though Franz and Lispworks tried to address that by making unlimited-length trials of their fine IDEs available. Agreed on Emacs+Slime not being a good answer for noobs.
"I focused on IDE's, but I'm also interested in the general question of what things can be done and not done to get more people into Lisp."
Not to worry, it is happening, just step back and look at where languages are going, not the people. Steele famously describes Java as dragging people halfway from C++ to Lisp. Not sure about the fraction. :) Python moved a little more, and Ruby moved a little more. If Arc catches on it will constitute completion of the land bridge from static typing imperative languages to dynamically typed functional languages, but it does not matter because Common Lisp is starting to catch on anyway. And if you look at the ideas behind Lisp and why languages like Python and Ruby are doing so well, Lisp has already prevailed. My 2.
In the future, everyone will use Lisp, but they won't know it's Lisp, because it will be called Microsoft Windows Dynamic Service-Oriented Parenthesis Construction Language for .NET.
Mr Dussud (a respected Lisper in a former life) said MS would be happy to host a Lisp if we would just add strong static typing and lose our silly little OO package, CLOS. A couple of other things, too.
Wow, yeah. I tuned him out when I saw he wanted us to declare our variable types. I did not listen to all of the audio, but in there somewhere you will hear some nutjob asking if, given that Lisp has finally started taking over the landscape and he now wants us to run up the white flag and adopt static typing, he also thinks it is time for mainland China surrender to Taiwan. That was me.
I went up later and shook his hand for being brave enough to come into the lion's den. Strong guy. <ouch>
Profusion of dialects - lack of libraries. There is an inverse relationship between the success of languages and their power. C is cumbersome to write; therefore, everyone treasures the libraries that exist and takes pains to remain compatible. Java, even assuming that it made it 'halfway to Lisp', is clearly still cumbersome enough. Python is already too powerful, but it benefits from its slowness. In many problem domains (GUI, databases, numeric computing), it is way too slow, forcing you to use C-coded extensions, which again help to prevent fracturing of the user base. Where it's fast enough, you get a bazillion of half-assed libraries and frameworks. E.g. Python web programming is a mess and, compared to PHP and Java, not popular at all. (There are also limitations to the VM that matter here but I'm not going into those.)
Lisp is so powerful that it lets everyone modify it, and so everyone goes and creates their own dialect of it. It compiles to fast, native code, so no need to adopt and maintain libraries shared by a large user base.
There is a nice language called D. Unfortunately, it comes with two incompatible standard libraries. How do you like that as a new user? Well, in Lisp it's the same, only times 257. Throw in the oatmeal cum nail-clippings visual appeal (L.Wall) and you might well wonder why it is not less popular than it currently is.
