I'm not a pro Lisper, but I think reasons offered for this include:
* The AI winter (with Lisp, at the time, being seen as "the AI language")
* The heavy-cost and closed nature of the toolchain, and lack of standardization between toolchains, when industry was searching for a language to rally behind several decades ago
* The continued lack of standardization on all the "included batteries" when the open source movement was looking for languages to adopt for scripting a decade ago
* The deep semantic rabbit-hole a Lisp codebase can turn into when any macro-like functionality is used, meaning that a programmer has to basically learn every project as its own language (that is, DSL) before they can get to work—further meaning that projects grow a sense of individual stewardship/artistic direction, and become harder to work on by hiring committees and one-off contractors
* The continued outside impression of Lisps as slow, pure-functional and academic (even though most today are none of those)
* The unwillingness for most Lisps to embrace the Unix philosophy (meaning that it's easier to write a web browser in e-lisp than to loosely couple to one already written in some other language. Clojure is basically the only Lisp that escapes this, cleanly interoperating with the APIs of other languages... on the JVM, which itself throws Unix design principles out the window.)
I'm sure there are more well-founded answers, but all of those came to mind on my road to accepting Lisp.
Your point about "embrac[ing] the Unix philosophy" is a good one. Probably not the biggest factor for mainstream adoption, but significant nonetheless. While there are some Lisps now that that do this well (particularly those that compile to C, such as Chicken Scheme), it has a big historical precedent.
I've been wondering how Lisp would look if it were redesigned from the ground up to include Unix's philosophy, and I'm beginning to think that Erlang would be a good candidate. (With a healthy dose of Prolog as well.)
Several points have been touched on already, I will mention one: DSL. I agree that the perception of it as a barrier to entry in a project may exist. I think it is as incorrect as tying Lisp to AI, albeit for a very different reason. Indeed, a new member of a team has to learn DSL if one is used in the project. However, new members always have to learn the conventions and patterns used in the project they are joining. DSL makes some of them explicit and more understandable.
Strong arguments there, I cannot even question them. Probably the two that stand out are DSLs and impression of Lisp as slow, academic in nature.
I'll go read up on AI winter again, shame I wasn't around to witness that era. On a related note, I just got an email a few days ago for "Third Conference on Artificial General Intelligence, AGI-10 (in Lugano, Switzerland, March 5-8 2010)".
* The AI winter (with Lisp, at the time, being seen as "the AI language")
* The heavy-cost and closed nature of the toolchain, and lack of standardization between toolchains, when industry was searching for a language to rally behind several decades ago
* The continued lack of standardization on all the "included batteries" when the open source movement was looking for languages to adopt for scripting a decade ago
* The deep semantic rabbit-hole a Lisp codebase can turn into when any macro-like functionality is used, meaning that a programmer has to basically learn every project as its own language (that is, DSL) before they can get to work—further meaning that projects grow a sense of individual stewardship/artistic direction, and become harder to work on by hiring committees and one-off contractors
* The continued outside impression of Lisps as slow, pure-functional and academic (even though most today are none of those)
* The unwillingness for most Lisps to embrace the Unix philosophy (meaning that it's easier to write a web browser in e-lisp than to loosely couple to one already written in some other language. Clojure is basically the only Lisp that escapes this, cleanly interoperating with the APIs of other languages... on the JVM, which itself throws Unix design principles out the window.)
I'm sure there are more well-founded answers, but all of those came to mind on my road to accepting Lisp.