In improv, the idiom one is playing in changes and either narrows or widens the number of choices available at any given moment. So a thorough bass improvisor's approach was different than a Lizst or Scriabin's was, than a dixieland jazz player's polyphony, a bebopper's soloing over chord changes, or a post bop player altering the changes against a given theme, than a free jazzer's action painting in sound, than an Indian Raga, Balinese Gamelan or Shona mbira player's. Is it against a fixed meter, non-metered pulse, arhythmic or whatever happens in the moment. How about so called non-idiomatic improvisation which requires virtuosic listening more than anything and while there may be notes and chords involved they are tangential to the proceedings. Though really in even the most rigid forms the intangibles of phrasing and timbral variation, voicings, etc. are seemingly endless. In my small experience thinking is the last thing you want to do in improvisation. The idea is to get your conscious mind out of the way and interact with what you're hearing. Perceptions about what is working and what isn't is not the same in the moment as in listening to a document of it later.
This may seem like nitpicking, but I look at the context of Poe's statement. To him hearing music would have been an ephemeral experience of a very limited amount of music; some popular songs of the time, parlor pieces, etc. Major pieces of symphonic music might be heard once or twice in a lifetime. He probably knew about Bach and Mozart, likely not Schubert or Beethoven, let alone African, Indian, Japanese or what have you. Today our experience of music is nearly all electronic, that to some degree or another intends to create the illusion of a real physical space, or sometimes emphatically doesn't. We have unprecedented opportunities for hearing sounds from all over the planet. The very definition of the boundaries of what is and isn't music has changed drastically. There are many more ideas in music today than there would have been in the music of Poe's time as the 20th C saw an explosion in the available sound world, conceptual underpinnings and the uses that were and are made of music.
As to where inspiration originates is anyone's guess; not from the ego and the conscious mind though in my opinion.