If a buyer is buying over $400K, or even over $300K, the layout of the home becomes very important to them.  That is not to say that it is unimportant for a $200K buyer -- but someone buying a more expensive home oftentimes plans to stay in it for a longer time frame.

If not the #1 feedback, the #2 feedback I receive from showings of homes priced over $400K is that the layout just didn't work for the buyers.  They wanted another bedroom here, instead of there.  They wanted a more open floor plan.  They wanted a more formal dining room.  They wanted a more spacious basement. 

Challengingly, the layout is something that is hard (nearly impossible) for a seller to change in order to appeal to a wider segment of buyers.  Thus, if your home has a layout that continues to not work for buyer after buyer, you must either wait (and potentially wait and wait and wait) for the buyer who loves that layout -- or adjust the list price to make the house (even with its layout) more appealing to more buyers.