If a buyer is buying over $400K, or even over $300K, the layout of the home becomes very important to them.  That is not to say that it is unimportant for a $200K buyer -- but someone buying a more expensive home is oftentimes planning to stay in it for a longer time frame.

Back in the red hot market of 2004-2008, this did not matter as much -- inventory was low, buyers were happy to just get a house at all -- but that has changed.  Now days, buyers are assuming that they may want to (or have to, if prices aren't increasing over time) stay in their home for quite a while.  As a result, they want a layout that works really well for their family.

If not the #1 feedback, the #2 feedback I receive from showings of homes priced over $400K is that the layout just didn't work for the buyers.  They wanted another bedroom here, instead of there.  They wanted a more open floor plan.  They wanted a more formal dining room.  They wanted a more spacious basement. 

Challengingly, the layout is something that is hard (nearly impossible) for a seller to change in order to appeal to a wider segment of buyers.  Thus, if your home has a layout that continues to not work for buyer after buyer, you must either wait (and potentially wait and wait and wait) for the buyer who loves that layout -- or adjust the list price to make the house (even with its layout) more appealing to more buyers. 

That said, as reflected in the conversation at the top of this post -- a great deal on a house is not always enough to sway a buyer beyond the layout challenges that a house offers.