Why is it so hard to find new construction for non-luxury buyers?

In the Greater Boston area, we have a built-in scarcity of land. Boston has the Atlantic Ocean on one side, limiting the range of its suburbs. The suburban ring around Boston is more like a suburban crescent. In town and in the suburbs, buildable lots are hard to come by. This has kept land prices high within a good commute to Boston. Many undeveloped lots have ledge or some other feature that makes them practically unbuildable. Frequently, houses are torn down for new construction. Many of the tear-downs have been post-War houses on suburban neighborhood lots.
In order to make a profit, that new construction will be either a mansion-level single family or as many townhouses as the builder can get permitted on that lot. The new build is not going to be a moderate family-sized single family house. It is just not profitable to build that.
In order to buy a buildable lot, a builder will need to spend $200,000 or more for land, then prepare it for building. That preparation can cost hundreds of thousands and take years in the approval process. Land has to meet standards and a good bit of research goes into figuring out what will pass muster in the towns and cities. These things cost money, as do remedies for uneven ground level, water problems, rock ledges, and pollution in the ground.
If the builder purchases a house to tear-down, the house purchase can be closer to $400 in suburban areas. Then, add the cost of the tearing down the old house and building the new house. This adds up fast and makes the investment high for the builder.
In short, in order to make enough money to bother with the project, most builders are building something that is not designed for a moderate-income family. Mores the pity. But that is how it is.