Fewer Americans are applying for mortgages — a sign that some buyers may be dismayed by rising home prices and mortgage rates.