The main drawback with a low down payment is higher mortgage payments. There will also likely be monthly “Private Mortgage Insurance” (PMI) required on any loan where there is less than 20% down payment. This can obviously increase your monthly payment by $100–300 a month depending on your credit rating and other factors.
The advantages are that you have a very high leveraging factor (potentially multiplying your returns), maximum potential tax deductions and very little risk. If, for some reason, you can no longer make the payments you’ll only lose your 3.5% down payment. If you qualify and are comfortable with the monthly payments – go for it!