When the custom homepage image gets uploaded, CONTENTdm creates a resized copy that fits within the default 150x150px container. This image is actually going through a couple of different resizes, from a height of 317px > 150px > 206px.
If you turn off your CSS, save the changes, and previewing the site, you will see the images used as the thumbnails are crisp.
You may want to reconsider how your CSS handles image resizing for thumbnails, and upload images that more closely match what will be the display size.