I am having this exact problem. CLicking on the checkout button popped up the lightbox-like white dialog box. I disabled "Enable Fancy Redirection On Checkout". I cleared my browser cache and also got a friend who has not been to the site before to try checking out. It's doing the same thing it has been doing for me, it redirects to http://www.coryellekramer.com/blog/wp-content/plugins/wp-cart-for-digital-products/eStore_payment_submission.php and gives a 500 Internal Server Error.
The estore URL is http://www.coryellekramer.com/blog/shop/flower-essences
To replicate the issue, add a product from the above page to the cart. On the checkout page click on the Checkout button and you'll see the problem. Same thing happens whether you use the Paypal or Manual checkout option.
Here is the shortcode that is used on the checkout page: [wp_eStore_cart_fancy1]
Plugin version 6.5.3
Wordpress version: 3.3.1
Same results even with all other plugins turned off and using the default Wordpress 2010 theme.
No content in the debugging logs.