If you have the "Use WordPress Mailing System" checkbox checked the plugin will use the WordPress mail function to send emails.
If you uncheck this option then the plugin will use a simple PHP mail script that comes with this plugin to email the purchase details.
Each of this option have it's advantages and disadvantages. The main advantage of unchecking this setting is that the plugin can use a PHP mail script which gives more control over the email that is being sent out (eg. you can customize the name that appears in the "From Name" field easily). Now, if you are on a dedicated server like myself then you will most likely go with this path and you won't have any issues but the truth is that some of you will be using a shared hosting plan and many shared hosting servers can block emails from random script.
To avoid this email blocking this "Use WordPress Mailing System" option was introduced so you can have WordPress do the emailing for you cause WordPress is a known platform and emails from it wont' be blocked. The only drawback is that the "From Name" in the email will have words like "Wordpress" which may not look very meaningful. There is a solution for this too. I created the following plugin which lets you customize the "From Name" of your outgoing mails from WordPress:
http://www.tipsandtricks-hq.com/wordpress-plugin-change-wp-email-from-details-1883
So, now you can customize the from name to say something like your company name (eg. Tips and Tricks HQ) and all outgoing mails from wordpress will have that name.