Mass Mailing Strategy with C# and Amazon SES

Read this first

From my experience Amazon SES might be the cheapest option out there for bulk mailing, but is it technically good? I consider it OK at best, but in some areas it is an epic failure.

SMTP is old

Yes ladies and gentlemen SMTP is really old, as in older than “Back to the future old” and in tech terms that’s pretty damn old. SMTP was introduced in 1982 and received its last revision in 1998. Let’s also be clear SMTP is not designed for mass mailing out of the box.

Why is SMTP not suitable for bulk mailing?

Well you’ve probably heard people talk about how chatty SMTP is. Yes it is extremely chatty, but that isn’t the biggest problem. SMTP was designed ideally for a 1:1 instruction per email, with CC and BCC addressing the need to send more than 1 email from 1 instruction.  This is acceptable for transactional email, but when it comes to bulk email, the idea is to minimize the total instructions as much as possible and by doing that reduce the network consumption, IO and time to bulk send.

The protocol simply isn’t up to scratch for mass mailing when you consider the following scenarios:

  • Personalised body per recipient
  • Varied attachments per recipient
  • Large volumes of recipients

in 2 of the above scenarios using CC and BCC is not an option, and so truly 1 instruction must be created per email! When it comes to bulk sending, this is outrageous. Imagine using c# having to create 1 mail object per email when you have 2 million emails to send off?

In the 3rd scenario you can lesson the problem if you are not offering personalized email, by reusing CC or BCC. However with CC other recipients are aware of each other, so that is out of the question for mass mailing, leaving BCC, and here again you encounter issues depending on your mail host, what limits they impose to the max number of BCC’s you can include. In the case of Amazon SES that number is 50!

The solution is custom APIs

Providing developers and users with a custom API to your mailing solution means you allow your developers to bypass the limitations of traditional SMTP, and deal with the limitations your side. I’m not implying this is cheap or easy. But this is the only way to implement a professional mass mailing solution – SMTP simply isn’t a good approach.

So for example – imagine if you were a mass mailing provider. You would want to provide your developers with an API that could for example allow uploading attachments upfront for reuse for a campaign. Or HTML and TXT templates, where they could reference the template and do server side replacements for personalization. All of this is not rocket science, a lot of existing mass mailing solutions offer this type of thing, why? Because they too came to the same conclusion that the SMTP interface is just too taxing on the end user.

Amazon API is nothing more than SMTP wrapped

The Amazon SES API might be slightly less chatty than SMTP, but realistically this offers no tangible performance value. The crux of the matter is that Amazon did not implement their API to solve mass mailing problems, but I think it had to do with their billing policies (speculation!). Eventually though Amazon started offering a standard SMTP interface which they might as well have done all along.

Multi-threading to the rescue

I know what you’re thinking, and I thought it too, and I went a step further I actually implemented a full blown dynamic multi-threaded C# app to send emails at max speed through Amazon SES, and here is what I found:

Based on all the limitations in the Amazon SES offering and limitations in SMTP (And this includes their SMTP wrapped API), 1 or 2 threads is all you are ever going to need.

Here is why:

1. Amazon impose a max send rate – starting at 5/sec and the max they’ll allocate is 99/sec – That is it! That essentially means 99 unique emails per second, not instructions!

2. Let’s say you went down the full blown multi-threaded route and created 1 thread * max rate. You might think this is a reasonable approach because this way you can personalize mails client side, and have a truly 1:1 mapping of instruction : email (which again is outrageous). At the end of the day the max you’re going to get is 99 emails per second at best (and most likely slightly less).

3. Since BCC is limited to 50, and assuming you can create a non personalized email instruction in a second, well that means technically with 2 threads you could push 100.

So as you can see from this basic example – batching instructions is much more efficient. 2 threads can do the work of 100. Furthermore if Amazon did not impose a 50 BCC limit, then 1 thread would suffice.

My recommendation

I eventually had my arm twisted to continue using Amazon SES, and so because of this ditched email personalization and went with a single thread for handling emails.

If you can avoid personalization then you can get 99 emails per second to Amazon with just 2 threads. If you really have to personalize then 1 thread * limit is your solution, but beware the more IO you are doing the more you can expect problems, timeouts, network issues, etc.

Ideally if you’re after personalization with high volumes, you DO NOT WANT TO BE USING AMAZON

 

 

Advertisements

WordPress and WPDeals

If you’re feeling adventurous you can head over to http://www.royalspadeals.com, and check out the potential of WordPress and WPDeals. Both are free open source solutions, and as you already know WordPress runs on PHP.

Royal Spa Deals (http://www.royalspadeals.com) offers luxury spa deals in Dubai. My initial idea was to use Joomla and a plugin, but after finding WPDeals, I decided to use that instead.

It perhaps wasn’t the easiest approach. WPDeals has some massive flaws out of the box, namely within its stock system. Also since Royal Spa Deals is located in Dubai, Paypal does not support AED. You can read how I went around solving this in an earlier blog post (or leave a comment).

All in all I found WPDeals to be a bit frustrating to work with. Needed a lot of fixing, but eventually with enough hacking and slashing, I managed to get it to the point where it was useable.

Does my geek wife exist?

So I’m recently divorced. When I told friends, family and people I work with their initial reaction was “Oh sorry to hear that”. I guess it is the polite thing to say. I guess “Wow, that’s awesome, congrats!” borders on presumptuous.

Best part about being single is you know that there is now great potential to find a suitable life partner. Also having been through a marriage of roughly 8 years. More than enough time to learn some valuable lessons.

This got me thinking instead of rushing into my next relationship based on attraction and just general muck about romance. I’ve come up with some initial criteria as to what attributes my next girlfriend -> possibly to be wife should have. Basically what I’m after is a “Linux chick”

debian girl

No, she doesn’t literally have to know Linux, but here are some guidelines:

  • Must be a geek. Not really after a very nerdy girl, but not ruling an extreme nerd out, but absolutely she has to be a geek.
  • Must be a gamer. Not talking about the casual game of solitaire or Zuma either. At least should be able to take me on head to head in a game of Unreal tournament. At least have played an MMO. Is interested in gaming, and doesn’t mind playing late into the night.
  • Must be a tech enthusiast. I want a girlfriend who doesn’t just look at the screen with blurry eyes and grin strangely at what I’m doing but who understands at least conceptually what I’m programming, or working on. Knowing how to code herself would be a strong advantage. Not knowing how to install programs or use a computer – totally unacceptable. Not after girlish giggles on this one – I really can’t have someone who 3 years in says “All you do is sit on that computer”.
  • Also must be into doing fun things. Not after a complete freak either. Walking, running, camping, biking, all these things are good. But not obsessed with fitness.
  • Reads occasionally, or reads considerably online.
  • Has a geeky dress sense.
  • Playing a musical instrument would be a strong advantage, at least should be willing to learn some basic guitaring.
  • Loves to cook – and here I stress – NEW THINGS…..
  • Is a creative thinker.

Wonder if such geek chicks exist?

Developers final take on writing an app in WPF (XAML) vs Windows Forms

Desktop apps came before Web Apps. This is a no brainer, the internet came into existence way after Desktop applications, and the initial internet consisted only of web pages. Web apps came later.

The presentation of Desktop apps have a history of being rigid, boring yet functional, for the most part strongly resembling the operating system, they’re installed on.

I know WPF is not a new kid on the block, but it is imo one of the most brilliant and under rated offerings so far released by Microsoft.

In essence a lot of developers have simply boycotted WPF and still opt for traditional Windows forms, but if the tech is so much better, then why?

Why WPF is not getting the attention it deserves?

This is the short list of reasons why WPF has yet to really capture the hearts of MS developers and become the defacto way of building modern UI’s.

  • WPF (XAML) and Silverlight were bundled at launch. Now I know Silverlight takes a lot of flack as being an epic failure. However consider the following. The internet presentation at the time had come from HTML 4, XHTML was the defacto way of coding, and everything was becoming web based. It was another golden era for browsers. Tons of new competition, and the W3C was finally the authority it was supposed to be all this time. Mobile application development was on the rise. Most of the front end web logic was being powered by Js.   Just to be clear, this was before JQuery!

    Everyone was questioning, do we really want to consider using this “messy” and “delinquent” scripting language to power the next generation of the web. Let’s call it “Web 3”. Let me remind you how fast technology changes, this is at the time when Active-X was still semi acceptable, and Flash was the cool kid on the block.

    So in essence, Microsoft back at the drawing board said “Look guys, our desktop apps look dated, Active-X has no future and HTML+JS isn’t ideal” Putting their heads together – their brightest engineers brought us WPF and XAML – XAML a descriptive XML based language. WPF and XAML would be used to build stunning desktop applications, Windows phone applications, and this would spill over to the “Write Once Deploy Everywhere” Web, in the form of a plugin called Silverlight, which could be theoretically installed in all major browsers. Interestingly enough IE was supposed to support XAML OOB, completely replacing the need for any HTML or Js.

    Sounded good in theory, however at the time not only Microsoft was at the drawing board. Elsewhere steps were being taking to improve upon the dated status quo. Many Js frameworks were on the rise, most noticeably JQuery. Google hard at work developing Android decided to harness the power of HTML for their “Write Once Deploy Everywhere” strategy. And HTML too was being challenged – HTML 5 was born. HTML 5 had 1 major mandate, to allow for responsive web design, to make it easier to cater for all screen sizes on any device. Getting inline for the myriads of devices to enter the market place.

    A few years in, and Microsoft realised that raw HTML5 had taken the lead for presentation, and since JQuery, I guess a lot of developers found new love for Js. Many advocate Js. Silverlight was retired.

    Microsoft never retired WPF or XAML, but since WPF was not widely used at the time, a lot of developers assumed it had been retired too.

  • There was confusion if XAML was a competitor of HTML. A lot of developers simply don’t trust the longevity of XAML.
  • Developers have learnt to embed web browser controls to do UI with. Thus truly harnessing write once, deploy all.
  • Desktop applications are on the decline.
  • Developers are familiar with Windows Form, and refuse to learn a new (and complex) WPF.
  • Deadlines 🙂
  • The confusion between Visual Studio and the need for an additional tool called Blend. Most developers take 1 look at Blend, and think yeah – this looks like Flash. There is no way in hell I have the ability to learn this!

 

 

 My Experience with my first WPF Application

I remember on 2 occasions I’ve opened up Blend before and indeed it did look complex. Both times I’ve gone back to Windows Forms just because I’ve had to get the job done quickly and no time to experiment. This time was different. I decided no matter what I would do what it takes to learn WPF.

What I found is that WPF was less complex than I imagined, and my app even has some animation effects and cool stuff.

It took me just 3 days to become familiar with Blend. I still find it a bit strange to have to use 2 IDE’s for 1 application, but really this is nothing to complain about.

So what do I really think about WPF?

What I’ve noticed before as a Win Forms developer is you develop much lower expectations for your user interface. You end up developing these formal business looking applications. The UI’s look boring, and you focus on the functionality.

When it comes time to develop a new screen, most of the design considerations are where you will place your components from the standard toolbox. – Welcome to Win Forms development! Ah the joys 🙂

One thing I’ve noticed about WPF is that it sets you free. In a way it is a bit scary, but at the same time very exciting.

When I develop a new screen in WPF much more possibilities are flying around in my head. It’s exactly like developing for the web. Suddenly you can get your designs down to the pixel level (if you want). Create interesting buttons with rounded corners and gradients, however overs, etc. Have a list of items in a WrapPanel, like div floating in HTML.

Simply put you can really fine tune the design experience. You also get storyboards and animations – so pretty much everything you need at your finger tips to create a UI exactly as you want it, all you need is time and imagination.

C# in the background

What I really like about WPF is that I can continue using C#. Let me be clear, I’ve seen a good few languages, and nothing beats the elegance and power of C#.

The controls however have not just been lazily ported across from Windows Forms.

Take a Winforms Datagrid, typically you go to the DB -> ORM your data across and then convert this to a Datarow.

In WPF each row IS your ORM type. You don’t type cast it in our out.

So for example : client myClient = DataGrid.selectedItem AND NOT client myClient = getClientByID(DataGrid.selectedItem.Columns[“ID”]);

Things like this make working with WPF a breeze.

Final thoughts

If you’re not working in WPF and still using Win Forms you’re missing out.

Go ahead, dive in and give it a try.

 

 

WPF : Access to the path ‘C:\Users\Michael\Documents\My Music’ is denied.

Recently I came across a strange issue. It would seem the WPF XAML designer for some reason wants to traverse through My Music folder.

While I am flattered the XAML Designer might be interested in “My Music”, I’m not amused that the designer actually crashes and shows this error:

Access to path denied

 

So what to make of this one? One minute I’m working normally in WPF, and the next this thing pops up and prevents me from meeting my deadline.

Google should know right? Wrong! Nothing much there besides a few other people who have come across this, and are frustrated as hell, left wondering if WPF and XAML is more of a prototype than an actual development tool.

So I decided to ask the question on Stackoverflow except the fine people over there didn’t appreciate my question. Neither did they think I deserved to ask the question:

 

Stack

 

 

So around this point in time, I have no idea what the problem is and the internet seems less than helpful. Now I know sooner or later, maybe even right now there is some other poor jerk having the same problem – who doesn’t have the luxury of Microsoft Gold Support.

That is why I am writing this, I’m going to try and investigate this problem and document my investigation process.

Step 1

The first thing I wanted to do is find out if it is a problem with just 1 XAML file or ALL XAML files in my project.

This is easy enough to do, I tried opening a few XAML files, all of them worked 5 minutes ago, none of them work now. Clearly this means:

Conclusion(s) :  Not the XAML markup causing the problem. 

Step 2

It is a system wide problem, or just a solution / project problem.

I created a new Grid Application. Lo and Behold the XAML editor worked just fine.

Conclusion(s) :  It’s something in the solution or project causing this.

Step 3

Permissions of the folder the project is in?

Added: ALL APPLICATION PACKAGES user to the folder my project is in : C:\Projects\XYZ…

Result:  Nothing

Added : My own account to the folder (even though I am administrator) and assigned full control

Result: Nothing

Conclusion(s) :   Don’t think its permissions can’t really be sure though (maybe I missed something?)

Step 4

Create a copy of the solution and project, and try and tear it apart to see what is wrong.

So I create a copy of C:\Projects\XYZ -> C:\Projects\ABC….

Now where to start tearing this thing down?

Well for starters the new copied solution and project throw the same error.

4.1 Delete images in \Images dir.  (I know its a bit lame, but last thing I did was add an image, so let’s see)

Result:  Nothing

4.2 Delete fonts – Yes I had some embedded fonts, Maybe that is the problem?

Result: Nothing

4.3 Mmm maybe it’s time to start looking at references.

4.3.1 Removed:  WPFAnimatedgif.dll

Result:  Worked…..

AHA…….

So its a reference that does this…. But I actually need that. So I have an idea…

I downloaded that DLL to myDocuments. I’m going to move it to my project directory instead and reference it from there.

HOLD THUMBS!

that worked!

THE SOLUTION:

Ladies and geeks, Do not reference a DLL from MyDocuments folder.

 

 

 

Thinking of working in IT in Dubai, read this first!

This is my experience job hunting in Dubai as an IT professional.

The caste system

The first thing I would like to discuss which is important, is the caste system in Dubai. Simply put at the very top of the ladder sit the local Emirati, after this the next group having the highest social status are western / European expats. Both of these groups have high status. On the other side – Indians, Pakistanis, and Asians have a lower social status. Filipinos are at the very bottom of the ladder.

Now why am I mentioning this? If you’re planning on moving to Dubai, then think about this. If you’re from India or Pakistan you can almost expect to earn a lower salary than someone from the UK. This caste system is blatant racism, but there are very good reasons why people tolerate it, and even uphold it. But to explain this goes beyond the scope of this post.

All I’m saying is when it comes to IT salaries and work conditions, perks and benefits. Westerners have a greater shot of making real money, and from my observations Indians work for much less money in Dubai.

Dubai is a city, not a country

I know you know this, but unless you’ve actually been to Dubai, you need to think long and hard about this.  For all intents and purposes the UAE consists of 2 cities that have IT work. Dubai and Abu Dhabi – that’s it. Dubai might be a large city, but that is all it is – 1 city. So there aren’t an unlimited amount of IT jobs in Dubai. Finding work in my opinion can be tough and challenging. As I’ve mentioned in the caste system intro, there is a lot of competition and a lot of Indians who will work for 4 times less than what you would expect to get paid in Dubai, the few companies who actively only hire Westerners do pay more, and these are the companies you want to seek out – if you are a westerner.

Dubai is a shifty place

Here is the thing, as an expat you will NEVER gain citizenship in the UAE, because of this you will always be a visitor, you and everyone else. This alone affects the way people do business in Dubai. But I think generally Dubai is a cut throat type of place. The way the people are is representative to the harsh desert environment they find themselves in.

DO NOT TRUST ANY ONE.

I made the mistake of trusting people from one company I was interviewing at. I had a promise of a job, where I would need to undergo a trial period. The whole way they proposed this trial period was sketchy to say the least. I had a promise of work, and in the end it fell through, with an apology, however trust me on this, the company was irresponsible to offer me work, if they had no plans on following through with it. Do you think they care?

There are lots of stories about people not getting paid. Lots of stories about expats being forced to leave because they’ve been fired without enough notice period.

On another occasion I applied for a job, and was told by some guy with an Indian accent that I should come for an interview the next day. When asking for more information about the job, he refused to disclose anything, not even an estimated salary. I noticed that not only 1 had been invited to the interview, but the email had many other recipients in the “To” field. All with Indian names. I decided not to attend, because I expected the salary to be around 5000 AED.

Dubai is expensive

Never, ever just pack your stuff and come over thinking you will find work within the 3 months visit visa. NEVER do this. Dubai is much more expensive than you might think. Always job hunt prior to leaving, and even then you can’t be guaranteed it will work out. A lot of people have been shafted on arrival. But at least then you can cut your loss without spending your life savings.

Dubai is hot

Very hot and because of this you can expect to be spending a lot of time indoors with air conditioning. Now I’ve always had some form of air conditioning in my car, but never in my house. I’ve experienced it for part of the day at work (sometimes), and in some malls (sometimes). But never had the full 24/7 aircon experience. Let me tell you do not underestimate this. I was sick. The people I was living with had developed a persistent cough. And aircon is not healthy. Opening windows isn’t an option in the summer. And so you end up feeling like you can’t escape the aircon. One of the best things for me was having real air again, after leaving Dubai.

Dubai is the easiest place in the world to find women

I know this has nothing to do with IT, but it is worth a mention. Let me tell you Eastern Europe is a difficult place to find girls in comparison to Dubai. Eastern Europe might be fill of nice looking girls who smile and seem friendly, but when you want to take it to the next level you’re mostly left high and dry. Not so in Dubai. Women are actively seeking decent men.

Why would you want to live in Dubai?

Well in America you have the American dream. The Dubai dream goes something like this: Living in a Villa, having a house maid, driving a SUV, living and eating well and getting a lot of sun, living in a grown up Disney land.

Is this approachable? I’m sure it is, but for an IT developer you’re going to need a bit of luck finding a well paid job to pull this off. 5000 AED a month won’t give you this. Neither will 15000 AED. What you’re looking at is 25000 AED to enjoy this. And not many companies in 2013 are willing to pay a developer 25000 AED. There was a time when those salaries were getting tossed around, but not anymore.

So let’s say you realistically earn 12500 AED. Well you can still enjoy an apartment with a gym, and swimming pool and drive your SUV, but there won’t be much money left for anything else.

Still not too bad right?

Why are so many apartments empty?

A LOT of apartments in the Dubai Marina are sitting empty. I was told this by taxi drivers, and it is easy to see at night when apartment lights should be visible. Why? Rent in Dubai is expensive, but Dubai Marina is ridiculously expensive. Simply put, the Marina was created to support a luxury level of clientele, and there simply are not enough people earning that kind of money to fill the apartments. A LOT have moved out to cheaper areas, or left Dubai completely.

This should tell you something about what you can realistically expect to earn. If salaries were any better, much more of those apartments would be filled with developers like you and me earning good money.

Is it all doom and gloom?

No, there are people who have made a success in Dubai, but they didn’t get there easily. Dubai is not crying out for IT developers, and mostly development is considered cheap labour in Dubai due to the influx of Indians, etc – who are willing to work for much less than their western counterparts.

But there are always those companies who offer a higher quality of living to their employees, they’re scarce but you can find them.