My issues with UoN SU SUMS

What came before

When I joined the University the students' union was using an archaic piece of software by the name of MSL. This wasn't great, the website looked like it was built for IE, it was painful to use on mobile, and overall it was clunky, but it worked. After being elected onto a couple society committees, over the summer of 2020 I started to use the management side of it a lot more. While it looked outdated it did work fine, and I was able to quickly learn the ropes. I may be looking back with somewhat rose tinted glasses given the shitshow that followed, but I believe that the old system was not too bad.

The great transition

As much as the old system wasn't too bad, it was clear a replacement was needed at some point. The lack of proper mobile support had become a real issue by 2020, and consequently the SU announced the move to a new system called "SUMS", or Student Union Management Service. You can find their website at sums.su (not actually run by the Soviet Union). The changed was scheduled to take place in early July, to give everyone a chance to get familiar with the new system and to iron out any issues before the start of term, and this seemed sensible enough. I expected there to be some settling in pains, so it would be great if they could happen over the summer when it isn't super necessary.

I started getting really worried when August came and went, leaving us with no sign of the new system as we went into September. Had the impact of the pandemic caused the SU to put their plans on hold, knowing how essential functioning online systems would be, or maybe that they knew they couldn't pull it off given that they had made a large portion of their relevant staff redundant to fill a £750,000 budget gap? Fuck no!

The initial deployment of the system was done around the 10th of September (plus or minus a few days for memory lapses, this applies to all dates in this article), sensibly done on a subdomain so the old system could keep running. Initially this was just a front end website, with no way to log in, and most links leading to a 404 error, but over the next week or so we gained basic abilities like being able to log in and some of the links now when to pages with significantly less information on them than the old site, but at least they were going somewhere. The real problem came on the 17th of September, when without warning they pulled the old site and moved SUMS over to the main domain. The transition was likely done then so as to have the new system in place for the start of term on the 21st of September, but SUMS was not ready.

At this point I was stuck using the new system which was not functional at this point. There was no documentation for the new site, and the training material for SU stuff unrelated to the site mostly broke as all of the links to useful information broke, just returning 404 errors. Clearly this has been rushed out, and we were only beginning to see the issues it would cause.

The email saga

My role on both of the committees I am on (AnimeSoc and HackSoc) mostly centres around the sending of the weekly newsletter. Because of this, one unfortunate Sunday (and the Saturday before and Monday afterwards) I had to figure out the way to send a newsletter using the new system. The AnimeSoc newsletter is very simple, and was just a text email. This I could write up into the newsletter sending box directly and it was mostly fine. The issue I encountered immediately was that I wanted to put some links into the email, but SUMS didn't give an option for that. (I also wanted a header image but gave up on that quickly). I considered links as an essential feature, as they boast a much higher click through rate than just pasting the URL as text, and they also look much nicer, particularity for long and/or machine generated URLs. I eventually managed to get my link in by using the inspector tool built into Firefox, and manually editing the HTML in the correct section of the page which made up the email body. This was a fairly horrible user experience, but it did work, so whatever.

The real issue came from sending the HackSoc newsletter, which was a fancy HTML email produced using mailchimp. The old system had a source button which would allow you to put in the HTML source code directly in, but SUMS has nothing like that. Fortunately the trick with the Firefox inspector worked here as well, so I was able to get my code into the email, but that was only the start...

When I sent the first test email to committee members (gone was the option to test with a specific email address, so they got all of the many test emails involved) it initially looke fine, until I scrolled down and saw that the email had been cut off half way, leaving only a single half complete table tag. Inspection of the HTML indicated that the email HTML stopped after a certain point, with only a couple lines of SUMS tracking code before the end. This immediately smelt like a length limit, and further testing relieved this to be the case. The initial email was only about 84 KiB of HTML, but apparently that was too much. I tried minifying the HTML and that got more of the email through, showing that I was on the right lines. Now mailchimp does not produce the smallest code, mainly to maintain compatibility with as many email clients as possible. (Thanks Outlook!) so I was able to squeeze some more email by removing comments and manually optimising the HTML. This probably broke Outlook compatdibility but at that point I was beyond caring. With this done I was still unable to avoid the email getting cut off, so I was forced to remove some content. This email, being the first one of the year, was a bit one, but with some regret I managed to remove enough from the email for it not to cut anything off. (It still actually cut off the footer a little but whatever.)

So now I just selected the drop down option for "All Members" and hit send. Yeah right.
When I did that SUMS immediately returned the message "Failed". This error message is worse than useless, and did not make my mood any better, having worked on this for many hours prior. At this point I gave up and sent a very simple email using the SUMS editor (and HTML inspector for the link) linking to a copy of the newsletter hosted on the web. Unsurprisingly, engagement was atrocious for that newsletter.

The next weekend I sat down to figure out what was going on, and this is when one of my senpai on the committee suggested an API limitation. We will never know for sure, but we though, and testing suggests, that SUMS makes a big old API request to some other service to send the emails. The content of the request is something along the lines of:

{ [
{"address": "[email protected]", "content": "<!DOCTYPE html> ..."},
{"address": "[email protected]", "content": "<!DOCTYPE html> ..."},
...
] }

Which seems to include a separate copy of the email source for each email address, making the request very large. Some back of the envelope calculations give this single request a size of about 20 MB, so it is no wonder that the emailing API is rejecting it. I suspect the reason I didn't encounter this issue with the AnimeSoc newsletter was the combination of the smaller newsletter with the much smaller membership base (around a third of HackSoc).

To get around this we had to send the email to smaller groups of our membership at a time. SUMS has a "Communication Groups" feature which you can add specific people too, and this is what I used to do this. I got the membership list in JSON form and wrote a python script to parse and output 20 names at a time, which I put into communication groups and was finally able to send the email to, through a lot of painful copy pasting. (The fact that I had to click on a popup with each one did not help.) Eventually, after about 10 hours that weekend spent on it, I managed to send out the newsletter. I did not want to repeat that.

To fix the problem of email size I ended up writing my own email generator, which took markdown and converts it into a fancy HTML email using a supplied template. It is open source and you can find it here. Writing this also taught me the many issues of HTML emails, which is a whole separate rant, but also highlighted another issue with SUMS, in that it strips <style> tags out of the head, despite them being supported by almost all email clients nowadays, so I had to add CSS inlining to my program.

Fortunately some things have gotten somewhat better, with them adding in the capability to do links and images in the SUMS editor, and most importantly, about a week ago they fixed the issue with sending the email, so I no longer have to spend my life copy-pasting membership lists.

As an aside I also learned from another society that used them that attachments were broken. They would be inserted as a link into the email but when you opened it you were stopped with a 403 Forbidden error, even if you were logged into the SU site.

The current state of things

What prompted me to write this rant was that when I went to send my newsletters on Monday, I couldn't. They would send and then just remain stuck on "Pending". At least we have had some communication from the SU, who posted on Facebook (because that is the only bloody way to get in contact with them) that the email system was not working, but would be fixed by the evening or perhaps the morning after. Now, four days later (And no replies to my status enquiries) it is still not working, and after having spend dozens of hours wrestling with this system only to have it continuously fail in new and exciting ways, I am finally sick of it. Also, currently treasurers have no way to access the societies accounts, which is causing all kinds of trouble, as can be imagined.

While a transition like this was needed at some point, the timing and execution of this change has been horrendous, and has been a constant source of pain for me.

Here's hoping that by next year SUMS will be sorted out so that it is not plagued by these issues, though I have my doubts.