Author: billpg

Posted on

POP3 – UIDL is a required command!

RFC 1957 observes, discussing mail reading software that implements the popular POP3 protocol: “two popular clients require optional parts of the RFC. Netscape requires UIDL, and Eudora requires TOP.”

This reads like a complaint, but this tell me that Netscape’s mail reader (which these days is called Thunderbird) is well designed.

The rot started with  RFC 1939, the standard for this protocol. This document specifies that UIDL is optional. This was a mistake. Without UIDL, the protocol is not reliable. I write this in the hope of persuading you that UIDL should not only be considered a requirement for a POP3 server, but that any client software that doesn’t require UIDL should not be trusted. I’m looking at you, Eudora!

What is UIDL and how does it fit into POP3?

UIDL is the “directory listing” command in POP3. When a client issues this request, the server responds with a list of “unique-id” strings that may as well be considered file names.

Opening a POP3 connection, authenticating and performing a “directory listing”.

Each unique-id is paired with a numeric id, starting from 1. The other commands to download and delete messages all use these numeric ids. Each time the client reconnects, it will need to repeat the UIDL command so it knows which numeric ids refer to which messages.

For something as fundamental as a directory listing, it seems odd for that to be optional.

Without UIDL, the client needs to fall-back onto those numeric message ids alone. Instead of UIDL, the STAT command returns the number of messages in a mailbox. With that, the client can loop from 1 to n, downloading and deleting each one, leaving the mailbox empty once they have all been downloaded. As POP3 is explicitly designed for download-and-delete operation and not keeping the messages on the server, you might consider that UIDL is not necessary. So let us follow that road where we don’t have UIDL.

Living in a world without UIDL.

Operating POP3 without UIDL only works in an ideal world. If you had 100% reliable connections to the server then you might get away with it. Reality tells us the world is not ideal.

Let’s think about the step of deleting a message once you’ve downloaded it. You might think that DELE is the request to delete messages you’ve downloaded (or don’t want), but the request to actually delete messages is QUIT.

The client flags the messages to delete with DELE, but those deletes aren’t committed until the client later issues a QUIT request. If the connection stops before a QUIT, the server has to forget about those DELE commands and the messages all have to remain in the mailbox for when you reconnect. This is by design as you wouldn’t want your messages deleted if your client is in an unstable environment that can’t keep a connection open.

Consider though, what would happen if the underlying connection was dropped just as the client issued a QUIT request. You sent the request but no response came back.

Download and delete a single message, but the connection fails a critical point.

What happened? We don’t know. We can’t know. There are three reasonable possibilities…

  • The QUIT command never arrived at the server. The server just saw the connection drop.
  • The server couldn’t process the delete and responded with an error, which got lost.
  • The server successfully deleted the messages, but the response got lost.

You asked for some messages to be deleted, but you don’t know if your instruction was processed or not. The only way to find out is to reconnect (when you can) and see if the messages you asked the server to delete has gone or not.

Let’s say that time has passed and the client is finally able to reconnect to the server again. Last time, the client downloaded a single message and may or may not have deleted it. Now we’ve reconnected we find a single message in the mailbox. Is this the one we deleted before or a new one that’s arrived in the interim? A handy directory listing would be real useful right about now!

This is why I would mistrust any mail reading software that didn’t require that a mail server implements UIDL. Messages might get downloaded twice or wrongly deleted if the wrong assumptions are made.

“Come back!”

The alternatives to UIDL are all unreasonable.

If the above doesn’t convince you that UIDL is necessary, this section is to answer anticipated responses that UIDL is not necessary. Nuh huh!

(If you are already convinced and you don’t want to read my responses to anticipated arguments, you can skip this section.)

“That scenario you describe won’t ever happen in reality.”

Stage one: Denial.

Where is this perfect world where connections don’t stop working at the worst possible time? Where database updates happen instantly? I want to live there!

Think about what a server needs to do to process a QUIT command. Many flagged messages will need to be modified in an atomic transaction such that they won’t be included next time. Indexes will need to be updated and the dust needs to settle before the server can send its acknowledgement. During this time, the underlying TCP connection will be sitting there idle, looking just like a timeout error.

“We wouldn’t have a problem if mail servers were better engineered!”

Stage two: Anger

If your requirements of a mail server include underlying connections over the public internet that never fail, I think your requirements are a little unreasonable.

“So I occasionally see two copies of a message in my mailbox. Big whoop!”

Stage three: Bargaining.

If that started happening in software I was using, I’d file a bug report.

“There are other ways POP3 can resolve this issue.”

Stage four: Depression.

Alas, all of these alternatives that POP3 provide are unreasonable.

You could use the response to LIST as a fall-back? This command requests the size in bytes of each message. Most messages are long enough that they will have a unique size, but this isn’t reliable. Messages are often going to have the same size as others just by accident.

You could use TOP to retrieve just the header and extract something from that to track messages? Problem there is that no single header is a reliable identity. Two adjacent messages might have the same date or the same subject. The closest candidate for a suitable identity is Message-ID but this is generated by the sender, who might not include it or might reuse IDs. If we’re relying on the POP3 server to add them or modify duplicates provided by a sender, we’re back to relying on optional features.

You could use the TOP response and hash the entire header? This could work except message headers can change. I first saw this when experimenting with a mail server and observed that if I connected to a mailbox using IMAP, it would leave IMAP’s version of a unique identity in the header which wasn’t there before. As well as that, anti-spam systems might re-examine a mailbox’s contents and update the anti-spam or anti-virus headers. Any of these changes would look like a new message.

(As well as all that, TOP is itself an optional command, just like UIDL.)

You could download the entire message again and ignore it if you already have it? This would be ultimate fall-back. While I’ve seen headers change, the message body seems to be immutable. This is still an unreasonable situation. We’re downloading the whole message again, just because the server chose not to implement a simple directory-listing command.

Am I certain that the message body is immutable? No, not at all. If someone commented that mail server XYZ updates messages in the form of a MIME attachment, I wouldn’t be at all surprised.

Update – A digression on the Message-ID header

(Added 28/Jan/2021)
I am grateful to commenter “theamk” on Hacker News, who responded to me when I shared this post. To my dismissal of Message-ID as a means of de-duplication, they noted that RFC standards require that Message-IDs must be generated as unique.

I have experienced senders who have broken the protocol, sending many different messages with the same Message-ID. I do not argue these senders were in the wrong but that the POP3 server is not in a reasonable position to correct the situation.

If the server actively corrected the situation and replaced the reused Message-ID header with its own unique value, the message would not be a faithful reproduction of the message as sent any more and further damage any scope for auditing.

If the server discarded or rejected the message with a reused Message-ID, it would open up means for an attacker to predict the Message-ID a legitimate sender is going to use and send a message with that ID first, causing the legitimate sender’s message to be lost. There’s nothing stopping a sender from using someone else’s Message-ID pattern. (Maybe senders should use only unpredictable strings, but wishing it so won’t make it happen.)

This is also to say nothing of the situation when the messages served up don’t have any Message-ID, which I’ve seen happen with messages exchanged within the local server only. (IE. Not routed over the public internet’s mail servers.) None of the small number of services inside the box from the original composer to the POP3 delivery agent supplied a Message-ID when it was missing, so the message turned up with the basic To/From/Subject/etc headers and a Received header, but no Message-ID.

Acceptance?

Because the alternatives are so unreasonable, I consider UIDL a requirement for handling POP3. Servers that don’t implement UIDL are bad servers. Clients that can work without UIDL are unreliable.

Still not convinced? Please leave a comment where you saw this piece posted.

“I’ve seen the future! I’ve seen the future! I’ve seen the future and it’s now!”

IMAP does it wrong.

The other popular mail-reading protocol is IMAP. In contrast to POP3’s download-and-delete model, IMAP’s model is that messages to stay on the server and are only downloaded when the client wishes to read it. This model enables mail readers on low-storage devices such as smartphones.

With IMAP, the IDs are restricted to numeric values and always go upwards, in contrast to the free-for-all “any printable ascii except spaces” allowed by POP3. While this may be nice for the client, by requiring a single source of incrementing ID numbers, it complicates matters for anyone wishing to implement an IMAP server using a distributed database as a back-end.

But the worse thing about IMAP’s message identity system is that the standard permits the server to discard any IDs it has assigned by updating a mailbox’s UIDVALIDITY property. If this value ever changes, it is a signal to the client that any unique IDs it may have remembered are no longer valid.

A client needs a reliable way to identify messages between connections to recover from an unknown state. It does not need for servers to have a license to be unreliable.

If a mail server that implements IMAP wants any respect from me, it would document that its UIDVALIDITY value is fixed and will never change and that the unique-ids it generates are reliable.

POP3 does it wrong too.

If I’m going to criticize IMAP for flaws in its unique ID system, I should address flaws in POP3’s system too, having spent most of this article praising it.

Quoth RFC 1939: “The server should never reuse an unique-id in a given maildrop,” (good) “for as long as the entity using the unique-id exists.” (no!)

Consider that worst case scenario. The client flags a single message to be deleted and finally issues a QUIT command to complete the translation. The server successfully processes the request but the response to the client is lost. As far as the server is concerned, the message is gone and there’s no problem, but as far as the client knows, the continued existence of that message is unknown.

Now consider a new message arrives on the mail server and because the RFC says it can, it assigns the same unique ID to this new message as the one that was just deleted. The client eventually reconnects and requests the list of unique IDs and finds the ID of the message it wanted to delete is still there. It doesn’t know the server used its right to reuse unique IDs and that this is actually a new message!

Now, I’ve never seen a mail server actually reuse a unique ID. The clever people who have developed mail servers in the real world seem to understand that reusing IDs is not something you ever want to do, even if the RFC says you can.

RFC 1939 also says, “this specification is intended to permit unique-ids to be calculated as a hash of the message. Clients should be able to handle a situation where two identical copies of a message in a maildrop have the same unique-id.”

Unique IDs don’t have be unique? Ugh.

This allowance only applies to identical messages. In reality, messages are never identical. After bouncing around the internet and going through various anti-spam and anti-virus servers, messages do accumulate a frightening number of Received: headers left behind from each intermediate hand-over. Each one with a time-stamp and its own ID number. Any one of these is enough to produce a distinct hash.

Picture Credits. (All Creative-Commons licensed.)
“Listening to Radio Karnali” by “BBC World Service”.
“List 84” by “Weisbaden 2010”.
“The Time of Sunset” by Joy Sarah Nawati.
“Future” by “Legosz”.
“PuTTY screen-shots” by me.

Posted on

The Brexit Chart War

There’s a movement among remainers to get Ode To Joy to number one in the charts for Brexit day. We know, even if successful, it won’t change anything. Johnson isn’t going to cancel Brexit because of this. (He totally should, nonetheless.)

So why is there a similar movement among Brexiters to get their own song “Seventeen Million F–k Offs” to number one instead?

They’ve won. Despite only getting about half the votes in 2016, Maximum Brexit is going ahead. But once the party’s over, they still won’t be happy and deep down, they know. Victory feels so much like defeat.

And among all this false revelry, there’s a bunch of smug Remoaners trying to spoil the party. “They were supposed to have been defeated by now. Our victory must be total!”

And so they get a song that’s literally telling remainers to f–k off to number one. Maybe that small victory on top of all their other victories will finally make them happy?

Sorry Brexiters. You lost. We all lost.

Get over it.

Posted on

Brexit – Is that it?

So it’s finally Brexit Day. After many repeated delays and false starts, Brexit is here!

What’s that? You’ve just posted that the sun is shining and the birds are singing?

Oh I see, you’re happy that we’re not in some apocalyptic doom. Brexit was supposed to cause the earth to split open and a literal hell on earth was about to envelop us all for a thousand years! Turns out, no, that didn’t happen.

So what? Is that it? Is that all you’ve got to show for three years of dragging our country through the grinder?

Sic transit gloria mundi

“No age is too early nor too late for the health of the soul.”
— Epicurus, Letter to Menoeceus.

First of all, this isn’t Brexit.

We’re in the transitional period laid out by the Withdrawal Agreement. Until we leave this period, everything pretty much stays the same.

What has happened, from a practical point of view…
🏰 We’ve lost our seat on The European Council.
🗳️ We’ve lost our representatives on The European Parliament.
🐘 We can no longer unilaterally revoke the Article 50 notification.

So not only have we lost our leverage, we get to follow all of the rules of the European Union without having any say in them. If that seems like something worth celebrating, I’m not going to stop you. (I might think you’re a bit weird though.)

Let’s wait until (if) we’re out of the transitional period and are actually out of the EU, not just from a legal point of view. We’ll talk then.

Fear itself

“The just man is most free from disturbance, while the unjust is full of the utmost disturbance.”
— Epicurus, Sovereign Maxims #17.

Oh yes. There was going to be World War III if we voted for Brexit.

(Let’s leave Iran to one side for a moment.)

David Cameron is normally credited with saying this, but he never actually said that. (And I hate the idea of defending the idiot that got us into this stupid mess in the first place.) The idea stems from a speech he made in May 2016, where he spoke of the EU bringing peace to a warring continent.

Look at France and Germany’s history: War, War, More War, EEC/EU, Not War.

That’s a long way from claiming Brexit would cause World War III, so I hope you’ll excuse me if I take such notions with a significant amount of salt.

Even if that were true, the remain campaign lost (as Brexiters are so eager to remind me) so it doesn’t really matter what they said. Let’s talk about what the winning party promised and if Brexit is living up to that promise.

Why are we here?

“Reality is that which, when you stop believing in it, doesn’t go away.”
— Philip K. Dick.

Please don’t go around proclaiming lack-of-hell as if it were a benefit of Brexit. It really isn’t. Would we be facing a hellish doom if we had stayed in the European Union? No. The sun would still be shining and birds would still be singing.

To be a benefit of Brexit, it would have to be:
😃Something so good or desirable that it realistically outweighs the drawbacks.
🥺Something that we can’t have as members of the EU.
😝Something we can have outside of the EU.

Let’s put on our reality hats and please tell me what realistic benefits of Brexit are. Tell me that instead of glib comments about the weather.

What are the benefits of Brexit?
🦄 The ones that are beneficial.
🦄 The ones that don’t require that you first “believe in Brexit”.
🦄 The ones that we can’t have inside the EU.
🦄 The ones that stand up to the cold, hard, unforgiving reality.

Credits:
📷 Tynemouth Priory, Castle & Moat by “WISEBUYS21”.
📷 Where good and bad weather meet by
Susanne Nilsson
📷 Lemonade Stand by John Mosbaugh
📷 Back to reality by Mike Carter.
👍 Thanks to my friends Maja and Simon and the members of “The 48%” group for their help.

Posted on

Should MPs who “cross the floor” face a by-election?

When standing for election a politician will usually run as the member of a political party. The quid-pro-quo of this arrangement is that the party will collectively campaign for all their candidates. In return, the party will expect the successfully elected MPs to support their leader’s policies and support that leader in any votes-of-confidence.

Occasionally, an MP will decide to leave their party. This is often called “Crossing The Floor” for rather archaic reasons. Sometimes that MP becomes independent, sometimes they join other parties. Most famously, wartime leader Winston Churchill left the Conservative Party in 1904 to join the Liberal Party, only to return in 1924.

When an MP does this, it will often be met with calls for that MP to put themselves up for re-election by their voters in a by-election. Normally, this only happens when an MP dies, resigns, or maybe appointed to the House of Lords.

“The voters elected a Conservative to be their MP, not a Liberal! You should let the voters have their say if they want to keep you as their MP.”

I disagree.

We Elect People, not Parties…

“These are things utterly unknown to the laws of this land, and which arise from a fundamental mistake of the whole order and tenor of our constitution.” – Edmund Burke, 1774.

In the Westminster system, we vote for people. We don’t vote for parties.

It might not seem that way, given how much of campaigns at election time are for the various political parties rather than for the individual candidates. Nonetheless, at the completion of an election, the end result is that a person will have been elected to serve as MP. That MP is an individual who has been empowered by the voters to be their representative in Parliament for the term they were elected for.

If it were the case that we do actually elect political parties, we shouldn’t need a by-election when an MP needs to be replaced. The party could simply select a replacement internally without needing to involve the electorate. This shouldn’t be a problem, after all, the voters elected that party for a five year term, not the MP.

In fact, why not just give parties the power to dismiss and replace their MPs at will? If a party leader doesn’t like how an MP is acting, they can tell that MP “You’re Fired” and appoint a replacement. If that idea seems like an undemocratic stitch-up, I quite agree. (This is also a reason why I prefer AV over PR, but that’s another article.)

Currently, the only thing that can force a by-election for an MP who doesn’t want one is if that MP is convicted of a crime and enough constituents demand a “recall”. Is leaving a political party really up there with being a convicted criminal?

… and that’s a good thing.

“It has long (perhaps throughout the entire duration of British freedom) been a common form of speech, that if a good despot could be insured, despotic monarchy would be the best form of government. I look upon this as a radical and most pernicious misconception of what good government is, which, until it can be got rid of, will fatally vitiate all our speculations on government.” – John Stuart Mill, 1861.

It’s a good thing that we elect people instead of the parties. MPs are elected and empowered to make the difficult decisions and be held responsible for them. That is the essence of Parliamentary Democracy. By so empowering MPs, they become one of the checks-and-balances. A party leader can’t overrule the objections of other MPs, precisely because dissenting MPs are so empowered. Those MPs know they will be held accountable to their voters, not to anyone else. 

Recently (as I write this in 2019), an unelected Prime Minister wished to bulldoze his Brexit policy through Parliament. MPs did the job they were elected to do and said “No”. That Prime Minister saw first-hand what happens when you try to throw your weight around like King Charles. He found himself restrained by the very checks and balances that exist in functioning democracies.

(I stand by “unelected”. As I write this, Boris Johnson has neither been elected by the voters at large nor by their representatives in Parliament. His predecessor, Theresa May, became Prime Minister while her party held a comfortable majority in Parliament and later won a vote of confidence of MPs. I would have liked to have seen a formal vote of MPs to elect a new Prime Minister when the position becomes vacant, but I accept in her case that would have been a formality.)

If we move the focus of that empowerment to the parties, those MPs will find themselves accountable to their party, not to the voters. We’ll have reduced the role of an MP to mere party functionary. If an MP’s first loyalty is to their party rather than their voters, we’ll have just populated Parliament with a bunch of toadying yes-men. We wouldn’t even need 600-odd MPs if all you need are party robots, you could get by with around 50.

What would we gain if MPs did have to contest a by-election for leaving their party. There’s a general election every five years when they will have to be re-elected anyway. When that re-election comes around they will find themselves facing someone selected by their old party. Party leaders are already rather powerful, they could do with feeling not quite so powerful once in a while.

Further Reading/Sources:

Edmund Burke, Speech to the electors of Bristol.

John Stuart Mill, Considerations on Representative Government

How MPs are elected

How to become an MP

Wikipedia, Recall of MPs Act, 2015

The Guardian, Boris Johnson to seek election after rebel Tories deliver Commons defeat

i, From ‘do or die’ to ‘dying in a ditch,’

BBC, May’s government survives no-confidence vote

Credits:

Thanks to Stuart Edmond for inspiring me to write this.

Thanks to Ollie Killingback and Andrew Phillips-Godfrey for their feedback and ideas.

Leaving Home picture by Hans Splinter.

Get Out picture by Ramy Raoof.

Posted on

A flag for Milton Keynes

I wrote this page before I found out that Milton Keynes City Council has held a flag design contest for children and already selected their new flag. This page is for the historic record.

Congratulations Milton Keynes on your new City status. All self respecting cities need a flag. London, Coventry, St Albans all have really good ones. You should have one too.

Having worked in Milton Keynes for over a decade, I was surprised to learn that Milton Keynes don’t have a flag. I was attending an event organised by a local group and they were using the “swan” flag of Buckinghamshire. This prompted me to set to work on a design.

I propose that Milton Keynes formally adopt this flag and wave it with pride. I will cheerfully gift all rights to the design to the City of Milton Keynes if they wish to adopt it as their official city flag.

Symbolism

The Cross of St George

The boldest element of the flag is the Cross of St. George, the flag of England. Milton Keynes is a part of England and found right around the middle.

The centre of the cross gives way to a circle, representing the grid system and roundabouts that Milton Keynes is famous for.

The Cubic Curve

The curve represents mathematics – the contribution of the Milton Keynes area to the war effort. The operation at Bletchley Park is credited with stopping the second world war by many years and became the foundation of the computer revolution, where Milton Keynes continues to play an integral part.

The curve itself is a formula that represents the year 1967, marking the year Milton Keynes was founded.
y = 1x³ + 9x² + 6x + 7

The Concrete Cow

The canton of the flag has a cow spot, representing the famous concrete cows. The spot itself comes from one of the cows themselves. I visited the site to look for a good spot that would fit and selected this one, inverting black and white to better fit the design.

Is this a joke?

No. I am quite serious. If you think symbolism that includes the roundabouts and concrete cows are mocking the City of Milton Keynes, remember that this is what made your city great. The grid system was built so that the sun would rise along the streets on the summer solstice as if the whole city is a Neolithic monument. That is so incredibly cool!

The cows are cool! Your local roller-derby team is named after the cows. Does London have concrete cows? No, they do not! (I think.) Be proud of your quirkiness.

“Milton Keynes has its own flag? What, a concrete cow and a roundabout?”
“Yes.”
“Cool.”

About Me

I’m Bill (P. Godfrey) and I’m a bit of an amateur vexillologist. (That’s the study of flags.)

I’ve been working for a Milton Keynes based technology company for a little over fifteen years. As I spend so much time there I joined and participate in a few local groups and societies.

There’s more about me and some of my writing on my website.

Posted on

Dear WordPress. Please stop using MySQL.

This may very well end up being my last Blogger based post as I’m slowly adopting (self-hosted) WordPress as a publishing platform. I have a set of websites running on a commodity cPanel-based shared host, with a view to moving to a dedicated VM in due course. While setting things up and playing about with WordPress, I kept tripping over an obstacle that just kept getting in the way of doing what I wanted to do.

MySQL was that obstacle.

Dear WordPress, please have an option to use a file-based database (such as SQLite) instead.


Why would you want to do such a thing?

First of all, simmer down, MySQL is a perfectly good database. It does the job it was designed to do very well. My problem is that MySQL exists on a server separated from the rest of WordPress.

Think about what makes up a single installation of WordPress. You’ve got a bunch of PHP files, the themes, the plugins, the images and media I’ve uploaded. All of these are in a single folder on the web server. I could ZIP the folder up, UNZIP it later, push the folder into GIT version control, all in the certainty I’m got everything. Except…

Some of my website is not in that folder. It’s on the database. I can’t just ZIP the website up because an essential component is off in another realm. That folder does not contain everything and I now need to keep database backups alongside the folder backups. Grrr…

I was considering adding a plug-in to one of my live websites. Because people were using it, I didn’t want any down-time. Accordingly, I made a copy of the website folder and also made a copy of the database. The new copy then had to be reconfigured to point to the new database and only then could I play about with whatever plug-in or theme I wanted to add. Whatever clones and copies I make, the database is always at arms length and I need to be very careful that the PHP is always linked with the right database and that I’ve not got any cross-overs.

If the data on the database were on a file in that folder, there would be nothing external to keep track of. Copy the whole folder and job done! Taking backups would mean zipping up the folder and everything is there without worrying about keeping the two parts in sync.

Wouldn’t that make the site inefficient?

Maybe, but I plan to use very aggressive caching. There’s a plug-in where the site contents end up as static files and the code accessing the database only has to run when I log into the admin panel or the cache engine decides its time to update itself.

I can imagine this might not work so well for a site where changes happen very frequently. Maybe, but I suspect those are in a minority. For them, MySQL would probably still be an option, but it feels like for such a website, WordPress itself probably isn’t the right tool for the job.

Incidentally, I don’t plan to support commenting because I do find moderation a bit of grind. Because I’m just one person and I have other stuff going on in my life, there would be a very long delay between someone posting a comment and my approving it. The better quality discussion tends to happen on sites like Hacker News where there is an active moderation team that I can’t even hope to match.

Why not use a WP site manager tool?

Since WordPress are probably not on the verge of releasing an update with SQLite option, I will probably end up doing exactly that.

I already have tools for managing folders and zip files. Cloning a website could be a simple folder-copy operation were it not for the separate database. Tools that know about the database are very nice but it all feels like the wrong answer to the question. We’re all in a world where things are in the state they are in and we have to stoically make it work. Site managers fix the symptoms but they don’t address the underlying issue.

Picture Credits. (All CC.)
• “Me And My Shadow” by “DaPuglet”.
• “Trees” by RichardBH.
• “Like a string of pearls” by Thomas Rousing.

Posted on

#DeniedMyVote at the 2019 European Elections

You may have read about the news that many voters have been turned away from voting over here in the UK. I’d like to share a little analogy abut what happened, why it is so controversial and how it goes beyond voter suppression.

Imagine you’re in California, and there’s a political party whose only policy is to deport anyone born on the other side of the Mississippi. (That and being paid donations through PayPal that are all conveniently under the £500 limit allowed for anonymous donations. Totally not a big donation from one person that would normally need to be declared.)

A group of people living in California but born in the east are rightfully worried that this party might gain power. They are entitled to vote as citizens so they make sure they are correctly registered.

Voting day arrives and the process goes smoothly for Californians. Voters from out-of-state, however, find they can’t vote.

Some are told they should have filled out a special form but were never told that was needed. Some did fill in the form but it got lost. Some did everything right but the people running the vote didn’t know this group was entitled to vote and just struck them off the register.

Election day passed and a significant number of people didn’t get to vote. Those people probably wouldn’t have voted for the deportation party so the situation may might end up unfairly inflating their share.

(Sounds ridiculous? This is basically what’s going on here. Only difference is our Mississippi River is a bit wider and we call ours The English Channel and The North Sea.)

Just to round out the story, on the day the news of voter suppression breaks, a senior politician chooses this day (quite coincidentally) to announce their resignation. News reports quickly start paying attention to this new distraction and soon forget about the dodgy funding and voter suppression.

Further Reading:
Channel 4
Guardian
BBC

Picture credit: Vote dans Crémazie
by Jean-Pierre Lavoie.

Posted on

My Crazy Software Engineer Tattoo (that I didn’t get)

I had an idea for a nerdy tattoo a few years ago. It would represent myself as a software engineer and I thought it was quite clever. I seriously considered having it done but decided against it in the end, despite its cleverness.

Ink’d

This is my idea, the “end comment” symbol in many programming languages:

*/

In C, and other languages that can trace their lineage to C, comments start with a /* and end with a */. Anything inside is ignored by the language, allowing the programmer to describe what’s going on. This is tremendously useful when reading other people’s code or even your own code from the past.

    /* This is a comment. */
    Code();

    /* This is another comment. */
    MoreCode();

Another way of looking at it is that these /* and */ symbols mark the change of state between comments and code. /* says “After this is comment” while */ says “After this is code.”

Or to put it another way, */ means “Enough talk, time for action.”

(This is where you exclaim to yourself how clever I am to have thought of that.)

I didn’t have the tattoo done in the end. Describing what it meant would have taken too much explanation. Even if a fellow programmer recognized the symbol, they would probably first think it looked like I’ve been “commented out”, as they wonder if I had the /* on the other side.

Also, rotated a little, it looks a bit like a squinting cyclops.

*/
Posted on

What’s a gallon?

A little while ago, I wrote a post on Facebook complaining that my new car’s fuel efficiency gauge was expressed in miles per gallon. “What’s a gallon?” I asked.

Over here in England, fuel has been sold by the litre for decades. Gallons are an archaic unit which no-one under 40 has ever had to deal with.

Unitary council

Two friends very helpfully commented with conversion rates. “3.78 litres in a gallon” said one friend. “4.54 litres in a gallon” said another. While both answers were significantly different, both my friends were equally quite correct.

One figure is for a United States standard gallon, while the other is for an Imperial standard gallon. Which one does my car use?

“They went metric without telling us!”

“The car was made for the English market – It’ll be the gallon that’s used in England.”

But we don’t use gallons in England and we’re back where we started. It could just as easily be argued that it probably uses the American gallon because that is the only unit called a “gallon” that is still in active use in 2019 and you can’t say that about the imperial gallon. (Unless maybe Liberia or Myanmar still uses it.)

But this all misses the point. The gallon has inserted itself into the process quite unnecessarily. The car’s electronics already tracks fuel levels in litres and would have to perform a calculation in order to show me fuel efficiency in gallons. That leaves me having to perform the opposite calculation to get the figure back into litres again. If the car designers had stuck with litres and didn’t bother with pointless calculations, we’d all be a lot happier.

“I would walk 500 miles…”

“But Bill, you want to measure fuel efficiency in miles per litre, but why not kilometres per litre?”

While England have moved on from gallons, we remain tied to miles, regretfully. Speed limit signs are in miles per hour, as are the big numbers in my car’s dashboard. I would love it if we joined the rest of the world and completed metrication. Our painless move from gallons shows we can do it.

Imagine if a filling station got it in their heads to start selling fuel by the gallon, as if it fell through a time vortex. Anyone stopping at that station is only going to ask “What’s that in litres?”. Most people would drive on, looking for a filling station that isn’t weird.

Our move to litres is a success story for England. We can make the switch. We barely noticed when food and drinks (other than in pubs) started being sold by the kilogram or litre. We got used to it as we joined the rest of the world. A few people grumbled but the rest of got on with our lives. Switching to kilometres on the road will be the same.

Picture Credits…
📸 “The Price You Pay” by Damian Gadal. (CC)
📸 “They went metric…” by me!

Posted on

GIT isn’t perfect. (And other blasphemies.)

I was embarrassingly late in the game coming to GIT as a version control system. Time has since passed and I’m now happily using it. The days when we had to lock files before we worked on them are thankfully a distant memory.

My road was a little bumpy…

Orphan git.

It’s a new dawn, it’s a new day, it’s new version control system and I’m ready to download a project from a GIT server.

The first step was to perform a clone of a project. “Clone? That’s a strange word for it. Maybe this is just it’s own way of saying Download.” Whatever you call it, my “clone” was ready and I can get to work.

Later, I’ve done my work, tested it my dev-environment and I’m ready to check it in. Before I do that I need to check on any changes the others have done. Looking for a “Get Latest” option, I find “Pull”, which seems to do the right thing. Those changes don’t conflict with mine so I can continue.

How do I get my changes back to the server? There’s a “push” option which sounds like it would be the opposite to pull, but that doesn’t do anything. Finally, I find “Commit” and it shows me all my changes with space to write my comments. Success!

I’ve committed my changes to the master branch! Sounds final. The red icons next to the modified files have now turned green. There’s no indication there’s more to do, or at least none that’s obvious to this beginner. My job is done so I can now switch my workstation off and disappear on vacation for a week.

Not quite. The thing called the “master” branch isn’t the master branch. When I had the “clone” operation and thought the name was a bit odd, it turned out I had just made a complete copy of the code repository and it was now separate from the original. To finish the job, I needed to also perform a push, but after having done the commit.

And this is what annoys me about GIT – we profoundly disagree about what’s important. As far as I’m concerned, my local hard disk is just a necessary staging point to the way to the central repository where release builds happen. You perform a commit and that has all the ceremony and pizzazz, but a push deals with the big picture.

Ask most GIT clients what’s important an it will answer that its own local repository is the focus of operations. The remote server is basically an after-thought. “Huzzah! We’ve done a commit! Oh, you want to push too. Okay then. Don’t forget to do the push again next time because I won’t remind you if you forget!”

(I have, many times, ran completely pointless builds having committed but forgotten to push. The build server builds without my changes and I end up wondering why nothing I did made any difference.)

There is no shared server only ZUUL!

Before I proceed, I should probably clear a few things up. Discussing GIT architecture can be a bit tricky because if you talk about a shared server or clients-and-servers as separate things, someone will step in and insist those don’t exist.

Okay. As my mathematics teacher once told me, “You can use what ever fruity language you like, as long as you define your terms.”

A GIT server never produces its own content (except maybe for house-keeping), but instead receives commits pushed from one or more clients.
A GIT client produces commits (by the user) and pushes them to the server.

“I was very happy with SVN thank-you-very-much.”

Enough negativity. What would my ideal client would look like? Maybe it already exists. In a nutshell, there would be no local repository. All operations would take place on the shared server with the client channelling my actions to that server.

  • When I commit, I’m committing to a branch on the server.
        (No more forgetting to push after commit.)
  • When I create a branch, I’m creating the branch on the server.
        (No more forgetting to pull before starting.)
  • When I switch branches, I’m switching to the latest current state of that branch on the server.
  • When I browse the history, I’m browsing the history on the server.
  • If there is anything stored locally, there’s maybe a cache to save time but that’s it.

As far as the server and the rest of the team are concerned, I’m still using traditional GIT. I work, I commit changes, I push them to a branch on the server. Just like everyone else. No-one has to change the way they work to accommodate what I’m doing.

How would it work?

Some clients already have a combined commit-and-push option. My ideal client would take that further into an atomic commit-and-push. The client would create the commit data internally and then attempt to push it to the server. If the push fails, (perhaps someone else has made a change or the server is down) then the error is reported to the user but the commit is rolled back. Once the user has resolved the problem, they may try to commit again.

Other actions would work in a similar way. Actions that would normally happen as actions upon the local branch, are instead applied to the remote branch using pushes and pulls where necessary. If there is a local “cloned” repository behind the scenes, it’s just there as a convenient cache.

What if you are offline?

One of the selling points of GIT is that you can work offline, perhaps while travelling on an airplane. Sometimes the server itself is down and it just keeps on working.

If you do find yourself offline and you can’t push your work, my ideal client would have a way to store an “offline commit”. This would effectively be like how commits work in traditional clients. The difference is that the UI wouldn’t hide that the commits are only offline. The changed files would have a different color and there would be a bright indicator somewhere, warning you that you’ve not really committed your work yet.

What if you send pull-requests or patches instead of pushing?

All of this would only work if you have push rights to the server. Some people don’t work that way and instead can only clone-and-pull, but need other means to get their work into the shared repository. For those people, fair enough, the traditional client is probably best. The best of all would be one that could work in either way, depending on if the user has push rights or not.

I wouldn’t be able to push updates to you directly.

When GIT people say that there’s no such thing as a shared GIT server, they mean it. When you do a “push” action, you don’t have to push to the origin server, you could push to one of your colleague’s workstations instead.

People do that? That sounds like a project management nightmare!

SVN is still available if you prefer that way of working.

I’d still rather be working with GIT than the previous generation such as SVN. Merging changes was a nightmare and I’m happy those days are behind me. Distributed teams couldn’t work well with SVN without a lot of administration.

All in all, I’m only really looking to replace one isolated part of the GIT system. If I switched to my ideal alternative client, the rest of the team wouldn’t have to and could continue using traditional GIT happily.

Picture Credit:
100_2223 by “paolo”. (CC)
Sir Walter Raleigh, by William Segar, 1598. (PD)
(Why yes, I am very clever. Thank you for noticing. I’m sure no-one else has thought of this exact joke.)