This is the second post in my series describing a number of extensions to the POP3 protocol. The main one is a mechanism to refresh an already opened connection to allow newly arrived messages to be downloaded, which I’ve described on a separate post. This one is a lot simpler in scope but if we’re doing work in this protocol anyway, this may as well come as a package.
I am grateful to the authors of POP4 for the original idea.
This post is part of series on POP3. See my project page for more.
Enumeration
To recap, when a client wishes to interact with a mailbox, it first needs to send a UIDL command to retrieve a list of messages in the form of pairs of message-id integers and unique-id strings. (I’ve written before how UIDL really should be considered a required command for both client and server.)
The numeric message-ids are only valid for the lifetime of this connection while the string unique-ids are persistent between connections. All of the commands (prior to this extension) that deal with messages use the numeric message-ids, requiring the client to store the UIDL response so it has a map from unique-id to message-id.
This extension allows the client to disregard the message-ids entirely, modifying all commands that have a message-id parameter (RETR, TOP, DELE, LIST, UIDL) to use a unique-id parameter instead.
“UID:”
If the server lists UID-PARAM in its CAPA response, the client is permitted to use this alternative form of referencing a message. If a message-id parameter to a command is all numeric, the server will interpret that parameter as a numeric message-id as it always has done. If the parameter instead begins with the four characters “UID:”, the parameter is a reference to a message by its unique-id instead.
C: DELE 1
S: +OK Message #1 (UID:AAA) flagged for deletion.
C: DELE UID:AAB
S: +OK Message #2 (UID:AAB) flagged for deletion.
(The POP4 proposal used a hyphen to indicate the parameter was a unique-id reference. I decided against adopting this as it could be confused for a negative number, as if numeric message-ids extended into the negative number space. A prefix is a clear indication we’re no longer in realm of numeric identifiers and may allow other prefixes in future.)
If a client has multiple connections to a single mailbox, it would normally need to perform a UIDL command and store the response for each connection separately. If the server supports unique-id parameters, the client is permitted to skip the UIDL command unless it needs a fresh directory listing. Additionally, the client is able to use a multiple connections without having to store the potentially different unique-id/message-id maps for each connection.
RFC 1939 requires that unique-ids are made of “printable” ASCII characters, 33 to 126. As the space (32) is explicitly excluded, there is no ambiguity where a unique-id parameter ends, either with a space (such as with TOP) or at the end of the line.
Not-Found?
If a requested unique-id is not present, the server will need to respond with a “-ERR” response. To allow the client to be sure the error is due to a bad unique-id rather than any other error, the error response should inside a [UID] response code. (The CAPA response should also include RESP-CODES.)
S: RETR UID:ABC
C: -ERR [UID] No such message with UID:ABC.
S: RETR UID:ABD
C: -ERR [SYS/PERM] A slightly more fundamental error.
It should be noted that a [UID] error might not necessarily mean the message with this unique-id has been deleted. If a new message has arrived since this particular connection opened, the server may or may not be ready to respond to requests for that message. A client should only make the determination that a message has gone only if it can confirm it with either a new or refreshed connection.
Extensions yet to come
I’ve pondered about if this extension, once its written up in formal RFC language, should modify any future extensions that use message-id parameters. Suppose next year, someone writes a new RFC without having read mine that adds a new command “RUTA” that rutabagas the message specified on its command line.
(What? To rutabaga isn’t a verb? Get that heckler out of here!)
The wording could be: “Any command available on a service that advertises this capability in its CAPA response, that accepts a message-id parameter that is bounded on both sides by either the space character or CRLF, and normally only allows numeric values in this position, MUST allow a uid-colon-unique-id alternative in place of the message=id parameter.”
(In other words, this capability, only changes commands where a unique-id with a prefix can unambiguously be distinguished from a numeric message-id.
My inclination is for the RFC defining this capability exhaustively lists the commands it modifies to just the ones we know about. (RETR, TOP, DELE, LIST, UIDL.) I would add a note that strongly encourages authors of future extensions to allow UID: parameters as part of their standard. If someone does add a RUTA command without such a note, then strictly speaking, the client shouldn’t try and use a UID: parameter with the RUTA command, but probably will.
I’m on the fence. What do you think?
Restrictions
RFC 1939 that defines POP3, makes a couple of allowances with the UIDL command that would make UID: parameters problematic. A server is allowed to reuse unique-ids in a single mailbox, but only if the contents of two messages are identical. A server is also allowed to reuse a unique-id once the original message using that unique-id has been deleted.
Since these allowances would introduce a complication to which message is being referenced, any server advertising this capability (in RFC language) MUST NOT exercise these two allowances. If a server advertises UID parameters, it is also promising that its unique-ids really are unique.
Fortunately, all mail servers I’ve looked at can already make this promise, either they use a hash but add their own “Received:” header or they assign an incrementing ID to each incomming message.
POP3 is a popular protocol for accessing a mailbox full of email messages. While small devices have moved their mail reading apps to IMAP and proprietary protocols, POP3 remains the preferred protocol for moving email messages between big servers where a no-frills, download-and-delete system is preferred.
A problem with this protocol is embodied in this question: “How often should we poll for new messages?” There’s a non-trivial overhead to connecting. Poll too quickly and you overload the system. Poll too far apart and messages take too long to arrive.
Over my head!
To recap, let’s take a look at what needs to happen every time a POP3 client wants to check for new messages.
The client and server handshake TCP as the underlying connection.
The client and server handshake TLS for security.
The client authenticates itself to the server.
The client finally gets to ask if there are any new messages.
“Oh, no new messages? Okay, I’ll go through all that again in five seconds.”
POP3 doesn’t have a way to avoid this continual opening and closing of connections, but it does have a mechanism to add extensions to the protocol. All it needs is for someone to write the new extension down and to develop a working prototype. Which I have done.
“I have no time for your prototype!”
billpg industries POP3 service
On my github account, you’ll find a prototype POP3 server that implements this extension. Download it, compile it, run it. Go nuts. The service is written using the “listener” model. You set it up to listen for incomming connections and it talks the protocol until you shut it down. The library deals with the complexities while requests for messages are passed onto your provider code.
You don’t need to write that provider code if you only want to try it out. I’ve written a basic Windows app where you can type in new messages into a form, ready for the client to connect and download them. If Linux is more your thing or you prefer your test apps to work autonomously, I’ve also written a command-line app that sits there randomly populating a mailbox with new messages, waiting for a client to come along and get them.
To Sleep, and Goodnight…
Now for the new extension itself. It comes in two parts, the SLEE command put a connection to sleep and the WAKE command brings it back.
If a server normally locks a mailbox while a connection is open, then SLEE should release that lock. In fact, SLEE is defined to do everything that QUIT does, except actually close down the connection. Crucially, this includes committing any messages deleted with DELE.
During a sleeping state, you’re no longer attached to the mailbox. None of the normal commands work. You can only NOOP to keep the underlying connection alive, QUIT to shut it down or WAKE to reconnect with your mailbox.
If the server responds to WAKE with a +OK response, a new session has begun. The refreshed connection needs to be viewed as if it is a new connection, as if the client had QUIT and reconnected. The numeric message IDs from before will now be invalid and so the client will need to send a new STAT or UIDL command to update them.
In order to save the client additional effort, the server should include a new response code in the +OK response.
[ACTIVITY/NEW] indicates there is are new messages in the mailbox that were not accessible in the earlier session on this connection.
[ACTIVITY/NONE] indicates there are no new messages this time, but it does serve as an indication that this server has actively checked and that it is not necessary for the client to send a command to check.
(No “ACTIVITY” response indicates the server is not performing this test and the client will need to send a STAT or UIDL command to retrieve this information.)
A server might give an error response to a WAKE command, which may include a brief error message. In this situation where a connection can’t be refreshed for whatever reason, a client might chose to close the underlying connection and open a new one.
The error might include the response code [IN-USE] to indicate that someone else is connected to the mailbox, or [AUTH] to indicate that the credentials presented earlier are no longer acceptable.
Note that the SLEE command is required to include a commit of DELE commands made. If the client does not want the server to commit message deletes, it should send an RSET command first to clear those out.
How would a client use this?
To help unpack this protocol extension, here is description of how the process would work in practice with a server that implements this extension.
The client connects to the server for the first time. It sends a CAPA request and the response includes SLEE-WAKE, which means this connection may be pooled later on.
The client successfully logs in and performs a UIDL which reveals three messages ready to be downloaded. It successfully RETRs and DELEs each message, one by one.
C: USER me@example.com
S: +OK Send password.
C: PASS passw0rd
S: +OK Logged in. You have three messages.
C: UIDL
(Remainder of normal POP3 session redacted for brevity.)
Having successfully downloaded and flagged those three messages for deletion, the client now sends a SLEE command to commit those three DELE commands sent earlier. The response acknowledges that the deleted messages have finally gone and the connection has entered a sleeping state.
C: SLEE
S: +OK Deleted 3 messages. Sleeping.
The client can now put the opened connection in a pool of opened connections until needed later. It is not a problem if the underlying connection is closed without ceremony in this state, but it may be prudent for the pool manager to periodically send a NOOP command to keep the connection alive and to detect if any connections have since been dropped.
C: NOOP
S: +OK Noop to you too!
Time passes and the client wishes to poll the mailbox for new messages. It looks in the pool for an opened connection to this mailbox and takes the one opened earlier. It sends a WAKE command to refresh the mailbox.
C: WAKE
S: +OK [ACTIVITY/NONE] No new messages.
Because the server supports the ACTIVITY response code and the client recognized it, the client immediately knows that there is nothing left to do. The client immediately sends a second SLEE command to put right back into the sleeping state.
C: SLEE
S: +OK Deleted 0 messages. Sleeping.
(Incidentally, the client is free to ignore the “ACTIVITY” response code and instead send a STAT or UIDL command to make its own conclusion. It will need to do this anyway if the server does not include such a response code.)
More time passes and the client is ready to poll the mailbox again. As before, it finds a suitable opened connection in the pool and it sends another WAKE command.
C: WAKE
S: +OK [ACTIVITY/NEW] You've got mail.
Having observed the notification of new mail, the client sends a new sequence of normal POP3 commands.
This was all within one connection. All the additional resources needed to repeatedly open and close TCP and TLS are no longer needed.
I’ll be doing the job of turning all this informal text into a formal RFC on my github project.
I advocate for the UK to democratically re-join the EU. Without shame or apology.
“We can’t re-join, there was a referendum!”
There was indeed. The referendum said the UK should leave the EU. On the 31st of January 2020, it left.
The referendum did not say “The UK should leave the EU and stay out forever”. It didn’t even give a minimum number of years. I checked the wording on the ballot paper just to be sure. It prescribed a single action and that action has been done.
The moment the UK left on the 31st of January, the 2016 referendum lost all power. The UK could have re-joined the next day. We do not need to continue to “respect the result” because it has already been respected, in full.
This piece is an attempt to persuade you, dear reader, that the UK should apply to re-join the EU. “Remaining” is not longer an option, but applying to re-join certainly is, and it is an option we should wholeheartedly take.
“We couldn’t get the same deal we had before.”
To re-join, the UK would be entering the EU as new members. The opt-outs negotiated by governments past would not be available. Let’s deal with those before going any farther.
“The UK would have to join The Schengen Area.”
Yes! Let’s join The Schengen Area! This was meant to be a section of responses to reasons not to join, but this is a benefit. We’d get to be in the Schengen Area! Huzzah!
This is an agreement that allows people to cross internal borders without having to show a passport or apply for a visa. You just cross the border, maybe glancing at the welcome sign that announces you’re in a new country now. There are exceptions allowed for emergencies and pandemics, but most of the time you cross over without ceremony.
I have an American extended family, via my American wife. One of those relatives was going to visit Paris as a second honeymoon and they asked us if they could come and visit us in England on a day-trip through the tunnel. We looked at the details and because the UK has never been in Schengen, they would have to formally leave Schengen in order to enter the UK. On their return to France they would have to go through the process again to re-enter the Schengen area again under a new visa, just to return to their hotel room where they left all their luggage.
In the end, the hassle of multiple entries and re-entries was too much and they decided against including England on their second honeymoon. I see this often in Americans travelling around Europe, skipping the islands because the inhabitants’ insistence on being special and having their own passport area.
But if you really want to keep the advantages of being outside Schengen, I have a plan. Next time you travel to France, take your passport, even though you wouldn’t have to. When you arrive, approach some passing French person and demand they look at your passport.
“Look at my passport, Frenchie! Look at it!”
“The UK would have to join the Euro.”
Great! Let’s join the Euro! (Why are benefits of EU membership keep getting named as down-sides?) We’ll be able to trade with other member states without having to bear the costs of exchanging currency rate all the ding dang time.
We won’t be missing much. The Pound we have today isn’t the same Pound from my childhood. Small change nowadays goes straight into a jar in the corner. 50p coins used to have an image of Britannia but now is only some design from a trendy design studio. ÂŁ1 coins used to have the Latin “Decus et Tutamen” engraved around the edge, a tradition that dates back to Isaac Newton, but no more.
Since Brexit caused the value of the pound to tank from which it has never recovered, people have no idea how much things are any more. The pound has given us neither history nor stability. We may as well join the Euro and retire the pound.
But even if you are not as enthusiastic for joining the Euro, the EU’s rules only require that new members join when the economy is ready. Those EU members who are not using the Euro seem quite relaxed about this status quo and are showing no hurry to switch. There’s no reason the UK can’t do the same.
“The UK wouldn’t get the rebate from its membership fees.”
Finally!
In the 2016 referendum campaign, busses were driven around with the claim that the UK sends ÂŁ350 million a week to the EU. This was false, because it didn’t take account of the rebate the UK negotiated in the 80s. The actual amount the UK sent to the EU was in fact significantly smaller. If the UK did re-join, it is with a bit of irony that this could actually end up being the amount we’d send to the EU.
It sounds like a lot of money, but is it really? There are around 42 million adults in the UK. Divide one figure by the other and the cost of the UK’s EU membership is on average, around ÂŁ8.33 per week per adult. That’s the big scary number on the side of a liar’s bus. Eight f…ing pounds and thirty three pence a week!
If that one pound and change per day is really the argument for staying out, Brexit itself has already cost the UK almost as much as it has sent the EU since joining the EEC in the 70s. Throwing good money after bad is not a viable economic strategy. Add all the costs and benefits together and the UK gained from EU membership!
The UK would benefit from membership.
Support for the Good Friday Agreement. Freedom for UK citizens to live/work/retire across the EU. Participation in Galileo satellite navigation. Driving licences and insurance valid accross the EU. Erasmus student exchanges. Mutual recognition of professional qualifications. (Just a few of my favourites from Edwin Hayward’s list.)
We’d get the benefits of EU membership. Why shouldn’t we pay at the same rate all other member states have to pay?
This is the core reason I advocate for re-joining the EU. We would be members and membership has its benefits. Thanks to the EU’s Single Market and Customs Union, any EU based business can trade all over the EU. Any regulatory tests can be done once without having to repeat them for each member state. Your couriers don’t need to wait for customs checks or to pay tariffs. Your invoices can be paid without additional paperwork.
It might be argued that the UK could join the Customs Union and Single Market, without joining the EU as full members, as many of the EU’s neighbours such as Norway have done.
But imagine if a club local to you offered you a couple of choices. Both choices included use of the club’s facilities in return for a membership fee, but one choice also gave you a democratic say in how the club is run.
That democratic say in how the EU is run comes with membership. Norway doesn’t get to sit at the European Council nor do their citizens get to elect members of the European Parliament. If we’re going to be in the club, let’s be in the club!
Epilogue
I wasn’t going to write this. A friend of mine asked me to make a case for re-joining and it felt like a fool’s errand. I know his feelings about the EU and I doubted any argument I could make would persuade him, a serious long-term euro sceptic. The arguments I’ve made here are what would convince me, but I’m already a flag-waving pro-European.
As such, I would like to consider this piece part one of a series. To both my euro-sceptic friend and anyone else who may be reading this, why do you want the UK to stay out of the EU? If I get enough responses, I’ll put a part-two together responding to those points.
Sources
BBC News, including image of 2016 opinion poll form.
RFC 1957 observes, discussing mail reading software that implements the popular POP3 protocol: “two popular clients require optional parts of the RFC. Netscape requires UIDL, and Eudora requires TOP.”
This reads like a complaint, but this tell me that Netscape’s mail reader (which these days is called Thunderbird) is well designed.
The rot started with RFC 1939, the standard for this protocol. This document specifies that UIDL is optional. This was a mistake. Without UIDL, the protocol is not reliable. I write this in the hope of persuading you that UIDL should not only be considered a requirement for a POP3 server, but that any client software that doesn’t require UIDL should not be trusted. I’m looking at you, Eudora!
What is UIDL and how does it fit into POP3?
UIDL is the “directory listing” command in POP3. When a client issues this request, the server responds with a list of “unique-id” strings that may as well be considered file names.
Opening a POP3 connection, authenticating and performing a “directory listing”.
Each unique-id is paired with a numeric id, starting from 1. The other commands to download and delete messages all use these numeric ids. Each time the client reconnects, it will need to repeat the UIDL command so it knows which numeric ids refer to which messages.
For something as fundamental as a directory listing, it seems odd for that to be optional.
Without UIDL, the client needs to fall-back onto those numeric message ids alone. Instead of UIDL, the STAT command returns the number of messages in a mailbox. With that, the client can loop from 1 to n, downloading and deleting each one, leaving the mailbox empty once they have all been downloaded. As POP3 is explicitly designed for download-and-delete operation and not keeping the messages on the server, you might consider that UIDL is not necessary. So let us follow that road where we don’t have UIDL.
Living in a world without UIDL.
Operating POP3 without UIDL only works in an ideal world. If you had 100% reliable connections to the server then you might get away with it. Reality tells us the world is not ideal.
Let’s think about the step of deleting a message once you’ve downloaded it. You might think that DELE is the request to delete messages you’ve downloaded (or don’t want), but the request to actually delete messages is QUIT.
The client flags the messages to delete with DELE, but those deletes aren’t committed until the client later issues a QUIT request. If the connection stops before a QUIT, the server has to forget about those DELE commands and the messages all have to remain in the mailbox for when you reconnect. This is by design as you wouldn’t want your messages deleted if your client is in an unstable environment that can’t keep a connection open.
Consider though, what would happen if the underlying connection was dropped just as the client issued a QUIT request. You sent the request but no response came back.
Download and delete a single message, but the connection fails a critical point.
What happened? We don’t know. We can’t know. There are three reasonable possibilities…
The QUIT command never arrived at the server. The server just saw the connection drop.
The server couldn’t process the delete and responded with an error, which got lost.
The server successfully deleted the messages, but the response got lost.
You asked for some messages to be deleted, but you don’t know if your instruction was processed or not. The only way to find out is to reconnect (when you can) and see if the messages you asked the server to delete has gone or not.
Let’s say that time has passed and the client is finally able to reconnect to the server again. Last time, the client downloaded a single message and may or may not have deleted it. Now we’ve reconnected we find a single message in the mailbox. Is this the one we deleted before or a new one that’s arrived in the interim? A handy directory listing would be real useful right about now!
This is why I would mistrust any mail reading software that didn’t require that a mail server implements UIDL. Messages might get downloaded twice or wrongly deleted if the wrong assumptions are made.
“Come back!”
The alternatives to UIDL are all unreasonable.
If the above doesn’t convince you that UIDL is necessary, this section is to answer anticipated responses that UIDL is not necessary. Nuh huh!
(If you are already convinced and you don’t want to read my responses to anticipated arguments, you can skip this section.)
“That scenario you describe won’t ever happen in reality.”
Stage one: Denial.
Where is this perfect world where connections don’t stop working at the worst possible time? Where database updates happen instantly? I want to live there!
Think about what a server needs to do to process a QUIT command. Many flagged messages will need to be modified in an atomic transaction such that they won’t be included next time. Indexes will need to be updated and the dust needs to settle before the server can send its acknowledgement. During this time, the underlying TCP connection will be sitting there idle, looking just like a timeout error.
“We wouldn’t have a problem if mail servers were better engineered!”
Stage two: Anger
If your requirements of a mail server include underlying connections over the public internet that never fail, I think your requirements are a little unreasonable.
“So I occasionally see two copies of a message in my mailbox. Big whoop!”
Stage three: Bargaining.
If that started happening in software I was using, I’d file a bug report.
“There are other ways POP3 can resolve this issue.”
Stage four: Depression.
Alas, all of these alternatives that POP3 provide are unreasonable.
You could use the response to LIST as a fall-back? This command requests the size in bytes of each message. Most messages are long enough that they will have a unique size, but this isn’t reliable. Messages are often going to have the same size as others just by accident.
You could use TOP to retrieve just the header and extract something from that to track messages? Problem there is that no single header is a reliable identity. Two adjacent messages might have the same date or the same subject. The closest candidate for a suitable identity is Message-ID but this is generated by the sender, who might not include it or might reuse IDs. If we’re relying on the POP3 server to add them or modify duplicates provided by a sender, we’re back to relying on optional features.
You could use the TOP response and hash the entire header? This could work except message headers can change. I first saw this when experimenting with a mail server and observed that if I connected to a mailbox using IMAP, it would leave IMAP’s version of a unique identity in the header which wasn’t there before. As well as that, anti-spam systems might re-examine a mailbox’s contents and update the anti-spam or anti-virus headers. Any of these changes would look like a new message.
(As well as all that, TOP is itself an optional command, just like UIDL.)
You could download the entire message again and ignore it if you already have it? This would be ultimate fall-back. While I’ve seen headers change, the message body seems to be immutable. This is still an unreasonable situation. We’re downloading the whole message again, just because the server chose not to implement a simple directory-listing command.
Am I certain that the message body is immutable? No, not at all. If someone commented that mail server XYZ updates messages in the form of a MIME attachment, I wouldn’t be at all surprised.
Update – A digression on the Message-ID header
(Added 28/Jan/2021) I am grateful to commenter “theamk” on Hacker News, who responded to me when I shared this post. To my dismissal of Message-ID as a means of de-duplication, they noted that RFC standards require that Message-IDs must be generated as unique.
I have experienced senders who have broken the protocol, sending many different messages with the same Message-ID. I do not argue these senders were in the wrong but that the POP3 server is not in a reasonable position to correct the situation.
If the server actively corrected the situation and replaced the reused Message-ID header with its own unique value, the message would not be a faithful reproduction of the message as sent any more and further damage any scope for auditing.
If the server discarded or rejected the message with a reused Message-ID, it would open up means for an attacker to predict the Message-ID a legitimate sender is going to use and send a message with that ID first, causing the legitimate sender’s message to be lost. There’s nothing stopping a sender from using someone else’s Message-ID pattern. (Maybe senders should use only unpredictable strings, but wishing it so won’t make it happen.)
This is also to say nothing of the situation when the messages served up don’t have any Message-ID, which I’ve seen happen with messages exchanged within the local server only. (IE. Not routed over the public internet’s mail servers.) None of the small number of services inside the box from the original composer to the POP3 delivery agent supplied a Message-ID when it was missing, so the message turned up with the basic To/From/Subject/etc headers and a Received header, but no Message-ID.
Acceptance?
Because the alternatives are so unreasonable, I consider UIDL a requirement for handling POP3. Servers that don’t implement UIDL are bad servers. Clients that can work without UIDL are unreliable.
Still not convinced? Please leave a comment where you saw this piece posted.
“I’ve seen the future! I’ve seen the future! I’ve seen the future and it’s now!”
IMAP does it wrong.
The other popular mail-reading protocol is IMAP. In contrast to POP3’s download-and-delete model, IMAP’s model is that messages to stay on the server and are only downloaded when the client wishes to read it. This model enables mail readers on low-storage devices such as smartphones.
With IMAP, the IDs are restricted to numeric values and always go upwards, in contrast to the free-for-all “any printable ascii except spaces” allowed by POP3. While this may be nice for the client, by requiring a single source of incrementing ID numbers, it complicates matters for anyone wishing to implement an IMAP server using a distributed database as a back-end.
But the worse thing about IMAP’s message identity system is that the standard permits the server to discard any IDs it has assigned by updating a mailbox’s UIDVALIDITY property. If this value ever changes, it is a signal to the client that any unique IDs it may have remembered are no longer valid.
A client needs a reliable way to identify messages between connections to recover from an unknown state. It does not need for servers to have a license to be unreliable.
If a mail server that implements IMAP wants any respect from me, it would document that its UIDVALIDITY value is fixed and will never change and that the unique-ids it generates are reliable.
POP3 does it wrong too.
If I’m going to criticize IMAP for flaws in its unique ID system, I should address flaws in POP3’s system too, having spent most of this article praising it.
Quoth RFC 1939: “The server should never reuse an unique-id in a given maildrop,” (good) “for as long as the entity using the unique-id exists.” (no!)
Consider that worst case scenario. The client flags a single message to be deleted and finally issues a QUIT command to complete the translation. The server successfully processes the request but the response to the client is lost. As far as the server is concerned, the message is gone and there’s no problem, but as far as the client knows, the continued existence of that message is unknown.
Now consider a new message arrives on the mail server and because the RFC says it can, it assigns the same unique ID to this new message as the one that was just deleted. The client eventually reconnects and requests the list of unique IDs and finds the ID of the message it wanted to delete is still there. It doesn’t know the server used its right to reuse unique IDs and that this is actually a new message!
Now, I’ve never seen a mail server actually reuse a unique ID. The clever people who have developed mail servers in the real world seem to understand that reusing IDs is not something you ever want to do, even if the RFC says you can.
RFC 1939 also says, “this specification is intended to permit unique-ids to be calculated as a hash of the message. Clients should be able to handle a situation where two identical copies of a message in a maildrop have the same unique-id.”
Unique IDs don’t have be unique? Ugh.
This allowance only applies to identical messages. In reality, messages are never identical. After bouncing around the internet and going through various anti-spam and anti-virus servers, messages do accumulate a frightening number of Received: headers left behind from each intermediate hand-over. Each one with a time-stamp and its own ID number. Any one of these is enough to produce a distinct hash.
There’s a movement among remainers to get Ode To Joy to number one in the charts for Brexit day. We know, even if successful, it won’t change anything. Johnson isn’t going to cancel Brexit because of this. (He totally should, nonetheless.)
So why is there a similar movement among Brexiters to get their own song “Seventeen Million F–k Offs” to number one instead?
They’ve won. Despite only getting about half the votes in 2016, Maximum Brexit is going ahead. But once the party’s over, they still won’t be happy and deep down, they know. Victory feels so much like defeat.
And among all this false revelry, there’s a bunch of smug Remoaners trying to spoil the party. “They were supposed to have been defeated by now. Our victory must be total!”
And so they get a song that’s literally telling remainers to f–k off to number one. Maybe that small victory on top of all their other victories will finally make them happy?
So it’s finally Brexit Day. After many repeated delays and false starts, Brexit is here!
What’s that? You’ve just posted that the sun is shining and the birds are singing?
Oh I see, you’re happy that we’re not in some apocalyptic doom. Brexit was supposed to cause the earth to split open and a literal hell on earth was about to envelop us all for a thousand years! Turns out, no, that didn’t happen.
So what? Is that it? Is that all you’ve got to show for three years of dragging our country through the grinder?
Sic transit gloria mundi
“No age is too early nor too late for the health of the soul.”
— Epicurus, Letter to Menoeceus.
First of all, this isn’t Brexit.
We’re in the transitional period laid out by the Withdrawal Agreement. Until we leave this period, everything pretty much stays the same.
What has happened, from a practical point of view…
🏰 We’ve lost our seat on The European Council.
🗳️ We’ve lost our representatives on The European Parliament.
🐘 We can no longer unilaterally revoke the Article 50 notification.
So not only have we lost our leverage, we get to follow all of the rules of the European Union without having any say in them. If that seems like something worth celebrating, I’m not going to stop you. (I might think you’re a bit weird though.)
Let’s wait until (if) we’re out of the transitional period and are actually out of the EU, not just from a legal point of view. We’ll talk then.
Fear itself
“The just man is most free from disturbance, while the unjust is full of the utmost disturbance.”
— Epicurus, Sovereign Maxims #17.
Oh yes. There was going to be World War III if we voted for Brexit.
(Let’s leave Iran to one side for a moment.)
David Cameron is normally credited with saying this, but he never actually said that. (And I hate the idea of defending the idiot that got us into this stupid mess in the first place.) The idea stems from a speech he made in May 2016, where he spoke of the EU bringing peace to a warring continent.
Look at France and Germany’s history: War, War, More War, EEC/EU, Not War.
That’s a long way from claiming Brexit would cause World War III, so I hope you’ll excuse me if I take such notions with a significant amount of salt.
Even if that were true, the remain campaign lost (as Brexiters are so eager to remind me) so it doesn’t really matter what they said. Let’s talk about what the winning party promised and if Brexit is living up to that promise.
Why are we here?
“Reality is that which, when you stop believing in it, doesn’t go away.”
— Philip K. Dick.
Please don’t go around proclaiming lack-of-hell as if it were a benefit of Brexit. It really isn’t. Would we be facing a hellish doom if we had stayed in the European Union? No. The sun would still be shining and birds would still be singing.
To be a benefit of Brexit, it would have to be:
😃Something so good or desirable that it realistically outweighs the drawbacks.
🥺Something that we can’t have as members of the EU.
😝Something we can have outside of the EU.
Let’s put on our reality hats and please tell me what realistic benefits of Brexit are. Tell me that instead of glib comments about the weather.
What are the benefits of Brexit?
🦄 The ones that are beneficial.
🦄 The ones that don’t require that you first “believe in Brexit”.
🦄 The ones that we can’t have inside the EU.
🦄 The ones that stand up to the cold, hard, unforgiving reality.
When standing for election a politician will usually run as the member of a political party. The quid-pro-quo of this arrangement is that the party will collectively campaign for all their candidates. In return, the party will expect the successfully elected MPs to support their leader’s policies and support that leader in any votes-of-confidence.
Occasionally, an MP will decide to leave their party. This is often called “Crossing The Floor” for rather archaic reasons. Sometimes that MP becomes independent, sometimes they join other parties. Most famously, wartime leader Winston Churchill left the Conservative Party in 1904 to join the Liberal Party, only to return in 1924.
When an MP does this, it will often be met with calls for that MP to put themselves up for re-election by their voters in a by-election. Normally, this only happens when an MP dies, resigns, or maybe appointed to the House of Lords.
“The voters elected a Conservative to be their MP, not a Liberal! You should let the voters have their say if they want to keep you as their MP.”
I disagree.
We Elect People, not Parties…
“These are things utterly unknown to the laws of this land, and which arise from a fundamental mistake of the whole order and tenor of our constitution.” – Edmund Burke, 1774.
In the Westminster system, we vote for people. We don’t vote for parties.
It might not seem that way, given how much of campaigns at election time are for the various political parties rather than for the individual candidates. Nonetheless, at the completion of an election, the end result is that a person will have been elected to serve as MP. That MP is an individual who has been empowered by the voters to be their representative in Parliament for the term they were elected for.
If it were the case that we do actually elect political parties, we shouldn’t need a by-election when an MP needs to be replaced. The party could simply select a replacement internally without needing to involve the electorate. This shouldn’t be a problem, after all, the voters elected that party for a five year term, not the MP.
In fact, why not just give parties the power to dismiss and replace their MPs at will? If a party leader doesn’t like how an MP is acting, they can tell that MP “You’re Fired” and appoint a replacement. If that idea seems like an undemocratic stitch-up, I quite agree. (This is also a reason why I prefer AV over PR, but that’s another article.)
Currently, the only thing that can force a by-election for an MP who doesn’t want one is if that MP is convicted of a crime and enough constituents demand a “recall”. Is leaving a political party really up there with being a convicted criminal?
… and that’s a good thing.
“It has long (perhaps throughout the entire duration of British freedom) been a common form of speech, that if a good despot could be insured, despotic monarchy would be the best form of government. I look upon this as a radical and most pernicious misconception of what good government is, which, until it can be got rid of, will fatally vitiate all our speculations on government.” – John Stuart Mill, 1861.
It’s a good thing that we elect people instead of the parties. MPs are elected and empowered to make the difficult decisions and be held responsible for them. That is the essence of Parliamentary Democracy. By so empowering MPs, they become one of the checks-and-balances. A party leader can’t overrule the objections of other MPs, precisely because dissenting MPs are so empowered. Those MPs know they will be held accountable to their voters, not to anyone else.
Recently (as I write this in 2019), an unelected Prime Minister wished to bulldoze his Brexit policy through Parliament. MPs did the job they were elected to do and said “No”. That Prime Minister saw first-hand what happens when you try to throw your weight around like King Charles. He found himself restrained by the very checks and balances that exist in functioning democracies.
(I stand by “unelected”. As I write this, Boris Johnson has neither been elected by the voters at large nor by their representatives in Parliament. His predecessor, Theresa May, became Prime Minister while her party held a comfortable majority in Parliament and later won a vote of confidence of MPs. I would have liked to have seen a formal vote of MPs to elect a new Prime Minister when the position becomes vacant, but I accept in her case that would have been a formality.)
If we move the focus of that empowerment to the parties, those MPs will find themselves accountable to their party, not to the voters. We’ll have reduced the role of an MP to mere party functionary. If an MP’s first loyalty is to their party rather than their voters, we’ll have just populated Parliament with a bunch of toadying yes-men. We wouldn’t even need 600-odd MPs if all you need are party robots, you could get by with around 50.
What would we gain if MPs did have to contest a by-election for leaving their party. There’s a general election every five years when they will have to be re-elected anyway. When that re-election comes around they will find themselves facing someone selected by their old party. Party leaders are already rather powerful, they could do with feeling not quite so powerful once in a while.
This may very well end up being my last Blogger based post as I’m slowly adopting (self-hosted) WordPress as a publishing platform. I have a set of websites running on a commodity cPanel-based shared host, with a view to moving to a dedicated VM in due course. While setting things up and playing about with WordPress, I kept tripping over an obstacle that just kept getting in the way of doing what I wanted to do.
MySQL was that obstacle.
Dear WordPress, please have an option to use a file-based database (such as SQLite) instead.
Why would you want to do such a thing?
First of all, simmer down, MySQL is a perfectly good database. It does the job it was designed to do very well. My problem is that MySQL exists on a server separated from the rest of WordPress.
Think about what makes up a single installation of WordPress. You’ve got a bunch of PHP files, the themes, the plugins, the images and media I’ve uploaded. All of these are in a single folder on the web server. I could ZIP the folder up, UNZIP it later, push the folder into GIT version control, all in the certainty I’m got everything. Except…
Some of my website is not in that folder. It’s on the database. I can’t just ZIP the website up because an essential component is off in another realm. That folder does not contain everything and I now need to keep database backups alongside the folder backups. Grrr…
I was considering adding a plug-in to one of my live websites. Because people were using it, I didn’t want any down-time. Accordingly, I made a copy of the website folder and also made a copy of the database. The new copy then had to be reconfigured to point to the new database and only then could I play about with whatever plug-in or theme I wanted to add. Whatever clones and copies I make, the database is always at arms length and I need to be very careful that the PHP is always linked with the right database and that I’ve not got any cross-overs.
If the data on the database were on a file in that folder, there would be nothing external to keep track of. Copy the whole folder and job done! Taking backups would mean zipping up the folder and everything is there without worrying about keeping the two parts in sync.
Wouldn’t that make the site inefficient?
Maybe, but I plan to use very aggressive caching. There’s a plug-in where the site contents end up as static files and the code accessing the database only has to run when I log into the admin panel or the cache engine decides its time to update itself.
I can imagine this might not work so well for a site where changes happen very frequently. Maybe, but I suspect those are in a minority. For them, MySQL would probably still be an option, but it feels like for such a website, WordPress itself probably isn’t the right tool for the job.
Incidentally, I don’t plan to support commenting because I do find moderation a bit of grind. Because I’m just one person and I have other stuff going on in my life, there would be a very long delay between someone posting a comment and my approving it. The better quality discussion tends to happen on sites like Hacker News where there is an active moderation team that I can’t even hope to match.
Why not use a WP site manager tool?
Since WordPress are probably not on the verge of releasing an update with SQLite option, I will probably end up doing exactly that.
I already have tools for managing folders and zip files. Cloning a website could be a simple folder-copy operation were it not for the separate database. Tools that know about the database are very nice but it all feels like the wrong answer to the question. We’re all in a world where things are in the state they are in and we have to stoically make it work. Site managers fix the symptoms but they don’t address the underlying issue.
You may have read about the news that many voters have been turned away from voting over here in the UK. I’d like to share a little analogy abut what happened, why it is so controversial and how it goes beyond voter suppression.
Imagine you’re in California, and there’s a political party whose only policy is to deport anyone born on the other side of the Mississippi. (That and being paid donations through PayPal that are all conveniently under the ÂŁ500 limit allowed for anonymous donations. Totally not a big donation from one person that would normally need to be declared.)
A group of people living in California but born in the east are rightfully worried that this party might gain power. They are entitled to vote as citizens so they make sure they are correctly registered.
Voting day arrives and the process goes smoothly for Californians. Voters from out-of-state, however, find they can’t vote.
Some are told they should have filled out a special form but were never told that was needed. Some did fill in the form but it got lost. Some did everything right but the people running the vote didn’t know this group was entitled to vote and just struck them off the register.
Election day passed and a significant number of people didn’t get to vote. Those people probably wouldn’t have voted for the deportation party so the situation may might end up unfairly inflating their share.
(Sounds ridiculous? This is basically what’s going on here. Only difference is our Mississippi River is a bit wider and we call ours The English Channel and The North Sea.)
Just to round out the story, on the day the news of voter suppression breaks, a senior politician chooses this day (quite coincidentally) to announce their resignation. News reports quickly start paying attention to this new distraction and soon forget about the dodgy funding and voter suppression.
I had an idea for a nerdy tattoo a few years ago. It would represent myself as a software engineer and I thought it was quite clever. I seriously considered having it done but decided against it in the end, despite its cleverness.
Ink’d
This is my idea, the “end comment” symbol in many programming languages:
*/
In C, and other languages that can trace their lineage to C, comments start with a /* and end with a */. Anything inside is ignored by the language, allowing the programmer to describe what’s going on. This is tremendously useful when reading other people’s code or even your own code from the past.
/* This is a comment. */ Code();
/* This is another comment. */ MoreCode();
Another way of looking at it is that these /* and */ symbols mark the change of state between comments and code. /* says “After this is comment” while */ says “After this is code.”
Or to put it another way, */ means “Enough talk, time for action.”
(This is where you exclaim to yourself how clever I am to have thought of that.)
I didn’t have the tattoo done in the end. Describing what it meant would have taken too much explanation. Even if a fellow programmer recognized the symbol, they would probably first think it looked like I’ve been “commented out”, as they wonder if I had the /* on the other side.
Also, rotated a little, it looks a bit like a squinting cyclops.
