This year I re-read G.K. Chesterton's The Man Who Was Thursday. I think I have a new favorite line:
"… there are no words to express the abyss between isolation and having one ally. It may be conceded to the mathematicians that four is twice two. But two is not twice one; two is two thousand times one."
I'm holding onto this a lot this year. Friends are so important to me.
Listened to The Silver Chair again this week.
"If only the door was open again!" said Scrubb as they went on, and Jill nodded. For at the top of the shrubbery was a high stone wall and in that wall a door by which you could get out onto open moor. This door was nearly always locked. But there had been times when people had found it open; or perhaps there had been only one time. But you may imagine how the memory of even one time kept people hoping, and trying the door; for if it should happen to be unlocked it would be a splendid way of getting outside the school grounds without being seen.
I love this picture of the nature of desperation and hope. It resonated with me this week in my life.
Jill and Eustace, now both very hot and very grubby from goingBefore we even get to Narnia, Lewis is already setting the stage for the themes of defeat and eucatastrophe - themes that echo through each chapter and crescendo in the climax.
along bent almost double under the laurels, panted up to the wall. And
there was the door, shut as usual.
"It's sure to be no good," said Eustace with his hand on the handle; and then, "O-o-oh. By Gum!!" For the handle turned and the door opened.
Last week I accepted an offer to work at Red Hat in the InstructLab engineering team. I can't wait to join this company again.
I use Python's Paramiko to automate many things at work. Recently I wrote a SFTP client library. It it has a helper method like this:
def connect(username):
password = getpass.getpass()
transport = paramiko.Transport((HOST, PORT))
transport.connect(username=username, password=password)
return paramiko.SFTPClient.from_transport(transport)
Callers can do "sftp = connect('kdreyer@example.com')", then do SFTP operations like uploading files, listing directories, etc.
Recently I hit a problem where the SSH server ("HOST") was not available over the network. A firewall was silently dropping TCP packets without rejecting them (RST). As a result, when we called this connect() method in a CI job, we had to wait a long time for this code to time out.
As in all good networking stories, this did not happen every time, only sometimes, and especially at important times when we really needed the connection to work. This CI job was already large and long, the perfect opportunity for losing a human operator's focus and costing even more time when it dies slowly like this.
I began researching how to make Paramiko time out faster.
To begin, I set up another network environment that drops SSH packets in a similar way so that I can reproduce it on my laptop. I found Python was hanging in socket.connect().
It turns out that I can initialize Paramiko's Transport() class two ways. The first way is to pass in my own socket object. This is powerful but too advanced for most Paramiko users, so Paramiko also allows me to initialize with a hostname + port tuple (as I've done above). Transport operates on sockets, so the constructor will create the socket and call socket.connect() for me. Critically, it calls connect() with no timeout. There is no way to pass in a timeout to the Transport() constructor.
At this point I began looking at the broader Paramiko API. It turns out there is a much better way to initialize an SFTP client. Instead of creating my own paramiko.Transport, I can start with paramiko.SSHClient(). This method has several important features (like pre-selecting allowed authentication mechanisms), but the most important is the "timeout" argument! Paramiko sets this timeout on the socket object so we get TimeoutError sooner. I wrote a patch and the new method connect() method looks like this:
def connect(username):
password = getpass.getpass()
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect(HOST, PORT, username, password, timeout=3.0,
allow_agent=False, look_for_keys=False)
return client.open_sftp()
In terms of solving the bigger issue, usually I'd reach for the excellent backoff module in cases where infrastructure is unreliable, but this particular network failure is not one where we can simply catch Paramiko's TimeoutError and retry - unfortunately today it requires human involvement to fix. Eventually I will move the client to a more stable network, but that will require some broader architecture work. This Paramiko fix incrementally improves our operations in the meantime.
(Cross-posting here from koji-devel)
This week I created an Introduction to Koji video. Koji is the build system we use for the Fedora Project and Red Hat products.
This video answers the following questions:
I met someone recently who just graduated college and was on the job search. I linked him to my favorite podcast episode on this topic. After listening to the episode, this person then asked me:
"What kind of person would succeed as a company such as Red Hat?"
I love this question. I asked around, and one colleague gave the following answer:
"I'd say someone that is passionate about enterprise technology and loves to learn in a fast moving environment."
I love this answer, and I'm going to break it into pieces to explain why this applies.
This means that you understand more than just code. It means you understand the business (who our customers are, and how do we make customers happy, and why would people pay us money). It also means that you care about your work: you understand who your users are, and you think about how to improve the experience of those users consuming the work that you produce.
What's Enterprise technology? It's technology that is stable, secure, and used in a wide variety of places. You will be a good fit if:
In Enterprise software, there is a natural pull to move slowly and stagnate. However, open-source is gigantic and moves incredibly fast. (In fact, the core of Red Hat's value proposition is that Red Hat drives innovation in "upstream" projects and then distills that down into stable products that customers can trust.)
I've been at Red Hat for six years, and the product and technology landscape has changed a lot. During that time, Red Hat acquired Ansible and we've integrated so many things with that now. Ceph used to just integrate with OpenStack, and now we've integrated Ceph with NFS, iSCSI and OpenShift as well. Every year I work on different things and find new ways to integrate new and different projects together.
When I joined Red Hat in 2014, I did not know Python or C++. One of the best things about accepting the job was that I knew I would be surrounding myself with people who knew these languages incredibly well. I picked up so much by osmosis, working alongside some senior developers on the team who could patiently explain things to me.
Technology aside, a few years ago I found the Manager Tools podcast and learned so much about how to communicate effectively (really important at Red Hat were so much work is remote work). I also found a coach and learned how to get really clear on what I want, how to set boundaries effectively, how to finish projects on time, and how to handle personal challenges at work.
"Being a passionate learner" does not mean the often-ridiculed caricature of the programmer who spends every spare night and weekend away from family slaving away on open-source for free. It means someone who spends their time effectively, someone who knows how to balance the critical fire-fighting of the moment with the time to make long-term investments (whether that's technical research or organizational relationship-building). In fact, one of the things I'm passionate about is quitting work on time (or even early) and encouraging folks on my teams to do the same. Quitting on time is how we keep coming back tomorrow.
