Deepanshu Mishra

I have been exploring infra and cloud recently and man it's so amazing — it's one thing to write code for a feature or fix a bug, but to see and analyze how infra works and how the systems the whole world relies on are architected. Unlike anyone who suddenly one morning got very intrigued by effect, cloud, distributed systems and peak designing of systems, I also picked up the holy grail DDIA. I will not be telling the full form because c'mon seriously?

So while I was going through the design stuff (Not UI Design ffs) I explored load balancers, containers, Jenkins, etc. Then I was exploring Redis and saw a data structure called RedisList. At first I thought it was yet another list but no — it was the backbone of Redis queues. Now I have used Redis as a caching layer in a lot of my applications but this use case was new. All this time I thought only RabbitMQ or BullMQ were the messaging queues (yeah I'm a noob, thanks) and the moment I knew we could use Redis for this, I knew I had to implement it somehow.

How we got the idea for a job queue?

Now the first thing I thought of building was like a messaging system: two people chat, messages are stored in a Redis queue and a worker pulls them one by one. This was a vague idea that came to my mind but it sucked because it had a bottleneck — a single worker processes messages one by one in FIFO order, so one slow message blocks every other conversation with zero parallelism — so I moved on. The job queue came to my mind but I didn't want to implement a full-fledged job queue because I know I would fuck up due to my lack of knowledge in design and infra. So I thought of building a job queue but the jobs were mock — like literally. So it would still be a queue with the only difference being the jobs would be fake and to simulate the processing I could add a random wait time (which I called the processing time). Genius I know.

So things were clear in my mind and the idea was locked, but I know from experience that the pro senior engineers don't code first — they design the architecture first. So I hopped on to excalidraw and let me tell you I suck at designing and you know making visual stuff. So I had the picture in my mind but I didn't know the correct way to depict it in my architectural diagram, so I did whatever I could and you can see that it's a rookie's work what I have done in my architecture diagram. But we only go up from here.

My New Love: Effect and Go

So I have been keeping an eye on Effect for quite some time and I have tried Go several times in the past but didn't quite find it to my taste. But my favorite programmers I've seen were going so hard on Effect, I also thought to give it a try. I watched the Effect talks, workshops, got some idea about it and to see if it's something I would love, I built an Apple Music MCP server in Effect and that was it. I was struggling, it was painful, but I loved that pain. For the first time I enjoyed writing TypeScript without hating it (noob once again). And for Go, since I was going deep into cloud and infra, most of the learning materials or codebases I was coming across were Go, so there was no choice but to use it. So we also started learning Go.

I would say I am neither good at Go nor at Effect at the moment, but we keep punching up and see where it will take us. So coming back to our job queue project — I had two choices: build it in Effect or learn some Go by building it, and you might have guessed it — I chose Go. And trust me, these two projects are the only two projects in which I hand-wrote a good amount of code after a long time. We ship and rely on agents and it's cool but writing code by hand was so satisfying. So we chose Go and went with it.

Let's Talk about some code part

The code is open source on github you can go and check my dirty code. Now the idea was simple: we would need users who would create jobs and that job would just be a map which would have a string as the key but the value could be anything because it's mock and we want users to add anything they want. So a simple struct for a job would somewhat look like this:

type Job struct {
  JobID     string
  JobName   string
  JobStatus Status
  UserId    string
  Payload   map[string]any
}

Every user can create multiple jobs so userId is a must. So after that it was just some auth endpoints, auth middleware, db connection, redis connection, etc. The only major critical part was the JobService and the Worker.

The user creates jobs. The moment a job is created, it is pushed to the Redis queue with LPUSH. A worker which starts when the server starts as a goroutine picks up a job via BRPOP (which removes it from the Redis queue immediately), we simulate a random processing time during which the job status is set to RUNNING, and once it's finished we update it to FINISHED in the database. BRPOP is a command which blocks when the queue is empty and as soon as we have data it wakes up and does the work. So this was the basic idea behind the system.

What's Next

Now, we are done with the simple mock queue. We will now move to a real queue using the tech that's actually built for the job. So the next version would be a job queue built with RabbitMQ in Go obviously, but we would have some actual jobs as well with their proper integrations. I haven't thought about it but once I am done with its planning and architecture, I will let you people know.

That was it for the first blog and I will see you guys in the next one. So a very good bye and peace.

Deepanshu 🥀 dipxsy.app