From CombinedOutput to line-by-line deployment logs

After not showing a demo for a while, I finally recorded one again…

At this point I already had deployment logs stored in the database, and I wanted the frontend to start showing them while docker pull nginx:latest was still running. I tried to push things in that direction by returning deployment_id from the deploy endpoint, adding the log polling UI in the frontend, and appending Docker output into the database.

At first I thought that polling from the frontend would be enough… but the result was still not truly streaming per line. The reason is that I was still using cmd.CombinedOutput(), which waits for the whole command to finish before returning the output. So even though the frontend kept polling, the backend was basically still holding the logs until the process was done. You can see in the video below how all the logs appear at once after the process finishes, instead of showing up gradually.

That was a useful mistake honestly, because it made the actual bottleneck much clearer. The problem was not the polling itself… the problem was that the backend was not producing log lines incrementally yet.

So I switched the approach to use cmd.StdoutPipe() and bufio.NewScanner(). Instead of waiting for the whole docker pull output, I started reading it line by line and immediately saving each line with AppendLog(...). The difference is pretty clear in the video below, where each log line now appears as it happens.

That small change made a huge difference. Once the backend started inserting logs per line while the command was still running, the frontend polling finally felt real-time. I also merged stderr into stdout so everything could flow through the same stream and be captured in one place.

The main insight here is pretty simple… frontend polling alone does not magically create streaming. If the backend still buffers everything first, the UI will always feel delayed. Real-time logs only started to work once I changed how the process output was being read.