The specifics: 90% keyword recall at 33% lower latency than my first working version, around 99% on FaceForensics++, and a clustering pipeline that took roughly 2,000 weekly fraud alerts down to about 22 cases a human could actually review.
This is what happened in between, including the four weeks I wasted and the bug I ignored that turned into my research direction.
How long does it actually take?
Eight months for me, building alongside a full-time job. The variable is not intelligence or maths background. It is whether you spend your hours building systems or completing courses. I did both, and only one of them moved me forward.
The four weeks I threw away
I started the way I think most engineers start. I enrolled in a deep learning specialisation and began at lesson one. Linear algebra. Gradient descent. Backpropagation worked through by hand.
Four weeks in, I could derive a loss function on paper and I had shipped nothing.
What actually broke the deadlock was small and slightly embarrassing. I abandoned the course mid-module and wrote a script that took a folder of PDFs, split them into chunks, embedded them, and let me ask questions over them. Around 120 lines. It worked badly. Roughly half my queries came back with confidently wrong answers pulled from the wrong section of the wrong document.
Debugging that taught me more about embeddings in three days than four weeks of lectures had. When your chunks are 2,000 tokens long and the user's question is eight words, cosine similarity does not care that you understand matrix multiplication. The chunk contains forty other topics and the embedding is an average of all of them. You can read that sentence in a course and nod. You understand it permanently the day it wrecks your own output.
If I were starting again: write the broken version in week one. Theory lands differently when you have a specific failure you need it to explain.
The project that actually taught me the job
Around month three I made a decision that mattered more than any individual thing I learned: build one real system properly instead of five toy ones.
That became my enterprise RAG pipeline, doing question answering over financial documents, built to production standards rather than notebook standards. Here is what broke, roughly in the order it broke.
Dense retrieval alone quietly fails
Pure vector search kept missing documents that contained the exact term the user had typed. Someone asks about a specific instrument or entity name, and the retriever returns three documents that are about the same general topic and none that actually mention it.
Semantic similarity and correctness are not the same thing, and the gap is worst precisely where it hurts most — rare terms, proper nouns, specific identifiers. Which in financial documents is most of what people search for.
The fix was hybrid retrieval: run BM25 keyword search alongside dense vector search and combine the two ranked lists with Reciprocal Rank Fusion. RRF is useful here because it needs no score calibration between the two systems. It only cares about rank position, so you are not trying to make a BM25 score and a cosine score comparable, which they are not.
Keyword recall went to 90%. That single change did more for answer quality than any model swap I tried.
The reranker helped, and also did something strange
Next I added a cross-encoder reranker — ms-marco-MiniLM — to reorder the top candidates before they went to the model. Answer quality improved clearly. Latency got worse, which I expected.
What I did not expect: the reranker started promoting rare documents. Specifically, documents with very few semantically similar neighbours in the corpus were getting scored higher than their actual relevance justified. A one-off memo would outrank a well-established document that answered the question better.
I noticed it, wrote a note in the repo, and moved on. I had a deadline and it was not breaking anything visibly.
That note is now the seed of the research I am writing up. Because the implication, once I sat with it a year later, is uncomfortable. If rerankers systematically over-score outlier documents, then a document injected into a corpus at inference time — low frequency by definition, few neighbours by definition — is disproportionately likely to be surfaced and reproduced. No attacker required. No prompt injection. Just the system working as designed, on documents it was never meant to reproduce.
I am now formalising that as retrieval-induced memorisation, testing
whether privacy protections applied at the model layer bound it at all. My working conclusion is that they do not, because the exposure happens at retrieval time, outside the boundary those protections cover.
The point for anyone transitioning: the weird thing your system does that you cannot explain is not noise to be tolerated. Write it down. Some of those notes are just bugs. One of mine turned into a paper.
Latency, which was not an ML problem at all
My first genuinely working version was too slow to use. Not catastrophically, just badly enough that nobody would tolerate it in a real workflow.
Nothing I did to fix this was machine learning. I made ingestion async so documents processed in parallel rather than sequentially. I cached embeddings for repeated queries. I batched the reranking calls instead of firing them one at a time. Standard backend engineering, the kind I had been doing for years.
That got me a 33% reduction and it took a fraction of the time the retrieval work had.
This is the part I would most want a transitioning developer to hear. A large share of AI engineering is systems work wearing a different hat. Async patterns, caching strategy, batching, connection pooling, timeout handling, graceful degradation when a model call fails. You already know how to do this. The people coming from a pure ML background largely do not, and it shows the moment their notebook has to become a service.
I had no idea whether any of it was good
For the first two months of that project I evaluated changes by feel. I would tweak the chunking strategy, run five queries I had memorised, decide it seemed better, and keep it.
That is worthless, and worse than worthless, because it feels like progress.
I added LangSmith tracing so I could see what was actually retrieved for each query, then wrote a fixed evaluation set — questions I knew the correct source document for — and measured recall and answer rate properly. Answer rate settled around 95%.
The moment I had numbers, my iteration speed roughly doubled. Not because the numbers were impressive, but because I stopped having arguments with myself. A change either moved the metric or it did not, and I could revert in thirty seconds instead of agonising.
Setting up that evaluation harness was the single highest-value hour of the entire eight months. It was not learning a new framework. It was measuring something I had already built.
Where the frontend years paid off unexpectedly
In months seven and eight I built DERIV SAR Copilot, a financial anomaly detection tool. The modelling side was DBSCAN clustering over transaction patterns to isolate genuinely unusual behaviour, with GPT-4 generating draft suspicious activity report narratives for the cases that surfaced.
The clustering took roughly 2,000 weekly alerts down to about 22 high-confidence cases. On paper, that is the product.
In practice it was not, and this is the part I did not anticipate. A list of 22 flagged cases with no explanation is not usable by a compliance analyst. They cannot action something they cannot interrogate. What made it work was the investigation interface — the analyst can see why a case was flagged, drill into the underlying transactions, read the generated narrative, and accept, edit, or reject it.
I built that interface in an afternoon, because I had spent years doing exactly that kind of work.
Most ML engineers I have worked with could not have built it. Most frontend engineers could not have built the clustering. During a transition it is easy to feel like you are half of two things. The overlap is not a deficit you are working off. For a lot of real AI products, the overlap is the whole job.
Alongside this I finished my MTech thesis on multimodal deepfake detection — CNN models across image, audio and video, around 99% on FaceForensics++, with Grad-CAM for explainability and MLflow tracking every run. Useful work, and I am proud of it. But I will be honest that it taught me less about being an AI engineer than the RAG pipeline did. What it did teach me was rigour about evaluation, and that fed straight back into everything else I built.
Three things I would do differently
I optimised for understanding before shipping. Four weeks of theory produced nothing I could point at. Three days of debugging a broken retriever produced durable understanding of the same concepts. The order was backwards, and the order is the whole thing.
I built in private for five months. Nobody knew what I was working on. When I finally started publishing repos with READMEs that explained what broke and why — not just what the project did — opportunities started arriving without me asking. The work had not changed. The visibility had.
I treated evaluation as a final step. It should have been the second step, right after "it runs." Every week I spent tuning without measurement was a week of guessing dressed up as engineering.
If you have one month, do this
The compressed version of what worked, minus the parts that did not.
Week 1. Build a retrieval script over documents you actually care about. Chunk, embed, query. It will be bad. That is the point.
Week 2. Fix the retrieval. Add keyword search next to your vector search and combine them. Watch which queries flip from wrong to right.
Week 3. Write twenty questions where you know the correct source document. Measure recall. Get a real number.
Week 4. Deploy it somewhere public and write 500 words on what broke.
You will understand more about AI engineering after those four weeks than after four months of structured courses. I say that with some confidence, because I did it in the wrong order first and I have both data points.
The part that is harder to write about
There was a stretch around months four and five where I felt fraudulent in both directions. Not enough of a researcher for the AI people, not shipping enough product for the engineering people. I seriously considered going back to pure frontend work twice, and the second time I got as far as updating my CV.
What got me through it was not motivation or discipline. It was having exactly one specific broken thing to fix each week. Not "learn AI." Fix the retriever that keeps missing entity names. That is a task with an end.
If you are in that middle stretch right now — far enough in to see the size of what you do not know, not far enough to feel competent — that is not evidence you chose wrong. That is just what month four feels like, and it does end.
I mentor developers making this exact transition, largely because I remember how much unnecessary wandering I did in months one and two. If that is where you are, you can find me here on MentorCruise.
