Kiran Kumar
Model Repositories, Deployment, and Infrastructure
This is part two of a pair of posts adapted from a talk on building a cost prediction platform for a two sided marketplace. Part one covered the model itself: TabTransformer, a distributional output, and reusing one base model across teams through fine tuning and transfer learning. None of that pays off if the model cannot be trained reliably, tested before it ships, deployed safely, and monitored once it is live. This post covers that side: repository structure, training automation, backtesting, promoting a model, data quality checks, and alerting.
Repository strategy: monorepo versus multi repo
The team started with a multi repo setup, a separate repository per model or per team. It reads as the more modular choice at first, each team owns its own code and can move independently, but in practice it made two things hard: keeping shared logic (feature engineering, the base model code, common testing utilities) consistent across every copy, and enforcing a baseline level of test coverage. Test coverage in particular had quietly become a real concern, since nothing forced a new repo to inherit the testing discipline of an older one.
The team migrated to a monorepo, a single repository holding every model and the shared tooling around them. The migration itself was not free, existing pipelines and import paths had to move, but it improved model management outcomes noticeably: one place to enforce continuous integration checks, one place to update a shared utility and have every model pick up the change, and one place for a reviewer to see the full picture of what a change touches instead of hoping a parallel change landed correctly in a different repository.
Automating model training
Training and deployment were wrapped behind a small internal command
line library so a data scientist did not need to hand write
infrastructure code to get a model trained or served. Scheduling a
recurring training job looked something like libname schedule
training '00 * * * *', a familiar cron style schedule, and
standing up a serving endpoint for a trained model looked like
libname deploy endpoint.
The goal of this layer was to abstract away the infrastructure complexity, provisioning compute, containerizing a model, wiring up a serving endpoint, without taking away control from the person who actually understands the model. A data scientist still decides when to train, when to retrain, and when a new version is ready to deploy; the library just removes the need to know how the underlying infrastructure works to make those decisions happen.
Backtesting before anything ships
Before a candidate model gets anywhere near production traffic, it is backtested against canaries, held out test cases used specifically to catch regressions. A typical setup trains on several years of historical data, for example 03/01/2018 through 03/01/2023, and evaluates on the period immediately after, for example 03/02/2023 onward, data the model has never seen and that reflects the most recent conditions.
This is really a form of policy evaluation, similar in spirit to the off policy evaluation problem described in the reinforcement learning post: the model is a policy that maps inputs to a predicted cost, and backtesting asks how that policy would have performed on data it did not influence, before ever letting it influence anything in production.
Promoting a model
Getting a model from a passing backtest to serving real traffic follows three steps: agree on the metrics that will decide success, stand up a serving endpoint, and roll out gradually through shadow mode before an A/B test. Of the three, agreeing on the metrics up front is consistently the hardest part. It is tempting to skip straight to comparing models and sort out what counts as success afterward, but that almost always leads to arguing about the result instead of trusting it.
Shadow mode
In shadow mode, real production requests are routed to both the existing model and the candidate model, but only the existing model's prediction is used to make any actual decision. The candidate's predictions are logged, not acted on, which makes it possible to run distributional checks and compare metrics between the two models under real production conditions with zero operational risk. Shadow mode is also where bugs that never show up in a backtest tend to surface, things like a feature that is populated differently in the live pipeline than in the historical training data. A shadow period typically runs about a week, long enough to see a full weekly cycle of traffic.
A/B testing
Once a candidate clears shadow mode, it moves to an actual A/B test with a defined control group and a treatment group. Before launching, a power simulation determines how long the test needs to run to reliably detect a meaningful difference in the metrics agreed on earlier, the same experimental design thinking covered in the data science process series on this site applies directly here. Once the test has run long enough, the result is evaluated for statistical significance across each of the key metrics, not just the headline one, since a model can look better on one metric while quietly getting worse on another.
Data quality checks and hypothesis testing
Model quality is bounded by data quality, so the pipeline includes explicit data validation using pandera, a library for defining and enforcing a schema on tabular data: expected types, value ranges, and nullability get checked automatically rather than discovered later as a modeling bug. These checks run alongside the hypothesis testing that supports the model comparisons above, so a broken upstream feed gets caught as a data quality failure instead of silently showing up as a confusing shift in model metrics.
Alerting when something breaks
Alerting was centralized into channels people already watched: Slack and email. This sounds like an unremarkable choice, but it followed a real lesson learned the harder way. The team had also wired alerts through Datadog, and in practice that made cross team alert navigation worse, not better: an alert relevant to another team could get buried in a dashboard that team did not regularly check, while a Slack message in a channel they already had open got seen immediately. The most effective monitoring setup was not the most sophisticated one, it was the one that put an alert in front of a person who was already looking in that direction.
Closing thoughts
The architecture work in part one, TabTransformer, a distributional output, and one base model shared across teams through fine tuning and transfer learning, is what made the platform worth building. The repository structure, training automation, backtesting, staged rollout, and alerting covered here are what made it something a team could actually run and trust day to day. In hindsight, more of the total effort went into this second half than into the modeling itself, which tends to be true of most production machine learning systems: the model is necessary but rarely sufficient.