In most companies of any age there is a scheduled job that writes a file, drops it somewhere, and another job that picks it up and loads it. It was the pragmatic answer when it was built and it has run for years. It is also, usually, the least observable and most failure-prone part of the estate - and the failures are silent, which is the part that costs money.
The characteristic failure is not that the job crashes. It is that the file arrives truncated, or encoded differently after a supplier upgrade, or with a column added in the middle, and the load succeeds while writing subtly wrong data. Nobody notices until a report looks strange days later, and by then the wrong numbers have propagated into other systems. The second characteristic failure is a run that simply does not happen, discovered when someone asks why yesterday's orders are missing.
Before replacing anything, make the current pipeline observable, because you may find you have several problems rather than one. Assert an expected row count and schema on every load rather than trusting the file. Alert on absence, not just on error - a job that does not run produces no error at all. And record, per run, what arrived and what was written, so a discrepancy can be traced rather than argued about. This is a day or two of work and it frequently reveals that the integration has been quietly wrong for months.
The replacement is contract-first. Define the interface explicitly - what an order looks like, which fields are required, what the identifiers mean - and publish it as a specification both sides build against, rather than each side inferring the format from example files. This one change eliminates the entire class of problem where a supplier adds a column and your parser shifts every field by one.
Then choose between request and event. If the consumer needs current state on demand, an API is right. If several systems need to know when something changes, events are better - the producer publishes that an order was placed, and any number of consumers react without the producer knowing they exist. Most estates end up with both, and the mistake to avoid is point-to-point wiring between every pair of systems, which produces a web that becomes impossible to change safely as it grows.
Three things must be designed in rather than added later. Idempotency: a message delivered twice must not create two orders, which means every message carries an identifier the consumer records. Retry with backoff, and a dead-letter destination for messages that will never succeed, so one bad record does not block the queue behind it. And reconciliation - a scheduled comparison that proves both sides agree on counts and totals, because the point of leaving batch files behind is confidence, and confidence needs evidence rather than assumption.
Migrate one interface at a time and run both in parallel while you compare outputs. The old file keeps flowing, the new path writes to a staging destination, and you diff them daily until they match for a couple of weeks. Only then do you switch and decommission. It is slower than a clean cutover and it is how you find the undocumented behaviour that the file-based process was quietly relying on - which, in every one of these projects we have done, has existed.
