Integrating 100Million Records into 1 database table

JayyyTeee

not sure how normal this is but I plan on adding 100M records into my database though an API. What is the best known way to go about this? What are the risks involved? what should I look out for? should I do it all at once or increments?

Thank you in advanced! :)

Michael Udinski

Where are these 100M records now?

JayyyTeee

In another database (don't have any specific info on this)

nocodetalks

Let's divide that into 2 ways-

1. Active user- That's a bit tricky. The best thing to do is to migrate once and set up webhooks on the old database. As a result, the data in your new database will be up-to-date. Test a few beta users on the new database to see if anything is missing.
2. No current active user- Suppose there is no active user or the data is not used by any of the active users. If that is the case, migration is simple. You just need to set up a one-time migration. You can do that using API batches or multiple CSV uploads.

@JayyyTeee hope it make sense.

Ray Deck

A lot of the devil is in the details, but a few data ideas that might make it all a little less stressful:

1) You can send over pages of data at a time - this is pretty common for bulk exports. For that many records you might upload tens of thousands at a go. Not a problem, just be patient for the multiple transactions to run.

2) Upload as a CSV or JSON into Xano. Xano can parse either easily. Some databases offer a "SQL statement" export, which is not going to be Xano's strong suit, so avoid that option if possible. This will depend on the database you are exporting from, which I appreciate is till TBD.

3) Use the bulk insert functionality in Xano to make the inserts go a lot faster and more reliably.

And let us know in the community how it goes and what questions you have! We also talk about data migrations during daily State Change Pro office hours