Database and background tasks doing weird things with large dataset

Gregoire Demoge

Hi Xano team,

I a building a Shopify app using Xano as a backend.

Basically, the app help e-commerce store have advanced metrics on their sales and marketing.

To do that, I download all their past orders, customers, and products.

It worked fine for smaller stores (less than $200k /y - around 10k rows) but a bigger customer came, and things started to be weird.

Setup

To download customers' data, I'm using a background task. Since I might download 100k's rows for a given store, I have broken down the import task this way:
1. Call the API, Import the first 5000rows, then stop and save when you left off.
2. Start the Background task again and pick where you left off.
3. Repeat until completion

Problem

When I'm looking at the task history, the task did not run at all.

[image.png]
BUT, the database keeps getting new data
[image.png]
[image.png]
But the pace seems slowww. like 15 records per seconds.

So, Is it possible that a background task is running, and not showing up in the task history?

And does it make sense that the data written is around 15 records per second ?



PS: Here is the CPU Graph
[image.png]

Sean Montgomery

The task is probably running out of memory. If you have a lot of records then you need to use paging with query all records and loop through it yourself by calling query all records multiple times. 

 can you make a video on this?

Brandon Hassler

 I'm late to the party on this one, but with today's features, is paginating a large data set still more effective than the newer Stream feature?