For each loop not processing everything (bug?)

tomas randomas

Hello, so I have a relatively simple task to reset the daily statistics for users, using a "task" to do it and it was working well, but now there is about 3.5K of rows to edit and the loop is not processing everything.

Strangest part, and why I think it is a bug is that, it does not result in an error, but it does not process all the 3.5k rows either, just 700-800~ of them.

Now I have changed the return type from list to 'stream', but result stayed the same.

This is how the accounts_1 are returned, only the id in stream mode

This is what happens when I run this loop( 3.5k records should be processed):

Running it again instantly:

And again:

And after a couple more times it is finally done processing all the records, and there is nothing left = done:

So my question is, why are these 'for each' loops completing without processing all the records, and why sometimes it only picks up 100, and sometimes 1.5k records 😲

Thank you!

Chris Coleman

Got it, thank you. The first thing I would say is to remove the stream return type and go back to list — the reason is that if you're editing records, it's typically not recommended in general, and if you are editing the fields that the stream uses, it will "skip" records (which seems to correlate with your sequence of screenshots). This isn't a 'bug', it's more of how streaming data works in general.

After we swap back to list, I would just disable the loop altogether and run it a few times, see if the number of items returned by the query is different. If it is, then we know the issue lies in the query itself.

If the number of records is the same, then the issue is more likely just too many records at once, and you'd want to implement some paging.

Chris Coleman

Hi @tomas randomas! Are you using a custom query?

tomas randomas

Ahoi @Chris Coleman yes, yes I was using a custom query, actually just changed it before your response (made it a little more simple).

But it was really basic like Where field1 > 0 OR field2 > 0 OR field3 > 0 OR field4 > 0

Thank you

tomas randomas

Gooot it, thanks! Yeah probably 3.5k records is too much for the 'list' to be processed. Kind of thought that 'stream' type is the quicker workaround for paging, will do the paging!