(Untitled)

Eli Beachy

When using streaming with a for each loop, does the stream ingest additional records that are added to the database after the for each loop has started?

For example, if I have 10 records when the loop starts and I add another record while the loop is running, will it process only the initial 10 records or will it process all 11 records?

Sean Montgomery

 this would depend on how paging is implemented. The same issue could easily happen when doing paging yourself.

Stream keeps fetching data until there is none left, so this would make sense in this example. Another approach would be to have Stream stop when it returns any count of items under the per_page size. That would have avoided your issue in your example, but still would have edge cases if your initial amount of items happened to be exactly the per_page size.

The Stream support was designed for a list that does not change during that for each loop. We also have had users running into problems where they delete items in that list. Another solution is needed for those examples.

Eli Beachy

Thanks Sean. For my use case I actually want it to process any available records but wasn't sure how the streaming worked. I am currently using a While loop but thought a Stream might be more efficient so we don't have to query the db on every iteration.

Sean Montgomery

If your Stream uses a conditional that will filter out newly added items (at least from that specific loop), then it would work great. However, if there is no difference that the existing items and the newly created item, then you may need another approach.

An example would be having a status field.

stream: query all where status = 'ready'
foreach loop
  create new item with status = 'created'

Something along those lines. This is just an idea - don't bend over backwards to make it work with stream if it isn't a good fit.