Am I in over my head?

Brandon Hassler

Hey everyone I'm new to Xano and this is very much a introduction/venting post in hopes that perhps there is someone who could point me in the right direction 👋

So I've got a live production site built with Bubble that allows my community to submit deals they find on the web within my niche and upvote them. To really scale my vision of this site, I want to built it more into a price comparison engine, think Trivago but for PC tech. The problem is I have spent endless nights trying to find the right solution to make this happen and I feel like Xano may be last hope.
What I'm Trying to BuildThe bread and butter of any price comparison site is obviously pulling in data (regularly) from various APIs and formating that mess into one easy-to-read database.

My plan is to start by indexing around 10,000 products and ultimately scale up to around 40,000. Starting out, each product will display the price from three different merchants (Amazon, Best Buy & Newegg) and then I'll slowly add more.

Searching 40,000 database entries in Bubble is a breeze, but running workflows to check the various external API feeds even just once a day is where the Bubble site can slow down quick. Workflows become messy and even if I mastered "recursive workflows" (which I seem to struggle with) the math on checking even 10k products x 3 merchants every single day tells me my site is basically going to need workflows running 24/7 that could easily break at anytime. Not good.

But the general UI/UX of the site will be simple: You're looking to buy a gaming monitor? Here's an up-to-date list of the best monitors at the best price right now. And how do I avoid duplicate products from these various merchants? Every product has a "UPC" number which is unique to that product and used across all merchants. 
I Almost Settled on Make/IntegromatI'm a visual learner and Make.com catered very well to that, and the pricing is reasonable as well. I was willing to pay the extra cash to have an easy-to-use workflow that could pull in all this messy data, organize it nicely in a Google Spreadsheet, and then sync that spreadsheet to my Bubble database. The problem? While cleaner, my Bubble site would still need to be constantlely running a backend workflow to pull in all the data thefore slowing the site. Not to mention the # of 'operations' involved with my large database would drive my Make subscription costs through the roof! There's got to be a better way.
Enter XanoAnytime I brought up working with large datasets that need to be updated regularly on the Bubble forums I'd always have at least one person say "why not just put all the data on an external DB and then just call the API every time? The problem was that these other DB solutions didn't have "nocode" methods of connecting to API's to retreieve data like Bubble had. Plus, even if I could manage to get all the data in one-place, having an API call to that database everytime a user goes to my site and every time they change the search criteria/load a new page, what am I going to do when even 100 people are on my site at once looking at products? Every single provider suddenly get expensive.

Except Xano which by some miracle has no rate limiting on an affordable plan and no record limit?! I was sold.

The problem? I'm so burnt out of taking a week to learn a platform only to find out it's not the right solution. It seems like from my research that having an external database is the best solution for my type of site. While I wish I could "be up in running within minutes" I feel like to build out all the API logic on Xano between learning everything is going to take weeks.

Is Xano the best approach for this? If I have say 200 people on my site at the same time all constantely pulling data from my Xano database, is this going to scale with that? Are there complexities/costs that I'm not aware of? My plan/hope is to accomplish this with the "Launch" plan.

/rant 

Chris Coleman

Hey, Brandon. Thanks for the thoughtful, detailed post on your use case. That really helps us point you in the right direction. I do think you're on the right path here -- relying on a backend like Xano in combination with your frontend (Bubble) is going to be the secret sauce to making sure you get the speed and consistency you need. Because Xano can be both your database and the API (the entire backend), you gain a lot of efficiency right off the bat, and we have the capability to scale with you. In regards to specific questions about "will x plan handle my use case", we're working on providing some hard data for questions like these, but because everyone's API is different, it's difficult to nail down exactly.

I'd love to hear more about the specific issues you feel like you're having actually learning the platform. We're always looking for ways to improve our education and I want to know if there's anything I can put into action right away to help you out.

Brandon Hassler

Appreciate the reply! So I've been watching a lot of tutorials on pulling in API data and then saving that data to the database and I've finally got that down (mapping fields, etc). Granted, I only know how to do save the first "page" of items, which for many of the APIs I work with, is 100 items per page. With just a few more things I feel like I'd be to the put where I can slap together an MVP. Here's what I need to learn:
1.  How to paginate through external API data (e.g. The Best Buy Product API uses the cursor method to paginate through pages) I'll presumably need to learn how to tell Xano what to look for and when to stop pulling the API when no items/pages are left.
2. How to tell Xano when to create a new DB entry or when to simply update a certain cell, when pulling external API data. When crawling the various APIs for price data I basically want the logic in my "Function" to look at the UPC field (a unique field that identifies the product) and say "IF this UPC does not already exist in the database, create a new database entry, OTHERWISE, update the cooresponding UPC entry with the latest price"
3. Learn how to generate access tokens via API. One of the feeds I have access two requires a new authentication token every 60 minutes. That's fine when you're pulling data manually and generate it via their website, but I need to pull this data daily and automatically. They provide documentation on how to do this, but I can't figure out how to impliment something like that in Xano. 
4. Learn more about "Background Tasks" and figure out the most efficeint way to automate these daily price refreshes. Even though starting out I'm only working with 3 different merchant APIs, some of them I need to make many calls to collect all of the needed data. For example, with Best Buy, I need to make a call for each category of products my site covers, whereas with Newegg I should be able to do it all in a single call (not including the pagination). I don't currently know if 5 Background Tasks is plenty or not enough to essentially run all these various API calls once a day. Is there a way to experiment with this feature before upgrading the Launch plan?


A lot of this just come down to more learning, more watching, and practicing, but if anybody knows of any specific material for any of the specific things above they could point me to, that would certainly speed up my process!

Brett Lewis

From one noob to another...

Point 1 - You will need to loop through each page and then loop through all of the 'records' in each page - at least this is how I handled with. It took me a while but was able to handle this nested loop function. Once set up, it works great.

Point 2 - Xano does an 'upsert' by default - it will insert a new record, or update an existing record if it finds one. This was a game changer over building this logic into integromat.

Point 3 - I'd think it's possible to run a background task every 60 mins (or less) to fetch a fresh token. You could store that in your environment variables and use that variable in your API calls. But  there may be a more eloquent way to handle this...

Point 4 - The backgrounds task vary in the amount of CPU and duration - you will need to experiment. You may event be able to build a number of different API calls into one function/task. You can buy more of these if you need them, so you can scale are you need to. To test them, I suspect you will need to buy a month subscription - I joined a onboarding session and earned a $100 credit, which I'm using for evaluation purposes - though I'm sold and will sign up when my credit runs out.

I hope I have helped.

Chris Coleman

 Thanks so much for the detailed response, Brett. Appreciate it!