Deny crawling of API via robots.txt
Hi there,
my API endpoints got crawled by public crawlers which resulted also in high cpu usage.
Is it possible to prevent that e.g. via a robots.txt file hosted on the root of our hosts?
Best Answer
-
deployed! give it a try
Answers
-
Hey @Stefan Pointecker - have you checked out the reCAPTCHA extension? Maybe that could help in this scenario. I can double-check with the team if that's an appropriate security solution for this scenario
-
Hey @Michael Udinski - thanks for the response.
havent checked out the recaptcha extension yet - but if that means I would need to change every request to my api it would not be a viable solution to me.
in my opinion a default robots.txt with disallow would be beneficial for xano too, as it would decrease unnecessary/unwanted requests to the whole infrastructure.
-
I really think it would be beneficial for XANO too to just disallow crawlers on all instances by default! See the usage of my instance for example when the crawler hits my endpoints.
-
@Stefan Pointecker we'll be fixing this early next week with our new release.
Very valid point. Thanks for bringing it up.
-
@Sean Montgomery Great to hear! Is it going to be automatically active or do we have to switch it on?
-
automatically - deploying in the next hour
-
Great news! BTW is there any way to password protect swagger documentation of the API endpoint or a group of APIs?
-
@wojtek you can disable swagger documentation but if it's enabled you can't password-protect the swagger site. It's recommended that your API endpoints require authentication at a minimum to protect them. You can add other custom security measures as well with combinations of things like preconditions.
Categories
- All Categories
- 53 ? Announcements
- 47 ? Releases
- 37 ? Welcome
- 983 ? Help! I'm a Noob
- 125 ? No-Code Front-Ends
- 633 ? Working with APIs
- 439 ? Transforming data
- 126 ? Connect Xano to ...
- 50 ?? Find an Expert
- 348 ❓Other questions
- 35 ? Security
- 22 ✂️ Snippets
- 19 ? Showcase
- 7 ?️ Xano Chatter
- 62 ? Video Tutorials
- 171 ? Request a feature
- 229 ? Report a Bug
- 19 ? Templates & Extensions
- 7 ? Feedback