Sporadic endpoint errors in the last few days

Options
Brett Lewis
Brett Lewis Member ✭
in ? Working with APIs

In the last 3 days I've started to see a variety of errors from Hookdeck - which acts as a webhook proxy - trying to reach some of my Xano endpoints.

I've seen a variety of 404, 500, 502 and 503, SELF_SIGNED_CERT, CONNECTION_RESET and CONNECTION_REFUSED errors on 3 different endpoints.

e.g…

<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx</center>
</body>
</html>

It is consistent/sporadic, and this is a small number of requests in amongst the many successful ones. But this is not something I've encountered before.

The endpoint logs don't show any evidence of any of the errors.

Anyone else seeng this?

· Share on Twitter

Comments

  • Gino
    Gino Member ✭
    edited February 2023
    Options

    Yes, I have a lot of them!

    Now I get a lot of "default backend - 404"

    I contacted support but it looks like you need to pay $750/month in order to have someone to talk in high emergency cases…

    I feel disappointed about the Xano experience so far…

    There is any Xano Status Page available? I don't see it anywhere. I think it's a basic thing they should have in place.

    And yes, the endpoint logs don't show any evidence of any of the errors.

    Sometimes the Request History is not even loading.


    · Share on Twitter
  • Brett Lewis
    Brett Lewis Member ✭
    Options

    I have faith that the Xano team will sort us out soon.

    I don't have any live systems running on Xano as yet, and I'd hate to have to relook my backend choice.

    · Share on Twitter
  • George
    George Member ✭
    Options

    I've been getting a lot of 500 errors in my Xano project last few days.

    · Share on Twitter
  • Vladimir Slavinsky
    Vladimir Slavinsky Member ✭
    Options

    Today I noticed same errors, just a few minutes ago everything went away.
    I would not be surprised if this happened on Build packages, but…

    · Share on Twitter
  • Anna Sho
    Anna Sho Member ✭
    Options

    Same for me. Yesterday all endproints returned 500, Xano admin was crashing and then tables showed no data, looks like some glitch.

    · Share on Twitter
  • Michael Udinski
    Michael Udinski Administrator

    ADMIN
    edited February 2023
    Options

    Hi all,

    We apologize for the sporadic errors you all experienced the past few days within the Germany region. 

    The Germany server cluster encountered hardware issues that impacted several users. We have since been in communication with Google support and were able to swap out the faulty machines to get things operating normally again. 

    We understand how frustrating this is, and is one part of why one of our key initiatives this year is to provide users with clear insights into server health and performance. Our roadmap includes a server status page that users can reference in the event of a similar incident. 

    Although we never anticipate such incidents, we want to provide our users the transparency about what is happening. We hope to roll this out along with other improvements throughout 2023. 

    Thanks for your patience and understanding.

    · Share on Twitter
  • Brett Lewis
    Brett Lewis Member ✭
    Options

    Thanks @Michael Udinski

    I'm still getting the odd retry message (though much reduced) on two of the end points - should the problem be completely gone now?

    If yes, I will troubleshoot from my end.

    · Share on Twitter
  • Michael Udinski
    Michael Udinski Administrator

    ADMIN
    Options

    I'm not familiar with the retry message - are these endpoints interacting with a third-party service by chance? Can you share it?

    · Share on Twitter
  • Brett Lewis
    Brett Lewis Member ✭
    Options

    Apologies Michael, I have troubleshooted the error and it's a different (non-Xano) endpoint causingthe glitch.

    Thanks for the care and concern.

    · Share on Twitter
  • Gavin McMurdo
    Gavin McMurdo Member ✭
    Options

    I am getting errors (502, 404) from a React frontend and also accessing the Swagger interface. Here is a screenshots for one the swagger errors.

    Screenshot 2023-03-01 at 9.02.16 AM.png

    · Share on Twitter
  • Michael Udinski
    Michael Udinski Administrator

    ADMIN
    Options

    Hi @Gavin McMurdo - if this is persisting, can you please open a ticket with support using the in-product support chat or support@xano.com?

    · Share on Twitter
  • Gavin McMurdo
    Gavin McMurdo Member ✭
    Options

    @Michael Udinski I opened a ticket earlier this morning - it's sitting in the queue

    thanks for the response

    · Share on Twitter
  • Michael Udinski
    Michael Udinski Administrator

    ADMIN
    Options

    Ok awesome, thank you!! We'll have someone take a look as soon as possible

    · Share on Twitter
  • Gavin McMurdo
    Gavin McMurdo Member ✭
    edited March 2023
    Options

    We've noticed a correlation between the API's that work and the ones having issues - just go into the Dashboard and look at the API details - If the API durations are abnormally long, those are the ones having issues.

    I also noticed that the 200 green label doesn't always show on the responses with long durations.

    Hope this helps others

    · Share on Twitter
  • Gavin McMurdo
    Gavin McMurdo Member ✭
    Options

    If anyone else having issues - I am seeing multiple issues with xano at the moment

    I have support tickets open.

    · Share on Twitter
  • Brett Lewis
    Brett Lewis Member ✭
    Options

    I'm not seeing issues.

    A while back I found a tendency for an endpoint to get 'saturated' when being hit by a queue of webhooks. The normal processing time of less than a second kicked up to over 10 seconds, causing a huge memory/CPU spike.

    I changed my strategy to having the raw webhook data being stored directly into a 'queue' table without the extra processing that was happening each time. And setting up a function to check and process that queue from time to time. The Xano platform seems much happier with this appoach.

    · Share on Twitter

Categories