[Discussion] Performance in search dates

[surkova]

I'm working on a Flask app which does some markup parsing. One of the things it does is it parses strings like Arriving tomorrow by 9pm or Delivered on Friday. All of the strings are in English and they are short. Today I bumped the version of dateparser from 0.7.6 to 1.0.0 and this is what I saw in the distribution metrics (p50, p95, p99) of the function calling search (function abridged):

STATUS_TEXT_DELIVERED = re.compile(r"delivered", re.IGNORECASE)
settings = {
    "PREFER_DATES_FROM": "past"
    if bool(STATUS_TEXT_DELIVERED.search(text))
    else "future",
}
search_results = search_dates(text, languages=["en"], settings=settings)

One thing which strikes me most is huge latency spikes when the app is rebooted on deploy and how it calms down after some significant amount of time. This function is currently called around 20 times per minute, but we are expecting this number to grow to at least 400 rpm. On the screenshot you can see three deploys (red stripes).

Now, I have a very limited insight into what performance instrumentation you've been using, but what would be the easiest way to pinpoint what's happening with the search right after it starts from scratch? And why does it take so long to figure out the happy state?

[noviluni]

Hi @surkova, thanks for opening this issue!

I don't have any idea of why is this happening. There are some things that are initialized when importing the library, but it shouldn't produce this effect. I will take a look when having time, if you find any other clue or a way to reproduce it easily, please, let me know.

Thanks!

P.S: I see that you are using this library in a Flask app. I just wanted to pinpoint that we have some open issues related to concurrency and multithreading and we don't support it 100%: #441, #276, #834