Author(s): Lily Chen Originally published on Towards AI. One common use case of working with time series data is trend detection, or trend analysis. A real life example is looking at memory data to see if a service is experiencing a memory leak. Some time ago, my team at Datadog released a Memory Leak Wizard, whose aim was to help users solve memory leak issues. When the feature was first released, we asked the users to manually answer a set of questions before bringing them to what they need to solve their potential memory leak issue. We first ask if there is a goroutine leak. If there isn’t (as in the example above), we ask about Live Go Memory. In this case, there is growth, and after users selects “Yes”, we show the relevant Comparison Flame Graph that they can use to investigate where the growth in memory is from. Of course, in this modern age of AI, you might be wondering “why does a human need to answer this?” And you are correct, this could very much be automated. Fast forward a few months, we released the automated version of the Memory Leak workflow. Automatic trend analysis However, the process of capturing an upward trend in a time series data has some complexities involved. Here are some pitfalls I ran into and lessons I learned that might help others or help my future self if she were to do this kind of work again. First of all, is this even an LLM problem? My conclusion after working on this project is “no.” This is not a problem best solved by the current generation of LLMs like ChatGPT-4o. But before we get to why that’s the case, let’s delve first into some of the complexities of working with time series data. The necessity of scaling the data The way to identify linear growth in data is by performing linear regression. Linear regression finds the slope of the line of best fit, and the R-Squared value. Typically, to assess whether a dataset has a linear upward trend, we first define a threshold for slope and a threshold for the R-Squared value. A reasonable threshold for the slope could be 0.1. If both values are greater or equal to the threshold, we say the dataset has a linear growth. Linear regression, however, is sensitive to the magnitude of the data. When performing time series analysis, it is important to take that into consideration. For all examples below, the x-axis values are in milliseconds since epoch, or what Date.now() would return in Javascript. Let’s look at a simple dataset with 2 points. In both examples below, the y doubled. But in the first case, a linear regression will return a slope that’s pretty much 0. A slope of zero (or below) means no upward trend. Case 1: y axis values go from 10 to 20 If you try to fit a linear regression line through this data for an hour time frame, you will get a slope close to zero, i.e. no growth, even though the data doubled. Case 2: y axis value go from 1 million to 2 million When looking at memory usage in bytes, the y-axis values could be in the order of millions or billions. In this case, you might not want to scale the x-axis, depending on the actual time frame and memory values are. The caveat with scaling data If you scale the x-axis, the value of the magnitude of the slope kind of becomes meaningless on their own, and not to be relied on alone. Let’s look at this following example. Let’s say the threshold for positive growth is 0.1, and you have 2 points where the y-values go from 100 to 101. Let’s say they’re taken 1 minute apart so the deltaX (without scaling) is 60 * 1000. Without scaling, the effective slope is 1 / 60,000 or 0.000017 (i.e. very close to zero), so it would not pass check for linear growth. Good. But let’s say we scale the x-axis. Let’s set t1 to be 0, and t2 to be t1 + 60000. Or in other words, if t2 is Date.now(), t1 would be t2 - 60000. To scale the timestamp, let’s divide the timestamps by 6000. The delta X after scaling would be 10. In this case, the effective slope is 1 / 10, or 0.1, and would pass a threshold of >=0.1 for linear growth. Thus, slope alone is not reliable when scaling of the axis is involved. To get around this, we also need to look at the min(y) (minimum of the y-values) and max(y) (maximum of the y-values) of the entire series. If the max(y) is not a certain threshold greater than min(y), we do not label the graph as trending upward. For example, if max(y) <= min(y) * 1.05, i.e. if the data has not increased by 5% between the min and max, we can call it not increasing in a significant manner, even if linear regression after scaling gives us a large slope with high confidence. This is not foolproof, because what if max(y) is actually before min(y) in the time series? We’re using the ideal scenario that max(y) is towards the end and min(y) towards the beginning to filter out non significant growth. Hopefully it is enough in the context of what we’re doing at Datadog in the context of the Memory Leak Workflow. However, it’s something to keep an eye out on, and we’ll deal with this problem if we ever cross the bridge. So far I’ve only talked about linear regression, which is good at detecting linear data. However, there could be many shapes of growth in the context of a memory leak, such as quadratic or exponential growth. I’ll write in a later blog post the challenges of working with other regressions in the context of real world trend analysis, but first, the big question you might be wondering: Can LLMs do better? […]
↧