Could you talk a little bit about how you got to the hypothesis that Agent was the bottleneck? You said you tested the endpoint and your memory usage went up, but how did you know it was your Rules agents? Observers?
I noticed the memory increasing so rapidly that I figured it had to be copying the full set of rules somewhere. So reading the code and the Agent docs I stumbled on this snippet in the docs [1]:
The first function blocks the agent. The second function copies all the state to the client and then executes the operation in the client. One aspect to consider is whether the data is large enough to require processing in the server, at least initially, or small enough to be sent to the client cheaply.
Before reading this I didn't realize that using an Agent like this copies the whole state upon reading. It actually makes a lot of sense, it's just something I hadn't ran into earlier.
That was a really cool read! Thanks for sharing!
Could you talk a little bit about how you got to the hypothesis that Agent was the bottleneck? You said you tested the endpoint and your memory usage went up, but how did you know it was your Rules agents? Observers?
Thanks, that's a great question!
I noticed the memory increasing so rapidly that I figured it had to be copying the full set of rules somewhere. So reading the code and the Agent docs I stumbled on this snippet in the docs [1]:
Before reading this I didn't realize that using an Agent like this copies the whole state upon reading. It actually makes a lot of sense, it's just something I hadn't ran into earlier.
[1] hexdocs.pm/elixir/Agent.html#modul...