I do have MaxMemInQueues 96 MB but it does not prevent Tor from being below 1GB of RAM. In my case Tor reports that lzma buffers are huge and attempts to free some memory by killing connections are negligible:
Oct 15 01:13:05.000 [notice] We’re low on memory (cell queues total alloc: 528 buffer total alloc: 0, tor compress total alloc: 683344725 (zlib: 0, zstd: 0, lzma: 683344613), rendezvous cache total alloc: 0). Killing circuits withover-long queues. (This behavior is controlled by MaxMemInQueues.)
Oct 15 01:13:05.000 [notice] Removed 528 bytes by killing 11 circuits; 0 circuits remain alive. Also killed 1 non-linked directory connections. Killed 0 edge connections
—
Oct 18 14:13:06.000 [notice] We’re low on memory (cell queues total alloc: 528 buffer total alloc: 12288, tor compress total alloc: 780965400 (zlib: 0, zstd: 0, lzma: 780965272), rendezvous cache total alloc: 0). Killing circuits withover-long queues. (This behavior is controlled by MaxMemInQueues.)
Oct 18 14:13:06.000 [notice] Removed 8720 bytes by killing 2 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections. Killed 2 edge connections
Recently I saw this post about compression bomb, so I thought that there are ways to attack/disturb Tor nodes by playing with / pumping decompression requirements. In another post Tor managed to allocate 2147654498 for lzma… insane! 
I don’t see currently a way to keep Tor from being OOMKilled unless it learns how to manage compression buffers. 