Is tail call optimization truly necessary?

When I was learning about a new functional programming language, one of the first things I would check would be if it supports tail call optimization. My thinking was that if this feature isn't available, then the tail-recursive functions could blow the stack.

Tail call optimization is a technique that makes recursive function calls efficient by eliminating the need for new stack frames. When the last expression of a function is a call to itself, this optimization reuses the current function's stack frame rather than creating a new one. So, it prevents stack overflow errors and improves performance.

Scala supports tail call optimization. However, it is not enabled by default. You need to use the @annotation.tailrec annotation to tell the compiler that a function should be optimized.

def factorial(n: Int): Int =
  @annotation.tailrec
  def go(m: Int, acc: Int): Int = m match
    case 0 => acc
    case _ => go(m - 1, m * acc)

  go(n, 1)

The above code is an implementation of the factorial function. The go function is tail-recursive because the last expression is a call to itself go(m - 1, m * acc). The @annotation.tailrec annotation ensures that the compiler will optimize this function.

Recently, I've started wondering whether tail call optimization is really necessary. Let me explain why. The majority of functional programming languages have collections whose elements can be processed lazily, without loading them all into memory at once. Among other things, we use these collections to process large datasets. For example, we treat the contents of a huge file as a lazy sequence of lines, and we perform operations like map, filter, fold, reduce, etc. to compute some result. Interestingly, we can generate these collections with methods like unfold or iterate. It seems to me that almost all tail-recursive functions can be rewritten using lazy collections.

def factorial(n: Int): Int =
    Iterator
      .iterate((n, 1)) { (m, acc) => (m - 1, m * acc) }
      .dropWhile { (m, _) => m > 0 }
      .next()
      ._2

In the code above, the iteration is managed without additional call-stack usage, and the memory footprint is low because the elements are created and consumed on the fly. Additionally, the code is easier to understand compared to the tail-recursive version.

We can achieve similar functionality with other programming languages that follow the functional paradigm. So, the question remains: is tail call optimization truly necessary?