How and why are new programming languages created?

One issue with programming languages is that there is an inherent need for them to be stable. If you've written a large application with hundreds of thousands of lines of code, you're not going to be pleased that the latest version of the language you're using has major breaking changes. Consider how controversial and fraught the rollout of Python 3 has been. I think this is a factor that has a tendency to make established languages somewhat inflexible to new ideas and new requirements. Even as established languages evolve, they tend to do so in a way that is backward compatible, which sometimes means things are not as elegant as they could be. Over time people get frustrated, and some are even willing to try creating something new to address these sorts of issues.

• Go seems to be a case where the designers wanted something light and portable, very much like C. C has a lot going for it, and it's been around forever, but it's also really easy to create bugs and security flaws in C.
• Rust is interesting. Here the main idea seems to be to try to innovate in the domain of memory management. The compiler provides some guarantees about memory that few other languages offer. Again, my impression is that Rust aims to fit into a similar niche currently occupied by C and C++. C++ has also been around seemingly forever, and it's a language that's renowned for its extreme complexity. I think that creates room for challengers to come along to offer a simpler approach.
• Kotlin is another example of a fairly new language. It is not a radical departure, but it's basically Java "cleaned up." Even if Java wanted to do this themselves, it would be difficult because of the problem of backward compatibility.

That being said, making a new language is no picnic. It's the same old story: If the language doesn't already have a lot of users, that makes it hard to get users. It's kind of like how you need experience to get a job, and you need a job to get experience! It also takes a lot of work to get a language to a point where a typical programmer would find it useful.

I am wondering why would anyone prefer to make a whole new programming language with so many options on the market already

Other have given very good and high reasons, I'm going to give you two less nice ones that I think are as valid:

• ego (usually the ego of the person or people creating the language)
• controlling a piece of the development ecosystem (creating a community of developers around your company likely pays off in the medium-long term)

:)

There are several reasons for creating a new language:

• You see deficiencies and what to create a language that is easier to use, or more robust
• The domain you're working in could benefit from having it's own language (domain specific languages)
• You want to challenge yourself or test a theory in language design
• You want to extend/create vendor lock-in to a platform

To create a new language you need to produce a compiler -- or interpreter, but the differences at the high-level aren't important. These consist of a few general parts:

• Parser: Processes text files and converts them to a language usable by the next step. This is the abstract syntax tree (AST).
• Apply Semantics: Go through the AST and figure out the proper type, as well as change the structure for various language features. I called this the "Typing" stage in my compiler, though I guess some languages emphasis more the structure here.
• Reduce: Convert the expanded AST into an intermediate langauge (IL). This may be done in one or two steps, depending on the language. I had two steps in my compiler, one for a custom IR, and then to LLVM-IR to make use of standard tools.
• Emit: Convert the IR into a target specific machine code -- which may be an actual machine code, or the byte code of a VM.

Compilers are written in other languages. A few end up as self-hosting, which means the compiler is eventually rewritten in the language it compiles. Though the vast majority of compilers are not self-hosted, and there's rarely a good reason to do self-hosting.

@mortoray I still think of you as our resident language creator, care to chat about it? 🙂