Let's stop using [a-zA-Z]+

#javascript #regex #a11y #todayilearned

If you, like me, regularly (see what I did here?) validate alphanumeric fields using Regex, you probably learned to do it like this:

'Till'.match(/[a-zA-Z0-9]+/gu)

This is technically correct, of course. And it's what most validation libraries will do when you tell them a field is alpha / alphanumeric / etc.

However, I have a problem with this approach and a lot (!) of other people do, too. Because I'm from Germany. More specifically, from a town called Lüdenscheid. And Lüdenscheid won't match the regular expression above because of the Umlaut. Same applies for languages like French, Spanish, Czech, just to name a few.

So how can we as developers be more inclusive towards languages other than English? Do we have to include all possible variations of the latin alphabet? That's a common suggestion, but of course, it doesn't scale well.

Luckily, Unicode has us covered:

'Lüdenscheid'.match(/[\p{Letter}\p{Mark}]+/gu)

The \p flag allows us to pick a so called Unicode Character Category. In Unicode, all characters are sorted into categories that we can use in our regular expression. The Letter category includes letters from all kinds of languages, not just A-Z. But it does not include, e.g. <, >, + or $ which is important for security. The Mark category – as lionelrowe pointed out in the comments (thanks) – contains combining marks. In Unicode, a letter like ü can be either one or two combined code points. So depending on how the character is coded, we need the Mark category.

More details on the Mark category

If we omit the Mark category and run the following Regex: 'Lüdenscheid'.match(/[\p{Letter}]+/gu) it will match Lüdenscheid, if the ü is encoded as a single character. On the other hand, if the ü is encoded as a letter-mark-combination (u + ̈), the regex will only match Lu, because it will stop at the ̈ mark.

Browser support

Browser support for this feature is good, IE (not Edge) being the only exclusion.

Bonus

// Match only letters
'Lüdenscheid'.match(/[\p{Letter}\p{Mark}]+/gu)

// Match letters and spaces
'Pražští filharmonici'.match(/[\p{Letter}\p{Mark}\s]+/gu)

// Match letters and hyphens
'Île-de-France'.match(/[\p{Letter}\p{Mark}-]+/gu)

// Match letters hyphens and spaces
'Île-de-France'.match(/[\p{Letter}\p{Mark}\s-]+/gu)

Top comments (6)

lionel-rowe • Mar 9 '21 • Edited

Great post! But unfortunately, this still fails for some languages. For example, Burmese:

const burmese = 'မြန်မာဘာသာ'

;/^\p{Letter}+$/u.test(burmese) // false

Or even certain normalized representations of "Lüdenscheid":

// visually the same, but splits the umlaut and the "u" into two characters
const town = 'Lüdenscheid'.normalize('NFD')

;/^\p{Letter}+$/u.test(town) // false

You can get around this by also allowing the Mark category:

const regex = /^[\p{Letter}\p{Mark}]+$/u

regex.test(burmese) // true
regex.test(town) // true

Till Sanders • Mar 9 '21 • Edited

Good remark! I will update the post to include this 👍

Pacharapol Withayasakpunt • Mar 9 '21 • Edited

A while ago, I also learnt that JavaScript regex has unicode support (via /.../u).

I always had to use XRegExp before.

Also, the syntax is a little different. XRegExp('\\p{Han}') vs /\p{sc=Han}/u.

For this, I also recommend this VSCode extension, so that you don't write an invalid regex. (Like, valid in some compilers, but not native JavaScript.)

For /\p{Letter}/u, I would normally write shorthand.