The true power of regular expressions (2012)

The distinction matters because regex absolutely can't match HTML, and because regex, unlike PCRE expressions, have guaranteed O(1) space and O(n) time complexity when matching a string of length n. When you use PCRE features for string matching, that may degrade to exponential time which makes it useless. For example, you can do denial of service PCRE attacks, but not denial of service regex attacks (unless you can query with some megabyte-large regex).

My "go to" solution for parsing (and validating/matching) non-trivial grammars is a library that wraps regexes and allows you to structure the grammar with entities above substrings of a string literal (including arbitrary code for transformations). PyParsing for Python, scala-parser-combinators for Scala, Grammar in Raku, PetitParser in Smalltalk, PEGs in Janet, parser combinators in F#, and so on. These are mostly internal/embedded DSLs, which makes them much easier to use than the typical lexer/parser generators, while giving you all the power to structure and evolve the grammar easily.

For simple grammars, a well-written library adds little overhead over plain regexes. However, grammars rarely stay simple - very often, during the course of development, you find edge cases or the need for extensions. If you started with a structured parser, you're fine: there are specific ways of evolving the grammar, and you can use normal refactoring tools to perform them. If you started with a regex, you quickly end up with a monster regex literal that becomes more brittle and harder to change with each modification.

One important property I look for in parsing libraries is the support for left-recursion. Memoizing/packrat parser generators can handle it gracefully, which is important, because if I'm implementing a published grammar, I want to encode it as closely to the original as possible. For the same reason, I prefer having dedicated tools for associativity and precedence (so that I don't have to invent names for intermediate levels).

TL;DR: yes, regexes are much more expressive than the "regular" in the name would imply, but they still have their limits. For parsing things, it's better to start with something that can work in the simple case fast (so no lex/yacc-style codegen from 2 separate external DSLs), but which also provides enough structure that adding good error handling, extending the grammar, attaching arbitrary code transformations, etc. won't be a big problem later.

Obviously they have their place, but I know a lot of the older guys seemed to love them way more than the young.

If I am trying to e.g. count div tags with a regex like "<div" or whatever, then clearly this would work in 99.9% of cases and probably achieve what the poster is looking for.

As soon as you also add character classes to ignore various parts of the document that you are not interested in like "<div[^>]*>" or whatever it is, then it is eminently useful even if the bit we are ignoring is not fully regular.

One lovely thing about regex is how fast it is. I was asked to parse a massive CAN Bus log file for how many times some event had logged. This was the early 2000s and the file was 6GB, which was pretty big. I tried .Net's string.StartsWith or something and that took ages to run through the file. I did the same thing with a regex and it finished in like 5 seconds (HDD, not SSD!). I don't know how the magic works but it is very impressive.

Now they have three problems.

OT but this me-problem makes me angry every time I read it. Nothing is simple, otherwise it is trivial and not worth mentioning. I can't read over this without thinking that I'm not smart enough to wrap my head around something instantly.