πFS

Some of that is also the domain. It's less that science is an extreme form of compression, and more that natural phenomenon are highly compressible. They're a small number of kinds of interactions repeated a bajillion times. How many equations does it take to explain electricity (ignoring equations that are derivatives of ones already included)? I think it's less than 5.

On some level, you could probably reduce all of the Standard Model down to models of atoms, their motion, and the basic subatomic particles (the non-quantum ones). That would explain almost everything that happens on Earth in a very short form, though few people would be able to go from that to explaining how lightning works.

(Then it depends on your concern: "Aagh, the aunt fell!" // "Oh yes, that'd be Newton")

GPT-2 for instance achieves roughly 1 bit per byte, so it can be used to compress (english) text 8-fold. Modern models are likely much better.

https://news.ycombinator.com/from?site=github.com/philipl

Spy agencies would not only have to store it all in case it was something valuable, but at some point they may try to crack it because it's indistinguishable from encrypted data and waste resources on it. If enough people did it, total web surveillance could become impractical.

This is what stream ciphers are

Find k candidate indices for your data, then locate each of them. If the smallest one is a significantly smaller index space, repeat.

But Pi's binary expansion is not very practical for this purpose, since it's 11.0010...

OTOH. e is 10.1011...

Let's stick to fractional digits (the ones right of the binary point) at index 0 we have 1 and at index 1 we have 0.

So, to encode a stream of bytes so that each bit is encoded as the index of that bit in the e, all you need to do is to xor it with 0xFF

So, instead of using pi, design an optimal number to encode with.

What you'll find is that the optimal sequence ends up being equally efficient as listing the blocks in order and indexing by block number itself. There are a number of other solutions; you could use superpermutations to get "all possible subsequences" with fewer digits in your target number, but you'll end up needing to provide the encoder and decoder a table of where the digit sequences appear since they are no longer regular and indexing into that table will cost exactly the same as just writing your number as the concatenation of all the blocks and its efficient method for indexing into them by indexing on the block rather than the digit number.

This actually has some natural overlap with the "normal numbers" in that one of the earlier normal numbers was: https://en.wikipedia.org/wiki/Champernowne_constant I'm not sure whether this is necessarily optimal for an arbitrary block size. (My quick intuitive check suggests it may be, but "my quick intuitive check" in the time of an HN post is not something I'd count on.) In this scheme, you can include the fact that the person using this constant to encode knows the nature of the constant, so they know that if you give index 0-9, it's single digit, and if you index into the two-length blocks, it must have a length of two. Since the encoder and decoder know that, they can also skip the middle of the block and just index into "the n'th number"... which degenerates into "the index of number N is N", which means this is not a compression scheme.

To put all that in a nutshell, if you want to deeply understand why this compression scheme doesn't work, I think you can attain a deep understanding of why by optimizing it.

obviously it contains every finite digit string in base 10. I can't prove the digits are uniformly distributed in every base - you'd have to be more clever but you see the idea.

Infinities of random sequences exist that can be shown not to contain all data, 0-8 (base 10) is one such random sequence that is trivially proven to never contain 9...

There are no known patterns to pi, but, (I am legitimately curious about this), are there any known sequences e.g. of 1 million 0s and a single other digit within the decimal sequence of pi?

Given how it (pi) looks, I'm of the strong suspicion is that the answer is "no". But of course, proving that requires that some property of the randomness is provable. Which it does feel as if, given there are different infinities, there are also different randomnesses, hence the conjecture is ill-formed and probably incorrect...

This is roughly same as saying: "If you rewrite 938933556 as a binary number / usize, it will need 30 bits".

Sanity check: 1101111111|0110111111|0100110100 (| delimits every 10 bigits).

> Since the file is 128 bits long, one would expect this place to be around the 2*128th bit.

This statement is a bit more subtle. As a first ord approximation, we can see pi sort of as a RNG.

If we write pi (ignore the decimal point), as a binary number, we get: 11011001111111011110010101011110001010101111101101110001001100001...

You can... kind of squint and pretend this is a random sequence of 1s and 0s.

Now, if you had a file that is 128 bits (so lots of intermingling 0s and 1s), and each next digit of pi is effectively a coin flip. Pretend 1s are heads, and 0s are tails. You basically have to get the exact 128 consecutive coin flips of the same result as your file to get your file back.

Imagine now, PI not as a number, but a sequence of experiments of flipping the coin 128 times.

  - (11011..01000)(10000...00100)....
  - ^attempt 1     ^attempt 2

  - 11011................00100...
  - (       128 flips     )
  -  (  another 128        )
  -   (                     )
  -     ... so on and so on