Obscure audio codec is a perfect fit for the N64

Posted by frostbyte7 1 day ago

Counter250Comment30OpenOriginal

Comments

Comment by midnightclubbed 1 day ago

This codec would have been great 27 years ago! I suspect it wouldn't have gotten past the patent lawyers in 1999 though. It seems cute today but the entire codec space was a patent minefield, the N64 mp3 decoder was implemented twice (once based on the reference codec, once clean) with Nintendo/Rare then paying RAD game tools to license their decoder... of which none of their code was used but they had a license from the mp3 patent holder (Fraunhofer) which then shielded Nintendo from lawsuits. Fraunhofer wanted something like $250k to license directly which wasn't going to fly.

I'll also note the frame rate issues in Conker were mainly down to raw RDP fillrate issues, which was for sure the weakest part of the N64 hardware, it could draw beautiful triangles (for the time) but oh so so slowly. There was a decent amount of speech during gameplay using the RSP mp3 codec. The cartridge size was still an issue, 64Mb got a good amount of mono speech but was still very much at a premium.

Comment by xxpor 1 day ago

>It seems cute today

Maybe in audio, but video still seems like an absolute nightmare. Look at the absolute silliness with like having to pay $.99 to play h265 video in Windows.

Comment by jcalvinowens 23 hours ago

AV1 is royalty free and better!

Real example, I recorded the world series using an ATSC antenna, and I re-encoded a 90 second sample using a bunch of different settings on a 32-thread znver5:

  # Codec        Flags                     Size         Speed
  - -----        -----                     ----         -----
    Original     (Raw ATSC mpeg2video)     77MB (100%)  N/A
  1 libx265      -preset slower -crf 24    37MB  (48%)  0.245x
  2 libsvtav1    -preset 3 -crf 44         36MB  (25%)  1.87x
  3 libsvtav1    -preset 1 -crf 35         19MB  (25%)  0.556x
(NB: The same CRF results in a higher quality video at slower presets, this was for archival so we're using the some of the slowest presets).

(1) and (2) are completely indistinguishable from the original unless I go frame by frame looking for transient motion artifacts. (3) is only distinguishable in one shot where a dark background was washed out. Not only does AV1 produce results just as good as H265, it encodes quite a bit faster for the same size/quality.

That said, I've watched so much h264-encoded video over the past decade I think I've been trained to perceive the subtle visual "snow noise" as a sign of a high quality video: I find how AV1 blends backgrounds more visually objectionable than the dithering in H265, even though I think it's objectively higher fidelity. Similar to how I find high frame rate video "fake" after having watched NTSC TV for the first half of my life.

These were the CRFs necessary to get the same visual quality with libx265 at faster presets and the resulting sizes:

  hevc-slower-24          37MB (used above)
  hevc-slow-21            56MB
  hevc-medium-20          57MB
  hevc-faster-19          52MB
  hevc-veryfast-18        60MB

Comment by layer8 20 hours ago

You can get it for free though: https://www.free-codecs.com/guides/how_to_download_hevc_vide...

Comment by jedbrooke 21 hours ago

you used to be able to get it for free on windows[0] but it seems that might not be available anymore.

Of course there is always VLC

0: https://apps.microsoft.com/detail/9n4wgh0z6vhq

Comment by alala-lala-lon 1 day ago

> having to pay $.99 to play h265 video in Window

what?

Comment by 6SixTy 19 hours ago

Almost nobody actually pays for this, since every OEM includes a volume license that's baked into the price tag. VLC includes FFMpeg so it just works ootb.

Video codecs have historically been like this, to some extent or another. It's really just a manner of if the manufacturer has absorbed it into the price tag or pushed it to the consumer.

No one likes paying for a codec license anyways, but since patents are at play, they kind of have to.

Comment by Loudergood 13 hours ago

That leads to absolute idiocy like this https://www.reddit.com/r/technology/comments/1p3p1hy/dell_an...

Comment by fleroviumna 9 hours ago

[dead]

Comment by Lerc 21 hours ago

>raw RDP fillrate issues, which was for sure the weakest part of the N64 hardware

I thought weakest part was the size of the texture buffer.

Nobody expects the N64! Our two weakest parts are our raw RDP fillrate, and our tiny Texture buffer.

Comment by midnightclubbed 17 hours ago

Ok, from a performance perspective the RDP fillrate was the biggest issue. For artists the texture buffer was a massive challenge.

If the RDP had been clocked 4x faster and had 4x the texture size then it would have been incredible… it makes beautiful pixels!

Comment by andrekandre 16 hours ago

i've heard the rambus dram latency was also a big deal but does that have any effect on the fillrate as well? or is that mostly a bottlneck affecting the cpu?

Comment by cowlevel 11 hours ago

and memory bandwidth. Our three weakest parts are our raw RDP fillrate, our tiny texture buffer, our paltry memory bottleneck...and compiling in debug mode... Our four... no... *Amongst our weakest parts... are such elements as... I'll come in again.

Comment by skhr0680 10 hours ago

Don't shit on the N64 from too high. It introduced 1998 (and 2026) real-time rendering concepts at a time when two years was an eternity in PC hardware.

It's why modern programmers can make it do things that would have given John Carmack sleepless nights if they had accomplished them back in 1996.

Comment by jonas21 22 hours ago

In case anyone else got to the demo thinking the quality is awful -- the vocals are incomprehensible and keep cutting out -- it turns out it's like that in the original track:

https://www.youtube.com/watch?v=i1gVxKhdGPs&t=57s

Comment by phoboslab 21 hours ago

Good point! Replaced the video.

Comment by aruametello 21 hours ago

i got confused at first, and then noted that this was an awful piece of music to pick for a compression demonstration.

not the music taste itself, but perhaps he should have gone for something more known that would be easy for A/B tests.

Comment by ethanbezz 16 hours ago

Tom’s Diner by Suzanne Vega would've been a neat choice, as it was the track used to assess and refine the MP3 compression algorithm during it's development.

Comment by Sesse__ 20 hours ago

I wonder if this would also be a great fit for the MAS3507D! Supposedly you can replace the program with your own in a 1-kiloword RAM (the default MP3 decoder is in ROM, and probably larger). If you can fit this codec in 1024 MIPS instructions, perhaps you can do the same in 1024 of these DSP instructions…

Too bad there is seemingly zero public documentation on the format of these programs, though (and no emulator or similar).

Comment by gatane 15 hours ago

It was all great until they mentioned LLM usage.

The demoscene and homebrew scene is dying as we speak.

Comment by cowlevel 11 hours ago

No it's not, but it's splitting into pro-AI and anti-AI segments.

Think about it, if someone embedded GPT-2 into a demo in 2019 the same scene would've found it hilarious. It's not the AI, it's how we use AI and everyone being sick of it.

Comment by jamesnorden 2 hours ago

Is it such a sin that he used an LLM to transpile some code?

Comment by hxorr 20 hours ago

Great read, thanks!

Pardon my ignorance but one thing I'm curious about, if the decoder fills almost the entire 4k RSP instruction memory (IMEM) - wouldn't that be something games need for graphics? If so, then is this still usable for say a game?

Comment by phoboslab 19 hours ago

Like the DMEM, you can replace the IMEM at any time. In this case, when you're done with processing a block of music, you can DMA the code for 3d transforms or anything else to the RSP.

Libdragon handles this transparently for you with a queue[1].

[1] https://libdragon.dev/ref/rspq_8h.html

Comment by neuroelectron 5 hours ago

Didn't Top Gear Overdrive (1998) use MP3 tracks? It had several tracks by Grindstone that would play during races. The audio quality isn't that great but lyrics are mostly comprehendible.

https://www.youtube.com/watch?v=kyXsWd7DpEs

Wipeout 64 had some music by Fluke and Propellerheads but I'm pretty sure it used some tracker format to re-create the tracks from short recordings and honestly, they sound awful, not just because the audio quality but because the repetitiveness and lack of progression.

Comment by pona-a 1 day ago

Beautiful! I long wanted to write my own audio codec, this might be a nice challenge.

Comment by pferdone 1 day ago

Sorry but what? Half-Life on an N64?? it looks incredible!

Comment by everdrive 1 day ago

There's an explosion of N64 ports lately, with a huge mix of new libraries (libdragon) pure fan passion, and LLM assistance.

Some I'm tracking:

- Duke Nukem 3D (ie, the fully-functional PC version) https://github.com/BigChat88/N64DukeNukem3DPort

- Portal 64 Still alive https://github.com/mwpenny/portal64-still-alive

- Resident Evil 64 (ie, not RE2, but the first one) https://www.youtube.com/watch?v=_xRiwy8Z6S0

- Silent Hill (still an early WIP) https://www.youtube.com/watch?v=v7fM06jmHso

I haven't personally tested, but these look interesting:

- Simpsons Arcade https://www.youtube.com/watch?v=zH9JYQVzKDE

- X-men Arcade https://www.youtube.com/watch?v=ah3nB8MP4CM

- TMNT Arcade Collection https://www.youtube.com/watch?v=CiR5-ZpLtyI

At this point, they're coming out faster than I can play them. I'm in the middle of the 2nd Episode of Duke 3D right now and it's quite nice. The original 64 version was always fun, but mid-level saves are really a game changer.

Comment by haunter 1 day ago

> There's an explosion of N64 ports lately

There's an explosion of ALL ports lately. It's crazy.

Ocarina of Time only got released today officially but it's already playable on ARM Macs. It's not even a port per se but executing the Switch ARM code natively. And of course it's also playable on emulators regardless of the platform

Comment by thequux 10 hours ago

libdragon certainly isn't new; I distinctly recall talking to dragonminded (the original author) about her efforts to reverse-engineer the RSP code that came with the n64 devkit back in 2008 or so and her subsequent efforts to reverse the RSP. I can totally believe the increased rate of games though

Comment by lanmu 15 hours ago

[flagged]

Comment by blenser4 16 hours ago

[flagged]