Using Rust with Elixir for code reuse and performance

Zababa 2021-08-19 16:57:46 +0000 UTC [ - ]

> Unsurprisingly, Rust outperforms Elixir by 2 orders of magnitude.

That seems very surprising to me. Is this a common result? I've heard that the BEAM is not the best for "number crunching" code, is this one of those scenarios? Is this really just the raw performance of the languages, or a difference in algorithms? There's also no mention of the version of Elixir and the OTP. Did the JIT change anything?

bb1234 2021-08-19 16:55:07 +0000 UTC [ - ]

For someone who does not know much about NIFs, will it be a lot slower if the Rust service is coded as a command line application and then invoked using System.cmd? Is that a good alternative?

latch 2021-08-19 16:36:47 +0000 UTC [ - ]

One thing to keep in mind is that even the simplest Rust NIF will significantly slow your builds and increase your repo/artifact sizes.

nickjj 2021-08-19 14:12:29 +0000 UTC [ - ]

I wonder if they looked into using https://github.com/asaaki/cmark.ex which is an already made Markdown Elixir NIF written in C. No glue code needed since the package already exists.

Back when I was writing Elixir, it's what I used to process Markdown and it was also substantially faster than the native Elixir Markdown library (Earmark).

NiklasBegley 2021-08-19 15:38:03 +0000 UTC [ - ]

Author here. I actually was not aware of cmark.ex - thanks for pointing it out.

In this case the code reuse was more important than pure native speed. We already had a Rust library that used pulldown-cmark [1] with some custom tweaks that we wanted to duplicate. Maybe this behavior could have been copied using cmark.ex too (we thought about doing this in pure Elixir, as mentioned in the post), but given how straightforward Rustler made integrating our existing code, this seems like the better choice.

[1] https://github.com/raphlinus/pulldown-cmark

brightball 2021-08-19 14:59:13 +0000 UTC [ - ]

I’ve seen a major preference toward Rust for NIFs in Elixir land because of the guarantees. People like their BEAM guarantees and exiting the BEAM without strong guarantees feels risky.

zaphar 2021-08-19 14:24:12 +0000 UTC [ - ]

Even if they did they may have preferred Rust because parsing arbitrary user provided input is one of the more risky things you can do in C. This is a perfect use case for Rust's memory safety model.

queuebert 2021-08-19 14:28:01 +0000 UTC [ - ]

Before Rust, it seemed there was a law that every sufficiently critical code path would eventually be rewritten in C or C++.

After Rust, giving up a bit of performance to not have to maintain the C code underneath seems preferred. And often you don't even sacrifice performance.

nickjj 2021-08-19 14:39:56 +0000 UTC [ - ]

> After Rust, giving up a bit of performance to not have to maintain the C code underneath seems preferred

Yeah no doubt about it, although in this case the C implementation has been a long running project that's under the official commonmark GitHub repo at https://github.com/commonmark/cmark.

But I think the most important thing here is an Elixir NIF already exists to use it. The blog post as is leaves readers having to implement ~100 lines of Elixir code to use the Rust version because the author of the blog post didn't include that code in the article, or open source it as a library for others to use.

From a reader's POV if your goal is to get a highly stable, fast and safe Markdown parser running in Elixir, the Elixir cmark library I linked in a parent comment solves that problem out of the box.

ihumanable 2021-08-19 16:45:22 +0000 UTC [ - ]

Yea, I'd probably of ended up using cmark too.

It is nice though to learn about NIFs and rustler, there are a lot of high quality crates out there that you can wrap and take advantage of from elixir code once you get comfortable with doing it.

So possibly a sub-optimal approach, but the author indicates in another comment they had customized the rust version and wanted to be able to share code, which seems legit. Always good to stock up that toolbox though with more neat stuff, and rustler NIFs are seriously neat.

lostcolony 2021-08-19 15:22:50 +0000 UTC [ - ]

Yeah; I think people responding are taking your response as "they did the wrong thing". It sounds more like "Hey, here's another option that I've used in the past to solve that problem", leaving the evaluation of the relevant tradeoffs as an exercise to the reader (where it should be since you'd wait them differently for different projects)