Hugo Hacker News

GoKart: A static analysis tool for securing Go code

the-smug-one 2021-08-18 22:09:12 +0000 UTC [ - ]

Go has some nice tooling which is quite easy to use w.r.t. static analysis. I started writing a nil pointer analysis tool which was going to take advantage of and provide some more advanced information*. I "unfortunately" had a lot more fun stuff to do during my vacation, but it was very easy to get started with! So kudos to the Go team for making this kind of stuff possible for a 1-man team.

* Just a forward-style abstract interpretation living on-top of Go's type system as an additional layer so you get explanations for why the tool believes that a nil-pointer dereference may occur, etc.

SquishyPanda23 2021-08-18 22:55:41 +0000 UTC [ - ]

> I started writing a nil pointer

It still boggles my mind that Go decided to force programmers to worry about nil pointers.

tptacek 2021-08-19 03:53:09 +0000 UTC [ - ]

Then it boggles your mind that Go functions the way most popular languages work. You don't see so much dunking on Python, Java, Clojure, Ruby, &c, over this, even though these languages dominate the leaderboards.

Which is fine, except that this is probably the second-most boring critique of Go, one virtually everyone has heard before, and it has little if anything to do with the story we're actually commenting on, despite having spawned a huge thread about option types.

If GoKart had been my project, I'd be annoyed.

nyanpasu64 2021-08-19 04:57:34 +0000 UTC [ - ]

- MyPy doesn't have pervasive nullability, but distinguishes nullable and non-nullable types in the type system. A function declared to return int but randomly returns None has a bug in its type hints.

- I dunk on Java for pervasive nullability too (though there are tools that add @Nullable xor @NonNull annotations used for analysis, possibly sound). But Go has over a decade more hindsight and should've known better.

I haven't used the other languages.

hnlmorg 2021-08-19 10:28:15 +0000 UTC [ - ]

> [Python] distinguishes nullable and non-nullable types in the type system. A function declared to return int but randomly returns None has a bug in its type hints.

Go distinguishes between the two too. You cannot pass nil as a value to int. In fact in Go you'd get a compiler warning[0] so you don't even need to rely on type hints and a properly set up CI/CD pipeline to catch said faults:

The problem with Go is that pointers can be nullable[1] as well as interfaces[2] (interfaces, crudely speaking, being Go's solution to generics and inheritance. Crudely speaking. So interfaces get used a lot).

There is some logic to them being nullable if you think about the code from a hardware perspective but given how opinionated the compiler and language is, I feel they could have done more to catch accidental nils to save the developer from having to consciously consider them each time.

[0] https://play.golang.org/p/BADNnw08hoo

[1] https://play.golang.org/p/b39tY1SDQtZ

[2] https://play.golang.org/p/Fsjsa_-o7Qb

maleldil 2021-08-19 13:08:14 +0000 UTC [ - ]

The int example was a bit misguided, seeing as int is a primitive type in Go, different from pointer types. In Python, both are references.

Basically, it's the difference between returning None from a function `def f() -> MyClass` in Python (which is type error) versus returning nil from `func f() &MyClass`, which is completely normal in Go.

Having `Optional` in the function signature makes it explicit that one has to check for Nones. Go lacks that.

hnlmorg 2021-08-19 14:35:43 +0000 UTC [ - ]

Optional parameters are a different thing. Yeah one workaround with the lack of optional parameters are to send nil values, but that's only going to work if the type you want optional is a nil'able type. Plus there's nothing stopping null values from being passed in Python code outside of optional parameters. My point is optional parameters are one reason why null values might creep in but not the cause nor reason for null values.

Plus I personally think if you need an optional parameter then 99% of the time the API is likely designed wrong from the outset (eg maybe you should instead be passing a class). The reason being is that optional parameters aren't always predictable (eg why is this value optional? When do I need to include a value for it?)

2021-08-19 05:20:37 +0000 UTC [ - ]

SquishyPanda23 2021-08-19 11:58:13 +0000 UTC [ - ]

Golang wasn't designed in the 1990s.

I'm sorry you're bored and also annoyed in the possible universe where GoKart is your project. But it seems like this is a thing people want to talk about in a post about Go static analysis tooling.

I don't see why you're trying to police HN conversations.

pjmlp 2021-08-19 12:12:35 +0000 UTC [ - ]

ML was designed in 1976.

logicchains 2021-08-19 07:16:41 +0000 UTC [ - ]

>You don't see so much dunking on Python, Java, Clojure, Ruby, &c,

One of the common arguments now for why C# is superior to Java is that it supports non-nullable references. As does C++, which for large latency-sensitive projects is generally picked over C.

pjmlp 2021-08-19 12:14:21 +0000 UTC [ - ]

Which almost no one uses, because it is more trouble than it is worth on existing code bases, and requires everything to be on latest versions.

scns 2021-08-19 09:58:18 +0000 UTC [ - ]

I'd say it has been a better designed language for sure, which wasn't that hard since they just needed to take a look where Java messed up. So many things are obvious in hindsight so it's not a fair comparison. Regarding platform and reach, Java still wins i guess.

Disclaimer: Using neither.

radicalbyte 2021-08-19 10:41:20 +0000 UTC [ - ]

C# has a higher rate of change than Java and a very strong ecosystem - albeit not quite as strong as Java. The tooling - if you pay for it - is in my opinion much better.

The thing is, the Java ecosystem is insane. What else comes close to it in breadth and quality? Python, Go, Ruby etc certainly don't. C++?

pjmlp 2021-08-19 12:18:38 +0000 UTC [ - ]

C++ yes. That is why despite its flaws and complexity, it will be around for decades to come.

Even on the places where Java and .NET languages took over C++ hegemony, it is still there on the implementation of native/extern methods and COM/UWP libraries.

Also don't forget if your favourite compiler is a LLVM/GCC frontend, many of its improvements require a bit of C++ code changes as well.

Nullabillity 2021-08-19 09:52:44 +0000 UTC [ - ]

Sure, but if they can't even get the boring stuff right then why bother moving on to a deeper evaluation?

drvd 2021-08-19 13:37:21 +0000 UTC [ - ]

Programming in Go since 10 years and I do not have to worry about nil pointers. You seem to assume that the possibility of a pointer being nil is something that is complicated, a burden to the programmer and a source of runtime bugs. It's not. At least not in Go. At least not something you have to worry about in practice.

SquishyPanda23 2021-08-19 14:04:21 +0000 UTC [ - ]

I'm glad you've had a good experience, but yes it is a source of runtime bugs.

One piece of code from a well established tech company would just crashloop if authentication failed. I've seen others just die if the RPC service couldn't make a connection.

So in practice, yes I do have to spend my time tracking down nil pointer runtime bugs both from colleagues and also from other organizations.

SPBS 2021-08-19 00:49:09 +0000 UTC [ - ]

It’s not as bad as Java’s NullPointerException because primitive types and compound primitive types are much more prevalent (which are guaranteed to never be nil).

dabfiend19 2021-08-18 23:09:35 +0000 UTC [ - ]

as opposed to?

streblo 2021-08-18 23:13:40 +0000 UTC [ - ]

Optionals would have been a way to solve this problem

magicalhippo 2021-08-19 10:45:37 +0000 UTC [ - ]

So you get an Optional which haven't been set instead of a nil pointer. What's better about that?

SquishyPanda23 2021-08-19 11:59:14 +0000 UTC [ - ]

The type system knows about it and you're forced to check it

magicalhippo 2021-08-19 12:11:53 +0000 UTC [ - ]

Right, but my point is, the code which would raise an error because the pointer is nil now raises an error because the Optional is not set.

Is there really that much of a difference between those cases?

I agree though that in an interface, Optional conveys a more explicit meaning than something pointer-like, which is always a good thing.

SquishyPanda23 2021-08-19 13:38:11 +0000 UTC [ - ]

In practice it does make a big difference because if the type system knows about it then it can enforce handling of the exception. (Or more generally, it can just force you to pattern match on the optional time and make sure you handle the empty case.)

Go programs crash at runtime. In general, program failures and bugs should surface as soon as possible. Ideally no later than compile time. Instead, Go makes you wait until the app is running.

For an app that has a lot of configuration options, for example, there can be a latent bug that crashes the binary for some options. And that bug may not be detected for months because nobody was using that combination of config options.

The only real defense of this is to pepper your code with a bunch of nil checks. But these nil checks are also hard to test, so Go devs just learn to ignore missing code coverage. In fact, your code coverage metrics look better if you don't check for nil.

I'm sure at some point Go or a library will offer a version of optional types that is well-adopted. But my point is that by the time Go was designed, null references were already widely considered a bad idea and the source of a huge class of computer bugs. Go still deliberately designed them into the type system.

drvd 2021-08-19 13:43:00 +0000 UTC [ - ]

Some people think Optional/Either/etc are the absolute cure to a certain sort of problems and -- I speak from experience -- it is impossible to convince them that it's not.

Well, of course, if your end users, the business users, are okay if you present them an "optional result" which might or might not be a result, then Optionals/Either _are_ the cure. Unfortunately most end users are pissed if you tell them that you optionally shipped their purchase or that the refund will either be credited to their CC or not.

logical42 2021-08-18 23:39:59 +0000 UTC [ - ]

You know what's fun?

Getting a nil where you are supposed to have an Optional.

mattnewton 2021-08-19 00:42:55 +0000 UTC [ - ]

Afaik this is impossible in swift and kotlin, only optional values can contain nill.

tbarbugli 2021-08-19 02:16:56 +0000 UTC [ - ]

Try Core Data with Swift and you will see that happening. Lazy objects (vaults) are mapped from objc into Swift and will happily crash on something like a = b where both are not optional.

mattnewton 2021-08-19 15:45:51 +0000 UTC [ - ]

This is happening in objc code or in the swift part? I'm not terribly surprised though, my one experience with core data was miserable once we strayed even a little from the happy path and I ended up rolling my own since we didn't need full functionality anyways. And this was for an internal app, at Apple ┐( ∵ )┌

joshdev 2021-08-18 23:45:35 +0000 UTC [ - ]

Somebody has used Scala

still_grokking 2021-08-19 00:51:05 +0000 UTC [ - ]

More likely a Java lib form Scala than Scala as such.

In "pure" Scala (not in the FP sense, but just without mixing with Java) something like that is almost impossible.

marwis 2021-08-19 01:55:21 +0000 UTC [ - ]

Unless something drastically changed in Scala 3, there is nothing to protect you from null in Scala. In fact even Java is effectively safer thanks to all the null checking done by IntelliJ

still_grokking 2021-08-19 02:11:47 +0000 UTC [ - ]

Null is basically non-existent in idiomatic Scala. So technically you're right but besides calling Java libs there is only an infinitesimal small chance to get NPEs form Scala code. (Scala's NPE is the MatchException ;-)).

For Scala 3 there are improvements. It's "null safe" as long as you opt-in (modulo Java libs, and of course doing stupid things like casting a null to some other type).

https://docs.scala-lang.org/scala3/reference/other-new-featu...

rowanG077 2021-08-19 01:04:37 +0000 UTC [ - ]

so... Just make that impossible. It's not like this is unprecedented at this point. It's a standard feature even C++ of all languages supports.

still_grokking 2021-08-19 00:54:02 +0000 UTC [ - ]

As opposed to some modern type system feature that would catch such bugs at compile time rather than let them happen at runtime.

SquishyPanda23 2021-08-18 23:39:30 +0000 UTC [ - ]

Most modern languages have solved the problem. There are a variety of ways to do it.

pkaye 2021-08-19 00:04:21 +0000 UTC [ - ]

Can you list the variety of ways?

SquishyPanda23 2021-08-19 00:44:20 +0000 UTC [ - ]

Basically if you need to manipulate pointers, use safe pointers and keep track of pointer ownership. This is how modern C++ and Rust work.

If you don't need to manipulate pointers then nil really just represents a degenerate or optional value. For optional values these can be encoded any number of ways depending on the type system. One common pattern is an optional type. Another is to annotate the type to indicate that it might be null.

The idea is that if a programmer doesn't check that an nullable or optional type is missing then the program should fail at compile time instead of crashing at runtime. Golang chose to crash programs at runtime.

So for whatever reason, Go has decided that null pointer dereferences are not a big deal. But God help you if you try to comment out a variable use without assigning it to "_". Then the program fails to compile.

Zababa 2021-08-19 10:01:13 +0000 UTC [ - ]

> So for whatever reason, Go has decided that null pointer dereferences are not a big deal. But God help you if you try to comment out a variable use without assigning it to "_". Then the program fails to compile.

I think that's a good explanation of why people are so frustrated with this. There are lots of features like go fmt, go vet, the compiler checking that you use all variables and all imports that can feel a bit restrictive. But for something like null pointers, there is nothing. It's incoherent.

masklinn 2021-08-19 06:49:52 +0000 UTC [ - ]

I don't know about "variety of ways". You just make it so pointers can't be null, then provide a mechanism for opt-in nullability, requiring an explicit check / conversion to get a non-nullable pointer (which you can dereference) and allowing free / cheap / implicit conversion from non-nullable to nullable. This can be:

* separate pointer types (e.g. C++ pointers v references)

* a built-in sigil / wrapper / suffix e.g. C#'s Nullable / `?` types

* a bog-standard userland sum type e.g. Maybe/Option/Optional

In modern more procedural languages the third option often will have language-level (non-userland) facilities tackled on for better usability but that's not a requirement.

For cases (2) and (3) it can (depending on language and implementation) also provides a mechanism for making other value types optional without necessarily having to heap-allocate them.

mattnewton 2021-08-19 00:33:51 +0000 UTC [ - ]

Usually it's forcing the programmer to handle the null case statically, by wrapping the underlying value in something like an optional type and defining the operations that access the underlying value. Think of swift's optional unwrapping in "if let" statements

izgzhen 2021-08-19 01:49:46 +0000 UTC [ - ]

Golang doesn’t have a proper generics support at its beginning. It is too late now.

masklinn 2021-08-19 06:52:47 +0000 UTC [ - ]

You don't even need generics to avoid nullable pointers, you can special-case it as they special-cased slices, maps, channels, etc… e.g. `*int` -> non-nullable pointer to int; `?int` -> nullable pointer to int, and a tiny bit of flow analysis in nil checks so e.g. `if foo != nil` implicitly creates a new non-nullable version of `foo` inside the block body.

brundolf 2021-08-19 00:16:07 +0000 UTC [ - ]

I've wondered what it would be like to write a thin language that compiles to Go and mainly serves to introduce a reasonable type system on top, while benefitting from its performance and garbage-collection. Could prevent null dereferencing, among other things

HALtheWise 2021-08-19 06:51:20 +0000 UTC [ - ]

Long ago I hacked together a weekend project with a friend of "JSX for go" that allowed embedding html tags into Go source like people do for react. We were pleasantly surprised how readable and flexible the Go parser source code was, even for the pretty dramatically different syntax we were trying to support.

https://github.com/8byt/gox

nemo1618 2021-08-19 02:31:20 +0000 UTC [ - ]

It is super weird to me that no big compile-to-Go languages have emerged. It doesn't seem like anyone is even trying! Why not?

jmnicolas 2021-08-19 08:38:28 +0000 UTC [ - ]

> Why not?

Why? What would be the point except as a personal challenge?

Zababa 2021-08-19 10:11:45 +0000 UTC [ - ]

Most criticisms of Go could be adressed by a language with a Hindley-Milner type system: https://go.dev/blog/survey2020/missing_features.svg from https://go.dev/blog/survey2020-results. Having a ML-like that compiles to Go could solve all those issues, while still keeping the great ecosystem that Go managed to build. Just like with Scala, this new language could allow people to see if that's what they really want, and offers to the Go team possibilities of evolution without the need to commit completly to them.

pjmlp 2021-08-19 12:20:43 +0000 UTC [ - ]

For that I already have OCaml, Haskell and F#, so why bother?

Zababa 2021-08-19 12:32:41 +0000 UTC [ - ]

OCaml and Haskell don't have Go's ecosystem in terms of quantity. For F#, can it be compile to a binary like Go? I searched for a bit and couldn't find a good answer. Another thing is that people like to stay in their ecosystems. If you already have a large codebase in Go, internal libraries, etc, switching could be difficult.

pjmlp 2021-08-19 12:50:43 +0000 UTC [ - ]

Quantity isn't a synonym for quality.

There are a couple of ways to compile .NET code into native code. Since version 1.0 NGEN was part of the SDK, although its main purpose was faster startup with dynamic linking.

On Microsoft side there has been CoreRT, NativeAOT, .NET Native.

Mono has had AOT support since ages and it is anyway required for iOS deployments.

Other than that several community efforts have taken place as well, for example WebAssembly.

https://fsbolero.io/

Zababa 2021-08-19 13:04:36 +0000 UTC [ - ]

You're right about quantity and quality, that's why I mentionned quantity precisely. Quantity usually means that most of the stuff has already been made by someone, and people seem to value that a lot. I'll add that the documentation of your average OCaml or Haskell library is not the best, while from experience Go is a bit better (usually there are a few basic examples).

Thank you for the information on .NET AOT.

noisy_boy 2021-08-19 01:37:05 +0000 UTC [ - ]

I wrote one component in go - it does everything that is expected from it with good performance without having to deal with virtualenv/jvm dependencies. However I don't want to write that much code again. Only of there was a language with ecosystem/brevity/garbage collection of Java, strong type system/pattern matching of Rust, excellent multi-threading of go and produced a dependency free binary.

still_grokking 2021-08-19 02:02:22 +0000 UTC [ - ]

Scala Native probably.

But I've never used it so not sure how mature it is.

https://scala-native.org/

The other Go alternative I see is D.

Close to the metal but with high level features. Runs in a managed runtime. Creates native code.

https://dlang.org/

noisy_boy 2021-08-19 07:29:01 +0000 UTC [ - ]

Thanks for reminding me about D - I've always heard about it but never checked it out. Will try it out to see how it feels.

pjmlp 2021-08-19 12:23:45 +0000 UTC [ - ]

Be prepared that D kind of struggles with having a small community, other than that, it is a very nice C# like language with systems programming capabilities and AOT compilation.

pjmlp 2021-08-19 12:22:11 +0000 UTC [ - ]

D is more than that, think of C# as version 1.0 should have been all along.

benhoyt 2021-08-19 04:50:27 +0000 UTC [ - ]

I suspect in part it's a combination of 1) Go is good enough as is (and some people don't mind Go's simple type system or the boilerplate-y but explicit error handling), and 2) the Go tooling is really good -- "go build", "go test", etc, would all have to be wrapped or rewritten with slower, buggier versions.

dzonga 2021-08-19 03:55:45 +0000 UTC [ - ]

v-lang ? https://vlang.io/

hardwaregeek 2021-08-19 00:39:40 +0000 UTC [ - ]

Like a TypeScript for Go? We could call it Tolang.

brundolf 2021-08-19 00:43:20 +0000 UTC [ - ]

That's the idea!

I use Rust for a lot of personal projects mainly because of the type system, not because it doesn't have GC. I think GC's totally livable for a great many things, and it would help iteration speed a lot to not have to deal with the borrow-checker, but I just can't stand working in a language with a shaky type system these days. So Go-with-good-types sounds fantastic to me.

smabie 2021-08-19 02:05:42 +0000 UTC [ - ]

How about OCaml?

scns 2021-08-19 10:07:40 +0000 UTC [ - ]

OCaml will be a viable alternative when multithreading works, hopefully soon.

brundolf 2021-08-19 02:21:19 +0000 UTC [ - ]

I've heard the tooling and general ecosystem are not great (similar to Haskell), though I don't know firsthand

smabie 2021-08-19 03:57:38 +0000 UTC [ - ]

I've personally found it really good, probably better than Haskell. Not Go level though I imagine.

Zababa 2021-08-19 06:43:41 +0000 UTC [ - ]

The tooling is good, the ecosystem is where it's lacking.

ggu 2021-08-19 03:53:06 +0000 UTC [ - ]

+1 After learning OCaml & experiencing algebraic data types and pattern matching, writing in other languages just makes me want to switch to OCaml

scns 2021-08-19 10:10:06 +0000 UTC [ - ]

Or Kotlin, KOlang.

jacques_chester 2021-08-19 01:18:30 +0000 UTC [ - ]

I think your difficulty would be in finding interested users. People who like Go, like Go, warts and all. Everyone else has heaved a sigh of relief and decamped to other languages.

brundolf 2021-08-19 01:27:16 +0000 UTC [ - ]

To be clear, I'm not suggesting that I personally would be able to bootstrap this (and the tooling, and the declarations for existing libraries, etc) and have it take off

But if I saw it pop up on HN I'd jump on board in a heartbeat

Go has a lot of compelling benefits around compiling, performance, concurrency, etc that I think would translate. It just made a couple of really unfortunate decisions around null pointers and default values that turn a lot of people off. I think salvaging it from those unfortunate decisions would be worth doing for the right party with the resources to do so, and I think it would appeal to a lot of people

I'm not even sure I would want generics to be added, since that's an elephant in this particular room. I just want to be able to have a mote of confidence in the values I'm working with.

shp0ngle 2021-08-19 02:00:47 +0000 UTC [ - ]

That seems nice. I stopped using gosec (or whatever was the name) because of all that noise

ggu 2021-08-19 03:45:06 +0000 UTC [ - ]

Yeah, I was on the team that came up with GoKart and one of the main motivating factors was that gosec is just waay too noisy to be useful

jannetalex 2021-08-19 10:56:52 +0000 UTC [ - ]

. I've suffered with HIV/AIDS ever since I was a child but it's only been the last few years I discovered that I also have herpes virus . So I started looking for a way to get cure permanently from this deadly virus I visited so many hospitals in search for a solution.Few month ago I came across a site where a lady was sharing a testimony about Dr Godwin and how he cured HIV/AIDS virus and all kinds of diseases with natural herbs so I decided to give it a try and i messaged Doctor Godwin he told me how I was going to get the herbs so I did as he instructed few days later I received the herbs and I started taking the herbs as instructed by the Dr.I was shocked two weeks later my doctor told me that I was free from HIV/AIDS so I decided to let the world know how I was cured from HIV/AIDS by Dr godwin. you can reach him through his drgodwinharbs@gmail.com Or WhatsApp/call +234 8089906968.If you also want cure for your herpes or any of the disease listed below Dr godwin also cure the listed diseases below 1,Weight loss 2,cancer 3,herpes virus 4,high blood presure 5,diabetes 6,Skin rashes 7,Swollen lymph glands 8,Pneumonia Memory loss 9,Sores of the mouth, anus, or genitals 10,HIV/AIDS 11,Penis enlargement etc Goodluck call or wattsapp [+2348089906968 ] dr godwin website link --- [ https://drgodwinharbalhomecure22.simdif.com/ ] God bless you dr Godwin for what you did for me and my family he is so real and reliable