A Quick Explanation of Project Panux

A while back I decided that I wanted to make my own Linux distro because I disliked many design choices. Different distros have gone in different directions and made decisions that benefited a specific use. I wanted to make a distro that would work well for everything - with a small footprint so it could be used on embedded devices/routers, parallel init so that desktops would load up fast, and a large amount of control over packages while making it easy for non-tech-savvy users to install packages. I originally tried to base it off of Debian, then Arch, and then I gave up. Eventually, I decided that there was nothing for me to base it off of, and it was time for me to make my own distro.

Package Generator
This was the most annoying part. I tried makefiles and shell scripts, and then gave up on panux. Several years later, having learned about Go and Docker, I wrote a system that translated yaml into makefiles using Go text/template to allow shortcuts and programmatic generation of commands (best use of this is in the Mesa packaging script). These makefiles are run in a docker container. Later I went back and optimized it, running the source-fetching makefile on the host so that the sources would be cached.
Package Manager (lpkg)
The package manager for panux has been completely rewritten at least 5 times (I lost track). Originally I wrote it in shell (shpkg), but then I ran into some issues due to limitations in Busybox shell. Later it was rewritten in Lua (hence the name lpkg). The Lua version was rewritten several times to add new features and make major changes to the package management system. Finally, I rewrote lpkg in shell again because lua was taking up too much space.

So now the package management system is written in shell, and uses minisign to verify package signatures. It only depends on minisign (which uses libsodium) and busybox - which allowed me to cut the base system down to 1 MB.
Coreutils & LibC
I originally planned to use uClibc for Panux, but realized it lacked important features and switched to musl. I use busybox for the core utilities (think sh, grep, echo, etc.). These are small - which is why I chose them.
Init System (linit)
Linit is currently the biggest difference between Panux and other Linux distros. LinitD is a daemon that runs init scripts. It uses epoll in order to deal with communication, instead of threads. The process of starting a process in linit is as follows:

1. LinitD fork/execs a shell script called linit-start
2. linit-start runs the corresponding init file in /etc/init.d with the start option
3. Before running the start shell function in the init file, the depends shell function is run
4. The depends shell function calls a shell function called dep which invokes linitctl start
5. linitctl start connects to linitd over a unix socket and requests that the dependencies be started
6. linitctl start waits for linitd to notify of completion
7. The depends function completes and the start function is run
8. The init script finishes and linit-start notifies LinitD of completion
9. LinitD forwards the notification. . .

When it starts, LinitD starts an init script called boot which pulls everything in /etc/rc.d as a dependency - which starts the system. As you can see, LinitD does not actually have to do very much - almost everything is done by the init scripts. Because of this, LinitD only needs ~40KB of memory to run (for comparison my SysD is using 2.2 MB of normal mem and 3.2 MB of shared memory right now).

Getting Started with Golang Part 5: Cleanup with defer

Often in code you need to have a cleanup run whenever a function returns. It is usually annoying to have to write the same piece of code everywhere you have to return. For this, go has a special statement called defer. For example let's say you want to open a file and then close it when you are done with it:

file, err := ioutil.ReadFile("/tmp/somefilehere")
if err != nil {
   panic(err) //Panic used here for simplicity in this example
}
defer file.Close()
//Some file operations here

Once all of your file operations are complete and your function exits, the file will be closed - even if you exit with a panic. We can also have multiple defer statements. If we have multiple defers they will be run from last to first. This is done because defers defined later often rely on variables which will be cleaned up by defers defined earlier.

An example of this:



(Note: this demo is through Go playground)

SJR4 and the Campaigns to Drive in Reality

Recently, the US congress published a bill to revoke certain privacy measures that prevented ISPs from selling user data. Now, several people have launched campaigns to raise funds to buy the histories of the related congress members. Together, two campaigns have raised over $175k so far (shown below). The more popular of the two has a website. It has passed the senate and house, and only remains to be signed by President Trump (which he probably would/will do). Luckily, there has been a motion to reconsider.

Getting Started with Golang Part 4: Your first package

This part of the tutorial covers how to make and document a basic package and put functions on a type.

Every language has some method of implementing libraries. In Go, these are called packages (like they are called in Java), and are mapped based on an internet location (e.g. "github.com/jadr2ddude/go-opencv" is the package path of my fork of the go-opencv package). In this tutorial, we will be creating a very basic package that implements the HTML5 Server Sent Events protocol.

Preparation
For this tutorial, you will need a GitHub account (or any other git hosting system - just change your paths appropriately). Create a repository (preferably called html5sse) and clone it on your computer. Start by creating a .go file to put your code in inside this repository. First, we need to specify our package name:
package html5sse
Note that you have to put this package specification at the top of every line. Now, we need to specify our imports as usual. For this tutorial, we are importing "net/http" and "fmt".
Defining a struct
Instead of using object oriented programming, Go extends the C notion of a struct - a simple group of values. Using structs instead of Objects eliminates dynamic type checking overhead caused by inheritance (which Go structs still have), making code run faster. To define a struct type we do:
type TypeName struct {
      //Struct members
      exampleMember int
}
In Go, capitalization of type/variable names determines whether they are "exported". If something is exported (e.g. TypeName in the example above), that means that it can be accessed by another package. Otherwise, it is only directly accessible in the package it is defined in (such as the exampleMember of the struct defined above). However, you can access unexported things in exported functions - allowing getters and setters for example.
In our code, we will have 2 struct members - one called out that is an http.ResponseWriter, and one called count which is an int. Call the struct type "EventSender".
Creating a constructor function
Constructor functions in Go are essentially just normal functions which return a new instance. We will call out function NewEventSender. We will be doing normal C-like returns for this function (Go also supports named returns which will be discussed later). Our function definition would start like this:
func NewEventSender(w http.ResponseWriter) *EventSender {
Note the * before EventSender. Structs are pass by value by default, and * tells the compiler that we want to pass by reference. Inside this function, we first want to allocate a memory location for the struct using the special function make:
es := make(EventSender)
The variable es now contains a reference to the allocated struct. When we are done with it, Go will automatically delete it (Garbage Collection). Now we want to initialize the out member with the http.ResponseWriter passed in as an argument, and count with 0. This is done just like it is in most programming languages (with the = operator).
Putting a method on the struct
In Go you can put a method on (almost) any type of value - even redefined primitive types (e.g. you could put a .Add method on an int). To do this, you do not need to make any changes to your struct. Instead, you prepend an input to the beginning of the function definition. For our SendEvent method, the declaration would become this:
func (es *EventSender) SendEvent(e string) {
The EventSender this is being called on is now stored in a variable in the SendEvent function called es.
Sending our data
The HTML5 Server Sent Events protocol is extremely simple - you send a string followed by two newline characters and that string is given to a JavaScript function running in the browser. We also have to flush our stream whenever we send the event to force the OS to immediately send the event data without buffering it. First, we need to write data. Use fmt.Fprintf to write the data (use "%s\n\n") to the out http.ResponseWriter. Now, we need to flush the stream - using the .Flush() method of the ResponseWriter. Finally, we must increment the count variable using the ++ operator.
Additional Functions
We also want to implement a getter called GetCount for the EventSender.
Trying your library
Run go build in your repository to check syntax and then commit and push your code. Now run go get github.com/usernamehere/reponamehere. Now you can use this library. Try modifying your code from the previous tutorial to add a handler for "/event" which creates an EventSender and sends an event with example text (e.g. "Hello SSE!"). Add the following code to the page served by your original handler:
<script>
var ev = new EventSource("event");
ev.onmessage = function(event) {alert(event.data);};
</script>
Now, run your code and you should see an alert saying "Hello, SSE!".
Documenting
Now, we want to document the html5sse library. We do this by simply adding a comment before the definition of every exported field. In order to distinguish this from other comments, it must start with the name of the exported field. Once you are done with this, commit & push your changes and go to GoDoc.org. Plug in your package import path, and you should see your documentation pop up!

Getting Started with Golang Part 3: Hello Web!

Typically when you start learning a programming language, you start by printing Hello World to a terminal. We will do that in this post. However, where Go really shines is in HTTP. So we will create a simple dynamically generated site which has a header saying Hello Web! with the request path below it.

First install Go:

pacman -S go

or

apt-get install golang

or

dnf install golang

Every Go file must start with a package name. If you want it to be runable you have to put it in the main package. So the first line should be: package main

Now we need to import the text formatting library "fmt". So we have:

import (

     "fmt"

)

As in C, the function called main is called when the program is started. In Go, it does not take any arguments or return anything. So the function declaration becomes:

func main() {

Now we want to call a function called Println from the fmt library. For acessing functions from libraries, we use dot accesses. Println takes a string and prints it out trailed by a newline. So our function call is:

fmt.Println("Hello World!")

Note that in Go you do not need semicolons at the end of lines (although they are valid syntax and you need them for most for loops).

Now you just add a close bracket.

Now lets format with golang: go fmt hello.go (assuming you saved it as hello.go)

And now we can run it: go run hello.go

Hello World!

Now here comes HTTP. This is surprisingly easy. First, just add "net/http" in your imports on a line after "fmt".

Now we need to define a function that will be called when there is an HTTP request. This takes two arguments: an HTTP response writer and an HTTP request. In Go, variable names come before the variable types. Thus, our function declaration becomes:

func handleRequest(w http.ResponseWriter, r http.Request) {

fmt in Go has a function called Fprintf, which does basically the same thing as it's C equivalent (substituting values into a string and writing it out to a stream). With Go we can use the ` character for multi-line strings. In Go, the := operator defines a variable and initializes it at the same time - with the variable being of the type it was initialized with. So to define our page variable:

page := `

<!DOCTYPE html>

<h1>Hello Web!</h1>

%s

`

Now, we need to use fmt.Fprintf to write this as an HTTP response:

fmt.Fprintf(w,page,r.URL.Path)

Close bracket the function and then we need to add code to main to start the server. Now we start the server. I am using port 8080 because it does not require root (normally) like 80 does. To start it:

http.ListenAndServe(":8080",handleRequest)

If you have an SSL certificate, you can use http.ListenAndServeTLS instead and put your cert file and key file paths after the ":8080" to use https (you must use your server domain then).

Now start your Go code and open a browser on the same machine and navigate to http://127.0.0.1:8080

You should now see your script working. Try adding an http path such as http://127.0.0.1:8080/hello.

Getting Started with Golang Part 2: Overview of Utilities

Golang, like most other programming languages, parses the code into an abstract syntax tree. Golang has a lot of utilities that use this AST to help with coding.
go fmt
This built in command parses golang source code into AST and translates it back into code with easily readable spacing/tabbing. In nano, it is keymapped as Control-t, and there are plugins for VIM and EMACS.
It is also accessible in the standard library as "go/format"
go doc
Like Java and many other languages, Go has an automatic documentation tool (godoc). Godoc is also accessible as a website that will automatically generate documentation given a package address (e.g. github.com/Shopify/sarama).
go get
Golang uses addresses for importing packages (basically take the URL and remove the protocol prefix). (Note: standard libraries don't follow this pattern). For example, to import github.com/Shopify/sarama, you would import it as:

import (
      "github.com/Shopify/sarama"
      //other imports
)

(note: Blogger wont let me tab, so I tried to space)
The command go get clones the library with git into your Go path so that you can use it.
GopherJS
This is not an official utility, but I included this because this is incredibly useful. This translates Go packages into extremely fast JavaScript - allowing it to be run in a browser at high speed:

(above: HTML5 game benchmark with GopherJS with >10k objects running at 41 FPS)
GopherJS also has a command (gopherjs serve) which opens a web server and dynamically translates installed go packages to JavaScript. GopherJS itself can be converted to JavaScript and run in a browser, hence the GopherJS Playground.

Getting Started with Golang Part 1: Overview of Language Features

Golang is a programming language with a strong focus on simplicity, performance, concurrency, readability, and utility. Golang is pre-compiled rather than JIT-compiled, has an extremely fast garbage collector (with sub-millisecond pauses), has a large standard library (including most popular crypto algorithms, a TLS implementation, and HTTP), and has many integrated tools which help with coding.

Language Features
There are a lot of features of Go, so I picked two of my favorites:
Goroutines (Concurrency)
Golang makes concurrency easy to implement - you can start a Goroutine almost the same way you would call a function - just putting it after the go keyword. For example:

go ExampleFunc(arg1,arg2)

That will run ExampleFunc on a new goroutine. Goroutines are extremely lightweight - using segmented stacks that grow and shrink as you append and remove data. So, you can use a large number of goroutines without using very much RAM. Instead of using system threads, they are put in a shared pool and instead a fixed number of system threads is used to run them. Go uses asynchronous syscalls, which allows it to run another goroutine while it is waiting for an operation to complete.
Easy Passing of Data In Multithreaded Code with Channels
Go uses the idea of thread communication by sharing data. You can create a channel, pass it to goroutines on creation and then use it for sending data between. Channels are statically typed.

If you want to use channels you first make a channel. Then you can write a value to a channel such as:
exampleChannel <- 1
The goroutine sending to the channel will block until something reads from the goroutine. You can have multiple goroutines listening to the same channel - which will load-balance the processing of the data from the channel between the goroutines. I used this technique in my small web crawler. You can also serialize data from a channel on one server and deserialize it to a channel on another server.
Standard Library
The Go standard library is filled with many commonly used utilities such as crypto (AES, RSA, ECDSA, SHA256/512, etc.). It has reflected encoding libraries - which will take any type (e.g. a struct) and encode it in your preferred encoding (XML/JSON/GOB). The standard library also includes various popular compression algorithms (bzip2, gzip, DEFLATE, zlib, and LZW). The standard library also has image encoders/decoders for popular formats (GIF, JPEG, PNG, and TIFF in experimental). Most importantly, it implements an HTTP(S) library (server & client) which is quick and easy to use.