o11c

@o11c@programming.dev
0 Post – 24 Comments
Joined 1 years ago

Stop reinventing the wheel.

Major translation systems like gettext (especially the GNU variant) have decades of tooling built up for "merging" and all sorts of other operations.

Even if you don't want to use their binary format at runtime, their tooling is still worth it.

As a practical matter it is likely to break somebody's unit tests.

If there's an alternative approach that you want people to use in their unit tests, go ahead and break it. If there isn't, but you're only doing such breakage rarely and it's reasonable for their unit tests to be updated in a way that works with both versions of your library, do it cautiously. Otherwise, only do it if you own the universe and you hate future debuggers.

Write-up is highly Windows-centric (though not irrelevant elsewhere).

One thing that is regretfully ignored in discussions of async, tasks, green threads, etc. is that there is no support/consideration for native (reliable/efficient) thread-local variables. If you're lucky you'll get a warning about "don't use them".

Some languages don't even support linking at all. Interpreted languages often dispatch everything by name without any relocations, which is obviously horrible. And some compiled languages only support translating the whole program (or at least, whole binary - looking at you, Rust!) at once. Do note that "static linking" has shades of meaning: it applies to "link multiple objects into a binary", but often that it excluded from the discussion in favor of just "use a .a instead of a .so".

Dynamic linking supports much faster development cycle than static linking (which is faster than whole-binary-at-once), at the cost of slightly slower runtime (but the location of that slowness can be controlled, if you actually care, and can easily be kept out of hot paths). It is of particularly high value for security updates, but we all known most developers don't care about security so I'm talking about annoyance instead. Some realistic numbers here: dynamic linking might be "rebuild in 0.3 seconds" vs static linking "rebuild in 3 seconds" vs no linking "rebuild in 30 seconds".

Dynamic linking is generally more reliable against long-term system changes. For example, it is impossible to run old statically-linked versions of bash 3.2 anymore on a modern distro (something about an incompatible locale format?), whereas the dynamically linked versions work just fine (assuming the libraries are installed, which is a reasonable assumption). Keep in mind that "just run everything in a container" isn't a solution because somebody has to maintain the distro inside the container.

Unfortunately, a lot of programmers lack basic competence and therefore have trouble setting up dynamic linking. If you really need frobbing, there's nothing wrong with RPATH if you're not setuid or similar (and even if you are, absolute root-owned paths are safe - a reasonable restriction since setuid will require more than just extracting a tarball anyway).

Even if you do use static linking, you should NEVER statically link to libc, and probably not to libstdc++ either. There are just too many things that can go wrong when you given up on the notion of "single source of truth". If you actually read the man pages for the tools you're using this is very easy to do, but a lack of such basic abilities is common among proponents of static linking.

Again, keep in mind that "just run everything in a container" isn't a solution because somebody has to maintain the distro inside the container.

The big question these days should not be "static or dynamic linking" but "dynamic linking with or without semantic interposition?" Apple's broken "two level namespaces" is closely related but also prevents symbol migration, and is really aimed at people who forgot to use -fvisibility=hidden.

6 more...

1, Don't target X11 specifically these days. Yes a lot of people still use it or at least support it in a backward-compatible manner, but Wayland is only increasing.

2, Don't fear the use of libraries. SDL and GTK, being C-based, should both be feasible from assembly; at most you might want to build a C program that dumps constants (if -dM doesn't suffice) and struct offsets (if you don't want to hard-code them).

Even logging can sometimes be enough to hide the heisgenbug.

Logging to a file descriptor can sometimes be avoided by logging to memory (which for crash-safety includes the possibility of an mmap'ed file, since the kernel will just take care of them as long as the whole system doesn't go down). But logging from every thread to a single section of memory can also be problematic (even without mutexes, atomics can be expensive and certainly have side-effects) - sometimes you need a separate per-thread log, and combine in the log-reader tool.

This is about the one thing where SQL is a badly designed language, and you should use a frontend that forces you to write your queries in the order (table, filter, columns) for consistency.

UPDATE table_name WHERE y = $3 SET w = $1, x = $2, z = $4 RETURNING *
FROM table_name SELECT w, x, y, z
2 more...

I don't remember the last time I used ctrl-C. It's always select or "+y.

1 more...

I haven't managed to break into the JS-adjacent ecosystem, but tooling around Typescript is definitely a major part of the problem:

  • following a basic tutorial somehow ended up spending multiple seconds just to transpile and run "Hello, World!".
  • there are at least 3 different ways of specifying the files and settings you want to use, and some of them will cause others to be ignored entirely, even though it looks like they should be used.
  • embracing duck typing means many common type errors simply cannot be caught. Also that means dynamic type checks are impossible, even though JS itself supports them (admittedly with oddities, e.g. with string vs String).
  • there are at least 3 incompatible ways to define and use a "module", and it's not clear what's actually useful or intended to be used, or what the outputs are supposed to be for different environments.

At this point I'm seriously considering writing my own sanelanguage-to-JS transpiler or using some other one (maybe Haxe? but I'm not sure its object model allows full performance tweaking), because I've written literally dozens of other languages without this kind of pain.

WASM has its own problems (we shouldn't be quick to call asm.js obsolete ... also, C's object model is not what people think it is) but that's another story.


At this point, I'd be happy with some basic code reuse. Have a "generalized fibonacci" module taking 3 inputs, and call it 3 ways: from a web browser on the client side, as a web browser request to server (which is running nodejs), or as a nodejs command-line program. Transpiling one of the callers should not force the others to be transpiled, but if multiple of the callers need to be transpiled at once, it should not typecheck the library internals multiple times. I should also be able to choose whether to produce a "dynamic" library (which can be recompiled later without recompiling the dependencies) or a "static" one (only output a single merged file), and whether to minify.

I'm not sure the TS ecosystem is competent enough to deal with this.

The thing is - I have probably seen hundreds of projects that use tabs for indentation ... and I've never seen a single one without tab errors. And that ignoring e.g. the fact that tabs break diffs or who knows how many other things.

Using spaces doesn't automatically mean a lack of errors but it's clearly easy enough that it's commonly achieved. The most common argument against spaces seems to boil down to "my editor inserts hard tabs and I don't know how to configure it".

For an extension like this - unlike most prior extensions - you're best off with essentially an entirely separately compiled copy of the program/library. So IFUNC is a poor fit, even with peer optimization.

Only if the library is completely shitty and breaks between minor versions.

If the library is that bad, it's a strong sign you should avoid it entirely since it can't be relied on to do its job.

I've done something similar. In my case it was a startup script that did something like the following:

  • poll github using the search API for PR labels (note that this has sometimes stopped returning correct results, but ...).
    • always do this once at startup
    • you might do this based on notifications; I didn't bother since I didn't need rapid responsiveness. Note that you should not do this for the specific data from a notification though; it's only a way to wake up the script.
    • but no matter what, you should do this after N minutes, since notifications can be lost.
  • perform a git fetch for your main development branch (the one you perform the real merges to) and all pull/ refs (git does not do this by default; you'll have to set them up for your local test repo. Note that you want to refer to the unmerged commits for these)
  • if the set of commits for all tagged PRs has not changed, wait and poll again
  • reset the test repo to the most recent commit from your main development branch
  • iterate over all PRs with the appropriate label:
    • ordering notes:
      • if there are commits that have previously tested successfully, you might do them first. But still test again since the merge order could be different. This of course depends on the level of tests you're doing.
      • if you have PRs that depend on other PRs, do them in an appropriate order (perhaps the following will suffice, or maybe you'll have some way of detecting this). As a rule we soft-forbid this though; such PRs should have been merged early.
      • finally, ordering by PR number is probably better than ordering by last commit date
    • attempt the merge (or rebase). If a nop, log that somewhere. If not clean, skip the PR for now (and log that), but only mark this as an error if it was the first PR you've merged (since if there's a conflict it could be a prior PR's fault).
    • Run pre-build stuff that might need to create further commits, build the product, and run some quick tests. If they fail, rollback the repo to the previous merge and complain.
    • Mark the commit as apparently good. Note that this is specifically applying to commits not PRs or branch names; I admit I've been sloppy above.
  • perform a pre-build, build and quick test again (since we may have rolled back and have a dirty build - in fact, we might not have ended up merging anything!)
  • if you have expensive tests, run them only here (and treat this as "unexpected early exit" below). It's presumed that separate parts of your codebase aren't too crazily entangled, so if a particular test fails it should be "obvious" which PR is relevant. Keep in mind that I used this system for assumed viable-work-in-progress PRs.
  • kill any existing instance and launch a new instance of the product using the build from the final merged commit and begin accepting real traffic from devs and beta users.
  • users connecting to the instance should see the log
  • if the launched instance exits unexpectedly within M minutes AND we actually ended up merging anything into the known-good branch, then reset to the main development branch (and build etc.) so that people at least have a functioning test server, but complain loudly in the MOTD when they connect to it. The condition here means that if it exits suddenly again the whole script goes up and starts again, which may be necessary if someone intentionally tried to kill the server to force a new merge sequence but it was too soon.
    • alternatively you could try bisecting the set of PR commits or something, but I never bothered. Note that you probably can't use git bisect for this since you explicitly do not want to try commit from the middle of a PR. It might be simpler to whitelist or blacklist one commit at a time, but if you're failing here remember that all tests are unreliable.

It's solving (and facing) some very interesting problems at a technical level ...

but I can't get over the dumb decision for how IO is done. It's $CURRENTYEAR; we have global constructors even if your platform really needs them (hint: it probably doesn't).

The problem with XCB is that it's designed to be efficient, not easy. If you're avoiding toolkits for some reason, "so what if I block the world" may be a reasonable tradeoff.

Unfortunately both of those are used in common English or computer words. The only letter pairs not used are: bq, bx, cf, cj, dx, fq, fx, fz, hx, jb, jc, jf, jg, jq, jv, jx, jz, kq, kz, mx, px, qc, qd, qg, qh, qj, qk, ql, qm, qn, qp, qq, qr, qt, qv, qx, qy, qz, sx, tx, vb, vc, vf, vj, vm, vq, vw, vx, wq, wx, xj, zx.

Personally I have mappings based on <CR>, and press it twice to get a real newline.

The problem with mailing lists is that no mailing list provider ever supports "subscribe to this message tree".

As a result, either you get constant spam, or you don't get half the replies.

The problem is that the application developer usually thinks they know everything about what they want from their dependencies, but they actually don't.

The problem is that GLIBC is the only serious attempt at a libc on Linux. The only competitor that is even trying is MUSL, and until early $CURRENTYEAR it still had worldbreaking standard-violating bugs marked WONTFIX. While I can no longer name similar catastrophes, that history gives me little confidence.

There are some lovely technical things in MUSL, but a GLIBC alternative it really is not.

3 more...

What you are missing, of course, is the Rc<Refcell<T>> that you have to stick everywhere to make a nontrivial Rust program. It's like monads in Haskell, parentheses in lisp, verbosity in Java, or warnings in C - they're the magic words you have to incant correctly to make things work in their weird paradigms.

That's misleading though, since it only cares about one side, and ignores e.g. the much faster development speed that dynamic linking can provide.

1 more...

The solution is quite simple though: dogfood.

Developers must test their website on a dialup connection, and on a computer with only 2GB of RAM. Use remote machines for compilation-like tasks.

True, but successfully doing dynamically-linked old-disto-test-environment deployments gets rid of the real reason people use static linking.

DNS-over-TCP (which is required by the standard for all replies over 512 bytes) was unsupported prior to MUSL 1.2.4, released in May 2023. Work had begun in 2022 so I guess it wasn't EWONTFIX at that point.

Here's a link showing the MUSL author leaning toward still rejecting the standard-mandated feature as recently as 2020: https://www.openwall.com/lists/musl/2020/04/17/7 ("not to do fallback")

Complaints that the differences are just about "bug-for-bug compatibility" are highly misguided when it's useful features, let alone standard-mandated ones (e.g. the whole complex library is still missing!)