Please tell me you look skyward, shake your fist and yell damn you!!!!
I wonder if there's an available OS that parity checks every operation, analogous to what's planned for Quantum computers.
Unrelated, but the other day I read that the main computer for core calculation in Fukushima's nuclear plant used to run a very old CPU with 4 cores. All calculations are done in each core, and the result must be exactly the same. If one of them was different, they knew there was a bit flip, and can discard that one calculation for that one core.
Interesting. I wonder why they didn't just move it to somewhere with less radiation? And clearly, they have another more trustworthy machine doing the checking somehow. A self-correcting OS would have to parity check it's parity checks somehow, which I'm sure is possible, but would be kind of novel.
In a really ugly environment, you might have to abandon semiconductors entirely, and go back to vacuum as the magical medium, since it's radiation proof (false vacuum apocalypse aside). You could make a nuvistor integrated "chip" which could do the same stuff; the biggest challenge would be maintaining enough emissions from the tiny and quickly-cooling cathodes.
That feeling when it is, in fact, computer ghosts.
Me: "Hmm... No... No the code is good, it's the compiler that's wrong."
It's even worse then: that means it's probably a race condition and do you really want to run the risk of having it randomly fail in Production or during an important presentation? Also race conditions generally are way harder to figure out and fix that the more "reliable" kind of bug.
Or it was an issue with code generation, or something in the environment changed.
Mmm, race conditions, just like mama used to make.
Good luck figuring out why it sometimes doesn't work đ
There was that kind of bug in Linux and a person restarted it idk how much (iirc around 2k times) just to debug it.
This is 100% valid when dealing with code generation sometimes and I hate it
Legit happens without a race condition if youâve improperly linked libraries that need to be built in a specific order. Iâve seen more than one solution that needed to be run multiple times, or built project by project, in order to work.
Isn't that the definition of a race condition, though? In this case, the builds are racing and your success is tied to the builds happening to happen at the right times.
Or do you mean "builds 1 and 2 kick off at the same time, but build 1 fails unless build 2 is done. If you run it twice, build 2 does "no change" and you're fine"?
Then that's legit.
Yup, itâs that second one. 0% chance of success until all dependencies are built, then the final run has a 100% chance to work.
We call this sort of test "fuzzy". If it's really bad they call it by my own personal identifier of "unstable".
i sometimes do that so i can inspect the error messages on a cleared terminal
Sometimes I forget what I was looking for and have to restart the mental loop when doing this.
One of my old programs produces a broken build unless you then compile it again.
Some code has bugs.
Some code has ghosts.
Just had that happen to me today. Setup logging statements and reran the job, and it ran successfully.
I've had that happen, the logging statements stopped a race condition. After I removed them it came back...
Thank you for playing Wing Commander!
======== 37/37 tests passing ========
That's when the real debug session begins
Great time to find out your tests are useless!
Theyâre not completely useless. Theyâre conditionally useless, and we donât know the condition yet.
So how do you write a good test? It's like you have to account for unknown unknowns, and I don't really have a good theory for doing that.
Right now, I usually end up writing tests after the code is broken, and most of them pass because they make the same mistakes as my original code.
For unit tests, you should know the input and expected output from the start. Really responsible devs write the unit tests first, because you should know what you're going to put in and what you'll get out before you start writing anything. If you find yourself making the same mistakes in your tests as you do your code, you might be trying to do too much logic in the test itself. Consider moving that logic to its own testable class, or doing a one-time generation of a static set of input/output values to test instead of making them on the fly.
How granular your tests should be is a matter of constant debate, but generally I believe that different file/class = different test. If I have utility method B that's called in method A, I generally test A in a way that ensures the function of B is done correctly instead of writing two different tests. If A relies on a method from another class, that gets mocked out and tested separately. If B's code is suitably complex to warrant an individual test, I'd consider moving to to its own class.
If you have a super simple method (e.g. an API endpoint method that only fetches data from another class), or something that talks with an external resource (filesystem, database, API, etc.) it's probably not worth writing a unit test for. Just make sure it's covered in an integration test.
Perhaps most importantly, if you're having a lot of trouble testing your code, think about if it's the tests or the code that is the problem. Are your classes too tightly coupled? Are your external access methods trying to perform complex logic with fetched data? Are you doing too much work in a single function? Look into some antipatterns for the language/framework you're using and make sure you're not falling into any pitfalls. Don't make your tests contort to fit your code, make your code easy to test.
If ever you feel lost, remember the words of the great Testivus.
First off, thanks for the help!
Really responsible devs write the unit tests first, because you should know what youâre going to put in and what youâll get out before you start writing anything.
I've obviously heard the general concept, but this is actually pretty helpful, now that I'm thinking about it a bit more.
I've written pretty mathy stuff for the most part, and a function might return an appropriately sized vector containing what looks like the right numbers to the naked eye, but which is actually wrong in some high-dimensional way. Since I haven't even thought of whatever way it's gone wrong, I can't very well test for it. I suppose what I could do is come up with a few properties the correct result should have, unrelated to the actual use of it, and then test them and hope one fails. It might take a lot of extra time, but maybe it's worth it.
How do you deal with side effects, if what you're doing involves them?
For your vector issue, I'd go the route of some static examples if possible. Do you have a way to manually work out the answer that your code is trying to achieve?
For side effects, that may indicate what I referred to as tightly coupled code. Could you give an example of what you mean by "side effect"?
For your vector issue, Iâd go the route of some static examples if possible. Do you have a way to manually work out the answer that your code is trying to achieve?
Not necessarily. In this scenario I'd imagine it's a series of numbers as opposed to something more human-friendly exactly because there's internal complexity that's important but hard to manually survey, let alone generate. If you've worked with GANs at all, maybe it's a point in a latent space.
For side effects, that may indicate what I referred to as tightly coupled code. Could you give an example of what you mean by âside effectâ?
I mean it in the standard functional language way, if you're familiar. There's an operation that happens at some step of an algorithm, and it changes a data structure which is referred to or updated at another step. Sometimes you can't really avoid it, because the problem itself has an interconnection like that.
::: spoiler A sorting algorithm example, if that doesn't make this too complicated.
Concurrency it's pretty much guaranteed to do it, so let's say we're trying to implement some sort of bespoke sorting algorithm, where each compare is large and complex enough we have bugs, and which runs in multiple threads.
If threads are interfering with each other in this program, how do you test for that? The whole thing won't give expected results, obviously, but another unsorted array or a failure to terminate doesn't tell you much. Each compare and each swap might look correct at first, and give properly typed results. Let's assume that each thread might traverse to anywhere in the array, so you can't just check when they're overlapping.
:::
If that doesn't work, sometimes your computer just needs a rest. Take the rest of the day off and try it again tomorrow.
The crazy thing is that sometimes this just works...
I often do this, but I always hit Ctrl-S before running it again. Shamefully, this probably works about 10% of the time. Does that technically count as changing nothing?
That and a make clean can work wonders.
Autosave on focus loss dude.
Well, duh! You need to use the right incantations!
I've only recently joined the dev world and I saw this post in the morning. Late this afternoon I'm doing a deployment that fails, couldn't determine the cause as I'm a noob, before bothering a more senior dev for help I run it again... It worked.
it's only dumb til it works
The definition of insanity is doing the same thing over and over and expecting different results.
Got to make sure it's not one of those phantom failures.
Sponsored by QA gang. Gotta make sure it's a 5/5 issue and not just a frequent issue
My way: wrap it in a shell script and put a condition if exit status is not 0 then say "try clear the cache and run it again"
AKA - the test suites at half the companies I've worked. Except they use a loop with retries as well
"Works in my environment."
Einstein did say...
Ever work?
Disturbingly, yes
Every sufficiently complicated system is indistinguishable from being alive, and living beings need some warm-up time.
All the time. Causes include:
Test depends on an external system (database, package manager)
Race conditions
Failing the test cleared bad state (test expects test data not to be in the system and clears it when it exits)
Failing test set up unknown prerequisite (Build 2 tests depends on changes in Build 1 but build system built them out of order)
External forces messing with the test runner (test machine going to sleep or running out of resources)
We call those "flaky tests" and only fail a build if a given test cannot pass after 2 retries. (We also flag the test runs for manual review)
I mean
maybe
Not with code, but i write a lot of LaTeX and that often starts working on second compilation
on xcode i would say it has a 50% chance of working
Well if it doesn't work once try again if it doesn't work twice or thrice maybe there's a problem. If it works it is resolved ... for now, least effort solution.
And the worst part is when it actually does and you have no fucking idea what went wrong before.
The pc had the hiccups and now it's fine. Problem solved!
Some times my game engine needs a wake up run, then an actual run.
Yeah happens from time to time.
Blame cosmic rays.
That's step zero: rule out black magic
Those damn cosmic rays flipping my bits
Please tell me you look skyward, shake your fist and yell damn you!!!!
I wonder if there's an available OS that parity checks every operation, analogous to what's planned for Quantum computers.
Unrelated, but the other day I read that the main computer for core calculation in Fukushima's nuclear plant used to run a very old CPU with 4 cores. All calculations are done in each core, and the result must be exactly the same. If one of them was different, they knew there was a bit flip, and can discard that one calculation for that one core.
Interesting. I wonder why they didn't just move it to somewhere with less radiation? And clearly, they have another more trustworthy machine doing the checking somehow. A self-correcting OS would have to parity check it's parity checks somehow, which I'm sure is possible, but would be kind of novel.
In a really ugly environment, you might have to abandon semiconductors entirely, and go back to vacuum as the magical medium, since it's radiation proof (false vacuum apocalypse aside). You could make a nuvistor integrated "chip" which could do the same stuff; the biggest challenge would be maintaining enough emissions from the tiny and quickly-cooling cathodes.
That feeling when it is, in fact, computer ghosts.
Me: "Hmm... No... No the code is good, it's the compiler that's wrong."
runs again
Yeah, but sometimes it works.
It's even worse then: that means it's probably a race condition and do you really want to run the risk of having it randomly fail in Production or during an important presentation? Also race conditions generally are way harder to figure out and fix that the more "reliable" kind of bug.
Or it was an issue with code generation, or something in the environment changed.
Mmm, race conditions, just like mama used to make.
Good luck figuring out why it sometimes doesn't work đ
There was that kind of bug in Linux and a person restarted it idk how much (iirc around 2k times) just to debug it.
This is 100% valid when dealing with code generation sometimes and I hate it
Legit happens without a race condition if youâve improperly linked libraries that need to be built in a specific order. Iâve seen more than one solution that needed to be run multiple times, or built project by project, in order to work.
Isn't that the definition of a race condition, though? In this case, the builds are racing and your success is tied to the builds happening to happen at the right times.
Or do you mean "builds 1 and 2 kick off at the same time, but build 1 fails unless build 2 is done. If you run it twice, build 2 does "no change" and you're fine"?
Then that's legit.
Yup, itâs that second one. 0% chance of success until all dependencies are built, then the final run has a 100% chance to work.
We call this sort of test "fuzzy". If it's really bad they call it by my own personal identifier of "unstable".
The first is a surprise; the second is testing.
could be a race condition
Hmm..you may be right. I'll get my Hispanic friend to run it and see if he gets the same result.
It works on my machine
ok, then we ship your machine.
i sometimes do that so i can inspect the error messages on a cleared terminal
Sometimes I forget what I was looking for and have to restart the mental loop when doing this.
One of my old programs produces a broken build unless you then compile it again.
Some code has bugs.
Some code has ghosts.
Just had that happen to me today. Setup logging statements and reran the job, and it ran successfully.
I've had that happen, the logging statements stopped a race condition. After I removed them it came back...
Thank you for playing Wing Commander!
That's when the real debug session begins
Great time to find out your tests are useless!
Theyâre not completely useless. Theyâre conditionally useless, and we donât know the condition yet.
So how do you write a good test? It's like you have to account for unknown unknowns, and I don't really have a good theory for doing that.
Right now, I usually end up writing tests after the code is broken, and most of them pass because they make the same mistakes as my original code.
For unit tests, you should know the input and expected output from the start. Really responsible devs write the unit tests first, because you should know what you're going to put in and what you'll get out before you start writing anything. If you find yourself making the same mistakes in your tests as you do your code, you might be trying to do too much logic in the test itself. Consider moving that logic to its own testable class, or doing a one-time generation of a static set of input/output values to test instead of making them on the fly.
How granular your tests should be is a matter of constant debate, but generally I believe that different file/class = different test. If I have utility method B that's called in method A, I generally test A in a way that ensures the function of B is done correctly instead of writing two different tests. If A relies on a method from another class, that gets mocked out and tested separately. If B's code is suitably complex to warrant an individual test, I'd consider moving to to its own class.
If you have a super simple method (e.g. an API endpoint method that only fetches data from another class), or something that talks with an external resource (filesystem, database, API, etc.) it's probably not worth writing a unit test for. Just make sure it's covered in an integration test.
Perhaps most importantly, if you're having a lot of trouble testing your code, think about if it's the tests or the code that is the problem. Are your classes too tightly coupled? Are your external access methods trying to perform complex logic with fetched data? Are you doing too much work in a single function? Look into some antipatterns for the language/framework you're using and make sure you're not falling into any pitfalls. Don't make your tests contort to fit your code, make your code easy to test.
If ever you feel lost, remember the words of the great Testivus.
First off, thanks for the help!
I've obviously heard the general concept, but this is actually pretty helpful, now that I'm thinking about it a bit more.
I've written pretty mathy stuff for the most part, and a function might return an appropriately sized vector containing what looks like the right numbers to the naked eye, but which is actually wrong in some high-dimensional way. Since I haven't even thought of whatever way it's gone wrong, I can't very well test for it. I suppose what I could do is come up with a few properties the correct result should have, unrelated to the actual use of it, and then test them and hope one fails. It might take a lot of extra time, but maybe it's worth it.
How do you deal with side effects, if what you're doing involves them?
For your vector issue, I'd go the route of some static examples if possible. Do you have a way to manually work out the answer that your code is trying to achieve?
For side effects, that may indicate what I referred to as tightly coupled code. Could you give an example of what you mean by "side effect"?
Not necessarily. In this scenario I'd imagine it's a series of numbers as opposed to something more human-friendly exactly because there's internal complexity that's important but hard to manually survey, let alone generate. If you've worked with GANs at all, maybe it's a point in a latent space.
I mean it in the standard functional language way, if you're familiar. There's an operation that happens at some step of an algorithm, and it changes a data structure which is referred to or updated at another step. Sometimes you can't really avoid it, because the problem itself has an interconnection like that.
::: spoiler A sorting algorithm example, if that doesn't make this too complicated.
Concurrency it's pretty much guaranteed to do it, so let's say we're trying to implement some sort of bespoke sorting algorithm, where each compare is large and complex enough we have bugs, and which runs in multiple threads.
If threads are interfering with each other in this program, how do you test for that? The whole thing won't give expected results, obviously, but another unsorted array or a failure to terminate doesn't tell you much. Each compare and each swap might look correct at first, and give properly typed results. Let's assume that each thread might traverse to anywhere in the array, so you can't just check when they're overlapping.
:::
If that doesn't work, sometimes your computer just needs a rest. Take the rest of the day off and try it again tomorrow.
The crazy thing is that sometimes this just works...
I often do this, but I always hit Ctrl-S before running it again. Shamefully, this probably works about 10% of the time. Does that technically count as changing nothing?
That and a make clean can work wonders.
Autosave on focus loss dude.
Well, duh! You need to use the right incantations!
I actually did this earlier today
I've only recently joined the dev world and I saw this post in the morning. Late this afternoon I'm doing a deployment that fails, couldn't determine the cause as I'm a noob, before bothering a more senior dev for help I run it again... It worked.
it's only dumb til it works
Got to make sure it's not one of those phantom failures.
Sponsored by QA gang. Gotta make sure it's a 5/5 issue and not just a frequent issue
My way: wrap it in a shell script and put a condition if exit status is not 0 then say "try clear the cache and run it again"
AKA - the test suites at half the companies I've worked. Except they use a loop with retries as well
"Works in my environment."
Einstein did say...
Ever work?
Disturbingly, yes
Every sufficiently complicated system is indistinguishable from being alive, and living beings need some warm-up time.
All the time. Causes include:
We call those "flaky tests" and only fail a build if a given test cannot pass after 2 retries. (We also flag the test runs for manual review)
I mean
maybe
Not with code, but i write a lot of LaTeX and that often starts working on second compilation
on xcode i would say it has a 50% chance of working
Well if it doesn't work once try again if it doesn't work twice or thrice maybe there's a problem. If it works it is resolved ... for now, least effort solution.
If that doesnât work, restart it and then try.
I only do it because it works!