DasFaultier

@DasFaultier@sh.itjust.works
0 Post – 38 Comments
Joined 10 months ago

~/src/${reponame}

This is my day job, so I'd like to weigh in.

First of all, there's a whole community of GLAM institutions involved in what is called Digital Preservation (try googling that specifically). Here in Germany, a lot of them have founded the Nestor Group (www.langzeitarchivierung.de) to further the case and share knowledge. Recently, Nestor had a discussion group on Personal Digital Archiving, addressing just your use case. They have set up a website at https://meindigitalesarchiv.de/ with the results. Nestor publishes mostly in German, but online translators are a thing, so I think you will be fine.

Some things that I want to address from your original post:

  • Keep in mind that file formats, just like hardware and software, become obsolete over time. Think about a migration strategy for your files to a more recent format of your current format falls out of style and isn't as widely supported anymore. I assume your photos are JPGs, which are widely not considered safe for preservation, as they decay with subsequent encoding runs and use lossy compression. A suitable replacement might be PNG, though I wouldn't go ahead and convert my JPGs right away. For born digital photo material, uncompressed TIFF is the preferred format.
  • Compression in general is considered a risk, because a damaged bit will potentially impact a larger block of compressed data. Saving a few bytes on your storage isn't worth listing your precious memories.
  • Storage media have different retention times. It's true that magnetic tape storage has the best chances for survival, and it's what we use for long term cold storage, but it's prohibitively expensive for home use. Also, it's VERY slow on random access, because tape has to be rewound to the specific location of your file before reading. If you insist on using it, format your tapes using LTFS to eliminate the need for a storage management system like IBM Spectrum Protect. The next best choice of storage media are NAS grade HDDs, which will last you upwards of five years. Using redundancy and a self correcting file system like ZFS (compression & dedup OFF!) will increase your chances of survival. Keep you hands off optical storage media; they tend to decay after a year already according top studies on the subject. Flash storage isn't much greater either, avoid thumb drives at all cost. Quality SSD storage might last you a little longer. If you use ZFS or a comparable file system that provides snapshots, you can use that to implement immutability.
  • Kudos for using Linux standard tooling; it will help other people understand your stack of anything happens to you. Digital Preservation is all about removing dependencies on specific formats, technologies and (importantly) people.
  • Backup is not Digital Preservation, though I will admit that these two tend get mixed into one another in personal contexts. Backups save the state of a system at a specific point in time, DigiPres tries to preserve only data that isn't specific to a system and tends to change very little. Also, and that is important, DigiPres tries to save context along with the actual payload, so you might want to at least save some metadata along with your photos and store them all in a structure that is made for preservation. I recommend BagIt; there's a lot of existing tooling for creating it, it's self-contained, secured by strong checksums and it's an RFC.
  • Keep complexity as low as possible!
  • Last of all, good on you for doing SOMETHING. You don't have to be perfect to improve your posture, and you're on the right track, asking the right questions. Keep on going, you're doing great.

Come back at me if you have any further questions.

3 more...

With a username like this, I'd give all my hosts and servers moon names. Like the moons of Jupiter (Io, Europa, Ganymede, and Callisto).

7 more...

Gitlab at work, because, well, it's there and it works just fine.

Forgejo at home, because it's far less resource hungry.

In the end Git is a) a command line tool for b) distributed working, so it really doesn't matter much which central web service you put in place, you can always get your local copy via git clone REPO.

My wife has gained roughly 60 kg since I meet her many years ago, which some might see as that kind of limitation, but you know, I don't even see that when I look at her. She is smart as fuck, a great engineer, a wonderful mother, and just overall the better half of me. It's just so easy to love her, and I would never even consider leaving her for looks. We all have our imperfections, but that's not to say we're not worthy of love. Oh yeah, I'm a terrible smart ass, overweight, impatient, not a great father, and I don't communicate very well. Beats me what she sees in me.

400 staff German state institution, Windows desktops are standard, but you can get a supported and standardized Linux Mint installation provided by IT on your personal computer upon request. A few dozen people do. We also provide some 150 publicly accessible PCs for research in or brach locations, all of which are Mint as well. And IT staff is allowed to install any system on their hardware they want, no questions asked; many run Linuxes. Linuces. Linnixees.

Try goharbor.io, that's what I use. I think (but I'm not sure) that Forgejo/Gitea and Gitlab can also cache images.

Very true! And once you've done it for a while, you start to notice other cachers by the way they are awkwardly standing in unusual places trying to look inconspicuous.

Toddler related as well. Wanted to slam his face into the edge of the sink until the screaming stops. It was pretty close a few times.

5 more...

Nope nopedi nope.

Metube might be right for you.

As proven by the fact that dogs exist. I bet wolves were constantly itchy as well, but receiving those excellent scritches over millennia, they turned into the gentle doggos that we know today. And now I can have one of these 30kg predators in my house, trusting me with their life and sleeping on me occasionally and not mauling me with their sharp teeth, just because of the power of scritches.

2 more...

I, too, share your hatred for horses. They are arrogant fucks who think they are better then everyone else. One exception: there's these large horses with fluffy hooves and fat asses that seem to be chill and more like large dogs. You're OK.

Clearly true. And yes, he IS a little shit.

  • I use bash, because I never had the time to learn anything else.
  • Like @jlsalvador@lemmy.ml said, I use the #!/usr/bin/env bash shebang.
  • Nope
  • Also nope
  • Nope. Shell scripts reside in Git repos on Gitlab/Gitea/Forgejo and are checked out using Ansible playbooks onto the servers as necessary.
  • For scripts? Python. Read this blog post by the great @isotopp@chaos.social. For interactive use? bash is just fine for me, though I've customized it using Starship and created some aliases to have colored/pretty output where possible.
  • Use shellcheck before running your scripts in production, err on the side of caution, set -o pipefail. There are best practices guides for Bash, use those and you'll probably be fine.
  • Be prepared to shave yaks. Take breaks, touch grass, pet a dog. Use set -x inside your Bash script or bash -x scriptname on the CLI for debugging. Remember that you can always fallback to interactive CLI to test/prepare commands before you put them into your script. Think before you type. Test. Optimize only what needs optimization. Use long options for readability. And remember: Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows your address.
  • Nope, it's absolutely not bad practice to create aliases to save you some typing in interactive shell. You shouldn't use them inside your scripts though, because they might/will not be available in other environments.

I switched to fish because it has tab completion Yeah, so does Bash, just install it.

Oh, I also "curate" a list of Linux tools that I like, that are more modern alternatives to "traditional" Linux tools or that provide information I would otherwise not easily get. I'll post i

::: spoiler Tools Debian-Packages available

  • mtr

  • iputils-tracepath

  • iproute2

  • zsh

  • httpie

  • aria2

  • icdiff

  • progress

  • diffoscope

  • atop

  • powertop

  • ntopng

  • ethtool

  • nethogs

  • vnstat

  • ss

  • glances

  • discus

  • dstat

  • logwatch

  • swatch

  • multitail

  • lynis

  • ncdu (du-clone), alias du="ncdu --color dark -rr -x --exclude .git --exclude node_modules"

  • nnn (fully-featured terminal file manager. It’s tiny, nearly 0-config and incredibly fast. https://github.com/jarun/nnn)

  • slurm

  • calcurse

  • newsbeuter

  • tig ("ncurses TUI for git. It’s great for reviewing and staging changes, viewing history and diffs.")

  • qalc -ttyrec

  • taskwarrior

  • ttytter

  • ranger

  • ipcalc

  • pandoc

  • moreutils

  • googler

  • weechat

  • pdftk

  • abcde

  • dtrx

  • tload

  • ttyload

  • cockpit

  • sar

  • ht (hte Hex Editor)

  • dhex

  • ack (grep-clone)

  • silversearcher-ag (grep-clone)

  • ripgrep ("recursively searches file trees for content in files matching a regular expression. It’s extremely fast, and respects ignore files and binary files by default.", https://github.com/BurntSushi/ripgrep)

  • exa (statt ls) https://the.exa.website/ ("replacement for ls with sensible defaults and added features like a tree view, git integration, and optional icons.")

  • fzf (CLI fuzzy finder), alias preview="fzf --preview 'bat --color "always" {}'"

  • fd (simple, fast and user-friendly alternative to 'find', https://github.com/sharkdp/fd) -entr (watch-clone)

  • csvkit (awk-clone)

  • ccze (log coloring)

  • surfraw -hexyl ("hex viewer that uses Unicode characters and colour", https://github.com/sharkdp/hexyl) -jq ("awk for JSON. It lets you transform and extract information from JSON documents", https://stedolan.github.io/jq/) -pass ("password manager that uses GPG to store the passwords", https://github.com/lunaryorn/mdcat)

  • restic ("backup tool that performs client side encryption, de-duplication and supports a variety of local and remote storage backends.", https://restic.net/)

  • mdp (Markdown Presentation on CLI) -grepcidr

  • qrencode

  • caca-utils (show images on the CLI)

  • fbi ( & fbgs) (show images in Framebuffer device)

  • fbcat (take screnshot on framebuffer device)

  • nmap

  • micro (CLI Text Editor, ab Debian 11, https://micro-editor.github.io)

  • masscan (https://github.com/robertdavidgraham/masscan)

  • socat (Nachfolger von netcat, https://www.heise.de/select/ix/2017/11/1509815804306324)

  • dc3dd (patched version of GNU dd with added features for computer forensics)

  • smem (memory reporting tool)

  • free (Show Linux server memory usage)

  • mpstat (Monitor multiprocessor usage on Linux, part of sysstat package)

  • pmap (Montor process memory usage on Linux, part of the procps)

  • monit (Process supervision)

  • oping & noping

  • saidar (Curses-basiertes Programm für die Anzeige von Live-Systemstatistiken)

  • reptyr (Tool for moving running programs between ptys)

  • gron (https://github.com/tomnomnom/gron, makes JSON greppable, kann HTTP-Requests absetzen)

  • jc (https://github.com/kellyjonbrazil/jc, CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.)

  • bat (cat-clone), alias cat='bat' ("alternative to the common (mis)use of cat to print a file to the terminal. It supports syntax highlighting and - git integration.", https://github.com/sharkdp/bat)

  • ioping (https://github.com/koct9i/ioping, simple disk I/0 latency measuring tool, auch für disk seek rate/iops/avg)

  • vd (Visidata, multipurpose terminal utility for exploring, cleaning, restructuring and analysing tabular data. Current supported sources are TSV, CSV, fixed-width text, JSON, SQLite, HTTP, HTML, .xls, and .xlsx)

  • pdfgrep

  • duf https://github.com/muesli/duf (combined df and du, ncurses-based)

  • nala (apt-alternate, https://gitlab.com/volian/nala, https://christitus.com/stop-using-apt/)

  • iprange

  • tldr

  • rmlint

  • nvtop (https://github.com/Syllo/nvtop, GPUs process monitoring for AMD, Intel and NVIDIA)

  • lf (lf (as in “list files”) is a terminal file manager written in Go with a heavy inspiration from ranger file manager)

no Deb pkg avail

3 more...

Nah, that can't be right, I'm pretty sure it was them sweet sweet scritches. I asked, former wolf agrees.

I have limited Python experience, but I always thought that's what virtualenvs and requirements.txt files are for? When I used those, I found it easy enough to use.

I'm surprised nobody said wank yet.

1 more...

Very, VERY true.

+1 for gpodder, and of you have a Nextcloud, you can sync your subscriptions using an NC plugin that I can't remember the name of.

2 more...

If my wife didn't have an abortion, she would be fucking dead by now, so how would that even be a consideration that a sane person would make? Does an abortion make a woman less of a person? No. But does it make her less loveable? Also no. But is it a red flag that she had sex before, maybe even with a different person? No, and fuck patriarchy.

2 more...

I always assumed whale would be too fatty.

Ken Shirrif's Blog is incredibly great!

If you have a network share available on your LAN, you might want to try the FolderSync App. It can make your phone sync its photos every time you're in your WiFi and plugged into the charger.

Alternatively, if you have NextCloud, the NextCloud App can do that for you.

+1 for Ben Eater. There will be things you won't understand, but for those at least you get a good starting point for reading them up.

EDIT: Main thing for me was to watch them when I was rested and could concentrate, and I didn't binge them in one sitting; these are probably best consumed when they're not just running in the background.

Rest of the list:

::: spoiler Tools pt. 2

DNS tools:

Good stuff for pentesters and security researchers:

  • contained.af
  • cryptohack.org
  • 0x00sec.org
  • hack.me
  • chall.stypr.com
  • crackmes.one
  • hackxor.net
  • tryhackme.com
  • ctftime.org
  • ctflearn.com
  • picoctf.org
### .bashrc
### CUSTOM FUNCTIONS
# https://www.linuxjournal.com/content/boost-productivity-bash-tips-and-tricks
ftext () {
        grep -iIHrn --color=always "$1" . | less -R -r
}
duplicatefind (){
        find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | \
                xargs -I{} -n1 find -type f -size {}c -print0 | \
                xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
}
generateqr (){
        # printf "$@" | curl -F-=\<- qrenco.de
        printf "$@" | qrencode -t UTF8 -o -
}

:::

237216938 logging off.

I love the clock, but it doesn't seem to be part of the launcher; at least I couldn't find it after installing the launcher. Where can I find it?

EDIT: I just realized you're running GrapheneOS. Does the clock come with Graphene? Nice background pic btw!

Uuuuh, weird, I love it!

Cloud-init. The config yaml is rather straight forward, but I can't convince my VM to execute it, and it's driving me nuts.

Good to hear! When you go with the National Archives UK, you can't fail. They have some very, VERY competent people in staff over there, who are also quite active in the DigiPres community. They are also the inventors of DROID and the maintainers of the widely used PRONOM database of file formats. https://www.nationalarchives.gov.uk/PRONOM/Default.aspx Absolute heroes of Digital Preservation.

Yeah, you can always go crazy with (off site) copies. There's a DigiPres software system literally called LOCKSS (Lots Of Copies Keep Stuff Safe).

The German Federal Office for Information Security recommends a distance of at least 200km between (professional) sites that keep georedundant copies of the same data/service, so depending on your upload capacity and your familiarity with encryption (ALWAYS backup your keys!), some cloud storage provider might even be a viable option to create a second site.

Spare drives do absolutely work as well, but remember that, depending on the distance, data there will get more or less outdated and you might not remember to refresh the hardware in a timely manner.

A safe deposit box is something that I hadn't considered for my personal preservation needs yet, but sounds like a good idea as well.

Whatever you use, also remember to read back data from all copies regularly and recalculate checksums for fixity checks to make sure your data doesn't get corrupted over time. Physical objects (like books) decay slowly over time, digital objects break more spontaneously and often catastrophically.

That's some proper fur, I love it!

Neither does mine, but, I keep it to test a new tool from time to time.

Oh absolutely. I got my first smartphone when I was 20 years or so, so you can imagine what hitting puberty was like.

Thx kind stranger.

"Effi Briest" all the way.

Yes, such a program is called an installer. /s

Sorry, I don't have an answer for you that's more helpful than the rest of the comments here, they all did well. I second booting a live system.