What are the recommended scripting languages for complex shell scripts beyond bash?

CoderSupreme@programming.dev to Programming@programming.dev – 101 points –

I've been struggling with a rather complex shell script, and it's becoming apparent that Bash might not be the best choice for this particular task. While I usually gravitate towards statically typed languages like Go or Rust, I've noticed that many people recommend alternative languages such as Lua or Python for scripting tasks.

I'm curious to know your opinions and experiences with scripting languages for larger or more intricate shell scripts. Have you ever encountered a situation where Bash just didn't cut it, and if so, which scripting languages did you turn to for a more effective solution? Are there any specific languages you found particularly suitable for debugging, testing, or handling complex logic in your shell scripts?

84

When I'm doing too much to maintain in bash, and not enough to merit Python, I use PowerShell.

My controversial take: if you're looking for a better scripting language and haven't tried PowerShell, you should give it a try.

It's weird that Microsoft made a real shell.

PowerShell is actually open source, and it runs everywhere, including Mac and Linux. On Windows and Ubuntu, it's already installed.

Powershell's quality JSON and CSV handling is a huge game changer for quick scripts. The webrequest module is high quality. File operations are a breeze. Unlike bash, PowerShell can be formated to be pretty readable, when you care. Environment variable handling is mildly improved. Resusable code via modules is huge for quality of life.

PowerShell is the bash rewrite with lessons learned we all have wanted, but it's not on a lot Linux folks radar because Microsoft published it.

I’m trying to understand where you’d want to use PowerShell over Python. What’s something that’s in-between bash and Python?

Good question.

I choose PowerShell over Python when I need to call out to an existing command line utility, because I find the Python subprocess module is a huge pain in the ass.

PowerShell has about 80% of the power and readability of Python, while actually being a native shell.

Yeah, that’s fair. I was wondering if you’d call that out. Popen is rather opaque. I don’t know that I’d go so far as to try to remember yet another language to avoid it. I respect the decision though, especially with the portability of modern PowerShell.

I think you’d like the alternatives that mix languages with Shell. See Google’s zx or (shameless plug) my pysh for instance.

This 100%. I liked bash scripting when I was in college. Took some time to actually learn powershell, and it's been amazing. Steep learning curve in the beginning, but it's worth it

Edit: I had also meant to mention the fact that powershell definitely feels a little similar to bash. I mean, not quite the same. Powershell is more object oriented, whereas bash is more text oriented. Powershell for structured data is nice.

not to mention that you have all the dotnet ecosystem at your fingertips. Wanna write a WPF application? Go ahead. You want to use ML.net? A bit clunky but doable.

Couldn't agree more. It's a great shell and scripting language. It's object-oriented nature, native support for virtually every text format (csv, json, xml) and great libraries for others (yaml, excel), awesome regex and web/rest services support... it's hard to beat and works on virtually every platform.

Too few people in the Linux community will even look at it though since it has MS name on it.

For years my go to (after Bash) was Python. However, in the last few years I’ve switched to Rust for any kind of shell command wrapper or CLI tool.

TL;DR I think Rust is best suited to more complex CLI work.

Exact same story here. Bash -> Python -> Rust.

Generally speaking, people should settle on a compiled language if they can. They can iterate as fast as interpreted languages these days.

Edit: If you want to try something different in scripting, try the execline language. Its interpreter processes the script and exits immediately even before the script execution begins. Traditional shell interpreters (like bash) stay active till the entire script is finished. Execline achieves this by a clever chaining of Unix execs, forks and variable substitutions. This makes execline scripts lighter (useful in embedded systems), more secure and less error-prone than traditional scripts. The downside is that writing them will feel a bit weird - since the fundamental paradigm is different from regular shells. However, that will be a refreshing change if you're someone who likes to experiment and try new things.

Yes, and Rust with incremental compilation is pretty fast to iterate as well, as long as you don't use massive libraries/build-scripts etc.

Would you have any example (not necessarily yours) to showcase this? I mean, how is it better suited than say, C++?

Not op, but I feel the same as them.

Compared to C++, Rust has a very good toolchain and libraries. With C++ setting up a project that has dependencies is... painful. I'm a full-time C++ programmer with over 8 years of experience and if I didn't have to, I would never choose it for something new.

With Rust creating a new project and adding dependencies is trivial. There are a lot of great libraries and the ease with which you can use them is very empowering.
Clap and serde are super powers for CLI programs 😀

For smaller scripts that don't yet "deserve" full rust treatment, I now use nushell for personal projects.

I too use Rust for what normal people use shell scripts for. But I have a feeling that Rust is falling into the same trap that other languages with similar easy dependency management fall into (Python and NPM are good examples). You end up with a dozen direct dependencies and hundreds of indirect ones with dozens of levels of hierarchy. C and C++ programs have fewer dependencies because each additional one adds more headache for the developer. Drew Devault's Hare language is giving language repo and package manager a skip for the same reasons. And I'm starting to think that he may have a point.

I think it's not that bad yet, when comparing with npm. Usually the dependencies I use are of very high quality. But I'm also very selective with dependencies. I'm rather writing a simple part myself, than using a not-really maintained low-quality dependency...

Btw. I have not looked into the Hare language yet (will do that now), but if it's similar as deno, I won't like it. You want to have some kind of package management IME...

Drew Devault’s Hare language

Ok, they say "use your distros package-manager", that's basically asking for the same disaster as C or C++. I think cargo is one of the selling points of Rust.

At least say something like we use "Nix" for default package-management (which does a lot of things right)...

I personally don’t have any real experience with Go. Lots of smart folks I work with love it. In general, most of what I have read suggests that Rust is better suited to CLI tooling. For my use case it came down to:

  • Rust’s cargo system
  • The clap crate (which supports building out bash shell completion scrips via a Rust build script). Basically means I can generate a completion script at compile time and include this in the package I distribute to users)
  • Rust’s out of the box performance
  • The heavy lifting done by the borrow checker in bringing safety

Just curious have you tried Go for this? Go was recently approved at work and I have seen articles about Go for things like this and just wondering if it is worth it. I have been using ansible and chef but need to explore other options. I want to use Rust but I know the road blocks I will have to work through at work. So just wondering if you had any insights to Go over Rust

Not him, but I much more like the type-system of rust (e.g. enums).

It's especially true when you want to parse some json/xml/whatever. Just describe your datastuctures with regular struct and enum, add serde and done! It's like magic!

If we remove words “serde” and “enum”, no one will be able to guess whether the argument is for rust or golang.

I personally don’t have any real experience with Go. Lots of smart folks I work with love it. In general, most of what I have read suggests that Rust is better suited to CLI tooling. For my use case it came down to:

  • Rust’s cargo system
  • The clap crate (which supports building out bash shell completion scrips via a Rust build script. Basically means I can generate a completion script at compile time and include this in the package I distribute to users)
  • Rust’s out of the box performance
  • The heavy lifting done by the borrow checker in bringing safety

for larger or more intricate shell scripts

Those are call applications. Use any language you like. If go/rust is what you know use them. I use rust all the time for things beyond run a bunch of commands and tends to be my go to when I need to process data in any way.

If the script gets too large or you think it will, write a full program.
With languages like python or go its not that hard either and you get alot of benifits aswell.

Other than that, use whatever languwge works and you know.

Unpopular opinion: I'm old school and would probably use perl.

This is the correct answer. Perl is shell-like with support for advanced data structures and data parsing capabilities. Modern Perl is very slick, especially with the new object system.

Modern Perl

Perl or Raku?

Both are good and they each have their uses.

Perl is very Unix-y, recent releases have a very good object system, and Perl is quite fast but the syntax can take some getting used to. CPAN is a huge database of Perl modules, you'll likely find what you need module wise.

Raku is amazingly flexible and I like its object and type systems more than other languages. The only only down side is compared Perl is that Raku on the slow side, even Python is faster at the moment. Raku has a much more consistent syntax than Perl but the module ecosystem is nowhere near as big.

I'd say try both and use what seems to be the most optomal for whatever task you're dealing with. Personally, I use both for quick scripts about equally with performance and module availability usually being the deciding factors.

--

I do most of my scripting in perl too. Python has always irked me.

Only semi-related: I recently switched from bash shell to fish. I should have don't that years ago. Don't know why I held on to bash so tight.

What do you like about fish? Have you tried zsh? I’m in the market for a new shell too.

I'm not the person to say anything about zsh vs. fish. I last tried zsh around 2008. Back then I decided to stick with bash over other shells. At the time (and for decades earlier) it was clear that sh was inadequate. So the Bourne Again Shell was (and still is) ubiquitous. Other shells fighting for user space seemed like the xkcd-927 problem.

Now, I'm basically seeing that if I'm stuck without my .bashrc file, installing fish gives me most of the niceties I like. It also gives me niceties that I wouldn't have been able to do, like nice multi-lining. And while I personally find fi and esac charming, they are pretty dumb. And I certainly don't miss the for-do-done construction.

I bet I would like zsh as much as fish, but I don't have any motivation to try it.

fish looks at my path to highlight mistyped commands. It autocompletes on the fly and autocompletes with attention paid to the usage in your history. The coolest thing is that it parses man pages. That allows autocomplete to know the options of a command as long as it has a proper man page (which it just should).

My personal favourite remains perl for anything text oriented or for simple-ish orchestration.

I love Raku (formerly perl 6) for this purpose as well.

I was in a similar situation not too long ago.

My criteria for another scripting language included that it should be preinstalled on all target systems (i. e. Debian and Fedora), it should be an interpreted language and it needs to have type safety.

Afterall I settled with Python due to its popularity, its syntax and features (type safety since v3.6, etc.) and the fact that it is preinstalled on many Linux distributions. System components often use Python as well, which means that libraries to interact with the system tend to be included by default.

pyinvoke.

You can create quick and dirty CLIs, invoke shell commands, and have all of python available for things like parsing config files, getting and setting environment variables, and making remote REST calls.

Totally second this.

But I 'd like to recommend starting with subprocess module which is built-in. Then go ahead and see what the extensions have to offer... There are other that may come handy, like Typer from Tiangolo, and Fire from Google.

Personally, I don't feel like it's worth learning a separate scripting language when you're comfortable with a full-fledged programming language.

Python, Lua etc. used to be on a whole different level of usability, when compared to C. But compared to modern, high-level languages, the difference is marginal. Not to mention that at least compared to Rust, they start to look rather antique, too, and lack in robust tooling.

If you don't feel like maintaining a whole git repo, use e.g. rust-script.

A shell script can be more concise if you're doing a lot of shell things. Keeps you from having os.system() all over the place.

Things like "diff the output of two programs" are just more complex in other languages.

I love rust, but replacing my shell scripts with rust is not something I would consider doing any more than I'd consider replacing rust with my shell scripts.

Oh, I didn't mean to say, you should throw out your shell scripts. For anything less than, say, 20 lines, they're perfectly appropriate.

I'm saying, Rust et al start to feel like a good choice from, say, 100 lines upwards, and I just don't think, it's worth bridging the gap between those two.

In particular, you can build a function that allows you to run commands without much boilerplate, e.g.: run("echo hello | tee out.txt");
(The implementation just appends that argument to Command::new("sh").arg("-c") and runs it.)

That way, you can do the more complex things in Rust, whether that's control flow or something like modifying a JSON file, without giving up the utility of all the CLI tools on your system...

Somewhat of a weird addendum, but I actually only realized, you could port directly over like that, while writing the above comment.

Now I actually tried it on a 22 lines long shell script that I've been struggling with, and holy crap, I love it.

Like, I should say that I have always been (and likely will always be) shit at shell scripting. Any time I wanted to do a basic if, I had to look up how that works.
As a result, even those 22 lines were ripe with code duplication and I always felt really unsure about what will actually happen during execution.

Well, rightfully so. While porting over, I realized I had a bug in there, which has been annoying me for a while, but I always thought, well, it is a shitty shell script. I still remember thinking, I should probably not implement it like that, but then leaving it anyways, because I felt it would become unreadable with the shell syntax.

Now it actually feels maintainable, like I can even easily expand on it.
And I have to say that rust-script is really smooth. I barely notice that it's compiling during the first run after changing the script file, and it's fully cached afterwards, so it executes instantly.

I'll still have to check for libraries that basically provide such a run() function/macro for me, but yeah, basically my threshold for not using shell scripts just dropped to any kind of control flow being involved.

Yeah the strict type-system of Rust is great at finding issues.

I think when understanding, that bash is basically only programs with parameters ([ is a program that takes all kinds of parameters and as last parameter ]) then bash is quite ok for stuff that doesn't need a lot of algorithms, i.e. passing the in and out from one program to another. But as soon as there's basic logic, You'll want to use a fully-fledged programming language.

Also the maintainability aspect: You can just start using fancy stuff you never want to use in bash and it can slowly grow into a library or application or something like that.

Btw. I have started a syntax-sugar library/crate that creates typing information for all kinds of programs via the builder-type-state-pattern, so that you don't always have to look up man etc. and that it should be more convenient to execute programs (not open sourced yet, and low priority for me as I'm working on various other exciting projects currently)

Yeah as weird as it sounds to use a "low"-level systems programming language such as Rust. Rust works surprisingly well as "script" language. (And you don't have to deal with the ugliness of bash, admittedly though, that bash is quite a bit more concise when using a lot of program executions and piping the results etc.)

I make bash scripts to automate the configuration of new servers. Stuff like install packages, create users, create groups, configure the database, manage permissions...

I feel like that sort of stuff would be a nightmare to do in high level languages but maybe I'm just too used to bash.

As I already responded to others, my comment was meant in the context of the question, so I would not learn a scripting language in addition to Bash + a programming language.

For just running commands one-after-another, Bash is basically a minimal encoding, so no reason not to use it.

When you do start to need if-elses, loops etc., that's where Bash starts to become somewhat difficult to read. And personally, as someone who's not fluent in Bash control flow, I found it quite useful to do the control flow in my programming language of choice, but still just calling commands like you'd do in Bash.

Of course, this is a non-standard setup, and most target hosts will have Bash pre-installed, not rust-script, so it does obviously make a lot of sense to continue using Bash for what you're doing.
In general, my comment was meant for programmers. An ops person might know a full-fledged programming language, but still want to learn Python, because they need to write tons of Ansible tasks or whatever.

Look at Raku. This seems to be where Raku shines relatively brightly.

I think vanilla perl is also worth looking at if you need the script to be portable

What are you trying to achieve in Bash that you’re struggling with? It’s hard to suggest alternatives without knowing your objectives since languages excel in different areas.

Temporary files management for a ~/tmp folder, with archiving and cleanup after x and y days.

The expression syntax for the GNU find command is very powerful. I would expect that it is up to the task. If you don't have the GNU find command with it's extensions I could see how it's would be difficult.

As @damium@programming.dev says you may be able to do this with find command. This command lists all PDF files under ~/tmp that were created more than 7 days ago and does a directory listing. You could use this as a basis to move create an archive of individual files.

find ~/tmp -ctime +7 -iname "*pdf" -exec ls -rlht {} \;

The find command also has a -delete flag.

I have in the past used this combination to implement file management. I don't have access to the script any more. I don't remember why we used a shell script rather than logrotate as per @oddityoverseer@lemmy.world

I usually use Awk to do the heavy lifting within my Bash scripts (e.g. arg parsing, filtering, stream transformation), or I'll embed a Node.JS script for anything more advanced. In some cases, I'll use eval to process generated bash syntax, or I'll pipe into sh (which can be a good way to set up multiprocessing). I've also wanted to try zx, but I generally just stick to inlining since it saves a dependency.

Whatever works for you. I use C# for work so if there's something I need to do that's more complex than I know to do in bash I'll just use that.

I've used C# in the past. There's a tool called dotnet-script that lets you run csharp files as scripts without having to compile.

It does require you to install dotnet though.

But tbh it's pretty easy to just chuck an executable together.

For raster graphics image processing, I'd highly recommend G'MIC. Otherwise, Python and especially for string using regex library. I wish there was a vector graphics version of G'MIC.

I still reach for Perl when bash isn't enough.

Same. I'm sure python, rust, and all the others are better/cooler/vegan/whatever but perl is what I'm fluent in. More than once have I started to hack together something in python, only to scrap it and start over in perl because I can get it done so much faster. Trust me, my hourly cronjob doesn't care that it takes half a second more to run. And the UPS doesn't care that it takes 1mW more to run it. But I care a lot about not dicking around with documentation just to figure out what is pythonic and what isn't when a shitty perl oneliner will do just fine.

Same. I work primarily in Go and a little bit of Rust these days, but I still throw together a Perl script every year or two to automate something without needing to install something else on the machine or whatnot.

Myself I know Bash and I know Go.

I would say it's Python, so I use ChatGPT to code it for me if bash is not enough. If something that I can call "project" instead of "script" - I just use Go for it.

I use Go a lot for this task, because it's language I know well and can get the job done. And with Go you get a single binary, which is just as easy to deploy as a shell script.

I almost always use PowerShell (Core) for automation/scripting things that don't warrant an entire "application". It's as powerful as you need it to be, but I wouldn't recommend it if you aren't already familiar with .NET and its ecosystem.

I ran into this exact situation at work - though for me it was more the case that getting approvals for new software / installing new dependencies in our system is a massive pain.

So I went with Python since it's already installed on basically any Linux system. It was fine - I mean Python is a good language and can certainly handle string processing and data manipulation with relative ease.

I still think the Python docs are pretty bad, and I wasn't thrilled with the options for calling a subprocess in Python - they all felt kinda clunky, though I was barred from using the newest versions since I had to run an older version of Python.

But I ultimately got something that worked and it was certainly better executed / shorter than the bash equivalent it was replacing.

I'm mostly programming C# at my job (and private too now), and am quite fond of the language and ecosystem/basis. I tend to create even small utilities and one-off things with it. (Since C# 9 top level statements allow you to create the program entry point without a Main method - it becomes implicit/generated.)

I also find nushell quite interesting. It provides both a shell and scripting interface, and some promising concepts. I still have difficulties creating more complex command queries and scripts because the syntax is still foreign to me - and because you fall back to other tools when there's a need.

If compatibility is a concern it's still back to bash, pwsh, bat. They have functions and files too, so at least some sectioning and factoring is possible.

As an android developer working with GitHub actions I just use Kotlin Script: but that's because I'm already writing in Kotlin all the time.

It's really nice to be able to run it as a scripting language though

Really surprised no one has mentioned Ruby. It's installed by default on almost every system out there (unlike python), it will have the same features on every platform (unlike python where you might get 2.7 or 3.x depending. It's simple and easy to read, and only slightly more verbose than bash. It's very well suited for scripting (please don't use it for application work). It also took a lot of its design from Perl, which a bunch of people are mentioning in this thread, and as a result has a ton of the features of perl, along with a ton of features from other languages. Rust is heavily based on Ruby's design, and i've used Rust to create cli programs and I wouldn't recommend it. It's good, but most cli programs don't need the difficulty of rust for the benefits that rust gives.

Anyway, python has a really really good cli library called Click, but that's about the only good thing about it. If you are looking to use this script on multiple systems then Ruby will be much easier to transfer between systems (it will just work). I've deployed complex python, rust, and ruby CLIs across an org and Ruby was the only easy one. Rust was second easiest, Python absolutely terrible.

If you're not deploying this to other platforms or sharing it across a team or something like that then a lot of the downsides and upsides here don't really matter. Just use the easiest language.

Aw man, you can't write all that and then not give an example!

Ruby makes scripting drop-dead simple. You can run any shell command by surrounding it with back ticks.

# simple example, just grab files: 
files = `ls`.split("\n")

# pipes work inside back ticks
files.map {|f| `cat #{f} | grep "can I use grep w/out cat"`}
  .compact
  .each { |match| puts match }
# easy to build a pipeline on the data in ruby, too!

That's it! No messing around with popen3, or figuring out pipes or signals. Those are there too if you really need them, but if you just wanna write a quick script with a less arcane syntax - try Ruby!

Haha sorry, I wrote it all on my phone while traveling. and yeah, if you're running just shell commands it looks almost the exact same as a bash script, and then when you need actual scripting capabilities you get them.

Rust is heavily based on Ruby’s design

I would not say "heavily based". Literally only the closure/lambda syntax, which is cosmetic. Rust is mainly inspired by ML-family languages and C++.

I think Ruby is a reasonable choice for small scripts which someone might otherwise use Python for. But Rust is very well suited to more complicated or long-lasting command-line tools, especially if performance is at all a concern. Clap alone is super nice, but there are a lot of awesome libraries for making rich CLI tools easily.

And like....a hundred more I could mention. Idk, for anything that's not completely trivial, which will be used and maintained by humans and not thrown away, Rust is really nice.

I would not say “heavily based”. Literally only the closure/lambda syntax, which is cosmetic. Rust is mainly inspired by ML-family languages and C++.

All of Cargo is based on (and created by) the same person that created bundler for ruby. That list also misses out on a lot of things, like !, automatic returns, honestly most of the actual language 'design' rather than the internals (that seems to be a list of where the architects got their ideas for internal implementation as well, rather than just the readability of the language).

But Rust is very well suited to more complicated or long-lasting command-line tools, especially if performance is at all a concern. Clap alone is super nice, but there are a lot of awesome libraries for making rich CLI tools easily.

I disagree. Like I said, I wrote command line apps in all of these, performance was a factor. But a much larger factor is getting other devs on your team to contribute. And that was just absolutely impossible with Rust. The learning curve is just too high. For something that isn't a hobby project, but that you might need a team member to roll out a fix in just a few hours, Rust will not cut it.

Yes, you will have way more bugs in all the other programs, but honestly I had a shit ton of bugs in my rust cli as well, because, it turns out, rust works really well when it has control over everything, but man does it suffer when you have to interface with the real world.. And oh boy did that make it incredibly difficult to write. Like I said, I deployed CLIs in all three of these languages. Ruby was the easiest of them all. Not just in development, but also maintenance.

i would say lua if possible, but python has more libraries

I'm curious on why you would recommend Lua if available. Is language speed a typical limiting factor for your scripts?

I'm mostly asking since when I was forced to use the language it was nothing but annoyance. The 1 indexing and "if then/for do" will annoy me till my grave.

just for simple tasks, where i need something between the simplicity of bash and the type safety of rust. its just my personal preference when it comes to scripting, although i do believe it should be the first consideration for choosing a user friendly but powerful embedded or config language, like for neovim, especially if performance is a concern, but it will ofc not always be the best option. the 1 index is certainly annoying though, but i would personally rather that than anything to do with python, especially whitespace. the if...then and for...do is the same as bash, so i dont think its that bad

Maybe it's becuase i grew up on python and bash still seems somewhat alien to me, but any time I'm crafting something with more than three or so nested functions, I use python as a wrapper for bash. Python initiates a bash script, parses the retval, inititates the next bash script from that data, etc.

It's really unfortunate, that the interaction between Bash and Python is so cumbersome. I would really like a simple Bash-class to invoke simple command strings directly.

It's pretty close already. I forget where I cribbed the technique from, but I embed python functions into my scripts very often...

E.g. see here