OILS / doc / idioms.md View on Github | oils.pub

1002 lines, 633 significant
1---
2default_highlighter: oils-sh
3---
4
5YSH vs. Shell Idioms
6====================
7
8This is an informal, lightly-organized list of recommended idioms for the
9[YSH]($xref) language. Each section has snippets labeled *No* and *Yes*.
10
11- Use the *Yes* style when you want to write in YSH, and don't care about
12 compatibility with other shells.
13- The *No* style is discouraged in new code, but YSH will run it. The [OSH
14 language]($xref:osh-language) is compatible with
15 [POSIX]($xref:posix-shell-spec) and [bash]($xref).
16
17[J8 Notation]: j8-notation.html
18
19<!-- cmark.py expands this -->
20<div id="toc">
21</div>
22
23## Use [Simple Word Evaluation](simple-word-eval.html) to Avoid "Quoting Hell"
24
25### Substitute Variables
26
27No:
28
29 local x='my song.mp3'
30 ls "$x" # quotes required to avoid mangling
31
32Yes:
33
34 var x = 'my song.mp3'
35 ls $x # no quotes needed
36
37### Splice Arrays
38
39No:
40
41 local myflags=( --all --long )
42 ls "${myflags[@]}" "$@"
43
44Yes:
45
46 var myflags = :| --all --long |
47 ls @myflags @ARGV
48
49### Explicitly Split, Glob, and Omit Empty Args
50
51YSH doesn't split arguments after variable expansion.
52
53No:
54
55 local packages='python-dev gawk'
56 apt install $packages
57
58Yes:
59
60 var packages = 'python-dev gawk'
61 apt install @[split(packages)]
62
63Even better:
64
65 var packages = :| python-dev gawk | # array literal
66 apt install @packages # splice array
67
68---
69
70YSH doesn't glob after variable expansion.
71
72No:
73
74 local pat='*.py'
75 echo $pat
76
77
78Yes:
79
80 var pat = '*.py'
81 echo @[glob(pat)] # explicit call
82
83---
84
85YSH doesn't omit unquoted words that evaluate to the empty string.
86
87No:
88
89 local e=''
90 cp $e other $dest # cp gets 2 args, not 3, in sh
91
92Yes:
93
94 var e = ''
95 cp @[maybe(e)] other $dest # explicit call
96
97### Iterate a Number of Times (Split Command Sub)
98
99No:
100
101 local n=3
102 for x in $(seq $n); do # No implicit splitting of unquoted words in YSH
103 echo $x
104 done
105
106OK:
107
108 var n = 3
109 for x in @(seq $n) { # Explicit splitting
110 echo $x
111 }
112
113Better;
114
115 var n = 3
116 for x in (1 .. n+1) { # Range, avoids external program
117 echo $x
118 }
119
120Note that `{1..3}` works in bash and YSH, but the numbers must be constant.
121
122## Avoid Ad Hoc Parsing and Splitting
123
124In other words, avoid *groveling through backslashes and spaces* in shell.
125
126Instead, emit and consume [J8 Notation]($xref:j8-notation):
127
128- J8 strings are [JSON]($xref) strings, with an upgrade for byte string
129 literals
130- [JSON8]($xref) is [JSON]($xref), with this same upgrade
131- [TSV8]($xref) is TSV with this upgrade (not yet implemented)
132
133Custom parsing and serializing should be limited to "the edges" of your YSH
134programs.
135
136### More Strategies For Structured Data
137
138- **Wrap** and Adapt External Tools. Parse their output, and emit [J8 Notation][].
139 - These can be one-off, "bespoke" wrappers in your program, or maintained
140 programs. Use the `proc` construct and `flagspec`!
141 - Example: [uxy](https://github.com/sustrik/uxy) wrappers.
142 - TODO: Examples written in YSH and in other languages.
143- **Patch** Existing Tools.
144 - Enhance GNU grep, etc. to emit [J8 Notation][]. Add a
145 `--j8` flag.
146- **Write Your Own** Structured Versions.
147 - For example, you can write a structured subset of `ls` in Python with
148 little effort.
149
150<!--
151 ls -q and -Q already exist, but --j8 or --tsv8 is probably fine
152-->
153
154## The `write` Builtin Is Simpler Than `printf` and `echo`
155
156### Write an Arbitrary Line
157
158No:
159
160 printf '%s\n' "$mystr"
161
162Yes:
163
164 write -- $mystr
165
166The `write` builtin accepts `--` so it doesn't confuse flags and args.
167
168### Write Without a Newline
169
170No:
171
172 echo -n "$mystr" # breaks if mystr is -e
173
174Yes:
175
176 write --end '' -- $mystr
177 write -n -- $mystr # -n is an alias for --end ''
178
179### Write an Array of Lines
180
181 var myarray = :| one two three |
182 write -- @myarray
183
184## New Long Flags on the `read` builtin
185
186### Read a Line
187
188No:
189
190 read line # Mangles your backslashes!
191
192Better:
193
194 read -r line # Still messes with leading and trailing whitespace
195
196 IFS= read -r line # OK, but doesn't work in YSH
197
198Yes:
199
200 read --raw-line # Gives you the line, without trailing \n
201
202(Note that `read --raw-line` is still an unbuffered read, which means it slowly
203reads a byte at a time. We plan to add buffered reads as well.)
204
205### Read a Whole File
206
207No:
208
209 read -d '' # harder to read, easy to forget -r
210
211Yes:
212
213 read --all # sets $_reply
214 read --all (&myvar) # sets $myvar
215
216### Read Lines of a File
217
218No:
219
220 # The IFS= idiom doesn't work in YSH, because of dynamic scope!
221 while IFS= read -r line; do
222 echo $line
223 done
224
225Yes:
226
227 while read --raw-line {
228 echo $_reply
229 }
230 # this reads a byte at a time, unbuffered, like shell
231
232Yes:
233
234 for line in (io.stdin) {
235 echo $line
236 }
237 # this reads buffered lines, which is much faster
238
239### Read a Number of Bytes
240
241No:
242
243 read -n 3 # slow because it respects -d delim
244 # also strips whitespace
245
246Better:
247
248 read -N 3 # good behavior, but easily confused with -n
249
250Yes:
251
252 read --num-bytes 3 # sets $_reply
253 read --num-bytes 3 (&myvar) # sets $myvar
254
255
256### Read Until `\0` (consume `find -print0`)
257
258No:
259
260 # Obscure syntax that bash accepts, but not other shells
261 read -r -d '' myvar
262
263Yes:
264
265 read -0 (&myvar)
266
267## YSH Enhancements to Builtins
268
269### Use `shopt` Instead of `set`
270
271Using a single builtin for all options makes scripts easier to read:
272
273Discouraged:
274
275 set -o errexit
276 shopt -s dotglob
277
278Idiomatic:
279
280 shopt --set errexit
281 shopt --set dotglob
282
283(As always, `set` can be used when you care about compatibility with other
284shells.)
285
286### Use `&` When Mentioning Variable Names
287
288YSH uses [places](variables.html#return-by-mutating-a-place-advanced) to make
289out-paramaters of procs more explicit.
290
291No:
292
293 read -0 record < file.bin
294 echo $record
295
296Yes:
297
298 read -0 (&myvar) < file.bin
299 echo $record
300
301
302### Consider Using `--long-flags`
303
304Easier to write:
305
306 test -d /tmp
307 test -d / && test -f /vmlinuz
308
309 shopt -u extglob
310
311Easier to read:
312
313 test --dir /tmp
314 test --dir / && test --file /vmlinuz
315
316 shopt --unset extglob
317
318## Use Blocks to Save and Restore Context
319
320### Do Something In Another Directory
321
322No:
323
324 ( cd /tmp; echo $PWD ) # subshell is unnecessary (and limited)
325
326No:
327
328 pushd /tmp
329 echo $PWD
330 popd
331
332Yes:
333
334 cd /tmp {
335 echo $PWD
336 }
337
338### Batch I/O
339
340No:
341
342 echo 1 > out.txt
343 echo 2 >> out.txt # appending is less efficient
344 # because open() and close()
345
346No:
347
348 { echo 1
349 echo 2
350 } > out.txt
351
352Yes:
353
354 redir > out.txt {
355 echo 1
356 echo 2
357 }
358
359The `redir` builtin is syntactic sugar -- it lets you see redirects before the
360code that uses them.
361
362### Temporarily Set Shell Options
363
364No:
365
366 set +o errexit
367 myfunc # without error checking
368 set -o errexit
369
370Yes:
371
372 shopt --unset errexit {
373 myfunc
374 }
375
376### Use the `forkwait` builtin for Subshells, not `()`
377
378No:
379
380 ( cd /tmp; rm *.sh )
381
382Yes:
383
384 forkwait {
385 cd /tmp
386 rm *.sh
387 }
388
389Better:
390
391 cd /tmp { # no process created
392 rm *.sh
393 }
394
395### Use the `fork` builtin for async, not `&`
396
397No:
398
399 myfunc &
400
401 { sleep 1; echo one; sleep 2; } &
402
403Yes:
404
405 fork { myfunc }
406
407 fork { sleep 1; echo one; sleep 2 }
408
409## Use Procs (Better Shell Functions)
410
411### Use Named Parameters Instead of `$1`, `$2`, ...
412
413No:
414
415 f() {
416 local src=$1
417 local dest=${2:-/tmp}
418
419 cp "$src" "$dest"
420 }
421
422Yes:
423
424 proc f(src, dest='/tmp') { # Python-like default values
425 cp $src $dest
426 }
427
428### Use Named Varargs Instead of `"$@"`
429
430No:
431
432 f() {
433 local first=$1
434 shift
435
436 echo $first
437 echo "$@"
438 }
439
440Yes:
441
442 proc f(first, @rest) { # @ means "the rest of the arguments"
443 write -- $first
444 write -- @rest # @ means "splice this array"
445 }
446
447You can also use the implicit `ARGV` variable:
448
449 proc p {
450 cp -- @ARGV /tmp
451 }
452
453### Use "Out Params" instead of `declare -n`
454
455Out params are one way to "return" values from a `proc`.
456
457No:
458
459 f() {
460 local in=$1
461 local -n out=$2
462
463 out=PREFIX-$in
464 }
465
466 myvar='init'
467 f zzz myvar # assigns myvar to 'PREFIX-zzz'
468
469
470Yes:
471
472 proc f(in, :out) { # : is an out param, i.e. a string "reference"
473 setref out = "PREFIX-$in"
474 }
475
476 var myvar = 'init'
477 f zzz :myvar # assigns myvar to 'PREFIX-zzz'.
478 # colon is required
479
480### Note: Procs Don't Mess With Their Callers
481
482That is, [dynamic scope]($xref:dynamic-scope) is turned off when procs are
483invoked.
484
485Here's an example of shell functions reading variables in their caller:
486
487 bar() {
488 echo $foo_var # looks up the stack
489 }
490
491 foo() {
492 foo_var=x
493 bar
494 }
495
496 foo
497
498In YSH, you have to pass params explicitly:
499
500 proc bar {
501 echo $foo_var # error, not defined
502 }
503
504Shell functions can also **mutate** variables in their caller! But procs can't
505do this, which makes code easier to reason about.
506
507## Use Modules
508
509YSH has a few lightweight features that make it easier to organize code into
510files. It doesn't have "namespaces".
511
512### Relative Imports
513
514Suppose we are running `bin/mytool`, and we want `BASE_DIR` to be the root of
515the repository so we can do a relative import of `lib/foo.sh`.
516
517No:
518
519 # All of these are common idioms, with caveats
520 BASE_DIR=$(dirname $0)/..
521
522 BASE_DIR=$(dirname ${BASH_SOURCE[0]})/..
523
524 BASE_DIR=$(cd $($dirname $0)/.. && pwd)
525
526 BASE_DIR=$(dirname (dirname $(readlink -f $0)))
527
528 source $BASE_DIR/lib/foo.sh
529
530Yes:
531
532 const BASE_DIR = "$this_dir/.."
533
534 source $BASE_DIR/lib/foo.sh
535
536 # Or simply:
537 source $_this_dir/../lib/foo.sh
538
539The value of `_this_dir` is the directory that contains the currently executing
540file.
541
542### Include Guards
543
544No:
545
546 # libfoo.sh
547 if test -z "$__LIBFOO_SH"; then
548 return
549 fi
550 __LIBFOO_SH=1
551
552Yes:
553
554 # libfoo.sh
555 module libfoo.sh || return 0
556
557### Taskfile Pattern
558
559No:
560
561 deploy() {
562 echo ...
563 }
564 "$@"
565
566Yes
567
568 proc deploy() {
569 echo ...
570 }
571 runproc @ARGV # gives better error messages
572
573## Error Handling
574
575[YSH Fixes Shell's Error Handling (`errexit`)](error-handling.html) once and
576for all! Here's a comprehensive list of error handling idioms.
577
578### Don't Use `&&` Outside of `if` / `while`
579
580It's implicit because `errexit` is on in YSH.
581
582No:
583
584 mkdir /tmp/dest && cp foo /tmp/dest
585
586Yes:
587
588 mkdir /tmp/dest
589 cp foo /tmp/dest
590
591It also avoids the *Trailing `&&` Pitfall* mentioned at the end of the [error
592handling](error-handling.html) doc.
593
594### Ignore an Error
595
596No:
597
598 ls /bad || true # OK because ls is external
599 myfunc || true # suffers from the "Disabled errexit Quirk"
600
601Yes:
602
603 try { ls /bad }
604 try { myfunc }
605
606### Retrieve A Command's Status When `errexit` is On
607
608No:
609
610 # set -e is enabled earlier
611
612 set +e
613 mycommand # this ignores errors when mycommand is a function
614 status=$? # save it before it changes
615 set -e
616
617 echo $status
618
619Yes:
620
621 try {
622 mycommand
623 }
624 echo $[_error.code]
625
626### Does a Builtin Or External Command Succeed?
627
628These idioms are OK in both shell and YSH:
629
630 if ! cp foo /tmp {
631 echo 'error copying' # any non-zero status
632 }
633
634 if ! test -d /bin {
635 echo 'not a directory'
636 }
637
638To be consistent with the idioms below, you can also write them like this:
639
640 try {
641 cp foo /tmp
642 }
643 if failed { # shortcut for (_error.code !== 0)
644 echo 'error copying'
645 }
646
647### Does a Function Succeed?
648
649When the command is a shell function, you shouldn't use `if myfunc` directly.
650This is because shell has the *Disabled `errexit` Quirk*, which is detected by
651YSH `strict_errexit`.
652
653**No**:
654
655 if myfunc; then # errors not checked in body of myfunc
656 echo 'success'
657 fi
658
659**Yes**. The *`$0` Dispatch Pattern* is a workaround that works in all shells.
660
661 if $0 myfunc; then # invoke a new shell
662 echo 'success'
663 fi
664
665 "$@" # Run the function $1 with args $2, $3, ...
666
667**Yes**. The YSH `try` builtin sets the special `_error` variable and returns
668`0`.
669
670 try {
671 myfunc # doesn't abort
672 }
673 if failed {
674 echo 'success'
675 }
676
677### Does a Pipeline Succeed?
678
679No:
680
681 if ps | grep python; then
682 echo 'found'
683 fi
684
685This is technically correct when `pipefail` is on, but it's impossible for
686YSH `strict_errexit` to distinguish it from `if myfunc | grep python` ahead
687of time (the ["meta" pitfall](error-handling.html#the-meta-pitfall)). If you
688know what you're doing, you can disable `strict_errexit`.
689
690Yes:
691
692 try {
693 ps | grep python
694 }
695 if failed {
696 echo 'found'
697 }
698
699 # You can also examine the status of each part of the pipeline
700 if (_pipeline_status[0] !== 0) {
701 echo 'ps failed'
702 }
703
704### Does a Command With Process Subs Succeed?
705
706Similar to the pipeline example above:
707
708No:
709
710 if ! comm <(sort left.txt) <(sort right.txt); then
711 echo 'error'
712 fi
713
714Yes:
715
716 try {
717 comm <(sort left.txt) <(sort right.txt)
718 }
719 if failed {
720 echo 'error'
721 }
722
723 # You can also examine the status of each process sub
724 if (_process_sub_status[0] !== 0) {
725 echo 'first process sub failed'
726 }
727
728(I used `comm` in this example because it doesn't have a true / false / error
729status like `diff`.)
730
731### Handle Errors in YSH Expressions
732
733 try {
734 var x = 42 / 0
735 echo "result is $[42 / 0]"
736 }
737 if failed {
738 echo 'divide by zero'
739 }
740
741### Test Boolean Statuses, like `grep`, `diff`, `test`
742
743The YSH `boolstatus` builtin distinguishes **error** from **false**.
744
745**No**, this is subtly wrong. `grep` has 3 different return values.
746
747 if grep 'class' *.py {
748 echo 'found' # status 0 means found
749 } else {
750 echo 'not found OR ERROR' # any non-zero status
751 }
752
753**Yes**. `boolstatus` aborts the program if `egrep` doesn't return 0 or 1.
754
755 if boolstatus grep 'class' *.py { # may abort
756 echo 'found' # status 0 means found
757 } else {
758 echo 'not found' # status 1 means not found
759 }
760
761More flexible style:
762
763 try {
764 grep 'class' *.py
765 }
766 case (_error.code) {
767 (0) { echo 'found' }
768 (1) { echo 'not found' }
769 (else) { echo 'fatal' }
770 }
771
772## Use YSH Expressions, Initializations, and Assignments (var, setvar)
773
774### Set an Environment Variable Globally
775
776No:
777
778 export PYTHONPATH=. # export is disabled in YSH
779
780Yes:
781
782 setglobal ENV.PYTHONPATH = '.'
783
784That is, enviroments use the [ENV][] object/namespace, rather than being global
785variables.
786
787[ENV]: ref/chap-special-var.html#ENV
788
789Note: the idiom for setting an env var locally is unchanged:
790
791 PYTHONPATH=. myscript.py
792
793### Initialize and Assign Strings and Integers
794
795No:
796
797 local mystr=foo
798 mystr='new value'
799
800 local myint=42 # still a string in shell
801
802Yes:
803
804 var mystr = 'foo'
805 setvar mystr = 'new value'
806
807 var myint = 42 # a real integer
808
809### Expressions on Integers
810
811No:
812
813 x=$(( 1 + 2*3 ))
814 (( x = 1 + 2*3 ))
815
816Yes:
817
818 setvar x = 1 + 2*3
819
820### Mutate Integers
821
822No:
823
824 (( i++ )) # interacts poorly with errexit
825 i=$(( i+1 ))
826
827Yes:
828
829 setvar i += 1 # like Python, with a keyword
830
831### Initialize and Assign Arrays
832
833Arrays in YSH look like `:| my array |` and `['my', 'array']`.
834
835No:
836
837 local -a myarray=(one two three)
838 myarray[3]='THREE'
839
840Yes:
841
842 var myarray = :| one two three |
843 setvar myarray[3] = 'THREE'
844
845 var same = ['one', 'two', 'three']
846 var typed = [1, 2, true, false, null]
847
848
849### Initialize and Assign Dicts
850
851Dicts in YSH look like `{key: 'value'}`.
852
853No:
854
855 local -A myassoc=(['key']=value ['k2']=v2)
856 myassoc['key']=V
857
858
859Yes:
860
861 # keys don't need to be quoted
862 var myassoc = {key: 'value', k2: 'v2'}
863 setvar myassoc['key'] = 'V'
864
865### Get Values From Arrays and Dicts
866
867No:
868
869 local x=${a[i-1]}
870 x=${a[i]}
871
872 local y=${A['key']}
873
874Yes:
875
876 var x = a[i-1]
877 setvar x = a[i]
878
879 var y = A['key']
880
881### Conditions and Comparisons
882
883No:
884
885 if (( x > 0 )); then
886 echo 'positive'
887 fi
888
889Yes:
890
891 if (x > 0) {
892 echo 'positive'
893 }
894
895### Substituting Expressions in Words
896
897No:
898
899 echo flag=$((1 + a[i] * 3)) # C-like arithmetic
900
901Yes:
902
903 echo flag=$[1 + a[i] * 3] # Arbitrary YSH expressions
904
905 # Possible, but a local var might be more readable
906 echo flag=$['1' if x else '0']
907
908
909## Use [Egg Expressions](eggex.html) instead of Regexes
910
911### Test for a Match
912
913No:
914
915 local pat='[[:digit:]]+'
916 if [[ $x =~ $pat ]]; then
917 echo 'number'
918 fi
919
920Yes:
921
922 if (x ~ /digit+/) {
923 echo 'number'
924 }
925
926Or extract the pattern:
927
928 var pat = / digit+ /
929 if (x ~ pat) {
930 echo 'number'
931 }
932
933### Extract Submatches
934
935No:
936
937 if [[ $x =~ foo-([[:digit:]]+) ]] {
938 echo "${BASH_REMATCH[1]}" # first submatch
939 }
940
941Yes:
942
943 if (x ~ / 'foo-' <capture d+> /) { # <> is capture
944 echo $[_group(1)] # first submatch
945 }
946
947## Glob Matching
948
949No:
950
951 if [[ $x == *.py ]]; then
952 echo 'Python'
953 fi
954
955Yes:
956
957 if (x ~~ '*.py') {
958 echo 'Python'
959 }
960
961
962No:
963
964 case $x in
965 *.py)
966 echo Python
967 ;;
968 *.sh)
969 echo Shell
970 ;;
971 esac
972
973Yes (purely a style preference):
974
975 case $x { # curly braces
976 (*.py) # balanced parens
977 echo 'Python'
978 ;;
979 (*.sh)
980 echo 'Shell'
981 ;;
982 }
983
984## TODO
985
986### Distinguish Between Variables and Functions
987
988- `$RANDOM` vs. `random()`
989- `LANG=C` vs. `shopt --setattr LANG=C`
990
991## Related Documents
992
993- [Shell Language Idioms](shell-idioms.html). This advice applies to shells
994 other than YSH.
995- [What Breaks When You Upgrade to YSH](upgrade-breakage.html). Shell constructs that YSH
996 users should avoid.
997- [YSH Fixes Shell's Error Handling (`errexit`)](error-handling.html). YSH fixes the
998 flaky error handling in POSIX shell and bash.
999- TODO: Go through more of the [Pure Bash
1000 Bible](https://github.com/dylanaraps/pure-bash-bible). YSH provides
1001 alternatives for such quirky syntax.
1002