| 1 | ---
|
| 2 | default_highlighter: oils-sh
|
| 3 | ---
|
| 4 |
|
| 5 | Known Differences Between OSH and Other Shells
|
| 6 | ==============================================
|
| 7 |
|
| 8 | This document is for **sophisticated shell users**.
|
| 9 |
|
| 10 | You're unlikely to encounter these incompatibilities in everyday shell usage.
|
| 11 | If you do, there's almost always a **simple workaround**, like adding a space
|
| 12 | or a backslash.
|
| 13 |
|
| 14 | OSH is meant to run all POSIX shell programs, and most bash programs.
|
| 15 |
|
| 16 | <!-- cmark.py expands this -->
|
| 17 | <div id="toc">
|
| 18 | </div>
|
| 19 |
|
| 20 | <!--
|
| 21 | TODO:
|
| 22 |
|
| 23 | - `` as comments in sandstorm
|
| 24 | # This relates to comments being EOL or not
|
| 25 |
|
| 26 | - Pipelines
|
| 27 | - PIPESTATUS only set when a pipeline is actually run.
|
| 28 | - zsh-like lastpipe semantics.
|
| 29 |
|
| 30 | -->
|
| 31 |
|
| 32 | ## Numbers and Arithmetic
|
| 33 |
|
| 34 | ### printf '%d' and other numeric formats require a valid integer
|
| 35 |
|
| 36 | In other shells, `printf %d invalid_integer` prints `0` and a warning. OSH
|
| 37 | gives you a runtime error.
|
| 38 |
|
| 39 | <!-- TODO: Probably should be strict_arith -->
|
| 40 |
|
| 41 | ### Dynamically parsed command subs disallowed unless `shopt -s eval_unsafe_arith`
|
| 42 |
|
| 43 | In shell, array locations are often dynamically parsed, and the index can have
|
| 44 | command subs, which execute arbitrary code.
|
| 45 |
|
| 46 | For example, if you have `code='a[$(echo 42 | tee PWNED)]'`, shells will parse
|
| 47 | this data and execute it in many situations:
|
| 48 |
|
| 49 | echo $(( code )) # dynamic parsing and evaluation in bash, mksh, zsh
|
| 50 |
|
| 51 | unset $code
|
| 52 |
|
| 53 | printf -v $code hi
|
| 54 |
|
| 55 | echo ${!code}
|
| 56 |
|
| 57 | OSH disallows this by default. If you want this behavior, you can turn on
|
| 58 | `shopt -s eval_unsafe_arith`.
|
| 59 |
|
| 60 | Related: [A 30-year-old security problem](https://www.oilshell.org/blog/2019/01/18.html#a-story-about-a-30-year-old-security-problem)
|
| 61 |
|
| 62 | ## Static Parsing Differences
|
| 63 |
|
| 64 | This section describes differences related to [static
|
| 65 | parsing](http://www.oilshell.org/blog/2016/10/22.html). OSH avoids the
|
| 66 | dynamic parsing of most shells.
|
| 67 |
|
| 68 | (Note: This section should encompass all the failures from the [wild
|
| 69 | tests](http://oilshell.org/cross-ref.html?tag=wild-test#wild-test) and [spec
|
| 70 | tests](http://oilshell.org/cross-ref.html?tag=spec-test#spec-test).
|
| 71 |
|
| 72 | ### Strings vs. Bare words in array indices
|
| 73 |
|
| 74 | Strings should be quoted inside array indices:
|
| 75 |
|
| 76 | No:
|
| 77 |
|
| 78 | "${SETUP_STATE[$err.cmd]}"
|
| 79 |
|
| 80 | Yes:
|
| 81 |
|
| 82 | "${SETUP_STATE["$err.cmd"]}"
|
| 83 |
|
| 84 | When unquoted, the period causes an ambiguity with respect to regular arrays
|
| 85 | vs. associative arrays. See [Parsing Bash is
|
| 86 | Undecidable](https://www.oilshell.org/blog/2016/10/20.html) (2016).
|
| 87 |
|
| 88 | - [OILS-ERR-101](error-catalog.html#oils-err-101) explains more ways to fix
|
| 89 | this.
|
| 90 |
|
| 91 | ### Subshell in command sub - `$((` versus `$( (`
|
| 92 |
|
| 93 | You can have a subshell `(` in a command sub `$(`, but it usually doesn't make
|
| 94 | sense.
|
| 95 |
|
| 96 | In OSH you need a space after `$(`, so it would be `$( (`.
|
| 97 |
|
| 98 | characters `$((` always start an arith sub.
|
| 99 |
|
| 100 | No:
|
| 101 |
|
| 102 | $((cd / && ls))
|
| 103 |
|
| 104 | Yes:
|
| 105 |
|
| 106 | $( (cd / && ls) ) # Valid but usually doesn't make sense.
|
| 107 | $({ cd / && ls; }) # Use {} for grouping, not (). Note trailing ;
|
| 108 | $(cd / && ls) # Even better
|
| 109 |
|
| 110 | ### Nested Subshells - `((` versus `( (`
|
| 111 |
|
| 112 | You should never need nested subshells with `((` in Bourne shell or Oils.
|
| 113 |
|
| 114 | If you do, you should add a space with `( (` instead of `((`, similar to the
|
| 115 | issue above.
|
| 116 |
|
| 117 | In OSH, `((` always starts bash-style arithmetic.
|
| 118 |
|
| 119 | ---
|
| 120 |
|
| 121 | The only place I see `((` arise is when shell users try to use `( )` to mean
|
| 122 | **grouping**, because they are used to C or Python.
|
| 123 |
|
| 124 | But it means **subshell**, not grouping. In shell, `{ }` is the way to group
|
| 125 | commands.
|
| 126 |
|
| 127 | No:
|
| 128 |
|
| 129 | if ((test -f a || test -f b) && grep foo c); then
|
| 130 | echo ok
|
| 131 | fi
|
| 132 |
|
| 133 | Allowed, but not what you want:
|
| 134 |
|
| 135 | if ( (test -f a || test -f b) && grep foo c); then
|
| 136 | echo ok
|
| 137 | fi
|
| 138 |
|
| 139 | Yes:
|
| 140 |
|
| 141 | if { test -f a || test -f b; } && grep foo c; then
|
| 142 | echo ok
|
| 143 | fi
|
| 144 |
|
| 145 | ### Extended glob vs. Negation of boolean expression
|
| 146 |
|
| 147 | The OSH parser distinguishes these two constructs with a space:
|
| 148 |
|
| 149 | - `[[ !(a == a) ]]` is an extended glob.
|
| 150 | - `[[ ! (a == a) ]]` is the negation of an equality test.
|
| 151 |
|
| 152 | In bash, the parsing of such expressions depends on `shopt -s extglob`. In
|
| 153 | OSH, `shopt -s extglob` is accepted, but doesn't affect parsing.
|
| 154 |
|
| 155 | ### Here doc terminators must be on their own line
|
| 156 |
|
| 157 | Lines like `EOF]` or `EOF)` don't end here docs. The delimiter must be on its
|
| 158 | own line.
|
| 159 |
|
| 160 | No:
|
| 161 |
|
| 162 | a=$(cat <<EOF
|
| 163 | abc
|
| 164 | EOF)
|
| 165 |
|
| 166 | a=$(cat <<EOF
|
| 167 | abc
|
| 168 | EOF # this is not a comment; it makes the EOF delimiter invalid
|
| 169 | )
|
| 170 |
|
| 171 | Yes:
|
| 172 |
|
| 173 | a=$(cat <<EOF
|
| 174 | abc
|
| 175 | EOF
|
| 176 | ) # this is actually a comment
|
| 177 |
|
| 178 |
|
| 179 | ### Spaces aren't allowed in LHS indices
|
| 180 |
|
| 181 | Bash allows:
|
| 182 |
|
| 183 | a[1 + 2 * 3]=value
|
| 184 |
|
| 185 | OSH only allows:
|
| 186 |
|
| 187 | a[1+2*3]=value
|
| 188 |
|
| 189 | because it parses with limited lookahead. The first line would result in the
|
| 190 | execution of a command named `a[1`.
|
| 191 |
|
| 192 | ### break / continue / return are keywords, not builtins
|
| 193 |
|
| 194 | This means that they aren't "dynamic":
|
| 195 |
|
| 196 | b=break
|
| 197 | while true; do
|
| 198 | $b # doesn't break in OSH
|
| 199 | done
|
| 200 |
|
| 201 | Static control flow will allow static analysis of shell scripts.
|
| 202 |
|
| 203 | (Test cases are in [spec/loop][]).
|
| 204 |
|
| 205 | ### OSH has more builtins, which shadow external commands
|
| 206 |
|
| 207 | For example, `append` is a builtin in OSH, but not in `bash`. Use `env append`
|
| 208 | or `/path/to/append` if you want to run an external command.
|
| 209 |
|
| 210 | (Note that a user-defined proc `append` takes priority over the builtin
|
| 211 | `append`.)
|
| 212 |
|
| 213 | ### OSH has more keywords, which shadow builtins, functions, and commands
|
| 214 |
|
| 215 | In contrast with builtins, **keywords** affect shell parsing.
|
| 216 |
|
| 217 | For example, `func` is a keyword in OSH, but not in `bash`. To run a command
|
| 218 | named `func`, use `command func arg1`.
|
| 219 |
|
| 220 | Note that all shells have extensions that cause this issue. For example, `[[`
|
| 221 | is a keyword in `bash` but not in POSIX shell.
|
| 222 |
|
| 223 | ## Later Parsing Differences
|
| 224 |
|
| 225 | These differences occur in subsequent stages of parsing, or in runtime parsing.
|
| 226 |
|
| 227 | ### Brace expansion is all or nothing
|
| 228 |
|
| 229 | No:
|
| 230 |
|
| 231 | {a,b}{ # what does the second { mean?
|
| 232 | {a,b}{1...3} # 3 dots instead of 2
|
| 233 |
|
| 234 | Yes:
|
| 235 |
|
| 236 | {a,b}\{
|
| 237 | {a,b}\{1...3\}
|
| 238 |
|
| 239 | bash will do a **partial expansion** in the former cases, giving you `a{ b{`
|
| 240 | and `a{1...3} b{1...3}`.
|
| 241 |
|
| 242 | OSH considers them syntax errors and aborts all brace expansion, giving you
|
| 243 | the same thing back: `{a,b}{` and `{a,b}{1...3}`.
|
| 244 |
|
| 245 | ### Brackets should be escaped within Character Classes
|
| 246 |
|
| 247 | Don't use ambiguous syntax for a character class consisting of a single bracket
|
| 248 | character.
|
| 249 |
|
| 250 | No:
|
| 251 |
|
| 252 | echo [[]
|
| 253 | echo []]
|
| 254 |
|
| 255 | Yes:
|
| 256 |
|
| 257 | echo [\[]
|
| 258 | echo [\]]
|
| 259 |
|
| 260 |
|
| 261 | The ambiguous syntax is allowed when we pass globs through to `libc`, but it's
|
| 262 | good practice to be explicit.
|
| 263 |
|
| 264 | ### [[ -v var ]] doesn't allow expressions
|
| 265 |
|
| 266 | In bash, you can use `[[` with `-v` to test whether an array contains an entry:
|
| 267 |
|
| 268 | declare -a array=('' foo)
|
| 269 | if [[ -v array[1] ]]; then
|
| 270 | echo 'exists'
|
| 271 | fi # => exists
|
| 272 |
|
| 273 | Likewise for an associative array:
|
| 274 |
|
| 275 | declare -A assoc=([key]=value)
|
| 276 | if [[ -v assoc['key'] ]]
|
| 277 | echo 'exists'
|
| 278 | fi # => exists
|
| 279 |
|
| 280 | OSH currently treats these expressions as a string, which means the status will
|
| 281 | be 1 (`false`).
|
| 282 |
|
| 283 | Workaround:
|
| 284 |
|
| 285 | if [[ "${assoc['key']:+exists}" ]]; then
|
| 286 | echo 'exists'
|
| 287 | fi # => exists
|
| 288 |
|
| 289 | In ysh, you can use:
|
| 290 |
|
| 291 | var d = { key: 42 }
|
| 292 | if ('key' in d) {
|
| 293 | echo 'exists'
|
| 294 | } # => exists
|
| 295 |
|
| 296 | ## Data Structures
|
| 297 |
|
| 298 | ### Arrays aren't split inside ${}
|
| 299 |
|
| 300 | Most shells split the entries of arrays like `"$@"` and `"${a[@]}"` here:
|
| 301 |
|
| 302 | echo ${undef:-"$@"}
|
| 303 |
|
| 304 | In OSH, omit the quotes if you want splitting:
|
| 305 |
|
| 306 | echo ${undef:-$@}
|
| 307 |
|
| 308 | I think OSH is more consistent, but it disagrees with other shells.
|
| 309 |
|
| 310 | ### Values are tagged with types, not locations (`declare -i -a -A`)
|
| 311 |
|
| 312 | Even though there's a large common subset, OSH and bash have a different model
|
| 313 | for typed data.
|
| 314 |
|
| 315 | - In OSH, **values** are tagged with types, which is how Python and JavaScript
|
| 316 | work.
|
| 317 | - In bash, **cells** (locations for values) are tagged with types. Everything
|
| 318 | is a string, but in certain contexts, strings are treated as integers or as
|
| 319 | structured data.
|
| 320 |
|
| 321 | In particular,
|
| 322 |
|
| 323 | - The `-i` flag is a no-op in OSH. See [Shell Idioms > Remove Dynamic
|
| 324 | Parsing](shell-idioms.html#remove-dynamic-parsing) for alternatives to `-i`.
|
| 325 | - The `-a` and `-A` flags behave differently. They pertain to the value, not
|
| 326 | the location.
|
| 327 |
|
| 328 | For example, these two statements are different in bash, but the same in OSH:
|
| 329 |
|
| 330 | declare -A assoc # unset cell that will LATER be an assoc array
|
| 331 | declare -A assoc=() # empty associative array
|
| 332 |
|
| 333 | In bash, you can tell the difference with `set -u`, but there's no difference
|
| 334 | in OSH.
|
| 335 |
|
| 336 | ### Indexed and Associative arrays are distinct
|
| 337 |
|
| 338 | Here is how you can create arrays in OSH, in a bash-compatible way:
|
| 339 |
|
| 340 | local indexed=(foo bar)
|
| 341 | local -a indexed=(foo bar) # -a is redundant
|
| 342 | echo ${indexed[1]} # bar
|
| 343 |
|
| 344 | local assoc=(['one']=1 ['two']=2)
|
| 345 | local -A assoc=(['one']=1 ['two']=2) # -A is redundant
|
| 346 | echo ${assoc['one']} # 1
|
| 347 |
|
| 348 | In bash, the distinction between the two is blurry, with cases like this:
|
| 349 |
|
| 350 | local -A x=(foo bar) # -A disagrees with literal
|
| 351 | local -a y=(['one']=1 ['two']=2) # -a disagrees with literal
|
| 352 |
|
| 353 | These are disallowed in OSH.
|
| 354 |
|
| 355 | Notes:
|
| 356 |
|
| 357 | - The `=` keyword is useful for gaining an understanding of the data model.
|
| 358 | - See the [Quirks](quirks.html) doc for details on how OSH uses this cleaner
|
| 359 | model while staying compatible with bash.
|
| 360 |
|
| 361 | ## Assignment builtins
|
| 362 |
|
| 363 | The assignment builtins are `export`, `readonly`, `local`, and
|
| 364 | `declare`/`typeset`. They're parsed in 2 ways:
|
| 365 |
|
| 366 | - Statically: to avoid word splitting in `declare x=$y` when `$y` contains
|
| 367 | spaces. bash and other shells behave this way.
|
| 368 | - Dynamically: to handle expressions like `declare $1` where `$1` is `a=b`
|
| 369 |
|
| 370 | ### `builtin declare x=$y` is a runtime error
|
| 371 |
|
| 372 | This is because the special parsing of `x=$y` depends on the first word
|
| 373 | `declare`.
|
| 374 |
|
| 375 | ### Args aren't split or globbed
|
| 376 |
|
| 377 | In bash, you can do unusual things with args to assignment builtins:
|
| 378 |
|
| 379 | vars='a=b x=y'
|
| 380 | touch foo=bar.py spam=eggs.py
|
| 381 |
|
| 382 | declare $vars *.py # assigns at least 4 variables
|
| 383 | echo $a # b
|
| 384 | echo $x # y
|
| 385 | echo $foo # bar.py
|
| 386 | echo $spam # eggs.py
|
| 387 |
|
| 388 | In contrast, OSH doesn't split or glob args to assignment builtins. This is
|
| 389 | more like the behavior of zsh.
|
| 390 |
|
| 391 | ## Pipelines
|
| 392 |
|
| 393 | ### Last pipeline part may run in shell process (zsh, bash `shopt -s lastpipe`)
|
| 394 |
|
| 395 | In this pipeline, the builtin `read` is run in the shell process, not a child
|
| 396 | process:
|
| 397 |
|
| 398 | $ echo hi | read x
|
| 399 | $ echo x=$x
|
| 400 | x=hi # empty in bash unless shopt -s lastpipe
|
| 401 |
|
| 402 | If the last part is an external command, there is no difference:
|
| 403 |
|
| 404 | $ ls | wc -l
|
| 405 | 42
|
| 406 |
|
| 407 | This is how zsh behaves, and how bash (sometimes) behaves with `shopt -s
|
| 408 | lastpipe`.
|
| 409 |
|
| 410 | ### Pipelines can't be suspended with Ctrl-Z
|
| 411 |
|
| 412 | Because the last part may be the current shell process, the entire pipeline
|
| 413 | can't be suspended.
|
| 414 |
|
| 415 | OSH and zsh share this consequence of the `lastpipe` semantics.
|
| 416 |
|
| 417 | In contrast, bash's `shopt -s lastpipe` is ignored in interactive shells.
|
| 418 |
|
| 419 | ### `${PIPESTATUS[@]}` is only set after an actual pipeline
|
| 420 |
|
| 421 | This makes it easier to check compound status codes without worrying about them
|
| 422 | being "clobbered".
|
| 423 |
|
| 424 | Bash will set `${PIPESTATUS[@]}` on every command, regardless of whether its a
|
| 425 | pipeline.
|
| 426 |
|
| 427 | ## More Differences at Runtime
|
| 428 |
|
| 429 | ### Alias expansion
|
| 430 |
|
| 431 | Almost all "real" aliases should work in OSH. But these don't work:
|
| 432 |
|
| 433 | alias left='{'
|
| 434 | left echo hi; }
|
| 435 |
|
| 436 | (cases #33-#34 in [spec/alias][])
|
| 437 |
|
| 438 | or
|
| 439 |
|
| 440 | alias a=
|
| 441 | a (( var = 0 ))
|
| 442 |
|
| 443 | Details on the OSH parsing model:
|
| 444 |
|
| 445 | 1. Your code is statically parsed into an abstract syntax tree, which contains
|
| 446 | many types of nodes.
|
| 447 | 2. `SimpleCommand` are the only ones that are further alias-expanded.
|
| 448 |
|
| 449 | For example, these result in `SimpleCommand` nodes:
|
| 450 |
|
| 451 | - `ls -l`
|
| 452 | - `read -n 1` (normally a builtin)
|
| 453 | - `myfunc foo`
|
| 454 |
|
| 455 | These don't:
|
| 456 |
|
| 457 | - `x=42`
|
| 458 | - `declare -r x=42`
|
| 459 | - `break`, `continue`, `return`, `exit` — as explained above, these are
|
| 460 | keywords and not builtins.
|
| 461 | - `{ echo one; echo two; }`
|
| 462 | - `for`, `while`, `case`, functions, etc.
|
| 463 |
|
| 464 | ### Extended globs are more static like `mksh`, and have other differences
|
| 465 |
|
| 466 | That is, in OSH and mksh, something like `echo *.@(cc|h)` is an extended glob.
|
| 467 | But `echo $x`, where `$x` contains the pattern, is not.
|
| 468 |
|
| 469 | For more details and differences, see the [Extended Glob
|
| 470 | section](word-language.html#extended-glob) of the Word Language doc.
|
| 471 |
|
| 472 | ### Completion
|
| 473 |
|
| 474 | The OSH completion API is mostly compatible with the bash completion API,
|
| 475 | except that it moves the **responsibility for quoting** out of plugins and onto
|
| 476 | the shell itself. Plugins should return candidates as `argv` entries, not
|
| 477 | shell words.
|
| 478 |
|
| 479 | See the [completion doc](completion.html) for details.
|
| 480 |
|
| 481 | ## Interactive Features
|
| 482 |
|
| 483 | ### History Substitution Language
|
| 484 |
|
| 485 | The rules for history substitution like `!echo` are simpler. There are no
|
| 486 | special cases to avoid clashes with `${!indirect}` and so forth.
|
| 487 |
|
| 488 | TODO: Link to the history lexer.
|
| 489 |
|
| 490 | <!--
|
| 491 | TODO: we want to make history more statically parsed. Should test the ZSH
|
| 492 | parser.
|
| 493 | -->
|
| 494 |
|
| 495 | ## Links
|
| 496 |
|
| 497 | - [OSH Spec Tests](../test/spec.wwz/survey/osh.html) run shell snippets with OSH and other
|
| 498 | shells to compare their behavior.
|
| 499 |
|
| 500 | External:
|
| 501 |
|
| 502 | - This list may seem long, but compare the list of differences in [Bash POSIX
|
| 503 | Mode](https://www.gnu.org/software/bash/manual/html_node/Bash-POSIX-Mode.html).
|
| 504 | That page tells you what `set -o posix` does in bash.
|
| 505 |
|
| 506 |
|
| 507 | [spec/command-sub]: ../test/spec.wwz/command-sub.html
|
| 508 | [spec/loop]: ../test/spec.wwz/loop.html
|
| 509 | [spec/alias]: ../test/spec.wwz/alias.html
|
| 510 |
|
| 511 |
|