Why Sponsor Oils? | source | all docs for version 0.23.0 | all versions | oilshell.org

Guide to Procs and Funcs

YSH has two major units of code: shell-like proc, and Python-like func.

• Roughly speaking, procs are for commands and I/O, while funcs are for pure computation.
• Procs are often big, and may call small funcs. On the other hand, it's possible, but rarer, for funcs to call procs.
• You can write shell scripts mostly with procs, and perhaps a few funcs.

This doc compares the two mechanisms, and gives rough guidelines.

Tip: Start Simple

Before going into detail, here's a quick reminder that you don't have to use either procs or funcs. YSH is a language that scales both down and up.

You can start with just a list of plain commands:

mkdir -p /tmp/dest
cp --verbose *.txt /tmp/dest

Then copy those into procs as the script gets bigger:

proc build-app {
  ninja --verbose
}

proc deploy {
  mkdir -p /tmp/dest
  cp --verbose *.txt /tmp/dest
}

build-app
deploy

Then add funcs if you need pure computation:

func isTestFile(name) {
  return (name => endsWith('._test.py'))
}

if (isTestFile('my_test.py')) {
  echo 'yes'
}

At a Glance

Procs vs. Funcs

This table summarizes the difference between procs and funcs. The rest of the doc will elaborate on these issues.

proc print-max (; x, y) {
  echo $[x if x > y else y]
}

func computeMax(x, y) {
  return (x if x > y else y)
}

print-max (3, 4)

print-max (3, 4) | tee out.txt

var m = computeMax(3, 4)

call computeMax(3, 4)

return ([42, {name: 'bob'}])

var x = obj.myMethod()
call obj->myMutatingMethod()

assert [42 === x]

Func Calls and Defs

Now that we've compared procs and funcs, let's look more closely at funcs. They're inherently simpler: they have 2 types of args and params, rather than 4.

YSH argument binding is based on Julia, which has all the power of Python, but without the "evolved warts" (e.g. / and *).

In general, with all the bells and whistles, func definitions look like:

# pos args and named args separated with ;
func f(p1, p2, ...rest_pos; n1=42, n2='foo', ...rest_named) {
  return (len(rest_pos) + len(rest_named))
}

Func calls look like:

# spread operator ... at call site
var pos_args = [3, 4]
var named_args = {foo: 'bar'}
var x = f(1, 2, ...pos_args; n1=43, ...named_args)

Note that positional args/params and named args/params can be thought of as two "separate worlds".

This table shows simpler, more common cases.

var x = myMax(3, 4)

func myMax(x, y) {
  return (x if x > y else y)
}

var args = [3, 4]
var x = myMax(...args)

var x = myPrintf("%s is %d", 'bob', 30)

func myPrintf(fmt, ...args) {
  # ...
}

var x = mySum(3, 4, start=5)

func mySum(x, y; start=0) {
  return (x + y + start)
}

var opts = {start: 5}
var x = mySum(3, 4, ...opts)

var x = f(start=5, end=7)

func f(; ...opts) {
  if ('start' not in opts) {
    setvar opts.start = 0
  }
  # ...
}

Proc Calls and Defs

Like funcs, procs have 2 kinds of typed args/params: positional and named.

But they may also have string aka word args/params, and a block arg/param.

In general, a proc signature has 4 sections, like this:

proc p (
    w1, w2, ...rest_word;     # word params
    p1, p2, ...rest_pos;      # pos params
    n1, n2, ...rest_named;    # named params
    block                     # block param
) {
  echo 'body'
}

In general, a proc call looks like this:

var pos_args = [3, 4]
var named_args = {foo: 'bar'}

p /bin /tmp (1, 2, ...pos_args; n1=43, ...named_args) {
  echo 'block'
}

The block can also be passed as an expression after a second semicolon:

p /bin /tmp (1, 2, ...pos_args; n1=43, ...named_args; block)

Some simpler examples:

my-cd /tmp

proc my-cd (dest) {
  cd $dest
}

my-cd -L /tmp

proc my-cd (...flags) {
  cd @flags
}

var flags = :| -L /tmp |
my-cd @flags

print-max (3, 4)

proc print-max ( ; x, y) {
  echo $[x if x > y else y]
}

print-max (3, 4, start=5)

proc print-max ( ; x, y; start=0) {
  # ...
}

my-cd /tmp {
  echo $PWD
  echo hi
}

proc my-cd (dest; ; ; block) {
  cd $dest (; ; block)
}

p 'word' (42, verbose=true) {
  echo $PWD
  echo hi
}

proc p (w; myint; verbose=false; block) {
  = w
  = myint
  = verbose
  = block
}

Common Features

Let's recap the common features of procs and funcs.

Spread Args, Rest Params

• Spread arg list ... at call site
• Rest params ... at definition

The error builtin raises exceptions

The error builtin is idiomatic in both funcs and procs:

func f(x) {   
  if (x <= 0) {
    error 'Should be positive' (status=99)
  }
}

Tip: reserve such errors for exceptional situations. For example, an input string being invalid may not be uncommon, while a disk full I/O error is more exceptional.

(The error builtin is implemented with C++ exceptions, which are slow in the error case.)

Out Params: &myvar is of type value.Place

Out params are more common in procs, because they don't have a typed return value.

proc p ( ; out) {
  call out->setValue(42)
}
var x
p (&x)
echo "x set to $x"  # => x set to 42

But they can also be used in funcs:

func f (out) {
  call out->setValue(42)
}
var x
call f(&x)
echo "x set to $x"  # => x set to 42

Observation: procs can do everything funcs can. But you may want the purity and familiar syntax of a func.

Design note: out params are a nicer way of doing what bash does with declare -n aka nameref variables. They don't rely on dynamic scope.

Proc-Only Features

Procs have some features that funcs don't have.

Lazy Arg Lists where [x > 10]

A lazy arg list is implemented with shopt --set parse_bracket, and is syntax sugar for an unevaluated value.Expr.

Longhand:

var my_expr = ^[42 === x]  # value of type Expr
assert (myexpr)

Shorthand:

assert [42 === x]  # equivalent to the above

Open Proc Signatures bind argv

TODO: Implement new ARGV semantics.

When a proc signature omits (), it's called "open" because the caller can pass "extra" arguments:

proc my-open {
  write 'args are' @ARGV
}
# All valid:
my-open
my-open 1 
my-open 1 2

Stricter closed procs:

proc my-closed (x) {
  write 'arg is' $x
}
my-closed      # runtime error: missing argument
my-closed 1    # valid
my-closed 1 2  # runtime error: too many arguments

An "open" proc is nearly is nearly identical to a shell function:

shfunc() {
  write 'args are' @ARGV
}

Methods are Funcs Bound to Objects

Values of type Obj have an ordered set of name-value bindings, as well as a prototype chain of more Obj instances ("parents"). They support these operators:

• dot (.) looks for attributes or methods with a given name.
  • Reference: ysh-attr
  • Attributes may be in the object, or up the chain. They are returned literally.
  • Methods live up the chain. They are returned as BoundFunc, so that the first self argument of a method call is the object itself.
• Thin arrow (->) looks for mutating methods, which have an M/ prefix.
  • Reference: thin-arrow

The __invoke__ method makes an Object "Proc-like"

First, define a proc, with the first typed arg named self:

proc myInvoke (word_param; self, int_param) {
  echo "sum = $[self.x + self.y + int_param]"
}

Make it the __invoke__ method of an Obj:

var methods = Object(null, {__invoke__: myInvoke})
var invokable_obj = Object(methods, {x: 1, y: 2})

Then invoke it like a proc:

invokable_obj myword (3)
# sum => 6

Usage Notes

3 Ways to Return a Value

Let's review the recommended ways to "return" a value:

1. return (x) in a func.
   • The parentheses are required because expressions like (x + 1) should look different than words.
2. Pass a value.Place instance to a proc or func.
   • That is, out param &out.
3. Print to stdout in a proc
   • Capture it with command sub: $(myproc)
   • Or with read: myproc | read --all; echo $_reply

Obsolete ways of "returning":

1. Using declare -n aka nameref variables in bash.
2. Relying on dynamic scope in POSIX shell.

Procs Compose in Pipelines / "Bernstein Chaining"

Some YSH users may tend toward funcs because they're more familiar. But shell composition with procs is very powerful!

They have at least two kinds of composition that funcs don't have.

See #shell-the-good-parts:

1. Shell Has a Forth-Like Quality - Bernstein chaining.
2. Pipelines Support Vectorized, Point-Free, and Imperative Style - the shell can transparently run procs as elements of pipelines.

Summary

YSH is influenced by both shell and Python, so it has both procs and funcs.

Many programmers will gravitate towards funcs because they're familiar, but procs are more powerful and shell-like.

Make your YSH programs by learning to use procs!

Appendix

Implementation Details

procs vs. funcs both have these concerns:

1. Evaluation of default args at definition time.
2. Evaluation of actual args at the call site.
3. Arg-Param binding for builtin functions, e.g. with typed_args.Reader.
4. Arg-Param binding for user-defined functions.

So the implementation can be thought of as a 2 × 4 matrix, with some code shared. This code is mostly in ysh/func_proc.py.

Related

• Variable Declaration, Mutation, and Scope - in particular, procs don't have dynamic scope.
• Block Literals (in progress)