| category | total lines | num files |
| mycpp Translator | 7,405 | 16 |
| mycpp Test Data | 4,356 | 48 |
| Zephyr ASDL | 3,289 | 11 |
| pgen2 Parser Generator | 1,854 | 8 |
| Other Code Generators | 3,052 | 11 |
| Yaks | 510 | 4 |
| Hand-written C++ Code | 3,274 | 29 |
| Garbage-Collected Runtime | 5,319 | 26 |
| Unit tests in C++ | 7,811 | 27 |
| Incremental C++ Build | 3,580 | 20 |
| OSH (and common libraries) | 44,303 | 84 |
| YSH | 10,788 | 26 |
| Data Languages | 2,231 | 9 |
| Tools | 1,330 | 5 |
| Spec Tests | 57,238 | 190 |
| Gold Tests | 2,775 | 27 |
| Generated C++ Code | 122,926 | 41 |
The README for oilshell/oil has another overview of the repository.
This prototype uses the MyPy frontend to translate statically-typed Python to C++. The generated code calls a small runtime which implements things like List[T], Dict[K, V], and Python's len().
mycpp/compare_pairs.py 60 mycpp/const_pass.py 532 mycpp/control_flow_pass.py 559 mycpp/cppgen_pass.py 3,107 mycpp/crash.py 38 mycpp/debug_pass.py 465 mycpp/format_strings.py 114 mycpp/iolib.py 165 mycpp/ir_pass.py 99 mycpp/mops.py 285 mycpp/mycpp_main.py 423 mycpp/mylib.py 533 mycpp/pass_state.py 610 mycpp/util.py 177 mycpp/visitor.py 238 7,405 lines in 16 files
Small Python examples that translate to C++, compile, and run.
mycpp/examples/__init__.py 0 mycpp/examples/asdl_generated.py 65 mycpp/examples/cartesian.py 78 mycpp/examples/cgi.py 62 mycpp/examples/classes.py 298 mycpp/examples/containers.py 289 mycpp/examples/control_flow.py 113 mycpp/examples/escape.py 70 mycpp/examples/fib_iter.py 53 mycpp/examples/fib_recursive.py 49 mycpp/examples/files.py 64 mycpp/examples/gc_stack_roots.py 187 mycpp/examples/invalid_condition.py 31 mycpp/examples/invalid_default_args.py 44 mycpp/examples/invalid_except.py 38 mycpp/examples/invalid_format_args.py 15 mycpp/examples/invalid_format_strings.py 20 mycpp/examples/invalid_global.py 18 mycpp/examples/invalid_print.py 13 mycpp/examples/invalid_python.py 61 mycpp/examples/invalid_switch.py 84 mycpp/examples/invalid_try_else.py 38 mycpp/examples/invalid_types.py 31 mycpp/examples/invalid_types_2.py 31 mycpp/examples/length.py 109 mycpp/examples/lexer_main.py 47 mycpp/examples/loops.py 177 mycpp/examples/modules.py 81 mycpp/examples/parse.py 271 mycpp/examples/pgen2_demo.py 106 mycpp/examples/scoped_resource.py 221 mycpp/examples/test_arith_ops.py 45 mycpp/examples/test_cast.py 211 mycpp/examples/test_classes_gc.py 212 mycpp/examples/test_conditional.py 66 mycpp/examples/test_ctx_pattern.py 65 mycpp/examples/test_default_args.py 75 mycpp/examples/test_globals.py 62 mycpp/examples/test_hoist.py 61 mycpp/examples/test_integers.py 75 mycpp/examples/test_io_os_error.py 131 mycpp/examples/test_iterators.py 101 mycpp/examples/test_small_str.py 33 mycpp/examples/test_strings.py 272 mycpp/examples/test_switch.py 80 mycpp/examples/tuple_return_value.py 54 mycpp/examples/varargs.py 49 4,356 lines in 48 files
A DSL for algebraic data types, borrowed from Python. Oils is the most strongly typed Bourne shell implementation!
asdl/asdl_main.py 299 asdl/ast.py 238 asdl/format.py 535 asdl/front_end.py 515 asdl/gen_cpp.py 826 asdl/gen_python.py 611 asdl/pybase.py 51 asdl/runtime.py 67 asdl/util.py 11 asdl/visitor.py 136 3,289 lines in 11 files
An LL(1) parser generator used to parse YSH expressions. Also borrowed from CPython.
pgen2/driver.py 101 pgen2/grammar.py 303 pgen2/parse.py 225 pgen2/pgen.py 430 pgen2/pnode.py 67 pgen2/token.py 149 pgen2/tokenize.py 579 1,854 lines in 8 files
In order to make Oils statically typed, we had to abandon Python reflection and use C++ source code generation instead. The lexer, flag definitions, and constants can be easily compiled to C++.
core/optview_gen.py 94 cpp/embedded_file_gen.py 66 doctools/help_gen.py 658 frontend/consts_gen.py 651 frontend/flag_gen.py 540 frontend/lexer_gen.py 500 frontend/option_gen.py 94 frontend/signal_gen.py 105 osh/arith_parse_gen.py 99 ysh/grammar_gen.py 245 3,052 lines in 11 files
Experimental replacement for mycpp
yaks/gen_cpp.py 104 yaks/transform.py 250 yaks/yaks_main.py 156 510 lines in 4 files
Includes OS bindings. Small C++ files like cpp/osh_arith_parse.{cc,h} correspond to larger Python files like osh/arith_parse.py.
cpp/core.cc 398 cpp/core.h 126 cpp/data_lang.cc 273 cpp/data_lang.h 34 cpp/embedded_file.h 18 cpp/fanos.cc 57 cpp/fanos.h 18 cpp/fanos_shared.h 49 cpp/frontend_flag_spec.cc 231 cpp/frontend_flag_spec.h 151 cpp/frontend_match.cc 155 cpp/frontend_match.h 92 cpp/frontend_pyreadline.cc 337 cpp/frontend_pyreadline.h 73 cpp/libc.cc 241 cpp/libc.h 38 cpp/osh.cc 161 cpp/osh.h 50 cpp/osh_tdop.cc 16 cpp/osh_tdop.h 52 cpp/pgen2.cc 44 cpp/pgen2.h 86 cpp/preamble.h 48 cpp/pylib.cc 58 cpp/pylib.h 22 cpp/stdlib.cc 258 cpp/stdlib.h 177 cpp/translation_stubs.h 11 3,274 lines in 29 files
Uses a fork-friendly Mark-Sweep collector.
mycpp/common.h 92 mycpp/comparators.h 90 mycpp/gc_alloc.h 258 mycpp/gc_builtins.cc 461 mycpp/gc_builtins.h 175 mycpp/gc_dict.h 471 mycpp/gc_iolib.cc 46 mycpp/gc_iolib.h 175 mycpp/gc_list.h 521 mycpp/gc_mops.cc 79 mycpp/gc_mops.h 120 mycpp/gc_mylib.cc 311 mycpp/gc_mylib.h 381 mycpp/gc_obj.h 179 mycpp/gc_slab.h 58 mycpp/gc_str.cc 645 mycpp/gc_str.h 219 mycpp/gc_tuple.h 151 mycpp/hash.cc 45 mycpp/hash.h 22 mycpp/mark_sweep_heap.cc 438 mycpp/mark_sweep_heap.h 294 mycpp/probes.h 39 mycpp/runtime.h 25 mycpp/test_common.h 24 5,319 lines in 26 files
The goal is to make the spec tests pass, but unit tests are helpful too.
cpp/core_test.cc 361 cpp/data_lang_test.cc 178 cpp/data_race_test.cc 206 cpp/frontend_flag_spec_test.cc 154 cpp/frontend_match_test.cc 87 cpp/libc_test.cc 403 cpp/obj_layout_test.cc 215 cpp/osh_test.cc 50 cpp/pgen2_test.cc 32 cpp/pylib_test.cc 52 cpp/stdlib_test.cc 158 cpp/unicode_test.cc 71 mycpp/bump_leak_heap_test.cc 62 mycpp/float_test.cc 150 mycpp/gc_builtins_test.cc 447 mycpp/gc_dict_test.cc 745 mycpp/gc_heap_test.cc 479 mycpp/gc_iolib_test.cc 109 mycpp/gc_list_test.cc 561 mycpp/gc_mops_test.cc 173 mycpp/gc_mylib_test.cc 327 mycpp/gc_str_test.cc 1,558 mycpp/gc_stress_test.cc 222 mycpp/gc_tuple_test.cc 139 mycpp/mark_sweep_heap_test.cc 370 mycpp/small_str_test.cc 502 7,811 lines in 27 files
NINJA-config.sh 137 asdl/NINJA_subgraph.py 56 bin/NINJA_subgraph.py 141 build/ninja-rules-cpp.sh 390 build/ninja-rules-py.sh 364 build/ninja_lib.py 555 build/ninja_lib_test.py 255 build/ninja_main.py 432 core/NINJA_subgraph.py 58 cpp/NINJA_subgraph.py 227 data_lang/NINJA_subgraph.py 66 display/NINJA_subgraph.py 16 frontend/NINJA_subgraph.py 175 mycpp/NINJA_subgraph.py 446 osh/NINJA_subgraph.py 32 pea/NINJA_subgraph.py 19 prebuilt/NINJA_subgraph.py 79 yaks/NINJA_subgraph.py 93 ysh/NINJA_subgraph.py 39 3,580 lines in 20 files
This is the input to the translators, written in statically-typed Python. Note that bash is at least 140K lines of code, and OSH implements a large part of bash and more.
bin/oils_for_unix.py 201 builtin/assign_osh.py 581 builtin/bracket_osh.py 292 builtin/completion_osh.py 508 builtin/dirs_osh.py 346 builtin/io_osh.py 182 builtin/misc_osh.py 127 builtin/printf_osh.py 552 builtin/process_osh.py 603 builtin/pure_osh.py 581 builtin/read_osh.py 528 builtin/readline_osh.py 142 builtin/trap_osh.py 322 core/alloc.py 320 core/comp_ui.py 578 core/completion.py 1,517 core/dev.py 767 core/error.py 345 core/executor.py 859 core/main_loop.py 413 core/num.py 24 core/optview.py 58 core/process.py 1,973 core/pyos.py 290 core/pyutil.py 225 core/runtime.asdl 188 core/sh_init.py 293 core/shell.py 1,225 core/state.py 2,799 core/util.py 168 core/value.asdl 220 core/vm.py 342 display/ansi.py 15 display/pp_value.py 513 display/pretty.asdl 30 display/pretty.py 303 display/ui.py 602 frontend/args.py 677 frontend/builtin_def.py 166 frontend/consts.py 399 frontend/flag_def.py 549 frontend/flag_spec.py 349 frontend/flag_util.py 68 frontend/id_kind_def.py 809 frontend/lexer.py 455 frontend/lexer_def.py 1,138 frontend/location.py 576 frontend/match.py 310 frontend/option_def.py 419 frontend/parse_lib.py 405 frontend/py_readline.py 117 frontend/reader.py 280 frontend/signal_def.py 94 frontend/syntax.asdl 651 frontend/syntax_abbrev.py 141 frontend/typed_args.py 684 frontend/types.asdl 61 osh/arith_parse.py 197 osh/bool_parse.py 272 osh/bool_stat.py 135 osh/braces.py 539 osh/cmd_eval.py 2,298 osh/cmd_parse.py 2,862 osh/glob_.py 494 osh/history.py 193 osh/prompt.py 357 osh/sh_expr_eval.py 1,242 osh/split.py 311 osh/string_ops.py 545 osh/tdop.py 342 osh/word_.py 805 osh/word_compile.py 309 osh/word_eval.py 2,449 osh/word_parse.py 2,219 pyext/fanos.c 119 pyext/fastfunc.c 133 pyext/fastfunc.pyi 11 pyext/fastlex.c 389 pyext/fastlex.pyi 21 pyext/libc.c 421 pyext/libc.pyi 18 pyext/line_input.pyi 32 pyext/posix_.pyi 210 44,303 lines in 84 files
Expression grammar, parser, evaluator, etc.
builtin/completion_ysh.py 69 builtin/error_ysh.py 308 builtin/func_eggex.py 186 builtin/func_hay.py 151 builtin/func_misc.py 794 builtin/func_reflect.py 234 builtin/hay_ysh.py 429 builtin/io_ysh.py 301 builtin/json_ysh.py 139 builtin/meta_oils.py 848 builtin/method_dict.py 95 builtin/method_io.py 215 builtin/method_list.py 100 builtin/method_other.py 32 builtin/method_str.py 587 builtin/method_type.py 141 builtin/module_ysh.py 141 builtin/pure_ysh.py 228 ysh/expr_eval.py 1,591 ysh/expr_parse.py 387 ysh/expr_to_ast.py 1,721 ysh/func_proc.py 597 ysh/grammar.pgen2 541 ysh/regex_translate.py 393 ysh/val_ops.py 560 10,788 lines in 26 files
JSON, J8 Notation, ...
data_lang/nil8.asdl 31 data_lang/j8.py 1,386 data_lang/j8_lite.py 54 data_lang/pyj8.py 27 data_lang/j8.h 314 data_lang/j8_libc.c 221 data_lang/j8_libc.h 28 data_lang/utf8.h 170 2,231 lines in 9 files
tools/deps.py 149 tools/fmt.py 25 tools/readlink.py 31 tools/ysh_ify.py 1,125 1,330 lines in 5 files
A comprehensive test suite that compares OSH against other shells. If OSH passes these tests in BOTH Python and C++, it means that the translation works.
spec/alias.test.sh 571 spec/append.test.sh 303 spec/arith-context.test.sh 216 spec/arith-dynamic.test.sh 95 spec/arith.test.sh 998 spec/array-basic.test.sh 45 spec/array-compat.test.sh 189 spec/array.test.sh 803 spec/assign-deferred.test.sh 105 spec/assign-dialects.test.sh 61 spec/assign-extended.test.sh 881 spec/assign.test.sh 674 spec/assoc-zsh.test.sh 53 spec/assoc.test.sh 797 spec/background.test.sh 184 spec/ble-features.test.sh 645 spec/ble-idioms.test.sh 334 spec/blog-other1.test.sh 57 spec/blog1.test.sh 95 spec/blog2.test.sh 60 spec/brace-expansion.test.sh 502 spec/bugs.test.sh 405 spec/builtin-bash.test.sh 192 spec/builtin-bind.test.sh 186 spec/builtin-bracket.test.sh 581 spec/builtin-cd.test.sh 333 spec/builtin-completion.test.sh 596 spec/builtin-dirs.test.sh 241 spec/builtin-echo.test.sh 248 spec/builtin-eval-source.test.sh 345 spec/builtin-getopts.test.sh 347 spec/builtin-history.test.sh 198 spec/builtin-meta.test.sh 620 spec/builtin-misc.test.sh 161 spec/builtin-printf.test.sh 1,284 spec/builtin-process.test.sh 588 spec/builtin-read.test.sh 770 spec/builtin-special.test.sh 141 spec/builtin-times.test.sh 16 spec/builtin-trap-bash.test.sh 657 spec/builtin-trap-err.test.sh 647 spec/builtin-trap.test.sh 366 spec/builtin-type-bash.test.sh 326 spec/builtin-type.test.sh 117 spec/builtin-vars.test.sh 707 spec/case_.test.sh 244 spec/command-parsing.test.sh 64 spec/command-sub.test.sh 285 spec/command_.test.sh 194 spec/comments.test.sh 11 spec/dbracket.test.sh 438 spec/dparen.test.sh 205 spec/empty-bodies.test.sh 24 spec/errexit-osh.test.sh 814 spec/errexit.test.sh 532 spec/exit-status.test.sh 280 spec/explore-parsing.test.sh 42 spec/extglob-files.test.sh 384 spec/extglob-match.test.sh 400 spec/fatal-errors.test.sh 186 spec/for-expr.test.sh 173 spec/func-parsing.test.sh 106 spec/glob.test.sh 388 spec/globignore.test.sh 159 spec/hay-isolation.test.sh 169 spec/hay-meta.test.sh 257 spec/hay.test.sh 655 spec/here-doc.test.sh 370 spec/if_.test.sh 55 spec/interactive-parse.test.sh 37 spec/interactive.test.sh 343 spec/introspect.test.sh 287 spec/let.test.sh 23 spec/loop.test.sh 474 spec/nameref.test.sh 637 spec/nix-idioms.test.sh 196 spec/nocasematch-match.test.sh 71 spec/nul-bytes.test.sh 158 spec/osh-bugs.test.sh 31 spec/parse-errors.test.sh 226 spec/pipeline.test.sh 234 spec/posix.test.sh 153 spec/process-sub.test.sh 204 spec/prompt.test.sh 253 spec/quote.test.sh 231 spec/redirect-command.test.sh 306 spec/redirect-multi.test.sh 308 spec/redirect.test.sh 485 spec/regex.test.sh 631 spec/serialize.test.sh 230 spec/sh-func.test.sh 152 spec/sh-options.test.sh 879 spec/sh-usage.test.sh 94 spec/shell-grammar.test.sh 206 spec/smoke.test.sh 126 spec/strict-options.test.sh 277 spec/subshell.test.sh 23 spec/tilde.test.sh 184 spec/toysh-posix.test.sh 344 spec/toysh.test.sh 143 spec/type-compat.test.sh 159 spec/unicode.test.sh 196 spec/var-num.test.sh 43 spec/var-op-bash.test.sh 397 spec/var-op-len.test.sh 220 spec/var-op-patsub.test.sh 413 spec/var-op-slice.test.sh 403 spec/var-op-strip.test.sh 357 spec/var-op-test.test.sh 520 spec/var-ref.test.sh 509 spec/var-sub-quote.test.sh 312 spec/var-sub.test.sh 60 spec/vars-bash.test.sh 30 spec/vars-special.test.sh 782 spec/whitespace.test.sh 114 spec/word-eval.test.sh 62 spec/word-split.test.sh 421 spec/xtrace.test.sh 369 spec/ysh-TODO-deprecate.test.sh 144 spec/ysh-assign.test.sh 423 spec/ysh-augmented.test.sh 245 spec/ysh-bin.test.sh 20 spec/ysh-blocks.test.sh 339 spec/ysh-bugs.test.sh 302 spec/ysh-builtin-ctx.test.sh 168 spec/ysh-builtin-error.test.sh 436 spec/ysh-builtin-eval.test.sh 661 spec/ysh-builtin-help.test.sh 124 spec/ysh-builtin-meta.test.sh 356 spec/ysh-builtin-module.test.sh 489 spec/ysh-builtin-process.test.sh 85 spec/ysh-builtin-shopt.test.sh 183 spec/ysh-builtins.test.sh 676 spec/ysh-case.test.sh 245 spec/ysh-closures.test.sh 179 spec/ysh-cmd-lang.test.sh 54 spec/ysh-command-sub.test.sh 147 spec/ysh-completion.test.sh 204 spec/ysh-control-flow.test.sh 182 spec/ysh-convert.test.sh 190 spec/ysh-demo.test.sh 59 spec/ysh-dev.test.sh 163 spec/ysh-dict.test.sh 120 spec/ysh-env.test.sh 309 spec/ysh-expr-arith.test.sh 614 spec/ysh-expr-bool.test.sh 185 spec/ysh-expr-compare.test.sh 427 spec/ysh-expr-sub.test.sh 42 spec/ysh-expr.test.sh 442 spec/ysh-for.test.sh 192 spec/ysh-func-builtin.test.sh 250 spec/ysh-func.test.sh 496 spec/ysh-funcs-external.test.sh 86 spec/ysh-int-float.test.sh 220 spec/ysh-interactive.test.sh 41 spec/ysh-json.test.sh 1,251 spec/ysh-keywords.test.sh 37 spec/ysh-list.test.sh 179 spec/ysh-method-io.test.sh 86 spec/ysh-methods.test.sh 595 spec/ysh-multiline.test.sh 179 spec/ysh-namespaces.test.sh 75 spec/ysh-object.test.sh 296 spec/ysh-options-assign.test.sh 106 spec/ysh-options.test.sh 764 spec/ysh-place.test.sh 136 spec/ysh-printing.test.sh 391 spec/ysh-proc-meta.test.sh 241 spec/ysh-proc.test.sh 814 spec/ysh-prompt.test.sh 147 spec/ysh-regex-api.test.sh 939 spec/ysh-regex.test.sh 711 spec/ysh-reserved.test.sh 48 spec/ysh-scope.test.sh 768 spec/ysh-slice-range.test.sh 278 spec/ysh-source.test.sh 29 spec/ysh-special-vars.test.sh 50 spec/ysh-stdlib-args.test.sh 37 spec/ysh-stdlib.test.sh 25 spec/ysh-string.test.sh 593 spec/ysh-tuple.test.sh 28 spec/ysh-unicode.test.sh 184 spec/ysh-usage.test.sh 78 spec/ysh-user-feedback.test.sh 141 spec/ysh-var-sub.test.sh 45 spec/ysh-with-sh.test.sh 221 spec/ysh-word-eval.test.sh 144 spec/ysh-xtrace.test.sh 591 spec/zsh-idioms.test.sh 45 57,238 lines in 190 files
Another suite that tests shells "from the outside". Instead of making explicit assertions, we verify that OSH behaves like bash.
test/gold/abuild.sh 9 test/gold/and-or.sh 27 test/gold/autoconf-backtick.sh 50 test/gold/big-here-doc.sh 1,531 test/gold/case-in-subshell.sh 37 test/gold/char-class.sh 33 test/gold/command-sub-2.sh 12 test/gold/command-sub.sh 39 test/gold/comments.sh 22 test/gold/complex-here-docs.sh 192 test/gold/configure-bug.sh 13 test/gold/de1.sh 24 test/gold/declare.sh 28 test/gold/dollar-sq.sh 42 test/gold/echo-e.sh 36 test/gold/errexit-confusion.sh 139 test/gold/errexit.sh 121 test/gold/export.sh 65 test/gold/glob.sh 26 test/gold/lineno.sh 23 test/gold/nix.sh 32 test/gold/readlink.sh 61 test/gold/readonly.sh 24 test/gold/scope.sh 126 test/gold/strip-op-char-class.sh 6 test/gold/word-eval.sh 57 2,775 lines in 27 files
mycpp generates the big file _gen/bin/oils-for-unix.mycpp.cc. Other programs like Zephyr ASDL and re2c generate other files.
_gen/_tmp/match.re2c-input.h 1,323 _gen/asdl/hnode.asdl.h 197 _gen/bin/oils_for_unix.mycpp.cc 59,990 _gen/bin/oils_for_unix.mycpp.h 5 _gen/bin/text_files.cc 2,023 _gen/core/optview.h 131 _gen/core/runtime.asdl.cc 1,284 _gen/core/runtime.asdl.h 1,038 _gen/core/value.asdl.cc 1,409 _gen/core/value.asdl.h 1,131 _gen/cpp/build_stamp.cc 1 _gen/cpp/build_stamp.h 1 _gen/data_lang/nil8.asdl.cc 253 _gen/data_lang/nil8.asdl.h 229 _gen/display/pretty.asdl.cc 279 _gen/display/pretty.asdl.h 286 _gen/frontend/arg_types.cc 625 _gen/frontend/arg_types.h 826 _gen/frontend/consts.cc 1,844 _gen/frontend/consts.h 57 _gen/frontend/help_meta.cc 497 _gen/frontend/help_meta.h 6 _gen/frontend/id_kind.asdl.cc 937 _gen/frontend/id_kind.asdl.h 481 _gen/frontend/id_kind.asdl_c.h 412 _gen/frontend/match.re2c.h 24,784 _gen/frontend/option.asdl.h 217 _gen/frontend/signal.cc 205 _gen/frontend/signal.h 16 _gen/frontend/syntax.asdl.cc 6,175 _gen/frontend/syntax.asdl.h 5,393 _gen/frontend/types.asdl.h 86 _gen/frontend/types.asdl_c.h 46 _gen/osh/arith_parse.cc 129 _gen/yaks/yaks.asdl.cc 1,187 _gen/yaks/yaks.asdl.h 1,034 _gen/yaks/yaks_main.mycpp.cc 3,754 _gen/yaks/yaks_main.mycpp.h 5 _gen/ysh/grammar_nt.h 76 _gen/ysh/grammar_tables.cc 4,554 122,926 lines in 41 files