| 1 | Oils Repo Overview
|
| 2 | ==================
|
| 3 |
|
| 4 | Try this to show a summary of what's in the repo and their line counts:
|
| 5 |
|
| 6 | $ metrics/source-code.sh overview
|
| 7 |
|
| 8 | (Other functions in this file may be useful as well.)
|
| 9 |
|
| 10 | <div id="toc">
|
| 11 | </div>
|
| 12 |
|
| 13 | ## Executable Spec
|
| 14 |
|
| 15 | ### A Collection of Interpreters
|
| 16 |
|
| 17 | Oils is naturally structured as a set of mutually recursive parsers and
|
| 18 | evaluators. These interpreters are specified at a high-level: with regular
|
| 19 | languages, Zephyr ASDL, and a statically-typed subset of Python.
|
| 20 |
|
| 21 | bin/ # Main entry points like bin/osh (source in bin/oils_for_unix.py)
|
| 22 | frontend/ # Input and lexing common to OSH and YSH
|
| 23 | osh/ # OSH parsers and evaluators (cmd, word, sh_expr)
|
| 24 | ysh/ # YSH parser and evaluator
|
| 25 | data_lang/ # Languages based on JSON
|
| 26 | core/ # Other code shared between OSH and YSH
|
| 27 | builtin/ # Builtin commands and functions
|
| 28 | pylib/ # Borrowed from the Python standard library.
|
| 29 | tools/ # User-facing tools, e.g. the osh2oil translator
|
| 30 | display/ # User interface
|
| 31 |
|
| 32 | ### DSLs / Code Generators
|
| 33 |
|
| 34 | Here are the tools that transform that high-level code to efficient code:
|
| 35 |
|
| 36 | asdl/ # ASDL implementation, derived from CPython
|
| 37 | pgen2/ # Parser Generator, borrowed from CPython
|
| 38 | mycpp/ # Experimental translator from typed Python to C++.
|
| 39 | # Depends on MyPy. See mycpp/README.md
|
| 40 | pea/ # Experiment: a cleaner version of mycpp?
|
| 41 | opy/ # Obsolete Python compiler
|
| 42 |
|
| 43 | ## Multiple Build Systems
|
| 44 |
|
| 45 | ### Dev Build Runs Under CPython
|
| 46 |
|
| 47 | The Oils interpreter can run under a regular Python interpreter! This build is
|
| 48 | **slow**.
|
| 49 |
|
| 50 | build/
|
| 51 | py.sh # For development builds, running CPython
|
| 52 | pyext/ # Python extension modules, e.g. libc.c
|
| 53 | Python-2.7.13/ # For the slow Python build
|
| 54 | Makefile # For the tarball
|
| 55 |
|
| 56 | ### Generate C++, and Build Native Code with a `ninja` Wrapper
|
| 57 |
|
| 58 | We have native code to support the `oils-for-unix` build, which is pure C++.
|
| 59 |
|
| 60 | We build it with a Bazel-like wrapper around `ninja`:
|
| 61 |
|
| 62 | NINJA-config.sh # Generates build.ninja
|
| 63 |
|
| 64 | build/ # High level build
|
| 65 | ninja_main.py # invoked by NINJA-config.sh
|
| 66 | ninja_lib.py # build rules
|
| 67 | ninja-rules-cpp.sh
|
| 68 | ninja-rules-py.sh
|
| 69 | cpp/ # C++ code which complements the mycpp translation
|
| 70 | NINJA_subgraph.py
|
| 71 | mycpp/ # Runtime for the translator
|
| 72 | NINJA_subgraph.py
|
| 73 |
|
| 74 | prebuilt/ # Prebuilt files committed to git, instead of in
|
| 75 | # _gen/
|
| 76 |
|
| 77 | # Temp dirs (see below)
|
| 78 | _bin/
|
| 79 | _build/
|
| 80 | _gen/
|
| 81 | _test/
|
| 82 |
|
| 83 | ### End User Build System Has Few Dependencies
|
| 84 |
|
| 85 | Distro maintainers or end users should build from the `oils-for-unix` tarball,
|
| 86 | not the repo. ([The Oils Repo Is Different From the Tarball
|
| 87 | Releases](https://github.com/oils-for-unix/oils/wiki/The-Oils-Repo-Is-Different-From-the-Tarball-Releases).)
|
| 88 |
|
| 89 | We ship these files in the tarball:
|
| 90 |
|
| 91 | configure
|
| 92 | _build/
|
| 93 | oils.sh # generated shell script
|
| 94 | install
|
| 95 |
|
| 96 | So instead of running `ninja`, end users run `_build/oils.sh`, which invokes
|
| 97 | the same "actions" as `ninja`.
|
| 98 |
|
| 99 | This means they don't need to install `ninja` — they only need a C++
|
| 100 | compiler and a shell.
|
| 101 |
|
| 102 | ### Build Dependencies
|
| 103 |
|
| 104 | TODO: this section is out of date. We now use "wedges" in `~/wedge`.
|
| 105 |
|
| 106 | These tools are built from shell scripts in `soil/`. The `oil_DEPS` dir is
|
| 107 | "parallel" to Oils because it works better with container bind mounds.
|
| 108 |
|
| 109 | ../oil_DEPS/
|
| 110 | re2c/ # to build the lexer
|
| 111 | cmark/ # for building docs
|
| 112 | spec-bin/ # shells to run spec tests against
|
| 113 | mypy/ # MyPy repo
|
| 114 | mycpp-venv/ # MyPy binaries deps in a VirtualEnv
|
| 115 |
|
| 116 | py3/ # for mycpp and pea/
|
| 117 | cpython-full/ # for bootstrapping Oils-CPython
|
| 118 |
|
| 119 | ## Dev Tools
|
| 120 |
|
| 121 | ### Several Kinds of Tests
|
| 122 |
|
| 123 | Unit tests are named `foo_test.py` and live next to `foo.py`.
|
| 124 |
|
| 125 | test/ # Test automation
|
| 126 | gold/ # Gold Test cases
|
| 127 | gold.sh
|
| 128 | sh_spec.py # shell spec test framework
|
| 129 | spec.sh # Types of test runner: spec, unit, gold, wild
|
| 130 | unit.sh
|
| 131 | wild.sh
|
| 132 | testdata/
|
| 133 | spec/ # Spec test cases
|
| 134 | bin/ # tools used in many spec tests
|
| 135 | testdata/ # scripts for specific test cases
|
| 136 | stateful/ # Tests that use pexpect
|
| 137 |
|
| 138 | ### Scripts
|
| 139 |
|
| 140 | We use a lot of automation to improve the dev process. It's largely written in
|
| 141 | shell, of course!
|
| 142 |
|
| 143 | benchmarks/ # Benchmarks should be run on multiple machines.
|
| 144 | metrics/ # Metrics don't change between machines (e.g. code size)
|
| 145 | client/ # Demonstration of OSH as a headless server.
|
| 146 | deps/ # Dev dependencies and Docker images
|
| 147 | devtools/ # For Oils developers (not end users)
|
| 148 | release.sh # The (large) release process.
|
| 149 | services/ # talk to cloud services
|
| 150 | demo/ # Demonstrations of bash/shell features. Could be
|
| 151 | # moved to tests/ if automated.
|
| 152 | old/ # A junk drawer.
|
| 153 | web/ # HTML/JS/CSS for tests and tools
|
| 154 | soil/ # Multi-cloud continuous build (e.g. sourcehut, Github)
|
| 155 |
|
| 156 | ### Temp Dirs
|
| 157 |
|
| 158 | Directories that begin with `_` are **not** stored in `git`. The dev tools
|
| 159 | above create and use these dirs.
|
| 160 |
|
| 161 | _bin/ # Native executables are put here
|
| 162 | cxx-dbg/
|
| 163 | _build/ # Temporary build files
|
| 164 | _cache/ # Dev dependency tarballs
|
| 165 | _devbuild/ # Generated Python code, etc.
|
| 166 | _gen/ # Generated C++ code that mirrors the repo
|
| 167 | frontend/
|
| 168 | _release/ # Source release tarballs are put here
|
| 169 | VERSION/ # Published at oilshell.org/release/$VERSION/
|
| 170 | benchmarks/
|
| 171 | doc/
|
| 172 | metrics/
|
| 173 | test/
|
| 174 | spec.wwz
|
| 175 | wild.wwz
|
| 176 | ...
|
| 177 | web/ # Static files, copy of $REPO_ROOT/web
|
| 178 | table/
|
| 179 | _test/ # Unit tests, mycpp examples
|
| 180 | tasks/
|
| 181 | _tmp/ # Output of other test suites; temp files
|
| 182 | spec/
|
| 183 | wild/
|
| 184 | raw/
|
| 185 | www/
|
| 186 | osh-parser/
|
| 187 | osh-runtime/
|
| 188 | vm-baseline/
|
| 189 | startup/
|
| 190 | ...
|
| 191 |
|
| 192 | ## Docs
|
| 193 |
|
| 194 | doc/ # A mix of docs
|
| 195 | doctools/ # Tools that use lazylex/ to transform Markdown/HTML
|
| 196 | data_lang/ # doctools/ builds upon the "HTM8" subset in this dir
|
| 197 | README.md # This page, which is For Oils developers
|
| 198 |
|
| 199 | LICENSE.txt # For end users
|
| 200 | INSTALL.txt
|
| 201 |
|
| 202 | ## Related
|
| 203 |
|
| 204 | - [README.md](oils-repo/README.html). If you want to modify Oils, start here.
|
| 205 | We welcome contributions!
|