OILS / README.md View on Github | oilshell.org

318 lines, 235 significant
1Oils Source Code
2================
3
4[![Build
5Status](https://github.com/oils-for-unix/oils/actions/workflows/all-builds.yml/badge.svg)](https://github.com/oils-for-unix/oils/actions/workflows/all-builds.yml) <a href="https://gitpod.io/from-referrer/">
6 <img src="https://img.shields.io/badge/Contribute%20with-Gitpod-908a85?logo=gitpod" alt="Contribute with Gitpod" />
7</a>
8
9[Oils][] is our upgrade path from bash to a better language and runtime!
10
11- [OSH][] runs your existing shell scripts.
12- [YSH][] is for Python and JavaScript users who avoid shell.
13
14(The project was [slightly renamed][rename] in March 2023, so there are still
15old references to "Oil". Feel free to send pull requests with corrections!)
16
17[Oils]: https://www.oilshell.org/
18
19[OSH]: https://www.oilshell.org/cross-ref.html#OSH
20[YSH]: https://www.oilshell.org/cross-ref.html#YSH
21
22[rename]: https://www.oilshell.org/blog/2023/03/rename.html
23
24[Oils 2023 FAQ][faq-2023] / [Why Create a New Unix Shell?][why]
25
26[faq-2023]: https://www.oilshell.org/blog/2023/03/faq.html
27[why]: https://www.oilshell.org/blog/2021/01/why-a-new-shell.html
28
29It's written in Python, so the code is short and easy to change. But we
30automatically translate it to C++ with custom tools, to make it fast and small.
31The deployed executable doesn't depend on Python.
32
33This README is at the root of the [git repo][git-repo].
34
35[git-repo]: https://github.com/oils-for-unix/oils
36
37<div id="toc">
38</div>
39
40## Contributing
41
42* Try making the **dev build** of Oils with the instructions on the
43 [Contributing][] page. This should take 1 to 5 minutes if you have a Linux
44 machine.
45* If it doesn't, let us know. You can post on the `#oil-dev` channel of
46 [oilshell.zulipchat.com][], or file an issue on Github.
47* Feel free to grab an [issue from
48 Github](https://github.com/oils-for-unix/oils/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
49 Let us know what you're thinking before you get too far.
50
51[Contributing]: https://github.com/oils-for-unix/oils/wiki/Contributing
52[oilshell.zulipchat.com]: https://oilshell.zulipchat.com/
53[blog]: https://www.oilshell.org/blog/
54
55### Quick Start on Linux
56
57After following the instructions on the [Contributing][] page, you'll have a
58Python program that you can quickly run and change! Try it interactively:
59
60 bash$ bin/osh
61
62 osh$ name=world
63 osh$ echo "hello $name"
64 hello world
65
66- Try running a shell script you wrote with `bin/osh myscript.sh`.
67- Try [YSH][] with `bin/ysh`.
68
69Let us know if any of these things don't work! [The continuous
70build](http://travis-ci.oilshell.org/) tests them at every commit.
71
72### Dev Build vs. Release Build
73
74Again, note that the **developer build** is **very different** from the release
75tarball. The [Contributing][] page describes this difference in detail.
76
77The release tarballs are linked from the [home
78page](https://www.oilshell.org/). (Developer builds don't work on OS X, so use
79the release tarballs on OS X.)
80
81### Important: We Accept Small Contributions!
82
83Oils is full of [many ideas](https://www.oilshell.org/blog/), which may be
84intimidating at first.
85
86But the bar to contribution is very low. It's basically a medium size Python
87program with many tests, and many programmers know how to change such programs.
88It's great for prototyping.
89
90- For OSH compatibility, I often merge **failing [spec
91 tests](https://www.oilshell.org/cross-ref.html#spec-test)**. You don't even
92 have to write code! The tests alone help. I search for related tests with
93 `grep xtrace spec/*.test.sh`, where `xtrace` is a shell feature.
94- You only have to make your code work **in Python**. Plain Python programs
95 are easy to modify. The semi-automated translation to C++ is a separate
96 step, although it often just works.
97- You can **influence the design** of [YSH][]. If you have an itch to
98 scratch, be ambitious. For example, you might want to show us how to
99 implement [nonlinear pipelines](https://github.com/oils-for-unix/oils/issues/843).
100
101### I aim for 24 hour response time
102
103Please feel free to ping `andychu` on Zulip or Github if you're **waiting** for
104a pull request review! (or to ask questions)
105
106Usually I can respond in 24 hours. I might be traveling, in which case I'll
107respond with something like *I hope to look at this by Tuesday*.
108
109I might have also **missed** your Github message, so it doesn't hurt to ping
110me.
111
112Thank you for the contributions!
113
114### Docs
115
116The [Wiki](https://github.com/oils-for-unix/oils/wiki) has many developer docs. Feel
117free to edit them. If you make a major change, let us know on Zulip!
118
119There are also READMEs in some subdirectories, like `opy/` and `mycpp/`.
120
121If you're confused, the best thing to do is to ask on Zulip and someone should
122produce a pointer and/or improve the docs.
123
124Docs for **end users** are linked from each [release
125page](https://www.oilshell.org/releases.html).
126
127## Repository Structure
128
129Try this to show a summary of what's in the repo and their line counts:
130
131 $ metrics/source-code.sh overview
132
133(Other functions in this file may be useful as well.)
134
135### A Collection of Interpreters
136
137Oils is naturally structured as a set of mutually recursive parsers and
138evaluators. These interpreters are specified at a high-level: with regular
139languages, Zephyr ASDL, and a statically-typed subset of Python.
140
141 bin/ # Main entry points like bin/osh (source in bin/oils_for_unix.py)
142 frontend/ # Input and lexing common to OSH and YSH
143 osh/ # OSH parsers and evaluators (cmd, word, sh_expr)
144 ysh/ # YSH parser and evaluator
145 data_lang/ # Languages based on JSON
146 core/ # Other code shared between OSH and YSH
147 builtin/ # Builtin commands and functions
148 pyext/ # Python extension modules, e.g. libc.c
149 pylib/ # Borrowed from the Python standard library.
150 tools/ # User-facing tools, e.g. the osh2oil translator
151 display/ # User interface
152
153### DSLs / Code Generators
154
155Here are the tools that transform that high-level code to efficient code:
156
157 asdl/ # ASDL implementation, derived from CPython
158 pgen2/ # Parser Generator, borrowed from CPython
159 mycpp/ # Experimental translator from typed Python to C++.
160 # Depends on MyPy. See mycpp/README.md
161 pea/ # Perhaps a cleaner version of mycpp
162 opy/ # Python compiler in Python (mycpp/ will replace it)
163
164### Native Code and Build System
165
166We have native code to support both the dev build (running under CPython) and
167the `oils-for-unix` build (pure C++):
168
169 NINJA-config.sh # Generates build.ninja
170
171 build/ # High level build
172 NINJA-steps.sh
173 NINJA_main.py # invoked by NINJA-config.sh
174 NINJA_subgraph.py
175 oil-defs/ # Files that define our slice of CPython.
176 py.sh # For development builds, running CPython
177 cpp/ # C++ code which complements the mycpp translation
178 NINJA-steps.sh
179 NINJA_subgraph.py
180 mycpp/ # Runtime for the translator
181 NINJA-steps.sh
182 NINJA_subgraph.py
183
184 prebuilt/ # Prebuilt files committed to git, instead of in _gen/
185
186 Python-2.7.13/ # For the slow Python build
187
188 # Temp dirs (see below)
189 _bin/
190 _build/
191 _gen/
192 _test/
193
194### Several Kinds of Tests
195
196Unit tests are named `foo_test.py` and live next to `foo.py`.
197
198 test/ # Test automation
199 gold/ # Gold Test cases
200 gold.sh
201 sh_spec.py # shell spec test framework
202 spec.sh # Types of test runner: spec, unit, gold, wild
203 unit.sh
204 wild.sh
205 testdata/
206 spec/ # Spec test cases
207 bin/ # tools used in many spec tests
208 testdata/ # scripts for specific test cases
209 stateful/ # Tests that use pexpect
210
211### Dev Tools and Scripts
212
213We use a lot of automation to improve the dev process. It's largely written in
214shell, of course!
215
216 benchmarks/ # Benchmarks should be run on multiple machines.
217 metrics/ # Metrics don't change between machines (e.g. code size)
218 client/ # Demonstration of OSH as a headless server.
219 deps/ # Dev dependencies and Docker images
220 devtools/ # For Oils developers (not end users)
221 release.sh # The (large) release process.
222 services/ # talk to cloud services
223 demo/ # Demonstrations of bash/shell features. Could be
224 # moved to tests/ if automated.
225 old/ # A junk drawer.
226 web/ # HTML/JS/CSS for tests and tools
227 soil/ # Multi-cloud continuous build (e.g. sourcehut, Github)
228
229### Temp Dirs
230
231Directories that begin with `_` are **not** stored in `git`. The dev tools
232above create and use these dirs.
233
234 _bin/ # Native executables are put here
235 cxx-dbg/
236 _build/ # Temporary build files
237 _cache/ # Dev dependency tarballs
238 _devbuild/ # Generated Python code, etc.
239 _gen/ # Generated C++ code that mirrors the repo
240 frontend/
241 _release/ # Source release tarballs are put here
242 VERSION/ # Published at oilshell.org/release/$VERSION/
243 benchmarks/
244 doc/
245 metrics/
246 test/
247 spec.wwz
248 wild.wwz
249 ...
250 web/ # Static files, copy of $REPO_ROOT/web
251 table/
252 _test/ # Unit tests, mycpp examples
253 tasks/
254 _tmp/ # Output of other test suites; temp files
255 spec/
256 wild/
257 raw/
258 www/
259 osh-parser/
260 osh-runtime/
261 vm-baseline/
262 oheap/
263 startup/
264 ...
265
266### Build Dependencies in `../oil_DEPS`
267
268These tools are built from shell scripts in `soil/`. The `oil_DEPS` dir is
269"parallel" to Oils because it works better with container bind mounds.
270
271 ../oil_DEPS/
272 re2c/ # to build the lexer
273 cmark/ # for building docs
274 spec-bin/ # shells to run spec tests against
275 mypy/ # MyPy repo
276 mycpp-venv/ # MyPy binaries deps in a VirtualEnv
277
278 py3/ # for mycpp and pea/
279 cpython-full/ # for bootstrapping Oils-CPython
280
281
282### Build System for End Users version.
283
284These files make the slow "Oils Python" build, which is very different than the
285**developer build** of Oils.
286
287 Makefile
288 configure
289 install
290
291These files are for the C++ `oils-for-unix` tarball (in progress):
292
293 _build/
294 oils.sh
295
296### Doc Sources
297
298 doc/ # A mix of docs
299 doctools/ # Tools that use lazylex/ to transform Markdown/HTML
300 lazylex/ # An HTML lexer which doctools/ builds upon.
301 README.md # This page, which is For Oils developers
302
303 LICENSE.txt # For end users
304 INSTALL.txt
305
306## More info
307
308There are README files in many subdirectories, like
309[mycpp/README.md](mycpp/README.md).
310
311* [The blog][blog] has updates on the project status.
312* [Oils Home Page](https://www.oilshell.org/)
313* [oilshell.zulipchat.com][] is for any kind of discussion
314* Subscribe for updates:
315 * [/r/oilshell on Reddit](https://www.reddit.com/r/oilshell/)
316 * [@oilsforunix on Twitter](https://twitter.com/oilsforunix)
317
318