Install

Each release has an archive for each platform, holding two executables, the optional libraries, this guide, the example applications and the tests: shen-jit, shen-jit-stlib, which has the standard library loaded, lib/, include/, guide/index.html, example-apps/ and tests/.

On Linux, take a dynamic build unless your system is too old for it or has a different C library: those are built against an old glibc so that they run on newer ones. A static build runs anywhere, but cannot call C. The executable is the whole installation, so copy it anywhere, such as a directory on your PATH, unless you use the libraries in lib/, which you then name by their path in the archive or source tree.

tar xzf shen-jit-VERSION-linux-x86_64.tar.gz
cd shen-jit-VERSION-linux-x86_64
./shen-jit --version

If macOS refuses to run a downloaded executable, clear its quarantine: xattr -dr com.apple.quarantine . in the directory you unpacked.

tar xzf shen-jit-VERSION.tar.gz
cd shen-jit-VERSION
make                              # build/shen-jit
make stlib                        # build/shen-jit-stlib
make install PREFIX=/usr/local    # copies build/shen-jit to PREFIX/bin

pacman -S --needed make diffutils mingw-w64-ucrt-x86_64-gcc
make release stlib    # from a source tarball: build/shen-jit.exe, build/shen-jit-stlib.exe

First steps

With no arguments, shen-jit starts the REPL.

(0-) (+ 1 2)
3

(1-) (define greet Name -> (cn "hello, " Name))
(fn greet)

(2-) (greet "world")
"hello, world"

(3-) (map (+ 1) [1 2 3])
[2 3 4]

Errors name the value they are about, and show where they happened:

(4-) (define test X Y -> (+ X Y))
(fn test)

(5-) (test 1 "A")
"A" is not a number
  in test

• Ctrl-C stops a running evaluation, or a program waiting for input, and returns to the prompt. At the prompt it drops what you have typed of an expression.
• Ctrl-D ends the REPL. On Windows, use Ctrl-Z then Enter, or (shen-jit.exit 0).
• Line editing: the REPL has none of its own. For history and arrow keys, run it under rlwrap: rlwrap shen-jit.

Running programs

shen-jit                               # the REPL
shen-jit script prog.shen a b          # *argv* is ["prog.shen" "a" "b"]
shen-jit eval -l prog.shen -e '(main)' # load a file, evaluate an expression

The commands come from the kernel's launcher (shen-jit --help):

An error that nothing catches, or an unknown argument, ends the process with status 1.

Options for Shen/JIT itself go before the command:

A script whose first line starts with #! can run directly, with Shen/JIT on your PATH, and load skips that line. This does not work on Windows.

#!/usr/bin/env -S shen-jit script
(output "~A~%" (tl (value *argv*)))

The language

Everything here is standard Shen, and the Shen site teaches the language. A short taste, starting with functions defined by pattern matching:

(define total
  [] -> 0
  [X | Xs] -> (+ X (total Xs)))

(total [1 2 3])    \\ 6

Static typing is optional, and (tc +) turns it on:

(0-) (tc +)
true

(1+) (define inc {number --> number} X -> (+ X 1))
(fn inc) : (number --> number)

(2+) (inc 1)
2 : number

(3+) (inc "a")
type error

Prolog and a compiler-compiler are built in:

(defprolog parent
  ann bob <--;
  bob cid <--;)

(defprolog grandparent
  X Z <-- (parent X Y) (parent Y Z);)

(prolog? (grandparent ann cid))    \\ true

(defcc <digits>
  0 <digits> := [0 | <digits>];
  1 <digits> := [1 | <digits>];
  <e> := [];)

(compile (fn <digits>) [1 0 1])    \\ [1 0 1]

The standard library

shen-jit leaves the kernel's standard library out. shen-jit-stlib has it already loaded: maths, rationals, complex numbers, numerals, lists, strings, vector macros, pretty printing, files and tuples. It is about 2 MB, and starts in about the same time.

shen-jit-stlib eval -e '(prime? 7)'    # true

The names it exports are system functions, so a program running on it cannot define its own take or drop. That is why it is not in the default image.

It is the kernel's precompiled klambda/stlib.kl, which a source tarball carries as generated/stlib/stlib.kl. The kernel's build of that file leaves out Encrypt, Regex, Calendar, Data and Tk, whose sources are in its lib/. Any binary can load that file and save itself, which is all make stlib does. To add a library or a program of your own, see Saving a program.

shen-jit script tools/stlib.shen stlib.kl shen-jit-stlib

Saving a program

A program can write its own binary, with nothing but the shen-jit it runs on. The result is about 1 MB; 2 on Linux unless the one running comes from make release, as gcc's debug information comes along.

(define main -> (output "hello ~A~%" (hd (tl (value *argv*)))))
(shen-jit.save-executable "hello" main)

shen-jit eval -q -l hello.shen
./hello world    # hello world

• It runs the entry point, a function of no arguments, instead of the launcher. All its arguments are its own, in *argv* after its name, and its exit status is 0 unless it calls shen-jit.exit or an error reaches the top. With [] as the entry point, it keeps the launcher: repl, script, eval.
• It does not return: it writes the file and ends the process. The path is relative to *home-directory*, as open's is.
• All the heap lasts: definitions, globals, property lists, macros, datatypes and type signatures, so code loaded later type checks against them. The stack and open files do not, and *home-directory* starts empty.
• It runs only on the kind of system it was made on. It holds bytecode, compiled to native code on first call as always.
• On macOS it keeps the linker's ad-hoc signature, which runs. codesign --verify rejects it, as it does shen-jit, for the image appended to the runtime, which matters only for notarized distribution.

To save the heap as an image instead, without an executable:

shen-jit --save app.img eval -q -l app.shen
SHEN_JIT_IMAGE=app.img shen-jit eval -e '(main)'

An image starts with your program already loaded, and loads only into the build that wrote it. After a build from source, one appended to the runtime makes an executable of its own:

cat build/runtime app.img > app && chmod +x app
./app eval -e '(main)'

Threads

(thread Frozen) runs a frozen expression in a new thread and returns true, as programs written for Shen/Scheme expect. Threads are real, and one runs at a time, holding a global lock. So they give concurrency, for work that waits or runs beside the main program, and no speed-up from more cores.

(set done false)
(define work 0 -> (set done true) N -> (work (- N 1)))
(thread (freeze (work 1000000)))

(define wait -> (if (value done) finished (wait)))
(wait)    \\ finished

• Switching: every 10 ms if another thread is waiting, at a function's entry or a loop's next turn, and whenever a thread waits for input on stdin or a socket. Between those points no other thread runs, though a program should not count on where they fall.
• Errors: one that nothing catches ends its thread and is printed to stderr. The other threads go on.
• The end: the process ends when the main thread does, or when any thread calls (shen-jit.exit N). An image saved with --save holds what the main thread reaches, and shen-jit.save-executable refuses to run while other threads do.
• Ctrl-C at the REPL stops the main thread's work, and threads beside it go on.
• Not provided, as in Shen/Scheme: join, mutexes and channels. A thread signals another through a global, which the other can spin on, as above.
• Waiting elsewhere: reading a file that blocks, such as a FIFO, holds the lock, and Ctrl-C does not end that wait.
• Cost: nothing until the first thread starts. All code then compiles again on its next call, with a test at each loop's turn. The time is shared with the threads that are running.

Sealing

Kernel and port functions are sealed: calls to them are direct, and redefining one raises an error: F is sealed and cannot be redefined, or for the names the kernel exports, F is a system function. Your definitions are open: you can redefine them at any time, and their callers see the new definition.

With --seal, SHEN_JIT_SEAL=1 or (shen-jit.seal +), new definitions are sealed too, so calls to them are direct, and they cannot be redefined. (shen-jit.seal -) turns that off. This gains little, as open calls are inlined too, and patched when a function is redefined.

Datalog

The optional lib/datalog.shen evaluates relations, facts and recursive rules bottom up, to a fixed point, with each rule compiled through the JIT and its join order chosen from the sizes of the relations as it runs. Datalog covers loading, the language, the API and an example.

Bytes and bits

Primitives for raw bytes, with every access checked. The collector frees the buffers, and saved executables keep them.

(define fnv1a
  _ I I H -> H
  B I N H -> (fnv1a B (+ I 1) N
               (shen-jit.bit-and (* (shen-jit.bit-xor H (shen-jit.u8 B I)) 16777619) 4294967295)))

(let B (shen-jit.string->bytes "a")
  (fnv1a B 0 (shen-jit.bytes-length B) 2166136261))    \\ 3826002220

On a fixnum's 62 bits: shen-jit.bit-and, bit-or, bit-xor, bit-not, shift-left, shift-right, rotate32, bswap32, popcount, clz and ctz.

Shen/JIT instructions and frames, "Byte buffers and bit operations", has the details.

Buffer views and cursors

The optional buffer/cursor libraries provide checked ranges over managed bytes or sized pointers, copying, logical equality and mutable positions. Buffers and cursors covers loading, the API, ownership, typed clients and the JSON/binary examples.

Chunk I/O

Optional lib/io.shen adds bounded readers and writers over byte views, with memory endpoints; lib/io-file.shen adds native file descriptors through the FFI. They use their own endpoints and buffers, independently of Shen's default I/O. See Chunk I/O for loading, file helpers, partial transfers, cleanup and typed results. Binary framing adds bounded messages over those readers; Socket I/O endpoints connect the same I/O API to TCP connections. Operation deadlines provide total monotonic budgets; HTTP head framing captures bounded header blocks while preserving body bytes. HTTP requests and responses adds complete request heads, bounded bodies and response encoding. Streaming HTTP responses support bounded producers and chunked output. Bounded HTTP servers covers connection limits and controlled shutdown, with a SQLite counter application in example-apps/sqlite-http/.

JSON provides bounded UTF-8 decoding and encoding, lossless number tokens and typed values. HTTP request bodies can use Content-Length or bounded chunked framing.

Calling C

lib/ffi.shen lets Shen call plain C functions, with nothing to compile. It is not in the image, and has to be loaded before the code that uses it, as a macro applies only to files read after it is defined.

(shen-jit.library libm ["libm.so.6" "libm.dylib" "ucrtbase.dll"])
(define-foreign c-cos libm "cos" (double) --> double)

(output "~A~%" (c-cos 0))    \\ 1.0

shen-jit eval -l lib/ffi.shen -l cos.shen

A library is a symbol and the file names to try, in order. [] means the executable and what it links, which on Windows reaches nothing, as the executable is static. A static Linux build cannot load libraries, and says so.

C's int is int32. A function returning int leaves the upper half of the result register undefined, so declaring it int64 reads garbage. There is deliberately no type called int or long, as long is 32 bits on Windows and 64 elsewhere.

A function takes at most 6 arguments that are not doubles and 8 that are, and no structs by value. After the result, two flags:

• blocking gives up the lock during the call, so other threads run. Its byte buffers are copied and back, as another thread's collection may move them.
• (varargs N) marks a variadic function with N fixed arguments, from 1 to 3, all integers.

(define-foreign c-recv sockets "recv" (int32 bytes int64 int32) --> int32 blocking)
(define-foreign c-fcntl sockets "fcntl" (int32 int32 int32) --> int32 (varargs 2))

Functions are found at their first call, so a saved program finds them again. One that cannot be found raises foreign: cannot find cos in libm. (shen-jit.errno) is what the last call on this thread left in errno, or on Windows in GetLastError, where (shen-jit.c-errno) is the C library's.

Calling C in full has the rest, and how it works.

Memory outside the heap

A byte buffer lives in the Shen heap, where the collector may move it. Memory outside the heap stays where it is, so C can keep its address, a blocking call needs no copy, and the collector never copies or scans it, however large.

It is a pointer with a length. A pointer from C has none, and cannot be read until you state how many bytes it reaches. Then it works with what works on a byte buffer, in place, each access checked against that length:

\\ struct point { double x, y; int32_t id; };   /* 24 bytes */
(define-foreign make-points mine "make_points" (int32) --> pointer)
(define-foreign free-points mine "free_points" (pointer) --> void)

(let P (make-points 1000)
     M (shen-jit.pointer-sized P 24000)
  (do (shen-jit.f64! M 72 1.5)                                     \\ x of point 3
      (output "~A ~A~%" (shen-jit.f64 M 72) (shen-jit.s32 M 88))   \\ and its id
      (shen-jit.pointer-kill M)
      (free-points P)))

lib/memory.shen allocates it, in Shen alone. Give it to C as a pointer:

(define-foreign c-read libc "read" (int32 pointer int64) --> int64 blocking)

(let M (memory.alloc 1073741824)    \\ 1 GB, all zero
  (do (c-read Fd M 1073741824)
      (output "~A~%" (shen-jit.u32 M 0))
      (memory.free M)))

shen-jit eval -l lib/ffi.shen -l lib/memory.shen -l program.shen

• Nothing frees it for you, unless you ask for that below. memory.free frees and kills it, and forgetting to leaks, as in C.
• The length you state is trusted, and it is the one thing not checked. A wrong one, or memory C has freed, can crash. Kill a pointer once its memory is gone, so that using it raises.
• Each sized pointer stands alone. Killing one kills no other, nor the pointer it came from.
• A saved image holds it dead, and the memory itself is not saved: a byte buffer is what lasts in an image.
• The same code works on both. u8, u16, u32, s32 and f64, and the stores u8!, u16!, u32! and f64!, compile inline for a buffer and for a pointer. The others call C for both.

(let M (mmap.read "data.bin")
  (do (output "~A bytes, starting ~A~%" (shen-jit.bytes-length M) (shen-jit.u8 M 0))
      (mmap.close M)))

shen-jit eval -l lib/ffi.shen -l lib/mmap.shen -l program.shen

(define buffer
  N -> (handle.wrap (memory.alloc N) (/. M (memory.free M))))

(define mapped
  Path -> (handle.wrap (mmap.read Path) (/. M (mmap.close M))))

shen-jit eval -l lib/ffi.shen -l lib/memory.shen -l lib/mmap.shen -l lib/handle.shen -l program.shen

SQLite

lib/sqlite.shen is written over the FFI, in Shen alone. It needs SQLite's library, which macOS and Windows 10 have, and most Linux systems do.

(let Db (sqlite.open "people.db")
  (do (sqlite.exec Db "CREATE TABLE IF NOT EXISTS people (name TEXT, age INTEGER)")
      (sqlite.run Db "INSERT INTO people VALUES (?, ?)" ["Ann" 30])
      (output "~S~%" (sqlite.query Db "SELECT name, age FROM people WHERE age > ?" [20]))
      (sqlite.close Db)))    \\ [["Ann" 30]]

shen-jit eval -l lib/ffi.shen -l lib/sqlite.shen -l people.shen

To use a statement many times: sqlite.prepare, sqlite.bind, sqlite.step, which gives row or done, sqlite.row, sqlite.reset and sqlite.finalize. The comments at the top of the file list every function.

Values go in and come out as Shen values: a number is an INTEGER or a REAL, a string is TEXT, a byte buffer is a BLOB, and [] is NULL. true and false go in as 1 and 0. Errors raise, with SQLite's message. A connection belongs to one thread: other threads run while a query does, and some builds of SQLite, Apple's among them, do not allow two threads on one.

Sockets

lib/socket.shen is TCP, over IPv4 and IPv6, also in Shen alone.

(let S (socket.connect "example.com" 80)
  (do (socket.send S "GET / HTTP/1.0c#13;c#10;c#13;c#10;")
      (output "~A~%" (socket.recv-string S 4096))
      (socket.close S)))

shen-jit eval -l lib/ffi.shen -l lib/socket.shen -l program.shen

A server, in a thread, with a client beside it:

(set l (socket.listen "127.0.0.1" 0))

(thread (freeze (let S (socket.accept (value l))
                  (do (socket.send S (socket.recv S 100))
                      (socket.close S)))))

(let C (socket.connect "127.0.0.1" (socket.port (value l)))
  (do (socket.send C "hello")
      (output "~A~%" (socket.recv-string C 100))    \\ hello
      (socket.close C)))

A socket is an integer. Waiting lets other threads run, and Ctrl-C at the REPL ends it. A send to a closed connection raises instead of ending the process. The comments at the top of the file list every function.

C written for Shen

C can also take and return Shen values and call Shen, with include/shen.h. It links against nothing, as the runtime hands it a table of functions.

#include "shen.h"

shen_value sum_list(shen_vm *vm, shen_value *args)
{
    int64_t sum = 0;
    for (shen_value l = args[0]; shen_is_pair(vm, l); l = shen_tail(vm, l)) {
        shen_value x = shen_head(vm, l);
        if (!shen_is_fixnum(x))
            return shen_raise_value(vm, "sum_list: ~S is not an integer", x);
        sum += shen_fixnum(x);
    }
    return shen_make_fixnum(sum);
}

cc -shared -fPIC -I include -o libmine.so mine.c

(shen-jit.library mine ["./libmine.so" "./libmine.dylib" "mine.dll"])
(define-foreign-procedure sum-list mine "sum_list" 1)

(sum-list [1 2 3])    \\ 6

• Values are valid until the next call that may allocate, unless in a root: shen_root(vm, &local) for a call, or shen_root_new for longer. Reads do not allocate, so the loop above needs no root.
• Errors come back by value, as SHEN_ERROR. shen_call calls Shen, and no longjmp crosses C, so a C library calling back is safe.
• Waiting: shen_release and shen_acquire let other threads run while C waits.
• Limits: up to 8 arguments, and a thread the runtime did not start cannot call in.

Test with SHEN_JIT_GC_STRESS=1, which collects at every allocation and finds most mistakes with values. shen.h opens with the rules. This API is not stable until 1.0: it may change before then, and SHEN_API_VERSION says when it has.

Native code by hand

lib/native.shen lets a program write a procedure in Shen/JIT's own instructions, choosing its slots, branches and calls, and the JIT compiles it like any other. It is not in the image, and has to be loaded before the code that uses it, under (tc -).

(define-native sum-list (Xs Acc)
  (prim T cons? Xs)
  (jump-if-false T done)
  (prim H hd Xs)
  (prim Acc + Acc H)
  (prim Xs tl Xs)
  (tail-call sum-list Xs Acc)
  (label done)
  (return Acc))

(sum-list [1 2 3] 0)    \\ 6

shen-jit eval -l lib/native.shen -l sum.shen

A variable is a slot, a symbol is itself, and [1 2] is a list. The code is checked first, by the rules the lowerer's own output is held to, so wrong code is refused instead of crashing. A program generating code at run time passes lists to shen-jit.assemble and shen-jit.install-code. Slots hold tagged values, as in all Shen code, and bytes and bits and memory outside the heap are primitives here as anywhere.

For numeric work, real keeps doubles unboxed in registers, in code that calls nothing. It reads an expression. word does the same for 64-bit integers, which wrap, for hashes and random numbers, and as an index, which then has no tag to check. (loop L) marks the head of a loop and (again L) goes back to it, and loops can nest:

(define-native dot (A B End)
  (real Acc 0)
  (word I 0)
  (loop terms)
  (jump-unless-word (< I End) done)
  (real Acc (+ Acc (* (f64 A I) (f64 B I))))
  (word I (+ I 8))
  (again terms)
  (label done)
  (return-real Acc))

Every read and store is still checked against its memory's length, and nothing in these loops allocates until return-real boxes the result. Machine registers go to what the deepest loops use most, whatever the order of the names. Nothing is moved out of a loop for you, nor is a repeated expression noticed, so a literal or a conversion that a loop uses each time round is worth a name before it, and a value a statement uses twice is worth a name too. An FNV-1a hash, say, is a loop of wrapping multiplications:

(define-native fnv (Bytes End)
  (word Prime 1099511628211)
  (word Hash (+ (* 3421674724 4294967296) 2216829733))
  (word I 0)
  (loop bytes)
  (jump-unless-word (< I End) done)
  (word Hash (* (bit-xor Hash (u8 Bytes I)) Prime))
  (word I (+ I 1))
  (again bytes)
  (label done)
  (return-word (shift-right Hash 8)))

if, when, unless, while, for, block, exit and next write the same loops without labels: they become them, so the code that runs is the same. The hash again:

(define-native fnv (Bytes End)
  (word Prime 1099511628211)
  (word Hash (+ (* 3421674724 4294967296) 2216829733))
  (for I 0 End
    (word Hash (* (bit-xor Hash (u8 Bytes I)) Prime)))
  (return-word (shift-right Hash 8)))

Shen/JIT instructions and frames, "Native code written by hand", has the instructions and these forms.

Debugging

An error that nothing catches prints a backtrace, even through inlined functions:

shen-jit: uncaught error: hd: 1 is not a pair
  in inner
  in outer

(0-) (define twice X -> (+ X X))
(fn twice)

(1-) (shen-jit.code twice)
[defun twice [code twice 1 4 [] [[prim 3 + 2 2] [return 3]] [] [[V8] [+ V8 V8]]]]

The kernel's track, spy and step work as in other ports. For where the time goes, SHEN_JIT_PROFILE=1 prints a flat profile at exit, and SHEN_JIT_STATS=1 prints the collector's work and the live heap by type.

Environment variables

SHEN_JIT_PROFILE, SHEN_JIT_STATS, SHEN_JIT_PERF_MAP and SHEN_JIT_SLOW_PRIMS are on when set to anything, even 0.

Differences from Shen/Scheme

• No stlib in the default image. Only the kernel and its extensions for features, programmable pattern matching, namespaces and the launcher are in use: see The standard library.
• Fixnums only. Integers are 62-bit, and a result outside that range raises integer overflow.
• Sealing. Internal kernel functions cannot be redefined, where Shen/Scheme accepts such definitions.
• Numbers hash by value. A real and an integer that are = have the same hash, where Shen/Scheme's tells them apart.
• Threads run one at a time, where Shen/Scheme's run in parallel.
• On Windows, a few things differ: see On Windows.

Shen/JIT semantics defines the behaviour in full, and marks each difference with Deviation.

Building and testing

From a checkout, make deps fetches the Shen kernel and an earlier release of Shen/JIT into .deps/, checked by hash. That release runs only to generate the files a source tarball already carries. It has executables for the platforms in Install, and on Windows make deps also needs unzip. Elsewhere, use a source tarball, or set BOOTSTRAP to a Shen of your own.

git clone https://github.com/tizoc/shen-jit
cd shen-jit
make deps
make

This writes build/shen-jit, a single file that carries its own image. How Shen/JIT works explains the stages.

Shen/Scheme can take that release's place, for a way in that does not start from Shen/JIT. It makes the same generated files, byte for byte, which check-generated shows:

sh tools/ci-deps.sh shen-scheme .deps/shen-scheme
make BOOTSTRAP=.deps/shen-scheme/bin/shen-scheme check-generated all

These can be set on the command line, or in a file local.mk, which git ignores:

make bench also needs Shen/Scheme, as above, and sh tools/ci-deps.sh shen-scheme-src .deps/shen-scheme-src fetches its sources.

The fast tier takes about 20 s, and catches nearly everything:

make test test-jit test-ops test-kl check-lowerer check-generated test-kernel test-stlib test-thorn test-native test-buffer test-json test-io test-http test-datalog test-ffi test-overrides

To run the test suites from a source tarball, point KERNEL_DIR at a directory with the kernel's tests and extensions, and its klambda for make test-overrides:

make stlib
make KERNEL_DIR=/path/to/kernel check-generated test test-jit test-ops test-kl test-kernel test-stlib test-thorn test-native test-buffer test-json test-io test-http test-datalog test-ffi test-overrides

On Windows, every suite passes, GC stress included, except what cannot run: tests/cli.sh leaves out the tests that send a signal, which make test-ctrlc-windows covers, or use the profiler. The sanitizers stay on Linux and macOS.

AGENTS.md has more on working on it.

More

• How Shen/JIT works: the build, frames and calls, values and the collector.
• Shen/JIT semantics: the behaviour it implements, and where that differs from Shen/Scheme.
• Shen/JIT instructions and frames: the instruction set and the lowering rules.
• Calling C in full: plain C, C written for Shen, and how it works.