🔮 Lisp Notes

The main difference between Python REPL and Lisp REPL is that Lisp REPL is more advanced and powerful than Python REPL. Lisp REPL allows developers to redefine core functions and change the syntax of the language on-the-fly, while Python REPL is simpler and more limited in its capabilities. This makes Lisp REPL a more flexible and customizable environment for advanced programming tasks.

• Read a text file line by line: Write a function that takes a file path and reads the file line by line, printing each line to the standard output.

  (defun read-file (filepath)
    (with-open-file (in filepath)
      (loop as line = (read-line in nil)
            while line do (format t "~A~%" line))))
  

  (read-file "loremipsum.txt")
  

  Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod
  tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At
  vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd
  gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum
  dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor
  invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero
  eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no
  sea takimata sanctus est Lorem ipsum dolor sit amet.
  

• Write a text file line by line: Write a function that takes a list of strings and a file path, then writes each string as a separate line in the specified file.

  (defun write-file (lst filepath)
    (with-open-file (out filepath :direction :output :if-exists :supersede)
      (dolist (e lst)
        (format out "~A~%" e))))
  

  (write-file '("hello" "my" "darling") "hello.txt")
  

  cat hello.txt
  

  hello
  my
  darling
  

• Concatenate multiple text files: Write a function that takes a list of file paths and a destination file path, then concatenates the contents of all the input files into the destination file.

  (defun concat-files (src des)
    (let ((lines nil))
      (dolist (e src)
        (with-open-file (in e)
          (loop as line = (read-line in nil)
                while line do (push line lines))))
      (write-file (reverse lines) des)))
  

  (concat-files '("hello.txt" "loremipsum.txt") "concat.txt")
  

  cat concat.txt
  

  hello
  my
  darling
  Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod
  tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At
  vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd
  gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum
  dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor
  invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero
  eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no
  sea takimata sanctus est Lorem ipsum dolor sit amet.
  

• Copy a binary file: Write a function that takes a source file path and a destination file path, then copies the contents of the source file to the destination file. This task should be done using binary streams.

  (defun write-binary-file (file-path data)
    (with-open-file (stream file-path
                            :direction :output
                            :element-type '(unsigned-byte 8)
                            :if-exists :supersede
                            :if-does-not-exist :create)
      (write-sequence data stream)))
  

  (write-binary-file "example.bin" (vector 72 101 108 108 111 32 87 111 114 108 100))
  

  72

  101

  108

  108

  111

  32

  87

  111

  114

  108

  100

  (defun read-binary-file (file-path)
    (with-open-file (stream file-path :element-type '(unsigned-byte 8))
      (let ((data (make-array (file-length stream))))
        (read-sequence data stream)
        data)))
  

  (read-binary-file "example.bin")
  

  72

  101

  108

  108

  111

  32

  87

  111

  114

  108

  100

  (defun copy-binary-file (src-file-path dest-file-path)
    (write-binary-file dest-file-path
                       (read-binary-file src-file-path)))
  

  (copy-binary-file "example.bin" "example2.bin")
  

  72

  101

  108

  108

  111

  32

  87

  111

  114

  108

  100

• Count lines, words, and characters in a text file: Write a function that takes a file path and returns the number of lines, words, and characters in the file. A word is considered a sequence of characters separated by whitespace.
• Filter lines from a text file: Write a function that takes a file path and a predicate function, then prints the lines in the file that satisfy the predicate.
• Implement a simple text-based search engine: Write a function that takes a directory path, a search term, and an optional maximum number of results. The function should search all text files in the given directory (recursively if desired) and return the file paths that contain the search term, ordered by the number of occurrences of the term.
• Implement a simple CSV reader and writer: Write two functions, one for reading CSV files and another for writing them. The reader function should take a file path and return a list of lists, where each inner list represents a row in the CSV file. The writer function should take a list of lists and a file path, then write the data to the specified file in CSV format.
• Implement a basic line numbering utility: Write a function that takes a file path and a destination file path, then copies the content of the input file to the destination file, adding line numbers at the beginning of each line.
• Implement a basic grep-like utility: Write a function that takes a file path, a regular expression pattern, and an optional flag to indicate whether to perform a case-insensitive search. The function should print all the lines in the file that match the given pattern.
• Implement a basic word frequency counter: Write a function that takes a file path and returns a list of the most frequent words and their frequencies in the file. The list should be sorted by frequency in descending order.
• Implement a simple stream-based encryption/decryption utility: Write two functions for encrypting and decrypting a file using a simple encryption algorithm, such as the Caesar cipher or XOR cipher. Both functions should work with binary streams.

To practice OOP, you can work on exercises that help you understand and apply its core concepts: classes, objects, inheritance, polymorphism, encapsulation, and abstraction.

• Simple Class and Object: Create a class representing a geometric shape (e.g., Circle, Rectangle, or Triangle). Define attributes to store the properties of the shape (e.g., radius, width, height) and methods to calculate the area and perimeter.

  (defclass rectangle ()
    ((with :initarg :width
           :initform 0
           :accessor rectangle-width)
     (height :initarg :height
             :initform 0
             :accessor rectangle-height)))
  
  (defgeneric calc-area (rectangle)
    (:documentation "Calc the area of rectangles"))
  
  (defmethod calc-area ((r rectangle))
    (/ (rectangle-width r) (rectangle-height r)))
  

  (defparameter *rect*
    (make-instance 'rectangle
                   :width 100
                   :height 50))
  (calc-area *rect*)
  

  5000
  

• Bank Account: Implement a Bank Account class with attributes for the account holder’s name, account number, and balance. Include methods for depositing, withdrawing, and displaying the account balance. Create multiple instances of the Bank Account class to represent different accounts.

  (defclass account ()
      ((name :initarg :name
             :initform "Anonymous"
             :accessor account-name)
       (number :initarg :number
               :initform 1234567
               :accessor account-number)
       (balance :initarg :balance
                :initform 0
                :accessor account-balance)))
  

  #<STANDARD-CLASS COMMON-LISP-USER::ACCOUNT>
  

  (defparameter *elizabeth*
    (make-instance 'account
                   :name "Elizabeth"
                   :number 123
                   :balance "test"))
  
  (defparameter *charles*
    (make-instance 'account
                   :name "Charles"
                   :number 456
                   :balance 200))
  

  *CHARLES*
  

  (account-balance *charles*)
  

  200
  

• Inheritance: Design a class hierarchy for different types of animals (e.g., Mammal, Reptile, Bird). Create a base class (Animal) with common attributes (e.g., name, age) and methods (e.g., speak). Derive subclasses for specific animal types, each with their own unique attributes and methods.
• Polymorphism: Expand the animal class hierarchy from the previous exercise. Implement a speak method in each subclass that outputs a unique sound for each animal type (e.g., a dog barks, a cat meows). Demonstrate how you can call the speak method on different animal instances without knowing their specific class, taking advantage of polymorphism.
• Encapsulation: Design a Car class with attributes like make, model, year, and speed. Implement methods to accelerate, decelerate, and display the current speed. Use encapsulation to prevent the speed from being set directly, ensuring it can only be modified through the acceleration and deceleration methods.
• Abstraction: Create an Employee class hierarchy with different types of employees, such as Manager, Developer, and Designer. Implement an abstract method calculate_salary in the base Employee class that each subclass must override to provide a specific salary calculation based on the employee type.
• Composition: Design a Computer class that has components like CPU, RAM, HardDrive, and GPU. Use composition to represent the relationship between the computer and its components. Implement methods in the Computer class to interact with its components, showcasing how composition allows for complex behavior.
• Design Patterns: Study common design patterns, such as Singleton, Factory, Observer, and Strategy, and implement them in your projects. Design patterns provide solutions to recurring problems in software design and can help you build more efficient and maintainable code.

Slime can be “installed” in many different ways. Which way is the best for you is up to you. But you need swank running on your remote machine to connect from your emacs client machine via slime.

Start your sbcl:

sbcl

Load the swank stuff into sbcl. I used git to “install” slime, thats why the path to slime is in my home directory:

(load "~/slime/swank-loader.lisp")
(swank-loader:init)

Start the swank server on port 4005:

(swank:create-server :port 4005  :dont-close t)

To autoload swank in the future, add the following code to the ~/.sbclrc file:

(let ((swank (merge-pathnames "slime/swank-loader.lisp" (user-homedir-pathname))))
  (when (probe-file swank)
    (load swank)))

For an easier start of swank save the following lines of code into the ~/start-swank.cl file:

cat ~/start-swank.cl

(swank-loader:init)
(swank:create-server :port 4005 :style :spawn :dont-close t)

This file can then be called using a shell script:

cat ~/start-swank.sh

sbcl --noinform --load start-swank.cl

It is neccessary to secure the connection using ssh before you should connect from your local Emacs to the remote lisp machine. When the connection is secured you can connect from your Emacs client via slime-connect.

(describe-function 'slime-connect)

slime-connect is an autoloaded interactive Lisp closure in ‘slime.el’.

(slime-connect HOST PORT &optional CODING-SYSTEM INTERACTIVE-P &rest
PARAMETERS)

Connect to a running Swank server. Return the connection.

To secure the connection to the remote lisp machine we need to use ssh port forwarding.

ssh -L 4005:localhost:4005 marcus@u1.metaebene.dev

When using the Kitty terminal, I recommend connecting with SSH using the +kitten flag, as Ubuntu does not have a terminfo entry for Kitty:

kitty +kitten ssh -L 4005:localhost:4005 marcus@u1.metaebene.dev

More information about ssh port forwarding:

• https://www.ibm.com/docs/en/zos/2.2.0?topic=examples-openssh-tcp-port-forwarding
• https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding
• https://www.golinuxcloud.com/setup-ssh-port-forwarding/
• https://www.ssh.com/academy/ssh/tunneling-example

If the server is running and you did connect to your remote machine via ssh, the last step is to run slime-connect from your local emacs, connect to localhost on port 4005. And voila, a slime repl is fireing up connected to your remote machine.

Quicklisp is a great tool to load and install other libraries. You can follow this or this tutorial to install quicklisp.

If you are looking for other libraries, have a look at quickdocs.

Before we go one let us have a last test and check if everything works. We will install spinneret and generate a HTML page.

First install the spinneret dependency using quicklisp:

(ql:quickload :spinneret)

To load "spinneret":
  Load 1 ASDF system:
    spinneret
; Loading "spinneret"

If the package is not yet available in your quicklisp directory under “~/quicklisp/dists/quicklisp/software”, quicklisp will download it automatically. You should now be able to call spinneret from the “CL-USER>” namespace.

(apropos 'spinneret:with-html)

SPINNERET:WITH-HTML (fbound)
SPINNERET:WITH-HTML-STRING (fbound)

You should see both symbols available in the spinneret package. Let us create our own namespace using defpackage. Use (in-package :homepage) in your lisp repl to switch to the :homepage namespace.

(defpackage :homepage (:use cl) (:import-from :spinneret :with-html))
(in-package :homepage)

Let us create a small HTML page. html-style and html-lang are global variables but we are set them locally. Type the following lisp in your repl and you should see the corresponding output.

(let ((spinneret:*html-style* :human)
      (spinneret:*html-lang* "en"))
  (with-html
    (:doctype)
    (:html
     (:head
      (:title "Hello World!"))
     (:body
      (:header
       (:h1 "Hello World"))
      (:p "My first hello world homepage")))))

<!DOCTYPE html>
<html lang=en>
 <head>
  <meta charset=UTF-8>
  <title>Hello World!</title>
 </head>
 <body>
  <header>
   <h1>Hello World</h1>
  </header>
  <p>My first hello world homepage
 </body>
</html>

One of the main problems with running swank remotely is that Emacs assumes the files can be found using normal filenames. if we want things like slime-compile-and-load-file (C-c C-k) and slime-edit-definition (M-.) to work correctly we need to find a way to let our local Emacs refer to remote files.

Here is an example what kind of setup I use for filename translations. You should change the keywords up to your environment and put into your emacs init.el file:

(let ((u1
       (slime-create-filename-translator
        :machine-instance "u1"
        :remote-host "u1.metaebene.dev"
        :username "marcus")))
  (push hunchentoot slime-filename-translations))

You can found more information on the slime manual.

The usocket library is a networking library for Common Lisp, providing an easy-to-use interface for creating and managing network sockets. It allows you to build servers and clients for various network protocols, such as TCP and UDP, and supports both IPv4 and IPv6 addressing. With usocket, you can create network applications like web servers, chat servers, or any other system that requires communication over a network. The library offers functions for creating, listening, accepting, and closing sockets, as well as handling data transfer using socket streams, making it a powerful and flexible tool for network programming in Common Lisp.

usocket:socket-listen

The socket-listen function is like a hotel receptionist preparing a check-in desk for incoming guests. The function takes the host and port as the hotel’s address and sets up the check-in desk (a socket) for guests (clients) to arrive. The backlog parameter determines how many guests can wait in line before new guests are asked to wait outside (the maximum number of pending connection requests). The reuseaddress and reuse-address parameters let the hotel reuse the same address (host and port) for multiple check-in desks, making it easier to manage several events or services simultaneously. When you use the socket-listen function, you’re telling your program (the hotel) to get ready for incoming guests (client connections) by opening a check-in desk (listening socket) at the specified address (host and port).

The socket-listen function creates a TCP listening socket bound to a specific host and port. It accepts keyword arguments to control address reuse, backlog size, and the element type of the socket stream. The function extracts the IP address, creates a socket, sets options, binds it to the address and port, and starts listening for connections. If successful, it returns a stream server-socket object for accepting connections and communicating with clients.

The element-type for sockets in Common Lisp typically refers to the type of data being transmitted over the socket stream. The most common element types are:

• character: This is the default element type for socket streams in many Common Lisp implementations. It represents textual data, and the socket stream is treated as a sequence of characters.
• (unsigned-byte 8): This element type represents binary data, and the socket stream is treated as a sequence of bytes. Each byte is an unsigned integer ranging from 0 to 255.
• (unsigned-byte 16), (unsigned-byte 32), and (unsigned-byte 64): These element types are used for transmitting binary data in larger chunks, such as 2-byte, 4-byte, or 8-byte units.
• t: This element type represents a general-purpose data type, and the socket stream can handle a mix of data types, such as characters, integers, and floating-point numbers.

(defparameter *example-listener*
  (usocket:socket-listen usocket:*wildcard-host*
                         8082
                         :reuse-address t
                         :element-type '(unsigned-byte 8)))

(describe *example-listener*)

#<USOCKET:STREAM-SERVER-USOCKET {1005295963}>
  [standard-object]

Slots with :INSTANCE allocation:
  SOCKET                         = #<SB-BSD-SOCKETS:INET-SOCKET 0.0.0.0:8082, fd: 3 {100524AEC3}>
  WAIT-LIST                      = NIL
  STATE                          = NIL
  ELEMENT-TYPE                   = (UNSIGNED-BYTE 8)

usocket:wait-for-input

Imagine you’re running a hotel with many rooms. The wait-for-input function is like a hotel receptionist who waits for guests to call from their rooms or for potential guests to call from outside to book a room. The receptionist has a list of phone lines (socket-or-sockets) connected to the rooms and the outside line. They can wait for a specific amount of time (timeout) before giving up on waiting for any calls. When the receptionist receives a call, they note down which phone line the call came from. If the ready-only option is set, the receptionist will only report the phone lines with incoming calls. Otherwise, they’ll report all the phone lines they were monitoring, regardless of whether they received a call or not. At the end, the receptionist returns two things: a list of phone lines (either all the lines or only those with incoming calls), and the time remaining until their shift ends (or nil if their shift goes on indefinitely).

The wait-for-input function is used to monitor one or more sockets (or streams) and check if any of them are ready for reading. It accepts a single socket or a list of sockets as its input, along with an optional timeout value and a ready-only flag. The function starts by creating a wait-list for the sockets if it hasn’t been created by the caller. If the input consists of a single socket, it reuses the per-socket wait-list if available. The function then loops through the wait-list’s waiters (i.e., sockets) and sets their state based on whether they have data available for reading. If there’s at least one socket ready for reading, the function calls wait-for-input-internal with a timeout value of 0, which essentially polls the internal routine. Otherwise, it waits for the specified timeout period or indefinitely if no timeout is specified. Once the waiting is done, the function calculates the remaining time (if any) within the timeout period and returns two values: a list of sockets (either all the original sockets or only those that are ready for reading, based on the ready-only flag), and the remaining time within the timeout period (or nil if no timeout was specified). This function is useful for handling multiple connections concurrently, such as in a server application that needs to respond to incoming client requests.

usocket:socket-accept

Imagine a hotel with a doorkeeper who is responsible for letting guests in. The socket-accept function is like the doorkeeper who is waiting to greet and allow new guests to enter the hotel (accepting new client connections). When a guest arrives (a client connects), the doorkeeper (the function) creates a room key (a new stream-socket object) for the guest. This key allows the guest to access their room and communicate with the hotel staff (send and receive data). The room key is set up based on the guest’s preferences (the element-type). Once the guest has been allowed entry and has received their room key (the stream-socket object is returned), the doorkeeper is ready to greet the next guest. If the doorkeeper is momentarily unavailable due to a distraction (like a non-blocking I/O or interrupted system call on Windows), they make sure to get back to their post and continue waiting for the next guest to arrive.

The socket-accept function accepts a new client connection from a listening server socket. It creates a higher-level stream-socket object with input and output streams, using the raw socket representing the established connection. The function returns the stream-socket object for communication with the connected client. It handles errors using the with-mapped-conditions macro and, on Windows systems, resets the server socket’s ready state if necessary.

usocket:socket-stream

This function can be seen as the hotel’s internal communication system, like a telephone line or a messaging service, that connects the guests to the hotel staff. Once a guest is checked in (a connection is established), the communication system (the socket stream) allows the guest and the hotel staff to exchange messages (send and receive data) throughout their stay. This function creates a stream object that enables smooth data transfer between the server and the client over the established connection.

This function creates a stream object associated with a connected socket. The stream object simplifies data transfer between the server and the client by providing an abstraction over the raw socket. Using the stream object, you can read from and write to the socket using familiar Common Lisp I/O functions like read, write, read-char, and write-char.

These functions work together to create a networking system in Common Lisp:

• First, the server sets up a listening socket using usocket:socket-listen.
• The server then waits for incoming connection requests or data using usocket:wait-for-input.
• When a connection request arrives, the server accepts the connection using usocket:socket-accept, which returns a new connected socket.
• The server then creates a stream object for the connected socket using usocket:socket-stream, enabling easy communication with the client using Common Lisp I/O functions.

These steps enable the server to establish connections with clients and exchange data using the established socket connections and associated streams.

• Echo Server and Client: Implement a simple TCP server and client that communicate using sockets. The server should echo back any message sent by the client. This will help you understand socket programming, TCP connections, and client-server architecture.
• Chat Application: Create a basic chat application using sockets that allows multiple clients to connect and send messages to each other. This will introduce you to networking concepts, concurrent programming, and message handling.
• File Transfer: Implement a file transfer protocol (FTP) client and server that can upload and download files. This exercise will help you understand data transmission, file I/O, and error handling in a networking context.
• Web Server: Build a simple web server that serves static files (e.g., HTML, CSS, images) and responds to HTTP requests. This will help you learn about HTTP, request/response handling, and MIME types.
• Web Client: Write a program that makes HTTP requests to a given URL, retrieves the web page, and parses specific information from the page (e.g., extracting links or images). This will help you practice HTTP requests, HTML parsing, and regular expressions.
• UDP Communication: Implement a UDP server and client that exchange messages using datagrams. This will help you understand the differences between TCP and UDP, as well as how to handle unreliable and connectionless communication.
• Network Scanner: Create a network scanner that scans a given IP address range or a list of hostnames to determine if specific ports are open. This will introduce you to network discovery, port scanning, and basic security concepts.
• DNS Resolver: Implement a basic DNS resolver that takes a domain name as input and retrieves the corresponding IP addresses by sending a DNS query to a DNS server. This will help you learn about the Domain Name System (DNS) and how domain names are resolved to IP addresses.
• Proxy Server: Develop a proxy server that acts as an intermediary between a client and a server, forwarding client requests and server responses. This will introduce you to the concept of proxies, request forwarding, and caching.
• Network Security: Explore network security by implementing a basic encryption/decryption system for data transmitted between a client and server. This will help you learn about cryptography, key exchange, and secure communication.

Hunchentoot is a Common Lisp web server that supports HTTP and HTTPS, and includes features for URL routing, cookies, sessions, and form handling. It is designed to be modular and extensible, and can be used with various Lisp implementations.

Load the hunchentoot package into your lisp environment:

(ql:quickload :hunchentoot)

Note: The package ’hunchentoot is also be accessible through the ’tbnl symbol.

• https://lisp-journey.gitlab.io/blog/web-development-in-common-lisp/
• https://lisp-journey.gitlab.io/web-dev/
• https://lispcookbook.github.io/cl-cookbook/web.html
• https://cliki.net/Web
• https://github.com/CodyReichert/awesome-cl#network-and-internet

• Practical Common Lisp
• Lisp Web Tales