path: root/README.md

---
title: joyce
subtitle: Record your thoughts as they come.
author: Federico Igne
date: \today
...

`joyce` is an attempt at building a tool for rapid notetaking, i.e., quick collection of short thoughts that can come to mind at any point.

On a high-level, this system should be:

- *Hubiquitous*, needs to be available (read/write) whenever one has internet connection (falling back to pen&paper, otherwise).
- *Searchable*, one has to be able to search and filter the pile of notes.
- *Out of the way*, sophistication will only get in the way of recording one's thoughts.

`joyce` is structured as a small tool written in Rust running on a server, loosely inspired by [`twtxt`](https://github.com/buckket/twtxt).
Clients interface themselves with the server through a simple REST API.


# Installation and setup

<!-- TODO -->

## Command Line Interface

`joyce` offers a very simple command line interface.
For an overview of the functionalities offered by the tool run

```sh
joyce --help
```

`joyce` uses the `clap` library to parse command line arguments

```{#dependencies .toml}
clap = { version = "3.1", features = ["derive"] }
```

```{#main_uses .rust}
use clap::Parser;
```

using the [Derive API](https://github.com/clap-rs/clap/blob/v3.1.18/examples/tutorial_derive/README.md) to define the exposed functionalities.
The `struct` holding the CLI information is defined as follow

```{#main_config .rust}
/// Record your thoughts as they come.
#[derive(Parser, Debug)]
#[clap(author, version, about, long_about = None)]
struct Config {
  <<config_db>>
  <<config_address>>
}
```

and the arguments are parsed as

```{#config_parse .rust}
let config = Config::parse();
```

`joyce` assumes a SQLite database named `notes.db` is present in the current working directory.

```{#main_constants .rust}
const DB: &str = "./notes.db";
```

A custom path to a SQLite database can be passed using the `-d/--database` flag.

```{#main_uses .rust}
use std::path::PathBuf;
```

```{#config_db .rust}
/// SQLite database [default: './notes.db']
#[clap(short, long)]
database: Option<PathBuf>,
```

Once started, `joyce` will be available on localhost (`127.0.0.1`) at port `8080`.

```{#main_constants .rust}
const ADDR: &str = "127.0.0.1";
const PORT: u16 = 8080;
```

This behaviour can be overridden with the `-a`/`--address` and `-p`/`--port` flags

```{#config_address .rust}
/// Address the `joyce` service is bound to [default: '127.0.0.1']
#[clap(short, long)]
address: Option<String>,
/// Port the `joyce` service is bound to [default: 8080]
#[clap(short, long)]
port: Option<u16>,
```


# Notes

Notes are the first-class citizen of `joyce` and are the main content exchanged between server and clients.

## Anatomy of a note

A *note* is a simple structure recording a unique ID and a timestamp, both assigned at creation, a list of tags, and the body of the note.

```{#note_struct .rust}
#[derive(Debug, Deserialize, Serialize)]
pub struct Note {
  id: i64,
  timestamp: DateTime<Utc>,
  tags: Vec<String>,
  body: String,
}
```

```{#note_impl .rust}
impl Note {
  pub fn new(id: i64, timestamp: DateTime<Utc>, tags: Vec<String>, body: String) -> Self {
    Self { id, timestamp, tags, body }
  }
}

```

A `Note` can also be create from a `Row`, result of a query over the database.

```{#note_uses .rust}
use rusqlite::Row;
```

```{#note_impl .rust}
impl From<&Row<'_>> for Note {
  fn from(row: &Row<'_>) -> Note {
    Note::new(
      row.get(0).expect("Failed to read column 1"),        // id
      row.get(1).expect("Failed to read column 2"),        // timestamp
      serde_json::from_value(row.get::<_,serde_json::Value>(2).expect("Failed to read column 3")).expect("Failed to parse JSON"), // tags
      row.get(3).expect("Failed to read column 4")) // body
  }
}
```

Similarly, a `NoteRequest` is what a client would request at note creation.
It contains the same information as a `Note` without the information assigned at creation by the server and the database.

```{#note_uses .rust}
use serde_json::Value;
```

```{#note_request_struct .rust}
#[derive(Debug, Deserialize, Serialize)]
pub struct NoteRequest {
  tags: Vec<String>,
  body: String,
}
```

The stucture can also be converted into a tuple of parameters to pass a SQL query (see [`Params`](https://docs.rs/rusqlite/latest/rusqlite/trait.Params.html) from `rusqlite`).

```{#note_request_impl .rust}
pub type NoteParams = (DateTime<Utc>, Value, String);

impl From<NoteRequest> for NoteParams {
  fn from(req: NoteRequest) -> NoteParams {
    (Utc::now(), Value::from(req.tags), req.body)
  }
}
```

### (De)serialization

Since notes need to be sent and received via HTTP, the structure needs to be *serializable* (from/to JSON format).

```{#dependencies .toml}
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
```

```{#note_uses .rust}
use serde::{Serialize, Deserialize};
```

### IDs

We are using a `i64` as unique identifier because the field is automatically handled by SQLite, and `i64` is the most convenient type to convert from/to SQLite's `INTEGER` (according to [`rusqlite`](https://docs.rs/rusqlite/latest/rusqlite/types/index.html#)).

### Timestamps

Timestamps adhere the *RFC 3339 date-time standard* with UTC offset.

```{#dependencies .toml}
chrono = { version = "0.4", features = ["serde"] }
```

```{#note_uses .rust}
use chrono::prelude::{DateTime, Utc};
```

# The REST API

`joyce` uses [`actix-web`](https://actix.rs/) to handle HTTP requests and responses.

```{#dependencies .toml}
actix-web = "4.1"
```

```{#note_uses .rust}
use actix_web::{HttpResponse,Responder,web,get,post};
use actix_web::http::header::ContentType;
use super::db::Pool;
```

Each request handlers is an *async* function that accepts zero or more parameters, extracted from a request (see [`FromRequest`](https://docs.rs/actix-web/latest/actix_web/trait.FromRequest.html) trait), and returns an [`HttpResponse`](https://docs.rs/actix-web/latest/actix_web/struct.HttpResponse.html).

Internally requests will be carried out by querying the underlying SQLite database.

```{#note_uses .rust}
use super::db;
```

## GET /notes

This handler allows to request the full list of notes currently in the system.
The function takes 1 parameters (the connection pool to the underlying SQLite database) and returns a collection of notes as a JSON array.

```{#req_get_notes .rust}
#[get("/notes")]
pub async fn get_notes(pool: web::Data<Pool>) -> HttpResponse {
  <<get_connection>>
  let notes: Vec<Note> = db::get_notes(&mut conn);
  HttpResponse::Ok()
    .content_type(ContentType::json())
    .json(notes)
}
```

## POST /notes

New notes can be added by POSTing a JSON array of `NoteRequest`s of the form

```json
{
    "body": "This is a funny note",
    "tags": [ "joyce", "funny", "example" ]
}
```

The function takes 2 parameters:

- the connection pool,
- the collection of `NoteRequests` as a JSON object.


```{#req_post_notes .rust}
#[post("/notes")]
pub async fn post_notes(pool: web::Data<Pool>, req: web::Json<Vec<NoteRequest>>) -> impl Responder {
  <<get_connection>>
  let res = db::post_notes(&mut conn, req.into_inner());
  format!("Successfully added {res} note(s)")
}
```

## GET /tag/{tags}

Notes can be retrieved by (sets of) tags.
One or more tags separated by `+` can be passed to the request and `joyce` will retrieve those notes marked with **all** the provided tags.

The function takes 2 parameters (the connection pool to the underlying SQLite database and the sequence of tags) and returns a collection of notes as a JSON array.

Passing an empty collection of tags results in an error.

```{#req_get_tags .rust}
#[get("/tag/{tags}")]
pub async fn get_tags(pool: web::Data<Pool>, tags: web::Path<String>) -> HttpResponse {
  <<get_connection>>
  let tags = tags.split('+').map(|t| t.to_string()).collect::<Vec<_>>();
  if !tags.is_empty() {
    let notes: Vec<Note> = db::get_tags(&mut conn, &tags);
    HttpResponse::Ok()
      .content_type(ContentType::json())
      .json(notes)
  } else {
    todo!()
  }
}
```

# SQLite backend

`Note`s are saved into a [SQLite](https://sqlite.org/) database.
The `notes` database contains a single table mirroring the [`Note`'s structure](#anatomy-of-a-note).

```{#notes.sql .sql}
CREATE TABLE notes (
    id        INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT    NOT NULL,
    tags      TEXT    NOT NULL,
    body      TEXT    NOT NULL
);
```

Note that, apart from [standard SQLite types](https://docs.rs/rusqlite/latest/rusqlite/types/index.html#), `DateTime<Utc>` is converted to/from `TEXT`, `Vec<String>` is first wrapped in a JSON `Value` (from the `serde_json` crate) and then converted from/to `TEXT`.

`id`s are handled automatically by SQLite and are not set on the Rust side.

A new database can be bootstraped in the current directory as follows:

```sh
sqlite3 ./notes.db < notes.sql
```

## Interfacing with SQLite from Rust

To interface with the underlying database we use [`rusqlite`](https://github.com/rusqlite/rusqlite) along with [`r2d2`](https://github.com/sfackler/r2d2) to create a connection pool.

```{#dependencies .toml}
r2d2 = "0.8"
r2d2_sqlite = "0.21"
rusqlite = { version = "0.28", features = ["chrono","serde_json"] }
```

The following code sets up a connection pool to the SQLite database.

```{#db_uses .rust}
use std::path::Path;
use r2d2_sqlite::SqliteConnectionManager;
```

```{#db_types .rust}
pub type Pool = r2d2::Pool<SqliteConnectionManager>;
```

```{#db_connection_pool .rust}
pub fn get_connection_pool<P: AsRef<Path>>(db: P) -> Pool {
    let manager = SqliteConnectionManager::file(db);
    r2d2::Pool::new(manager).expect("Unable to connect to {db}")
}
```

When needed, one can get a connection from the pool.
This is a *blocking function* as as such is wrapped in a `web::block` to offload the task to one of the `actix` thread workers.

```{#get_connection .rust}
let mut conn = web::block(move || pool.get())
  .await
  .expect("Blocking error")
  .expect("Error getting from connection pool");
```

### Retrieving notes

We first build the query (caching it for later) and then execute it to retrieve all requested notes.

We return a `Vec<Note>` built from the query result.

```{#db_uses .rust}
use rusqlite::Connection;
use super::note::Note;
```

```{#db_operations .rust}
pub fn get_notes(conn: &mut Connection) -> Vec<Note> {
    let mut query = conn.prepare_cached("SELECT * FROM notes")
        .expect("Failed to prepare SELECT query");
    query
        .query_map([], |row| Ok(Note::from(row)))
        .and_then(Iterator::collect)
        .expect("Failed to collect query results")
}

```

### Creating notes

When inserting new `Note`s in the database, we loop over the requests, attaching a timestamp and executing an `INSERT` SQL query.
SQLite will take care of attaching an ID to the new entry.

Operations are executed into a single transaction [to achieve better performances](https://github.com/rusqlite/rusqlite/issues/262#issuecomment-294895051).

```{#db_uses .rust}
use super::note::{NoteParams,NoteRequest};
```

```{#db_operations .rust}
pub fn post_notes(conn: &mut Connection, reqs: Vec<NoteRequest>) -> usize {
    let insert = "INSERT INTO notes (timestamp, tags, body) VALUES (?, ?, ?)";
    let tx = conn.transaction().expect("Failed to start transaction");
    {
        let mut stmt = tx.prepare_cached(insert).expect("Failed to prepare INSERT query");
        reqs.into_iter().for_each(|req| { stmt.execute::<NoteParams>(req.into()).expect("Failed to execute INSERT query"); });
    }
    tx.commit().expect("Commit failed");
    0
}

```

### Retrieving tagged notes

We can query for specific tags by using the `LIKE` SQL operator.
Such an SQL query will look like this

    SELECT * FROM notes
    WHERE tags LIKE '%<tag>%'
    [ AND tags LIKE '%<tag>%' ...]

where the escaped expression `'%<tag>%'` matches anything containing the word `<tag>`, i.e., the `%` operator matches a (possibly empty) sequence of characters.

A query needs to be dynamically generated for the specific number of provided tags.

```{#build_query_tags .rust}
let mut query = String::with_capacity(32 + 16 * tags.len());
query.push_str("SELECT * FROM notes WHERE tags LIKE ? ");
for _ in 0..(tags.len()-1) {
    query.push_str("AND tags LIKE ? ");
}
```

Once built, we execute the query with the sequence of tags as parameters.
Note that we need to use [`rusqlite::params_from_iter`](https://docs.rs/rusqlite/latest/rusqlite/fn.params_from_iter.html) in order to handle a dynamic set of parameters.

```{#db_operations .rust}
pub fn get_tags(conn: &mut Connection, tags: &[String]) -> Vec<Note> {
  <<build_query_tags>>
  let mut query = conn.prepare_cached(&query)
      .expect("Failed to prepare SELECT query");
  let params = tags.iter().map(|tag| format!("%\"{}\"%", tag));
  query
      .query_map(rusqlite::params_from_iter(params), |row| Ok(Note::from(row)))
      .and_then(Iterator::collect)
      .expect("Failed to collect query results")
}

```


# Main service

The main service will instantiate a new `App` running within a `HttpServer` bound to *localhost* on port 8080.

```{#main_uses .rust}
use actix_web::{App, HttpServer, web};
```

The `App` will register all request handlers defined above as *services*.

```{#main_service .rust}
#[actix_web::main]
async fn main() -> std::io::Result<()> {
  <<config_parse>>
  let addr = config.address.unwrap_or_else(|| ADDR.to_string());
  let port = config.port.unwrap_or(PORT);
  let db_file = config.database.unwrap_or_else(|| PathBuf::from(DB));
  let db_pool = db::get_connection_pool(db_file);
  HttpServer::new(move || {
    App::new()
      .app_data(web::Data::new(db_pool.clone()))
      .service(note::get_notes)
      .service(note::get_tags)
      .service(note::post_notes)
  })
  .bind((addr, port))?
  .run()
  .await
}
```

# The program structure

The main program is structured as follows

```{#main.rs .rust path="src/"}
mod note;
mod db;

<<main_uses>>

<<main_constants>>

<<main_config>>

<<main_service>>
```

Notes, along with their REST API are defined in their own `note` module.

```{#note.rs .rust path="src/"}
<<note_uses>>

<<note_struct>>

<<note_impl>>

<<note_request_struct>>

<<note_request_impl>>

<<req_get_notes>>

<<req_get_tags>>

<<req_post_notes>>
```

Communication with SQLite is grouped under the `db` module.

```{#db.rs .rust path="src/"}
<<db_uses>>

<<db_types>>

<<db_connection_pool>>

<<db_operations>>
```

# TODOs

- Better error handling
- Logging
- add examples with cURL

## Open questions

- Should one be able to delete notes? Or mark them as read/processed?
- Authentication method?
- Custom filters on retrieval.
- `rusqlite` ships with SQLite bundled if necessary (useful to bootstrap the db?)

# Credits

`joyce v0.1.0` was created by Federico Igne ([git@federicoigne.com](mailto:git@federicoigne.com)) and available at [`https://git.dyamon.me/projects/joyce`](https://git.dyamon.me/projects/joyce).

```{#Cargo.toml .toml}
[package]
name = "joyce"
version = "0.1.0"
edition = "2021"

[dependencies]
<<dependencies>>
```