path: root/README.md

---
title: joyce - record your thoughts as they come
author: Federico Igne
date: \today
---

`joyce` is an attempt at building a tool for rapid notetaking, i.e., quick collection of short thoughts that can come to mind at any point.

On a high-level, this system should be:

- *Hubiquitous*, needs to be available (read/write) whenever one has internet connection (falling back to pen&paper, otherwise).
- *Searchable*, one has to be able to search and filter the pile of notes.
- *Out of the way*, sophistication will only get in the way of recording one's thoughts.

`joyce` is structured as a small tool written in Rust running on a server, loosely inspired by [`twtxt`](https://github.com/buckket/twtxt).
Clients interface themselves with the server through a simple REST API.

# Notes

Notes are the first-class citizen of `joyce` and are the main content exchanged between server and clients.

Notes, along with their REST API are defined in their own module

```{#main_mods .rust}
mod note;
```

```{#note.rs .rust path="src/"}
<<note_uses>>

<<note_struct>>

<<note_impl>>

<<note_request_struct>>

<<note_request_impl>>

<<req_get_notes>>

<<req_post_notes>>
```

## Anatomy of a note

A *note* is a simple structure recording a unique ID and a timestamp, both assigned at creation, a list of tags, and the body of the note.

```{#note_struct .rust}
#[derive(Debug, Deserialize, Serialize)]
pub struct Note {
  id: i64,
  timestamp: DateTime<Utc>,
  tags: Vec<String>,
  body: String,
}
```

```{#note_impl .rust}
impl Note {
  pub fn new(id: i64, timestamp: DateTime<Utc>, tags: Vec<String>, body: String) -> Self {
    Self { id, timestamp, tags, body }
  }
}

```

A `Note` can also be create from a `Row`, result of a query over the database.

```{#note_uses .rust}
use rusqlite::Row;
```

```{#note_impl .rust}
impl From<&Row<'_>> for Note {
  fn from(row: &Row<'_>) -> Note {
    Note::new(
      row.get(0).expect("Failed to read column 1"),        // id
      row.get(1).expect("Failed to read column 2"),        // timestamp
      serde_json::from_value(row.get::<_,serde_json::Value>(2).expect("Failed to read column 3")).expect("Failed to parse JSON"), // tags
      row.get(3).expect("Failed to read column 4")) // body
  }
}
```

Similarly, a `NoteRequest` is what a client would request at note creation.
It contains the same information as a `Note` without the information assigned at creation by the server and the database.

```{#note_uses .rust}
use serde_json::Value;
```

```{#note_request_struct .rust}
#[derive(Debug, Deserialize, Serialize)]
pub struct NoteRequest {
  tags: Vec<String>,
  body: String,
}
```

The stucture can also be converted into a tuple of parameters to pass a SQL query (see [`Params`](https://docs.rs/rusqlite/latest/rusqlite/trait.Params.html) from `rusqlite`).

```{#note_request_impl .rust}
pub type NoteParams = (DateTime<Utc>, Value, String);

impl From<NoteRequest> for NoteParams {
  fn from(req: NoteRequest) -> NoteParams {
    (Utc::now(), Value::from(req.tags), req.body)
  }
}
```

### (De)serialization

Since notes need to be sent and received via HTTP, the structure needs to be *serializable*.

```{#dependencies .toml}
serde = { version = "1.0", features = ["derive"] }
```

```{#note_uses .rust}
use serde::{Serialize, Deserialize};
```

### IDs

We are using a `i64` as unique identifier because the field is automatically handled by SQLite, and `i64` is the most convenient type to convert from/to SQLite's `INTEGER` (according to [`rusqlite`](https://docs.rs/rusqlite/latest/rusqlite/types/index.html#)).

### Timestamps

Timestamps adhere the *RFC 3339 date-time standard* with UTC offset.

```{#dependencies .toml}
chrono = { version = "0.4", features = ["serde"] }
```

```{#note_uses .rust}
use chrono::prelude::{DateTime, Utc};
```

# The REST API

`joyce` uses [`actix-web`](https://actix.rs/) to handle HTTP requests and responses.

```{#dependencies .toml}
actix-web = "4.1"
```

```{#note_uses .rust}
use actix_web::{HttpResponse,Responder,web,get,post};
use actix_web::http::header::ContentType;
use super::db::Pool;
```

Each request handlers is an *async* function that accepts zero or more parameters, extracted from a request (see [`FromRequest`](https://docs.rs/actix-web/latest/actix_web/trait.FromRequest.html) trait), and returns an [`HttpResponse`](https://docs.rs/actix-web/latest/actix_web/struct.HttpResponse.html).

Internally requests will be carried out by querying the underlying SQLite database.

```{#main_mods .rust}
mod db;
```

```{#note_uses .rust}
use super::db;
```

## Resources

- [Tutorial](https://web.archive.org/web/20220710213947/https://hub.qovery.com/guides/tutorial/create-a-blazingly-fast-api-in-rust-part-1/)
- [Introduction to `actix`](https://actix.rs/docs/getting-started/)

## GET /notes

This handler allows to request the full list of notes currently in the system.
The function takes 0 parameters and returns a JSON object.

```{#req_get_notes .rust}
#[get("/notes")]
pub async fn get_notes(pool: web::Data<Pool>) -> HttpResponse {
  let mut conn = web::block(move || pool.get())
    .await
    .expect("Blocking error")
    .expect("Error getting from connection pool");
  let notes: Vec<Note> = db::get_notes(&mut conn);
  HttpResponse::Ok()
    .content_type(ContentType::json())
    .json(notes)
}
```

## POST /notes

New notes can be added by POSTing a JSON array of `NoteRequest`s of the form

```json
{
    "body": "This is a funny note",
    "tags": [ "joyce", "funny", "example" ]
}
```

```{#req_post_notes .rust}
#[post("/notes")]
pub async fn post_notes(pool: web::Data<Pool>, req: web::Json<Vec<NoteRequest>>) -> impl Responder {
  let mut conn = web::block(move || pool.get())
    .await
    .expect("Blocking error")
    .expect("Error getting from connection pool");
  let res = db::post_notes(&mut conn, req.into_inner());
  format!("Successfully added {res} note(s)")
}
```

## GET /tag/{tags}

This handler allows to query the set of notes for specific tags.
One or more tags separated by `+` can be passed to the request.

```{#req_get_tags .rust}
#[get("/tag/{tags}")]
pub async fn get_tags(tags: web::Path<String>) -> HttpResponse {
  let tags = tags.split('+').map(|t| t.to_string()).collect::<Vec<_>>();

  let notes: Vec<Note>;
  <<notes_retrieve>>
  let tagged = notes.into_iter().filter(|n| tags.iter().all(|t| n.tags.contains(t))).collect::<Vec<_>>();

  HttpResponse::Ok()
    .content_type(ContentType::json())
    .json(tagged)
}
```

# The program

The main program is structured as follows

```{#main.rs .rust path="src/"}
<<main_uses>>

<<main_mods>>

<<main_service>>
```

# Main service

The main service will instantiate a new `App` running within a `HttpServer` bound to *localhost* on port 8080.

```{#main_uses .rust}
use actix_web::{App, HttpServer, web};
```

The `App` will register all request handlers defined above.

```{#main_service .rust}
#[actix_web::main]
async fn main() -> std::io::Result<()> {
    let db_pool = db::get_connection_pool("notes.db");
    HttpServer::new(move || {
        App::new()
            .app_data(web::Data::new(db_pool.clone()))
            .service(note::get_notes)
            .service(note::post_notes)
    })
    .bind(("127.0.0.1", 8080))?
    .run()
    .await
}
```

# SQLite backend

`Note`s are saved into a [SQLite](https://sqlite.org/) database.
The `notes` database contains a single table mirroring the `Note`'s structure.

```{#notes.sql .sql}
CREATE TABLE notes (
    id        INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT    NOT NULL,
    tags      TEXT    NOT NULL,
    body      TEXT    NOT NULL
);
```

Note that, apart from [standard SQLite types](https://docs.rs/rusqlite/latest/rusqlite/types/index.html#), `DateTime<Utc>` is converted to/from `TEXT`, `Vec<String>` is first wrapped in a `Value` (from the `serde_json` crate) and then converted from/to `TEXT`.

`id`s are handled automatically by SQLite and are not set on the Rust side.

A new database can be bootstraped in the current directory as follows:

```sh
sqlite3 ./notes.db < notes.sql
```

## Interfacing with SQLite from Rust

To interface with the underlying database we use [`rusqlite`](https://github.com/rusqlite/rusqlite) along with [`r2d2`](https://github.com/sfackler/r2d2) to create a connection pool.

```{#dependencies .toml}
r2d2 = "0.8"
r2d2_sqlite = "0.21"
rusqlite = { version = "0.28", features = ["chrono","serde_json"] }
```

The following code sets up a connection pool to the SQLite database.

```{#db_uses .rust}
use r2d2_sqlite::SqliteConnectionManager;
```

```{#db_types .rust}
pub type Pool = r2d2::Pool<SqliteConnectionManager>;
```

```{#db_connection_pool .rust}
pub fn get_connection_pool(db: &str) -> Pool {
    let manager = SqliteConnectionManager::file(db);
    r2d2::Pool::new(manager).expect("Unable to connect to 'notes.db'")
}
```

For the sake of convenience, all operations on the database are stored on a separate file.

```{#db.rs .rust path="src/"}
<<db_uses>>

<<db_types>>

<<db_connection_pool>>

<<db_operations>>
```

### Retrieving notes

We first build the query (caching it for later) and then execute it to retrieve all requested notes.

We return a `Vec<Note>` built from the query result.

```{#db_uses .rust}
use rusqlite::Connection;
use super::note::Note;
```

```{#db_operations .rust}
pub fn get_notes(conn: &mut Connection) -> Vec<Note> {
    let mut query = conn.prepare_cached("SELECT * FROM notes")
        .expect("Failed to prepare SELECT query");
    query
        .query_map([], |row| Ok(Note::from(row)))
        .and_then(Iterator::collect)
        .expect("Failed to collect query results")
}

```

### Creating notes

When inserting new `Note`s in the database, we loop over the requested notes, attaching a timestamp and executing an `INSERT` SQL query.

SQLite will take care of attaching an ID to the new entry.

Operations are executed into a single transaction [to achieve better performances](https://github.com/rusqlite/rusqlite/issues/262#issuecomment-294895051).

```{#db_uses .rust}
use super::note::{NoteParams,NoteRequest};
```

```{#db_operations .rust}
pub fn post_notes(conn: &mut Connection, reqs: Vec<NoteRequest>) -> usize {
    let tx = conn.transaction().expect("Failed to start transaction");
    {
        let mut stmt = tx.prepare_cached(
            "INSERT INTO notes (timestamp, tags, body) VALUES (?, ?, ?)"
        ).expect("Failed to prepare INSERT query");
        reqs.into_iter().for_each(|req| { stmt.execute::<NoteParams>(req.into()).expect("Failed to execute INSERT query"); });
    }
    tx.commit().expect("Commit failed");
    0
}

```

# Testing

## Using a file as a backend

This is a temporary solution until an interface to a database (most likely SQLite) is added.

```{#dependencies .toml}
serde_json = "1.0"
```

### Retrieving notes

```{#notes_retrieve .rust}
notes = {
  let db = File::open("notes.db").expect("Unable to open 'notes.db'");
  serde_json::from_reader(&db).unwrap_or(vec![])
};
```

### Adding notes

```{#notes_add .rust}
let mut notes: Vec<Note>;
<<notes_retrieve>>
notes.append(&mut new_notes);

let db = File::create("notes.db").expect("Unable to create/open 'notes.db'");
serde_json::to_writer(&db,&notes).expect("Unable to write to 'notes.db'");
```

# TODOs

- Better error handling

## Open questions

- Should one be able to delete notes? Or mark them as read/processed?
- Authentication method?
- Custom filters on retrieval.
- `rusqlite` ships with SQLite bundled if necessary (useful to bootstrap the db?)

# Credits

`joyce v0.1.0` was created by Federico Igne ([git@federicoigne.com](mailto:git@federicoigne.com)) and available at [`https://git.dyamon.me/projects/joyce`](https://git.dyamon.me/projects/joyce).

```{#Cargo.toml .toml}
[package]
name = "joyce"
version = "0.1.0"
edition = "2021"

[dependencies]
<<dependencies>>
```