--- title: joyce - record your thoughts as they come author: Federico Igne date: \today --- `joyce` is an attempt at building a tool for rapid notetaking, i.e., quick collection of short thoughts that can come to mind at any point. On a high-level, this system should be: - *Hubiquitous*, needs to be available (read/write) whenever one has internet connection (falling back to pen&paper, otherwise). - *Searchable*, one has to be able to search and filter the pile of notes. - *Out of the way*, sophistication will only get in the way of recording one's thoughts. `joyce` is structured as a small tool written in Rust running on a server, loosely inspired by [`twtxt`](https://github.com/buckket/twtxt). Clients interface themselves with the server through a simple REST API. # Notes Notes are the first-class citizen of `joyce` and are the main content exchanged between server and clients. Notes, along with their REST API are defined in their own module ```{#main_mods .rust} mod note; ``` ```{#note.rs .rust path="src/"} <> <> <> <> <> <> <> ``` ## Anatomy of a note A *note* is a simple structure recording a unique ID and a timestamp, both assigned at creation, a list of tags, and the body of the note. ```{#note_struct .rust} #[derive(Debug, Deserialize, Serialize)] pub struct Note { id: i64, timestamp: DateTime, tags: Vec, body: String, } ``` ```{#note_impl .rust} impl Note { pub fn new(id: i64, timestamp: DateTime, tags: Vec, body: String) -> Self { Self { id, timestamp, tags, body } } } ``` A `Note` can also be create from a `Row`, result of a query over the database. ```{#note_uses .rust} use rusqlite::Row; ``` ```{#note_impl .rust} impl From<&Row<'_>> for Note { fn from(row: &Row<'_>) -> Note { Note::new( row.get(0).expect("Failed to read column 1"), // id row.get(1).expect("Failed to read column 2"), // timestamp serde_json::from_value(row.get::<_,serde_json::Value>(2).expect("Failed to read column 3")).expect("Failed to parse JSON"), // tags row.get(3).expect("Failed to read column 4")) // body } } ``` Similarly, a `NoteRequest` is what a client would request at note creation. It contains the same information as a `Note` without the information assigned at creation by the server and the database. ```{#note_uses .rust} use serde_json::Value; ``` ```{#note_request_struct .rust} #[derive(Debug, Deserialize, Serialize)] pub struct NoteRequest { tags: Vec, body: String, } ``` The stucture can also be converted into a tuple of parameters to pass a SQL query (see [`Params`](https://docs.rs/rusqlite/latest/rusqlite/trait.Params.html) from `rusqlite`). ```{#note_request_impl .rust} pub type NoteParams = (DateTime, Value, String); impl From for NoteParams { fn from(req: NoteRequest) -> NoteParams { (Utc::now(), Value::from(req.tags), req.body) } } ``` ### (De)serialization Since notes need to be sent and received via HTTP, the structure needs to be *serializable*. ```{#dependencies .toml} serde = { version = "1.0", features = ["derive"] } ``` ```{#note_uses .rust} use serde::{Serialize, Deserialize}; ``` ### IDs We are using a `i64` as unique identifier because the field is automatically handled by SQLite, and `i64` is the most convenient type to convert from/to SQLite's `INTEGER` (according to [`rusqlite`](https://docs.rs/rusqlite/latest/rusqlite/types/index.html#)). ### Timestamps Timestamps adhere the *RFC 3339 date-time standard* with UTC offset. ```{#dependencies .toml} chrono = { version = "0.4", features = ["serde"] } ``` ```{#note_uses .rust} use chrono::prelude::{DateTime, Utc}; ``` # The REST API `joyce` uses [`actix-web`](https://actix.rs/) to handle HTTP requests and responses. ```{#dependencies .toml} actix-web = "4.1" ``` ```{#note_uses .rust} use actix_web::{HttpResponse,Responder,web,get,post}; use actix_web::http::header::ContentType; use super::db::Pool; ``` Each request handlers is an *async* function that accepts zero or more parameters, extracted from a request (see [`FromRequest`](https://docs.rs/actix-web/latest/actix_web/trait.FromRequest.html) trait), and returns an [`HttpResponse`](https://docs.rs/actix-web/latest/actix_web/struct.HttpResponse.html). Internally requests will be carried out by querying the underlying SQLite database. ```{#main_mods .rust} mod db; ``` ```{#note_uses .rust} use super::db; ``` ## Resources - [Tutorial](https://web.archive.org/web/20220710213947/https://hub.qovery.com/guides/tutorial/create-a-blazingly-fast-api-in-rust-part-1/) - [Introduction to `actix`](https://actix.rs/docs/getting-started/) ## GET /notes This handler allows to request the full list of notes currently in the system. The function takes 0 parameters and returns a JSON object. ```{#req_get_notes .rust} #[get("/notes")] pub async fn get_notes(pool: web::Data) -> HttpResponse { let mut conn = web::block(move || pool.get()) .await .expect("Blocking error") .expect("Error getting from connection pool"); let notes: Vec = db::get_notes(&mut conn); HttpResponse::Ok() .content_type(ContentType::json()) .json(notes) } ``` ## POST /notes New notes can be added by POSTing a JSON array of `NoteRequest`s of the form ```json { "body": "This is a funny note", "tags": [ "joyce", "funny", "example" ] } ``` ```{#req_post_notes .rust} #[post("/notes")] pub async fn post_notes(pool: web::Data, req: web::Json>) -> impl Responder { let mut conn = web::block(move || pool.get()) .await .expect("Blocking error") .expect("Error getting from connection pool"); let res = db::post_notes(&mut conn, req.into_inner()); format!("Successfully added {res} note(s)") } ``` ## GET /tag/{tags} This handler allows to query the set of notes for specific tags. One or more tags separated by `+` can be passed to the request. ```{#req_get_tags .rust} #[get("/tag/{tags}")] pub async fn get_tags(tags: web::Path) -> HttpResponse { let tags = tags.split('+').map(|t| t.to_string()).collect::>(); let notes: Vec; <> let tagged = notes.into_iter().filter(|n| tags.iter().all(|t| n.tags.contains(t))).collect::>(); HttpResponse::Ok() .content_type(ContentType::json()) .json(tagged) } ``` # The program The main program is structured as follows ```{#main.rs .rust path="src/"} <> <> <> ``` # Main service The main service will instantiate a new `App` running within a `HttpServer` bound to *localhost* on port 8080. ```{#main_uses .rust} use actix_web::{App, HttpServer, web}; ``` The `App` will register all request handlers defined above. ```{#main_service .rust} #[actix_web::main] async fn main() -> std::io::Result<()> { let db_pool = db::get_connection_pool("notes.db"); HttpServer::new(move || { App::new() .app_data(web::Data::new(db_pool.clone())) .service(note::get_notes) .service(note::post_notes) }) .bind(("127.0.0.1", 8080))? .run() .await } ``` # SQLite backend `Note`s are saved into a [SQLite](https://sqlite.org/) database. The `notes` database contains a single table mirroring the `Note`'s structure. ```{#notes.sql .sql} CREATE TABLE notes ( id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp TEXT NOT NULL, tags TEXT NOT NULL, body TEXT NOT NULL ); ``` Note that, apart from [standard SQLite types](https://docs.rs/rusqlite/latest/rusqlite/types/index.html#), `DateTime` is converted to/from `TEXT`, `Vec` is first wrapped in a `Value` (from the `serde_json` crate) and then converted from/to `TEXT`. `id`s are handled automatically by SQLite and are not set on the Rust side. A new database can be bootstraped in the current directory as follows: ```sh sqlite3 ./notes.db < notes.sql ``` ## Interfacing with SQLite from Rust To interface with the underlying database we use [`rusqlite`](https://github.com/rusqlite/rusqlite) along with [`r2d2`](https://github.com/sfackler/r2d2) to create a connection pool. ```{#dependencies .toml} r2d2 = "0.8" r2d2_sqlite = "0.21" rusqlite = { version = "0.28", features = ["chrono","serde_json"] } ``` The following code sets up a connection pool to the SQLite database. ```{#db_uses .rust} use r2d2_sqlite::SqliteConnectionManager; ``` ```{#db_types .rust} pub type Pool = r2d2::Pool; ``` ```{#db_connection_pool .rust} pub fn get_connection_pool(db: &str) -> Pool { let manager = SqliteConnectionManager::file(db); r2d2::Pool::new(manager).expect("Unable to connect to 'notes.db'") } ``` For the sake of convenience, all operations on the database are stored on a separate file. ```{#db.rs .rust path="src/"} <> <> <> <> ``` ### Retrieving notes We first build the query (caching it for later) and then execute it to retrieve all requested notes. We return a `Vec` built from the query result. ```{#db_uses .rust} use rusqlite::Connection; use super::note::Note; ``` ```{#db_operations .rust} pub fn get_notes(conn: &mut Connection) -> Vec { let mut query = conn.prepare_cached("SELECT * FROM notes") .expect("Failed to prepare SELECT query"); query .query_map([], |row| Ok(Note::from(row))) .and_then(Iterator::collect) .expect("Failed to collect query results") } ``` ### Creating notes When inserting new `Note`s in the database, we loop over the requested notes, attaching a timestamp and executing an `INSERT` SQL query. SQLite will take care of attaching an ID to the new entry. Operations are executed into a single transaction [to achieve better performances](https://github.com/rusqlite/rusqlite/issues/262#issuecomment-294895051). ```{#db_uses .rust} use super::note::{NoteParams,NoteRequest}; ``` ```{#db_operations .rust} pub fn post_notes(conn: &mut Connection, reqs: Vec) -> usize { let tx = conn.transaction().expect("Failed to start transaction"); { let mut stmt = tx.prepare_cached( "INSERT INTO notes (timestamp, tags, body) VALUES (?, ?, ?)" ).expect("Failed to prepare INSERT query"); reqs.into_iter().for_each(|req| { stmt.execute::(req.into()).expect("Failed to execute INSERT query"); }); } tx.commit().expect("Commit failed"); 0 } ``` # Testing ## Using a file as a backend This is a temporary solution until an interface to a database (most likely SQLite) is added. ```{#dependencies .toml} serde_json = "1.0" ``` ### Retrieving notes ```{#notes_retrieve .rust} notes = { let db = File::open("notes.db").expect("Unable to open 'notes.db'"); serde_json::from_reader(&db).unwrap_or(vec![]) }; ``` ### Adding notes ```{#notes_add .rust} let mut notes: Vec; <> notes.append(&mut new_notes); let db = File::create("notes.db").expect("Unable to create/open 'notes.db'"); serde_json::to_writer(&db,¬es).expect("Unable to write to 'notes.db'"); ``` # TODOs - Better error handling ## Open questions - Should one be able to delete notes? Or mark them as read/processed? - Authentication method? - Custom filters on retrieval. - `rusqlite` ships with SQLite bundled if necessary (useful to bootstrap the db?) # Credits `joyce v0.1.0` was created by Federico Igne ([git@federicoigne.com](mailto:git@federicoigne.com)) and available at [`https://git.dyamon.me/projects/joyce`](https://git.dyamon.me/projects/joyce). ```{#Cargo.toml .toml} [package] name = "joyce" version = "0.1.0" edition = "2021" [dependencies] <> ```