aboutsummaryrefslogtreecommitdiff

title: joyce subtitle: Record your thoughts as they come. author: Federico Igne date: \today ...

joyce is an attempt at building a tool for rapid notetaking, i.e., quick collection of short thoughts that can come to mind at any point.

On a high-level, this system should be:

  • Hubiquitous, needs to be available (read/write) whenever one has internet connection (falling back to pen&paper, otherwise).
  • Searchable, one has to be able to search and filter the pile of notes.
  • Out of the way, sophistication will only get in the way of recording one's thoughts.

joyce is structured as a small tool written in Rust running on a server, loosely inspired by twtxt. Clients interface themselves with the server through a simple REST API.

Installation and setup

Command Line Interface

joyce offers a very simple command line interface. For an overview of the functionalities offered by the tool run

joyce --help

joyce uses the clap library to parse command line arguments

clap = { version = "3.1", features = ["derive"] }

use clap::Parser;

using the Derive API to define the exposed functionalities. The struct holding the CLI information is defined as follow

/// Record your thoughts as they come.
#[derive(Parser, Debug)]
#[clap(author, version, about, long_about = None)]
struct Config {
  <<config_db>>
  <<config_address>>
}

and the arguments are parsed as

let config = Config::parse();

joyce assumes a SQLite database named notes.db is present in the current working directory.

const DB: &str = "./notes.db";

A custom path to a SQLite database can be passed using the -d/--database flag.

use std::path::PathBuf;

/// SQLite database [default: './notes.db']
#[clap(short, long)]
database: Option<PathBuf>,

Once started, joyce will be available on localhost (127.0.0.1) at port 8080.

const ADDR: &str = "127.0.0.1";
const PORT: u16 = 8080;

This behaviour can be overridden with the -a/--address and -p/--port flags

/// Address the `joyce` service is bound to [default: '127.0.0.1']
#[clap(short, long)]
address: Option<String>,
/// Port the `joyce` service is bound to [default: 8080]
#[clap(short, long)]
port: Option<u16>,

Notes

Notes are the first-class citizen of joyce and are the main content exchanged between server and clients.

Anatomy of a note

A note is a simple structure recording a unique ID and a timestamp, both assigned at creation, a list of tags, and the body of the note.

#[derive(Debug, Deserialize, Serialize)]
pub struct Note {
  id: i64,
  timestamp: DateTime<Utc>,
  tags: Vec<String>,
  body: String,
}

impl Note {
  pub fn new(id: i64, timestamp: DateTime<Utc>, tags: Vec<String>, body: String) -> Self {
    Self { id, timestamp, tags, body }
  }
}

A Note can also be create from a Row, result of a query over the database.

use rusqlite::Row;

impl From<&Row<'_>> for Note {
  fn from(row: &Row<'_>) -> Note {
    Note::new(
      row.get(0).expect("Failed to read column 1"),        // id
      row.get(1).expect("Failed to read column 2"),        // timestamp
      serde_json::from_value(row.get::<_,serde_json::Value>(2).expect("Failed to read column 3")).expect("Failed to parse JSON"), // tags
      row.get(3).expect("Failed to read column 4")) // body
  }
}

Similarly, a NoteRequest is what a client would request at note creation. It contains the same information as a Note without the information assigned at creation by the server and the database.

use serde_json::Value;

#[derive(Debug, Deserialize, Serialize)]
pub struct NoteRequest {
  tags: Vec<String>,
  body: String,
}

The stucture can also be converted into a tuple of parameters to pass a SQL query (see Params from rusqlite).

pub type NoteParams = (DateTime<Utc>, Value, String);

impl From<NoteRequest> for NoteParams {
  fn from(req: NoteRequest) -> NoteParams {
    (Utc::now(), Value::from(req.tags), req.body)
  }
}

(De)serialization

Since notes need to be sent and received via HTTP, the structure needs to be serializable (from/to JSON format).

serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"

use serde::{Serialize, Deserialize};

IDs

We are using a i64 as unique identifier because the field is automatically handled by SQLite, and i64 is the most convenient type to convert from/to SQLite's INTEGER (according to rusqlite).

Timestamps

Timestamps adhere the RFC 3339 date-time standard with UTC offset.

chrono = { version = "0.4", features = ["serde"] }

use chrono::prelude::{DateTime, Utc};

The REST API

joyce uses actix-web to handle HTTP requests and responses.

actix-web = "4.1"

use actix_web::{HttpResponse,Responder,web,get,post};
use actix_web::http::header::ContentType;
use super::db::Pool;

Each request handlers is an async function that accepts zero or more parameters, extracted from a request (see FromRequest trait), and returns an HttpResponse.

Internally requests will be carried out by querying the underlying SQLite database.

use super::db;

GET /notes

This handler allows to request the full list of notes currently in the system. The function takes 1 parameters (the connection pool to the underlying SQLite database) and returns a collection of notes as a JSON array.

#[get("/notes")]
pub async fn get_notes(pool: web::Data<Pool>) -> HttpResponse {
  <<get_connection>>
  let notes: Vec<Note> = db::get_notes(&mut conn);
  HttpResponse::Ok()
    .content_type(ContentType::json())
    .json(notes)
}

POST /notes

New notes can be added by POSTing a JSON array of NoteRequests of the form

{
    "body": "This is a funny note",
    "tags": [ "joyce", "funny", "example" ]
}

The function takes 2 parameters:

  • the connection pool,
  • the collection of NoteRequests as a JSON object.
#[post("/notes")]
pub async fn post_notes(pool: web::Data<Pool>, req: web::Json<Vec<NoteRequest>>) -> impl Responder {
  <<get_connection>>
  let res = db::post_notes(&mut conn, req.into_inner());
  format!("Successfully added {res} note(s)")
}

GET /tags/{tags}

Notes can be retrieved by (sets of) tags. One or more tags separated by + can be passed to the request and joyce will retrieve those notes marked with all the provided tags.

The function takes 2 parameters (the connection pool to the underlying SQLite database and the sequence of tags) and returns a collection of notes as a JSON array.

Passing an empty collection of tags results in an error.

#[get("/tags/{tags}")]
pub async fn get_tags(pool: web::Data<Pool>, tags: web::Path<String>) -> HttpResponse {
  <<get_connection>>
  let tags = tags.split('+').map(|t| t.to_string()).collect::<Vec<_>>();
  if !tags.is_empty() {
    let notes: Vec<Note> = db::get_tags(&mut conn, &tags);
    HttpResponse::Ok()
      .content_type(ContentType::json())
      .json(notes)
  } else {
    todo!()
  }
}

SQLite backend

Notes are saved into a SQLite database. The notes database contains a single table mirroring the Note's structure.

CREATE TABLE notes (
    id        INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT    NOT NULL,
    tags      TEXT    NOT NULL,
    body      TEXT    NOT NULL
);

Note that, apart from standard SQLite types, DateTime<Utc> is converted to/from TEXT, Vec<String> is first wrapped in a JSON Value (from the serde_json crate) and then converted from/to TEXT.

ids are handled automatically by SQLite and are not set on the Rust side.

A new database can be bootstraped in the current directory as follows:

sqlite3 ./notes.db < notes.sql

Interfacing with SQLite from Rust

To interface with the underlying database we use rusqlite along with r2d2 to create a connection pool.

r2d2 = "0.8"
r2d2_sqlite = "0.21"
rusqlite = { version = "0.28", features = ["chrono","serde_json"] }

The following code sets up a connection pool to the SQLite database.

use std::path::Path;
use r2d2_sqlite::SqliteConnectionManager;

pub type Pool = r2d2::Pool<SqliteConnectionManager>;

pub fn get_connection_pool<P: AsRef<Path>>(db: P) -> Pool {
    let manager = SqliteConnectionManager::file(db);
    r2d2::Pool::new(manager).expect("Unable to connect to {db}")
}

When needed, one can get a connection from the pool. This is a blocking function as as such is wrapped in a web::block to offload the task to one of the actix thread workers.

let mut conn = web::block(move || pool.get())
  .await
  .expect("Blocking error")
  .expect("Error getting from connection pool");

Retrieving notes

We first build the query (caching it for later) and then execute it to retrieve all requested notes.

We return a Vec<Note> built from the query result.

use rusqlite::Connection;
use super::note::Note;

pub fn get_notes(conn: &mut Connection) -> Vec<Note> {
    let mut query = conn.prepare_cached("SELECT * FROM notes")
        .expect("Failed to prepare SELECT query");
    query
        .query_map([], |row| Ok(Note::from(row)))
        .and_then(Iterator::collect)
        .expect("Failed to collect query results")
}

Creating notes

When inserting new Notes in the database, we loop over the requests, attaching a timestamp and executing an INSERT SQL query. SQLite will take care of attaching an ID to the new entry.

Operations are executed into a single transaction to achieve better performances.

use super::note::{NoteParams,NoteRequest};

pub fn post_notes(conn: &mut Connection, reqs: Vec<NoteRequest>) -> usize {
    let insert = "INSERT INTO notes (timestamp, tags, body) VALUES (?, ?, ?)";
    let tx = conn.transaction().expect("Failed to start transaction");
    {
        let mut stmt = tx.prepare_cached(insert).expect("Failed to prepare INSERT query");
        reqs.into_iter().for_each(|req| { stmt.execute::<NoteParams>(req.into()).expect("Failed to execute INSERT query"); });
    }
    tx.commit().expect("Commit failed");
    0
}

Retrieving tagged notes

We can query for specific tags by using the LIKE SQL operator. Such an SQL query will look like this

SELECT * FROM notes
WHERE tags LIKE '%<tag>%'
[ AND tags LIKE '%<tag>%' ...]

where the escaped expression '%<tag>%' matches anything containing the word <tag>, i.e., the % operator matches a (possibly empty) sequence of characters.

A query needs to be dynamically generated for the specific number of provided tags.

let mut query = String::with_capacity(32 + 16 * tags.len());
query.push_str("SELECT * FROM notes WHERE tags LIKE ? ");
for _ in 0..(tags.len()-1) {
    query.push_str("AND tags LIKE ? ");
}

Once built, we execute the query with the sequence of tags as parameters. Note that we need to use rusqlite::params_from_iter in order to handle a dynamic set of parameters.

pub fn get_tags(conn: &mut Connection, tags: &[String]) -> Vec<Note> {
  <<build_query_tags>>
  let mut query = conn.prepare_cached(&query)
      .expect("Failed to prepare SELECT query");
  let params = tags.iter().map(|tag| format!("%\"{}\"%", tag));
  query
      .query_map(rusqlite::params_from_iter(params), |row| Ok(Note::from(row)))
      .and_then(Iterator::collect)
      .expect("Failed to collect query results")
}

Main service

The main service will instantiate a new App running within a HttpServer bound to localhost on port 8080.

use actix_web::{App, HttpServer, web};

The App will register all request handlers defined above as services.

#[actix_web::main]
async fn main() -> std::io::Result<()> {
  <<config_parse>>
  let addr = config.address.unwrap_or_else(|| ADDR.to_string());
  let port = config.port.unwrap_or(PORT);
  let db_file = config.database.unwrap_or_else(|| PathBuf::from(DB));
  let db_pool = db::get_connection_pool(db_file);
  HttpServer::new(move || {
    App::new()
      .app_data(web::Data::new(db_pool.clone()))
      .service(note::get_notes)
      .service(note::get_tags)
      .service(note::post_notes)
  })
  .bind((addr, port))?
  .run()
  .await
}

The program structure

The main program is structured as follows

mod note;
mod db;

<<main_uses>>

<<main_constants>>

<<main_config>>

<<main_service>>

Notes, along with their REST API are defined in their own note module.

<<note_uses>>

<<note_struct>>

<<note_impl>>

<<note_request_struct>>

<<note_request_impl>>

<<req_get_notes>>

<<req_get_tags>>

<<req_post_notes>>

Communication with SQLite is grouped under the db module.

<<db_uses>>

<<db_types>>

<<db_connection_pool>>

<<db_operations>>

TODO: road to v1.0.0

  • Better error handling
  • Better logging
  • add examples with cURL

Open questions

  • Should one be able to delete notes? Or mark them as read/processed?
  • Authentication method?
  • Custom filters on retrieval.
  • rusqlite ships with SQLite bundled if necessary (useful to bootstrap the db?)

Credits

joyce v0.1.0 was created by Federico Igne (git@federicoigne.com) and available at https://git.dyamon.me/projects/joyce.

[package]
name = "joyce"
version = "0.1.0"
edition = "2021"

[dependencies]
<<dependencies>>
generated by cgit v1.2.3 (git 2.25.1) at 2025-07-18 14:17:38 +0000