Livebook

I've learned a few things using Livebook that took a minute to uncover, so I wanted a place to keep it all together. I'll list them as a table of contents and explain further in each section, with links to my stumbling through issues and PRs.

1. Version management: if you use a version manager like asdf or rtx, the Livebook version will be bound to your global or local instance, depending on the directory.
2. MacOS app bundles Elixir: the Livebook desktop app is a great case study for bundling Erlang and Elixir.
3. Reopening Livebook: if you close the Livebook window, you can open a new one by clicking on the Menu Bar icon and selecting Open.
4. Menu Bar quirks: when running applications like Vanilla or Bartender, the icon only shows up if I restart Vanilla.
5. Remote file management: the docker image defaults the working directory to /data, so if you use relative pathing in your notebooks or apps for storing files, you either have to infer this or change the WORKDIR in Docker.

Version Management

I spent maybe 20-30 minutes installing Livebook 0.9.3 and running it for my collection of notebooks. I would execute the command livebook server index.livemd in the directory, which ran version 1.14.4 at the time of discovery. When I upgraded Livebook, I was outside this directory which took in the global version 1.14.5. Because rtx or asdf completely isolates versions, I wasn't aware it was executing the 0.9.2 build with 1.14.4. Changing .tool-versions by using rtx use elixir@1.14.5-otp-25 brought my local environment up to global, and the Livebook version was what I expected.

I suspect this may bite me and others consistently if we're not careful. I'm sure I've had similar issues in the past that I did not equate to this at the time.

Desktop Elixir Bundler

A ton has changed since January 2022, but I opened this issue, which spurred a subsequent pull request. This work has slowly morphed into what is being called "ElixirKit" and is in the elixirkit/ directory in the repository. ElixirKit has a different surface area than Elixir Desktop, which is somewhat of a wrapper or orchestrator for wxWidgets in Elixir.

Reopening Livebook

In the past, when this was a Dock icon only, if you closed the browser window, it was less intuitive for me to open it again. I could reopen the window by clicking the Dock icon (see this issue for history). Now the app stays resident in the Menu Bar, and we can reopen windows or view logs to see the hidden console window. The view logs option is perfect for getting the auth token again. The dock also had issues fully closing the runtime but was ironed out <3.

Menu Bar Quirks

I use the free version of the Vanilla app. If Livebook is on the always-visible right side, the icon will display whenever I close and reopen the desktop app. If it's on the default left side, I must quit Vanilla for the icon to appear. The long-term fix of keeping it visible is enough for me, but it may bite other people that use similar menu bar applications.

This issue spells out what I saw and may be helpful. I kept trying to run the app again, not knowing that it was there in the menu bar but I just couldn't see it.

Remote File Management

I had issues understanding how to work with notebooks or public apps on Huggingface. When I used something like data = "./data/file-sorted.csv" where the notebook-relative data/ directory existed, I would get problems like ** (File.Error) could not open "./data/file-sorted.csv": no such file or directory. My fix at the time was for apps to use something like data = System.get_env("PWD") to get the current working directory. This change posed a problem locally because my relative data/ directory is in .gitignore. Using PWD would save the data in the current directory, which would not fall under .gitignore rules.

The long-term fix, which you can see here, was to include WORKDIR "/notebooks" at the end of the file. WORKDIR tells Docker the working directory so that /data is now the place for only Livebook configuration, and any relative paths will work as expected.

There are other parts to this Dockerfile, such as pulling down my notebooks and the ash_tutorial so that I could work on Huggingface for things like Whisper or other ML models. Adding git or FFmpeg as dependencies is trivial. I explicitly copy the public-apps directory from my repository so that the Huggingface repo stays pure and only cares about setting up the environment.

One of my goals is to flesh out a system where I can work on notebooks remotely and periodically synchronize changes up to the repository. That's why I've included git-sync but I haven't worked out how to leverage it. A public or private app could leverage Kino.start_child/1 to start a GenServer that watches the filesystem for changes and presents a UI to commit and push changes. I believe something like egit could do this, I would need to create the UI for it. I'd certainly take this approach over shelling out to the git CLI though that's not extraordinarily difficult either.

I had the privilege of hanging around Brooklin Myers before he joined DockYard as an instructor in early 2022. A unique Elixir community slowly coalesced with the first video of the beta cohort starting on September 21st, 2022. I wouldn't join the group until October 15th as I wasn't sure what to make of it at first. I figured I would audit the class like I was some college kid.

The Curriculum

The academy skews toward junior developers or other Elixir newbies without previous formal instruction. Despite that, the curriculum and the commitment of 2 hours per day was an exceptional resource regardless of experience level.

The curriculum is not as lightweight as Elixir koans, and it is not as self-paced as Exercism's Elixir track. I hadn't been a part of the Exercism Elixir cohort on Discord, but I suspect it may have been similar.

What sets the curriculum apart is that it starts in Livebook, a low barrier to entry for learning Elixir. Eventually, it moves to bare mix new projects, graduating to full-on mix phx.new Phoenix applications. The beta curriculum experience was different than the first cohort, and there are upcoming changes for the second cohort. It's helpful to know the curriculum changes when pain points surface. There is no sleight of hand or abandonware as the official repository is what is taught from start to finish.

As someone that can have analysis paralysis at times when it comes to what and how to learn, having the path chosen for me was extremely helpful. Exercism gates the syllabus, but that can be daunting to decipher when you're starting. I also rushed through the concepts I was interested in rather than taking the time to enjoy the journey. I firmly believe the curriculum and Exercism complement each other very well.

The curriculum culminated in a capstone project, a chance to bundle all the skills we learned to produce our applications. The capstone sets it apart from other learning materials.

The Cohort

The beta cohort was a mix of Elixir newbies, seasoned Elixir developers and mentors, and people that hadn't touched a programming language. We experimented with teaching styles and nailed a cadence "locked in" at the last minute. Everyone I paired with showed remarkable improvement between October and the demo day on January 20th. That level of improvement is a testament to Brooklin's teaching style. Fundamentals became second nature very quickly. I would be lucky to work with anybody I met in the cohort or Discord server, as everyone grew into a developer. Elixir has a way of binding cohesive communities, but Brooklin truly has his superpower with the people around him. As much as I love DockYard, this felt like "The Brooklin Show" *sponsored by DockYard(tm)

Demo Day

I was one of the fewer resident developers to present on Demo Day, and that almost didn't happen. My capstone project, Beatseek, was hastily thrown together by duct tape. I had a working prototype at least a month before the deadline, but I had only given myself ten days from mix phx.new to what I presented. I thought it went well without a script, working through some prior presentations, but it was unpolished. I used sleight of hand as I do on some demos, but as a magician, I wanted to show all the tricks.

I didn't cut a public release until two months after demo day because I wasn't happy with what I produced. I had to retrofit tests, which exposed several shortcomings. If I had to do it again, I would choose anything other than id3 tags because the edge cases are absurdly complex.

What Would I Change?

I had a few issues working through the curriculum or with other cohort members. Tracking progress was difficult, but I used an Obsidian daily standup journal template to check off the table of contents manually. The standup journal became a good way of tracking changes over time, though there were few. The ramp-up to Phoenix for people with no web development or API exposure was pretty steep for the beta cohort, but I don't know if this is still true. Web development fundamentals span a breadth of knowledge, but the curriculum helps cement these concepts. People new to web development may wish to spend more time going through the same sections a few times until the concepts of things like MVC are less foreign. It'll make the later parts much easier to push through.

The End Result

I am 100% glad I had access to an instructor and mentor, even in a limited capacity. Everyone on the Discord server is excellent and a joy to be around. I would do this again in a heartbeat, but 2 hours was a sweet spot for someone like me with a full-time position to juggle. I can see how much more beneficial the 6-hour full day could be with more immersion, but that is a lot of material to cram. We had some luxury in drawing the material out and taking some time to keep everyone on the same pace.

Tell me if you've done this before. You write up a nice little prototype of an idea in Livebook. You then get distracted by life situations like eating, writing an email, or taking a nap. You feel the need to close Livebook or prune the multiple sessions you've had running for weeks now. Because you have a million tabs open (with a session manager) and too many in Livebook to individually check, you restart your computer and let it crash(TM). When you open up Livebook again, "Oh. Shiiiiit" you exclaim. Where the hell did that notebook go? I'm 100% sure I clicked the disk icon, what the hell? If you're like me, you may have created this forked Livebook from memory, possibly taking a better approach.

There is a better way to handle this scenario. Livebook has had autosaves since 0.4:

The feature was added in this PR according to the changelog:

https://github.com/livebook-dev/livebook/pull/736

To find your autosave files:

• For the Desktop application and CLI in production: ~/Library/Application Support/livebook/autosaved/.

  • On my machine this expands to the absolute path /Users/jbrayton/Library/Application Support/livebook/autosaved/.

• For the dev environment: in config/dev.exs, this is set as config :livebook, :data_path, Path.expand("tmp/livebook_data/dev".

  • On my machine this expands to the absolute path /Users/Shared/repositories/personal/elixir/livebook/tmp/livebook_data/dev/autosaved/.

• For the test environment: in config/test.exs this is set as Path.expand("tmp/livebook_data/test").

  • On my machine this expands to the absolute path /Users/Shared/repositories/personal/elixir/livebook/tmp/livebook_data/test/autosaved/.

Notebooks are saved by day in the autosave directory and the date corresponds to when they were created (when you immediately click the New notebook button).

To view or change your autosave directory in the CLI:

• Go to http://localhost:8080/settings
• Or, if you're already in a notebook, click the Livebook icon in the top left and click Settings under the Home and Learn links.

For the Desktop application, the port will be randomized but you can either change the URL to tack on /settings after the port or click around to the settings page as described earlier.

Tracing the Default Setting

If you are curious as to how this setting gets configured, we can start by looking at Livebook.Settings.default_autosave_path() in https://github.com/livebook-dev/livebook/blob/main/lib/livebook/settings.ex#L32-L34. We follow Livebook.Config.data_path() to https://github.com/livebook-dev/livebook/blob/main/lib/livebook/config.ex#L76-L78 then the Erlang function :filename.basedir(:user_data, "livebook").

Running this in Livebook we get the output "/Users/jbrayton/Library/Application Support/livebook", precisely where the desktop app stores its files.

Finding Files

What lead me to this discovery, after vaguely remembering autosave was a thing, was looking for files on my computer. I purposefully install and use the locate command because I find it far easier to use than remembering the find -name syntax.

Here's the output for checking that the word autosave is in any directory or file name:

⋊> ~ locate autosaved/ 
/Users/Shared/repositories/personal/elixir/livebook/tmp/livebook_data/dev/autosaved/2022_10_31/18_25_03_mapset_drills_hedh.livemd
/Users/Shared/repositories/personal/elixir/livebook/tmp/livebook_data/dev/autosaved/2022_11_03/18_12_21_teller_bank_challenge_pv4e.livemd
/Users/Shared/repositories/personal/elixir/livebook/tmp/livebook_data/dev/autosaved/2022_11_03/18_13_39_untitled_notebook_pidb.livemd
/Users/Shared/repositories/personal/elixir/livebook/tmp/livebook_data/dev/autosaved/2022_11_03/19_31_57_dockyard_academy_amas_p75r.livemd
/Users/Shared/repositories/personal/elixir/livebook/tmp/livebook_data/dev/autosaved/2022_11_03/20_02_17_intro_to_timescale_jm7r.livemd
/Users/Shared/repositories/personal/elixir/livebook/tmp/livebook_data/dev/autosaved/2022_11_08/11_10_21_untitled_notebook_ervg.livemd
/Users/Shared/repositories/personal/elixir/livebook/tmp/livebook_data/dev/autosaved/2022_11_22/19_15_12_untitled_notebook_p75e.livemd

What I found interesting was that my files in ~/Library/Application Support/livebook/autosaved/ did not show up. Had I not realized there could be different locations, I may have overlooked the notebook I was looking for all along. I have no clue why locate doesn't scour the directories in ~/Library it should have access to but that's a problem for another day.

Originally, I wrote up a post trying to give a 2020 - 2021 overview that got hosed with a local git repo of this blog. I'm using the moment to remind myself that backups are important. It's also important to complete ideas for posts or journals quickly, even if something doesn't feel complete. Letting those linger for days without a git commit that hit the server is a genuine problem and I need to at the very least create and push to a new branch often.

One change that happened at the end of 2020, I started the journal section to try to capture bite-sized rough ideas. I had started a journal at work with notes in files like Phoenix Developer Diary.txt and I looked for a solution to merge my different diaries. The excellent Claire Codes has an extremely consistent diary at clairecodes and served as my main source of inspiration.

I've gone all-in learning Elixir by participating in my first Advent of Code in 2020. I tapered off pretty quickly as I had serious problems working through loops and control flow. Seeing other examples on Elixir Forum helped immensely as I had slowly gotten better at reading the code. Later on in the year, I decided to take a TodoMVC sample through to a LiveView version with a little help from other resources on the internet. I had also started a diary where I wanted to capture the approaches I took each day I worked on the example. I have a plan to try to tackle my version from scratch but I'm also looking at other application ideas.

While the Advent of Code and TodoMVC was good to get my feet wet, I learned far more by pushing through Exercism exercises. If you're on Exercism and curious, my solutions can be found here. I highly recommend using Exercism to learn any language it covers as the recently released version 3 makes for a great experience. Exercises feel a bit more "real world" and less like brain teasers that happen to use programming concepts. Even if I happened to look at the HINTS.md file, it never felt like cheating as it would only guide us toward a solution, not implement it.

After attending the excellent ElixirConf 2021 virtually, I've started working with Livebook in a few examples. I wanted to highlight the 3 notebooks that use the excellent spider_man package to crawl 3 websites: Elixir Jobs, Elixir Radar Jobs, and Elixir Companies. Parsing the DOM of each required slowly stretching far outside my comfort zone. It's also worth mentioning that in the Elixir Jobs example, I left a problem I found under the Sorting the Results section. Due to the zero-width space, the section throws the message ** (SyntaxError) nofile:5:1: unexpected token: "​" (column 1, code point U+200B).

Coming to the end of 2021, I'm looking forward to immersing myself deeper in the Elixir ecosystem. Livebook is also a great way to get your feet wet with Elixir concepts, like a powerful language scratchpad. There have been other life changes since January 2020 but those deserve separate posts when I can get to them. Fortunately, the pandemic hasn't been harsh on my family or extended family at all, which I consider an extreme blessing. I can't say we weren't impacted by the last 2 years but things could've been much worse.