I’m letting my Pinboard subscription expire. I’ve added 3,500 bookmarks since 2010 and the service was valuable when actively engaged in research. I haven’t looked up a bookmark in the past few years though. Now I’ll rely on Apple Notes and Micro.blog Bookmarks


Time to stop tracking my personal life ⏰

Through 2020, I built up an ornate system for tracking my time for both work and personal projects (like this one for reading). For most of 2021, I found this tracking really helpful.

I need to track my hours at work anyway, so using Timery and Shortcuts to automate much of this has been great. Having a strong sense of how long things take and ensuring good balance across projects are all benefits of time tracking.

For personal projects, though, I’ve been starting to feel a bit stressed by having a timer always running whenever I’m doing something, almost like I’m always in a race. At first, knowing how much time I was spending on particular things was great for my Year of the Tangible intention. This is well established now, and I haven’t been using the time reports for any personal projects. So, why am I creating anxiety for no benefit?

I’ve turned off all of my time tracking automations for personal projects. Despite some annoying bugs, ScreenTime is a good-enough replacement for keeping an eye on time spent on things like YouTube and social networking. A nice side benefit is that this also reduces the number of Shortcuts and other automation that I need to manage, allowing me to just enjoy my personal time.

Of course, I’ll keep tracking work projects, since the benefits far outweigh the costs there.


As an update to my earlier post about using MindNode for task management, I’ve refreshed my areas of focus and projects for work. I still find MindNode really helpful for this, especially for seeing the balance of projects across the areas of focus. In this case, I can see that I have many Process Improvement projects, which makes sense, given my company has a big push on Lean at the moment.

Screenshot of areas of focus and projects as a mindmap in MindNode

Now I can fill in next actions for each project and then sync with Reminders.


2021 iPhone Home Screens

My iPhone Home Screen continues to evolve and, now that we have Focus Modes, I’ve made some further adjustments.

Screenshot of three iOS HomeScreens based on Focus Modes

From left to right, I’m using three different Focus Modes: Personal, Work, and Fitness. The first two are entirely widget focused, while Fitness has a few app icons as well.

The dock has Drafts which, as the tagline says, is where text starts. This hasn’t changed from my earlier setups. The second icon launches a front-end Shortcut for Apple Notes. As I described in my Apple Note overview, this is an idea that I’ve borrowed from Matthew Cassinelli and providers a flexible interface to the app.

Personal

From top left, my Personal Home Screen starts with a stack of Reminders filtered to my Personal list, Fantastical, and Streaks. This is essentially my “what should I be doing” stack.

Next is a stack with Photos and Siri Suggestions. The Photos widget consistently surfaces delightful photos, so I’ve given it a prominent spot. While the usefulness of Siri Suggestions are variable, I like the idea of my phone learning my habits and showing me relevant actions.

Through the middle, I have Weather on the left and the right is a stack of Apple Music and Overcast, which are my options for listening to something.

On the bottom left is a stack of Timery and Screen Time. These are there to keep me mindful of what I’m actually doing, especially on weekends. The Timery widget shows me a summary view of my projects. So, in this screenshot I’ve put in an hour on exercise, another hour on reading, and 20 minutes with some household chores. The Screen Time widget helps keep me honest about how much I’m using my devices, especially on weekends when I really should be looking at something besides a screen.

And on the bottom right is a stack of Day One and Notes, filtered to my Personal folder. Day One is there for capturing family events and reflections. While the Notes folder often has some useful reference material for our weekend activities.

Work

Curiously, my Work Home Screen is less complicated than my Personal one.

The top is a stack with Fantastical and Mail’s VIP widget. I’m not entirely convinced that the Mail widget is useful here. I almost always just want Fantastical reminding me of my next meeting or task.

Given the more variable number of tasks I tend to be doing while in work mode, I’ve got the Siri Suggestions widget in the middle. I took this screenshot on the weekend, so it isn’t indicative of what it usually shows, which tends to be one of the Shortcuts that I’m often launching to manage my workday.

Intentionally mirroring my Personal HomeScreen, the bottom row is a stack with two Timery widgets and a stack of two Notes widgets, one filtered to my Work folder and the other to my Meetings folder.

Fitness

The Fitness Home Screen is mostly an experiment. I spend the vast majority of my time in one of the other two Focus Modes, so I’m not yet convinced that I need any other Home Screens.

Regardless, this one has the Fitness widget at the top for seemingly obvious reasons.

The middle row has the Training Today widget to help keep me honest about rest. And then a cluster of icons on the right. The only one that is non-standard is “Workout mix”, which is just a Shortcut to launch a good playlist in Apple Music.

The bottom row has Carrot Weather to make sure I’m not about to get rained on when heading out for a run. I’ve also added the Batteries widget there to make sure my Apple Watch and AirPods are ready for action.


I’ve set up Personal Automations to automatically switch between my Personal and Work Home Screens at 8:45 and 17:30. I’ve found these good reminders to keep my work activities within reasonable office hours. Starting a Workout automatically switches to the Fitness Home Screen.

There’s almost endless scope for fiddling with these. So, by writing them here, I’m adding some accountability to just stop that and use them for a while before making further changes.


Lessons from using Apple Notes for three months

Back in September, I committed to using Apple Notes for three months. The goal was to focus on my use cases for writing, rather than fiddling with new apps continuously.

Here’s what I’ve identified so far. Many of the approaches and features that I’m using in these use cases are readily available in other apps and often Notes is not the most efficient choice. Now that I’ve documented these use cases, I’d like to use them to assess alternative apps.

Meeting notes

Thanks to Timery, I know that 60% of my working time is spent in meeting. So many meetings!

For each one, I create a note to capture ideas, useful information, and tasks. I’ve automated this with a couple of Shortcuts. The one I use the most is “Start My Next Meeting”. This presents me with a list of upcoming meetings. I choose from the list and it creates a meeting note, starts a Timery timer, and opens the link to start the video call (typically Teams). The meeting note has the name of the meeting as the title, adds tags for #meeting and the Timery project, adds the date and time of the meeting, a list of attendees, and any notes from the calendar event. From this structure, I can then add notes throughout the meeting and extract any tasks into Reminders later.

I used Agenda for these sorts of notes before, which was powerful.

Daily summaries

Occasionally, I find myself at the end of a week with no clear sense of what I actually accomplished. To help with this, for the past year I’ve been recording the top three things I’ve done in a day into Day One (the 5 Minute PM template has been great for this).

To augment this, I’ve been using another Shortcut to create a Daily Work Report. This makes a note of the meetings I attended, tasks I completed, and tasks I created. These all get saved to a Daily Notes folder. I then use the day’s work report to pull out the highlights for Day One. There’s some redundancy here, though I find the process of choosing just three things for Day One is helpful.

Overall, I think that Day One is a better app for this use case.

Project notes

For each of my projects, I create a project note that states the purpose or objective of the project, key stakeholders, and timelines. Then I accumulate relevant notes and documents while making progress on the project. Creating these are also done via a simple Shortcut. I’ve experimented with using checklists for tasks in these notes, but find it isn’t as effective as my approach with MindNode and Reminders.

Once I finish a project, the associated note gets cleaned up and moved to an Archive folder to keep it out of the way.

Research

This is a rather broad category and, unlike the previous use cases, is for both work and personal notes. Much of this is capturing facts, quotes, and sources. If it is project specific, they go to the project note. Some are more generic and are kept as a standalone note. All of them get tags to help provide some structure. This is where Apple Notes ability to accept almost anything from the share sheet is powerful.

The new Quick Notes feature has been interesting for research. The ability to quickly highlight and then resurrect content on websites is great. I find actually working with the quick notes is pretty clumsy though. They have to stay in the Quick Notes folder and choosing which one to send content to can be tricky. I think there’s some great potential here and will keep experimenting.

For any webpages that I want to archive, I use another Shortcut that creates a plain-text note of the webpage along with some metadata and then adds the link to Pinboard. This has been surprisingly useful for recipes, when all I really want are the ingredients and steps, rather than the long history of the recipe’s development.

Other nice features

In addition to these use cases, there are a few nice features of Apple Notes that are worth mentioning.

  • As should be apparent from above, creating useful Shortcuts for Apple Notes is straightforward. In some sense, it is the Shortcuts that I’m finding really useful. Apple Notes is just the final destination for the content.
  • I’ve set up widgets by focus mode so that the most recent notes are shown on my Home Screen in the right context. These are restricted to a particular folder and sorted by date modified.
  • The formatting options are comprehensive, including table support.
  • I think I like the feature where checking off tasks moves them to the bottom of the list. Most of the time, this is what I want.
  • The iCloud web app is convenient for using notes on my Windows work PC. Unfortunately, I’ve found the syncing to be rather unreliable here, where notes just don’t show up in the web app sometimes.
  • I don’t share notes as often as I expected. When I do, it works really well.
  • Not really specific to Apple Notes, but I stole an idea from Matthew Cassinelli to aggregate all of these Notes Shortcuts into one super Shortcut that creates a list of Shortcuts to choose from.

Challenges

There are a few things that don’t work as well as they should:

  • Searching is too limited. In particular, you can’t narrow searches to particular folders. Most of the time I either only want to search my meeting notes or not include them. I had to set up a Shortcut that takes a search term as an input and then asks me to specify which folder to search. This should be built into the app’s search field.
  • Linking among notes isn’t really supported. You can sort of do this with url searches for note titles. Pretty clunky though.
  • Given how much I use Shortcuts for Apple Notes, it is frustrating how little you can do when creating a note. In particular, you can’t style text or add tags. Every time I use Shortcuts to create a note, the first thing I have to do is apply title and heading styles and convert any words prefixed with a # into an actual tag.

Other use cases

I’ve found a few use cases that don’t yet fit in with Apple Notes. I’m using Drafts for all of these:

  • Blog posts
  • Drafting long emails
  • Capturing and processing transitory texts, which Drafts is really optimized for

Restricting myself to Apple Notes was a helpful trick for crystallizing my use cases. Now that I’m three months in, I think I’ll stick with Apple Notes for a while longer. I’ve built up a good ecosystem of Shortcuts for working with the app and, of course, now have lots of content in the app.


Switching to iCloud+ Custom Email Domain 📧

I’ve switched my personal email over to Apple’s custom email domain with iCloud Mail. A roughly ranked list of reasons for the switch is:

  1. One less account to worry about. Not that it was a big deal, but now I don’t need to know the various setup details for my personal email. Once I’ve logged into my iCloud account, my email is ready.
  2. I appreciate Apple’s commitment to privacy and trust that they’ll apply this commitment to my email account.
  3. I’m already paying for iCloud+ and, so, might as well use this feature and save some money by not paying for separate email hosting.
  4. I’m actively using Reminders and Notes in iCloud.com and the Mail interface there is decent, certainly better than the rudimentary one offered by my previous email host.

Setup was straightforward with clear instructions. Having said that, the only issue I had was that initiating the setup process simply didn’t work for a few weeks. I tried a couple of times a week and each time I just got a generic error. Then, for no apparent reason, one day it worked. I suspect this was just an issue with rolling out a new service.

I should point out that my email needs are very basic for this personal account. I don’t need many automated rules, tagging, or filtering. So, iCloud Mail is fine. I wouldn’t switch over my work account (even if corporate IT would allow it). I get something like 100x the email at work and need more sophisticated tools.

Besides the initial trouble with initiating the setup, everything has been working well for the past week. I’m well aware of Apple’s well-earned reputation for challenges with internet services and will be staying vigilant for at least the next few weeks. One of the great benefits of having my own domain name is the ease with which I can switch mail hosts.


Switching from Agenda to Craft, for now 📅🗒

In my corner of the internet, there’s a well trodden, twisted path of searching for the one true notes app. I’ve reached a fork in the path between Agenda and Craft. As I wrote earlier, I’ve been using Agenda for a while now and its date-based approach really suits my meeting-dominated work. Now, though, Craft has added calendar integration and I’m testing it out.

There are several things I really like about Craft, relative to Agenda:

  1. Document syncing is far more reliable. This isn’t entirely Agenda’s fault. I’m restricted by corporate policy from using iCloud Documents, so have been using Dropbox sync for Agenda. I often have to wait an indeterminate, though long, time before documents sync across my devices. Craft sync has been instantaneous and very reliable.
  2. Having access to my documents from a web browser is great. I’ll be back to working from the office on a Windows laptop soon and won’t have access to my iPad. So, web access will be important.
  3. Performance is much better on Craft. Agenda often freezes in the middle of typing and suffers from random crashes. This could very well be something about my particular setup, though it doesn’t happen in any other apps.

On the downside, I do miss Agenda’s simplicity. Craft has lots of ways to organize notes (such as cards and subpages). Of course, you can mostly ignore this, but I like Agenda’s well-thought-through approach that didn’t require much deliberation about where to put things.

Of course, having just made this switch, Apple announced Quick Notes and I may well be back on Apple Notes in a few months.


Scheduling random meetings with a Shortcut ⚙️🗓

Staying in touch with my team is important. So, I schedule a skip-level meeting with someone on the team each week. These informal conversations are great for getting to know everyone, finding out about new ideas, and learning about recent achievements.

Getting these organized across a couple of dozen people is logistically challenging and I’ve developed a Shortcut to automate most of the process.

Borrowing from Scotty Jackson, I have a base in AirTable with a record for each team member. I use this to store all sorts of useful information about everyone, including when we last had a skip-level meeting. The Shortcut uses this field to pull out team members that I haven’t met with in the past four months and then randomizes the list of names. Then it passes each name over to Fantastical while also incrementing the date by a week. The end result is a recurring set of weekly meetings, randomized across team members.

The hardest part of the Shortcut development was figuring out how to get the names in a random order. A big thank you to sylumer in the Automators forum for pointing out that the Files action can randomly sort any list, not just lists of files.

I’m not sharing the Shortcut here, since the implementation is very specific to my needs. Rather, I’m sharing some of the thinking behind the code, since I think that it demonstrates the general utility of something like Shortcuts for managing routine tasks with just a small amount of upfront effort.


MindNode is the best mind mapping app for iOS

Continuing my plan to update App Store reviews for my favourite apps, up next is MindNode.

MindNode is indispensable to my workflow. My main use for it is in tracking all of my projects and tasks, supported by MindNode’s Reminders integration. I can see all of my projects, grouped by areas of focus, simultaneously which is great for weekly reviews and for prioritizing my work.

I’ve also found it really helpful for sketching out project plans. I can get ideas out of my head easily with quick entry and then drag and drop nodes to explore connections. Seeing connections among items and rearranging them really brings out the critical elements.

MindNode’s design is fantastic and the app makes it really easy to apply styles across nodes. The relatively recent addition of tags has been great too. Overall, one of my most used apps.


Trying out a new iPhone Home Screen 📱

With the release of iOS 7, I’m reconsidering my earlier approach to the Home Screen. So far I’m trying out a fully automated first screen that uses the Smart Stack, Siri Suggestions, and Shortcut widgets. These are all automatically populated, based on anticipated use and have been quite prescient.

My second screen is all widgets with views from apps that I want to have always available. Although the dynamic content on the first screen has been really good, I do want some certainty about accessing specific content. This second screen replaces how I was using the Today View. I’m not really sure what to do with that feature anymore.

I’ve hidden all of the other screens and rely on the App Library and search to find anything else.

I still like the simplicity behind my earlier approach to the Home Screen. We’ll see if that is just what I’m used to. This new approach is worth testing out for at least a few weeks.


Reading Shortcut for the iPad 👓⚙️

I haven’t yet adopted the minimalist style of my iPhone for my iPad. Rather, I’ve found that setting up “task oriented” Shortcuts on my home screen is a good alternative to arranging lots of app icons.

The one I use the most is a “Reading” Shortcut, since this is my dominant use of the iPad. Nothing particularly fancy. Just a list of potential reading sources and each one starts up a Timery timer, since I like to track how much time I’m reading.

Here’s a screenshot of the first few actions:

I like the bit of whimsy from using emoticons:

A nice feature of using a Shortcut for this is that I can add other actions, such as turning on Do Not Disturb or starting a specific playlist. I can also add and subtract reading sources over time, depending on my current habits. For example, the first one was Libby for a while, since I was reading lots of library books.

This is another example of how relatively simple Shortcuts can really help optimize how you use your iOS devices.


Reflection journal in Day One with an Agenda assist

I’ve been keeping a “director’s commentary” of my experiences in Day One since August 2, 2012 (5,882 entries and counting). I’ve found this incredibly helpful and really enjoy the “On This Day” feature that shows all of my past entries on a particular day.

For the past few months, I’ve added in a routine based on the “5 minute PM” template which prompts me to add three things that happened that day and one thing I could have done to make the day better. This is a great point of reflection and will build up a nice library of what I’ve been doing over time.

My days seem like such a whirlwind sometimes that I actually have trouble remembering what I did that day. So, my new habit is to scroll through my Today view in Agenda. This shows me all of my notes from the day’s meetings. I’ve also created a Shortcut that creates a new note in Agenda with all of my completed tasks from Reminders. This is a useful reminder of any non-meeting based things I’ve done (not everything is a meeting, yet).

I’m finding this new routine to be a very helpful part of my daily shutdown routine: I often identify the most important thing to do tomorrow by reviewing what I did today. And starting tomorrow off with my top priority already identified really helps get the day going quickly.


Different watch faces for work and home

watchOS 7 has some interesting new features for enhancing and sharing watch faces. After an initial explosion of developing many special purpose watch faces, I’ve settled on two: one for work and another for home.

Both watch faces use the Modular design with the date on the top left, time on the top right, and Messages on the bottom right. I like keeping the faces mostly the same for consistency and muscle memory.

My work watch face than adds the Fantastical complication right in the centre, since I often need to know which meeting I’m about to be late for. Reminders is on the bottom left and Mail in the bottom centre. I have this face set to white to not cause too much distraction.

My home watch face swaps in Now Playing in the centre, since I’m often listening to music or podcasts. And I have Activity in the bottom centre. This face is in orange, mostly to distinguish it from the work watch face.

Surprisingly, I’ve found this distinction between a work and home watch face even more important in quarantine. Switching from one face to another really helps enforce the transition between work and non-work when everything is all done at home.

The watch face that I’d really like to use is the Siri watch face. This one is supposed to intelligently expose information based on my habits. Sounds great, but almost never actually works.


I'm not analyzing COVID data, though I'm impressed with Ontario's open data

I’m neither an epidemiologist nor a medical doctor. So, no one wants to see my amateur disease modelling.

That said, I’ve complained in the past about Ontario’s open data practices. So, I was very impressed with the usefulness of the data the Province is providing for COVID: a straightforward csv file that is regularly updated from a stable URL.

Using the data is easy. Here’s an example of creating a table of daily counts and cumulative totals:

data_source <- "[data.ontario.ca/dataset/f...](https://data.ontario.ca/dataset/f4112442-bdc8-45d2-be3c-12efae72fb27/resource/455fd63b-603d-4608-8216-7d8647f43350/download/conposcovidloc.csv)"
covid_data <- read_csv(data_source) %>% 
  select(ACCURATE_EPISODE_DATE) %>% 
  rename(date = ACCURATE_EPISODE_DATE) %>% 
  group_by(date) %>% 
  summarise(daily_count = n()) %>% 
  mutate(cumulative_count = cumsum(daily_count))

From there we can make some simple plots to get a sense of how the case load is changing.

And, I’ll leave it at that, at least for public posting 🤓


Simple brew tea shortcut

Since I’m mostly stuck inside these days, I find I’m drinking more tea than usual. So, as a modification of my brew coffee shortcut, I’ve created a brew tea shortcut.

This one is slightly more complicated, since I want to do different things depending on if the tea is caffeinated or not.

We start by making this choice:

Then, if we choose caffeine, we log this to the Health app:

Uncaffeinated tea counts as water (at least for me):

And, then, regardless of the type of tea, we set a timer for 7 minutes:

Running this one requires more interactions with Siri, since she’ll ask which type we want. We can either reply by voice or by pressing the option we want on the screen.


A simple Shortcut for tracking workout time

I’ve been tracking my time at work for a while now, with the help of Toggl and Timery. Now that I’m working from home, work and home life are blending together, making it even more useful to track what I’m doing.

Physical exercise is essential to my sanity. So, I wanted to integrate my Apple Watch workouts into my time tracking. I thought I’d be able to leverage integration with the Health app through Shortcuts to add in workout times. Turns out you can’t access this kind of information and I had to take a more indirect route using the Automation features in Shortcuts.

I’ve setup two automations: one for when I start an Apple Watch workout and the other for when I stop the workout:

The starting automation just starts an entry in Timery:

The stopping automation, unsurprisingly, stops the running entry:

As with most of my Shortcuts, this is a simple one. Developing a portfolio of these simple automations is really helpful for optimizing my processes and freeing up time for my priorities.


Brew coffee shortcut

Shorcuts in iOS is a great tool. Automating tasks significantly boosts productivity and some really impressive shortcuts have been created.

That said, it is often the smaller automations that add up over time to make a big difference. My most used one is also the simplest in my Shortcuts Library. I use it every morning when I make my coffee. All the shortcut does is set a timer for 60 seconds (my chosen brew time for the Aeropress) and logs 90mg of caffeine into the Health app.

All I need to do is groggily say “Hey Siri, brew coffee” and then patiently wait for a minute. Well, that plus boil the water and grind the beans.

Simple, right? But that’s the point. Even simple tasks can be automated and yield consistencies and productivity gains.


Things cost more than they used to

I’m delivering a seminar on estimating capital costs for large transit projects soon. One of the main concepts that seems to confuse people is inflation (including the non-intuitive terms nominal and real costs). To guide this discussion, I’ve pulled data from Statistics Canada on the Consumer Price Index (CPI) to make a few points.

The first point is that, yes, things do cost more than they used to, since prices have consistently increased year over year (this is the whole point of monetary policy). I’m illustrating this with a long-term plot of CPI in Canada from 1914-01-01 to 2019-11-01.

Figure showing the long-term increase in CPI from 1914 to 2019. An image of an old fashioned cany bar is on the left and large pile of modern candy on the right.

I added in the images of candy bars to acknowledge my grandmother’s observation that, when she was a kid, candy only cost a penny. I also want to make a point that although costs have increased, we also now have a much greater diversity of candy to choose from. There’s an important analogy here for estimating the costs of projects, particulary those with a significant portion of machinery or technology assets.

The next point I want to make is that location matters, which I illustrate with a zoomed in look at CPI for Canada, Ontario, and Toronto.

Figure showing CPI from 2000 to 2019 with separate lines for Toronto, Ontario, and Canada. They mostly align until about 2015, when Ontario starts increasing faster than Canada and Toronto faster than Ontario.

This shows that over the last five years Toronto has seen higher price increases than the rest of the province and country. This has implications for project costing, since we may need to consider the source of materials and location of the project to choose the most appropriate CPI adjustment.

The last point I want to make is that the type of product also matters. To start, I illustrate this by comparing CPI for apples and alcoholic beverages (why not, there are 330 product types in the data and I have to pick a couple of examples to start).

Figure showing CPI from the 1950s with separate lines for apples and alcoholic beverages. Both are relatively flat through the 1950s and 1960s then start to increase rapidly. Apples fluctuate alot in each year.

In addition to showing how relative price inflation between products can change over time (the line for apples crosses the one for alcoholic beverages several times), this chart shows how short-term fluctuations in price can also differ. For example, the line for apples fluctuates dramatically within a year (these are monthly values), while alcoholic beverages is very smooth over time.

Once I’ve made the point with a simple example, I can then follow up with something more relevant to transit planners by showing how the price of transportation, public transportation, and parking have all changed over time, relative to each other and all-items (the standard indicator).

Figure showing CPI from 2000 to 2019 with separate lines for parking costs, transportation, public transit, and all-items. All-items is the lowest and closely tracked by transportation. Public transportation diverges around 2015 and rises dramatically. Parking fees are by far the highest.

At least half of transit planning seems to actually be about parking, so that parking fees line is particularly relevant.

Making these charts is pretty straightforward, the only real challenge is that the data file is large and unwieldy. The code I used is here.


Task management with MindNode and Agenda

For several years now, I’ve been a very happy Things user for all of my task management. However, recent reflections on the nature of my work have led to some changes. My role now mostly entails tracking a portfolio of projects and making sure that my team has the right resources and clarity of purpose required to deliver them. This means that I’m much less involved in daily project management and have a much shorter task list than in the past. Plus, the vast majority of my time in the office is spent in meetings to coordinate with other teams and identify new projects.

As a result, in order to optimize my systems, I’ve switched to using a combination of MindNode and Agenda for my task managment.

MindNode is an excellent app for mind mapping. I’ve created a mind map that contains all of my work-related projects across my areas of focus. I find this perspective on my projects really helpful when conducting a weekly review, especially since it gives me a quick sense of how well my projects are balanced across areas. As an example, the screenshot below of my mind map makes it very clear that I’m currently very active with Process Improvement, while not at all engaged in Assurance. I know that this is okay for now, but certainly want to keep an eye on this imbalance over time. I also find the visual presentation really helpful for seeing connections across projects.

MindNode has many great features that make creating and maintaining mind maps really easy. They look good too, which helps when you spend lots of time looking at them.

Agenda is a time-based note taking app. MacStories has done a thorough series of reviews, so I won’t describe the app in any detail here. There is a bit of a learning curve to get used to the idea of a time-based note, though it fits in really well to my meeting-dominated days and I’ve really enjoyed using it.

One point to make about both apps is that they are integrated with the new iOS Reminders system. The new Reminders is dramatically better than the old one and I’ve found it really powerful to have other apps leverage Reminders as a shared task database. I’ve also found it to be more than sufficient for the residual tasks that I need to track that aren’t in MindNode or Agenda.

I implemented this new approach a month ago and have stuck with it. This is at least three weeks longer than any previous attempt to move away from Things. So, the experiment has been a success. If my circumstances change, I’ll happily return to Things. For now, this new approach has worked out very well.


RStats on iPad

Among the many good new features in iPadOS, “Desktop Safari” has proven to be surprisingly helpful for my analytical workflows.

RStudio Cloud is a great service that provides a feature-complete version of RStudio in a web browser. In previous versions of Safari on iPad, RStudio Cloud was close to unusable, since the keyboard shortcuts didn’t work and they’re essential for using RStudio. In iPadOS, all of the shortcuts work as expected and RStudio Cloud is completely functional.

Although most of my analytical work will still be on my desktop, having RStudio on my iPad adds a very convenient option. RStudio Cloud also allows you to setup a project with an environment that persists across any device. So, now I can do most of my work at home, then fix a few issues at work, and refine at a coffee shop. Three different devices all using the exact same RStudio project.

A screenshot of RStudio Cloud on the iPad

One complexity with an RStudio Cloud setup is GitHub access. The usual approach of putting your git credentials in an .REnviron file (or equivalent) is a bad idea on a web service like RStudio Cloud. So, you need to type your git credentials into the console. To avoid having to do this very frequently, follow this advice and type this into the console:

git config --global credential.helper 'cache --timeout 3600'

My iPhone Home Screen

My goal for the home screen is to stay focused on action by making it easy to quickly capture my intentions and to minimize distractions. With previous setups I often found that I’d unlock the phone, be confronted by a screen full of apps with notification badges, and promptly forget what I had intended to do. So, I’ve reduced my home screen to just two apps.

iPhone home screen

Drafts is on the right and is likely my most frequently used app. As the tag line for the app says, this is where text starts. Rather than searching for a specific app, launching it, and then typing, Drafts always opens up to a blank text field. Then I type whatever is on my mind and send it from Drafts to the appropriate app. So, text messages, emails, todos, meeting notes, and random ideas all start in Drafts. Unfortunately my corporate iPhone blocks iCloud Drive, so I can’t use Drafts to share notes across my devices. Anything that I want to keep gets moved into Apple Notes.

Things is on the left and is currently my favoured todo app. All of my tasks, projects, and areas of focus are in there, tagged by context, and given due dates, if appropriate. If the Things app has a notification badge, then I’ve got work to do today. If you’re keen, The Sweet Setup has a great course on Things.

A few more notes on my setup:

  • If Drafts isn’t the right place to start, I just pull down from the home screen to activate search and find the right app. I’ve found that the Siri Suggestions are often what I’m looking for (based on time of day and other context).
  • Some apps are more important for their output than input. These include calendar, weather, and notes. I’ve set these up as widgets in the Today View. A quick slide to the right reveals these.
  • I interact with several other apps through notifications, particularly for communication with Messages and Mail. But, I’ve set up VIPs in Mail to reduce these notifications to just the really important people.

I’ve been using this setup for a few months now and it certainly works for me. Even if this isn’t quite right for you, I’d encourage you to take a few minutes to really think through how you interact with your phone. I see far too many people with the default settings spending too much time scrolling around on their phones looking for the right app.


Fixing a hack finds a better solution

In my Elections Ontario official results post, I had to use an ugly hack to match Electoral District names and numbers by extracting data from a drop down list on the Find My Electoral District website. Although it was mildly clever, like any hack, I shouldn’t have relied on this one for long, as proven by Elections Ontario shutting down the website.

So, a more robust solution was required, which led to using one of Election Ontario’s shapefiles. The shapefile contains the data we need, it’s just in a tricky format to deal with. But, the sf package makes this mostly straightforward.

We start by downloading and importing the Elections Ontario shape file. Then, since we’re only interested in the City of Toronto boundaries, we download the city’s shapefile too and intersect it with the provincial one to get a subset:

download.file("[www.elections.on.ca/content/d...](https://www.elections.on.ca/content/dam/NGW/sitecontent/2016/preo/shapefiles/Polling%20Division%20Shapefile%20-%202014%20General%20Election.zip)", 
              destfile = "data-raw/Polling%20Division%20Shapefile%20-%202014%20General%20Election.zip")
unzip("data-raw/Polling%20Division%20Shapefile%20-%202014%20General%20Election.zip", 
      exdir = "data-raw/Polling%20Division%20Shapefile%20-%202014%20General%20Election")

prov_geo <- sf::st_read(“data-raw/Polling%20Division%20Shapefile%20-%202014%20General%20Election”, layer = “PDs_Ontario”) %>% sf::st_transform(crs = “+init=epsg:4326”)

download.file("opendata.toronto.ca/gcc/votin…", destfile = “data-raw/voting_location_2014_wgs84.zip”) unzip(“data-raw/voting_location_2014_wgs84.zip”, exdir=“data-raw/voting_location_2014_wgs84”) toronto_wards <- sf::st_read(“data-raw/voting_location_2014_wgs84”, layer = “VOTING_LOCATION_2014_WGS84”) %>% sf::st_transform(crs = “+init=epsg:4326”)

to_prov_geo <- prov_geo %>% sf::st_intersection(toronto_wards)

Now we just need to extract a couple of columns from the data frame associated with the shapefile. Then we process the values a bit so that they match the format of other data sets. This includes converting them to UTF-8, formatting as title case, and replacing dashes with spaces:

electoral_districts <- to_prov_geo %>%
  dplyr::transmute(electoral_district = as.character(DATA_COMPI),
                   electoral_district_name = stringr::str_to_title(KPI04)) %>%
  dplyr::group_by(electoral_district, electoral_district_name) %>%
  dplyr::count() %>%
  dplyr::ungroup() %>%
  dplyr::mutate(electoral_district_name = stringr::str_replace_all(utf8::as_utf8(electoral_district_name), "\u0097", " ")) %>%
  dplyr::select(electoral_district, electoral_district_name)
electoral_districts
## Simple feature collection with 23 features and 2 fields
## geometry type:  MULTIPOINT
## dimension:      XY
## bbox:           xmin: -79.61919 ymin: 43.59068 xmax: -79.12511 ymax: 43.83057
## epsg (SRID):    4326
## proj4string:    +proj=longlat +datum=WGS84 +no_defs
## # A tibble: 23 x 3
##    electoral_distri… electoral_distric…                           geometry
##                                                 
##  1 005               Beaches East York  (-79.32736 43.69452, -79.32495 43…
##  2 015               Davenport          (-79.4605 43.68283, -79.46003 43.…
##  3 016               Don Valley East    (-79.35985 43.78844, -79.3595 43.…
##  4 017               Don Valley West    (-79.40592 43.75026, -79.40524 43…
##  5 020               Eglinton Lawrence  (-79.46787 43.70595, -79.46376 43…
##  6 023               Etobicoke Centre   (-79.58697 43.6442, -79.58561 43.…
##  7 024               Etobicoke Lakesho… (-79.56213 43.61001, -79.5594 43.…
##  8 025               Etobicoke North    (-79.61919 43.72889, -79.61739 43…
##  9 068               Parkdale High Park (-79.49944 43.66285, -79.4988 43.…
## 10 072               Pickering Scarbor… (-79.18898 43.80374, -79.17927 43…
## # ... with 13 more rows

In the end, this is a much more reliable solution, though it seems a bit extreme to use GIS techniques just to get a listing of Electoral District names and numbers.

The commit with most of these changes in toVotes is here.


Elections Ontario official results

In preparing for some PsephoAnalytics work on the upcoming provincial election, I’ve been wrangling the Elections Ontario data. As provided, the data is really difficult to work with and we’ll walk through some steps to tidy these data for later analysis.

Here’s what the source data looks like:

Screenshot of raw Elections Ontario data

Screenshot of raw Elections Ontario data

A few problems with this:

  1. The data is scattered across a hundred different Excel files
  2. Candidates are in columns with their last name as the header
  3. Last names are not unique across all Electoral Districts, so can’t be used as a unique identifier
  4. Electoral District names are in a row, followed by a separate row for each poll within the district
  5. The party affiliation for each candidate isn’t included in the data

So, we have a fair bit of work to do to get to something more useful. Ideally something like:

## # A tibble: 9 x 5
##   electoral_district  poll candidate   party votes
##                <chr> <chr>     <chr>   <chr> <int>
## 1                  X     1         A Liberal    37
## 2                  X     2         B     NDP    45
## 3                  X     3         C      PC    33
## 4                  Y     1         A Liberal    71
## 5                  Y     2         B     NDP    37
## 6                  Y     3         C      PC    69
## 7                  Z     1         A Liberal    28
## 8                  Z     2         B     NDP    15
## 9                  Z     3         C      PC    34

This is much easier to work with: we have one row for the votes received by each candidate at each poll, along with the Electoral District name and their party affiliation.

Candidate parties

As a first step, we need the party affiliation for each candidate. I didn’t see this information on the Elections Ontario site. So, we’ll pull the data from Wikipedia. The data on this webpage isn’t too bad. We can just use the table xpath selector to pull out the tables and then drop the ones we aren’t interested in.

```
candidate_webpage <- "https://en.wikipedia.org/wiki/Ontario_general_election,_2014#Candidates_by_region"
candidate_tables <- "table" # Use an xpath selector to get the drop down list by ID

candidates <- xml2::read_html(candidate_webpage) %>% rvest::html_nodes(candidate_tables) %>% # Pull tables from the wikipedia entry .[13:25] %>% # Drop unecessary tables rvest::html_table(fill = TRUE)

</pre>
<p>This gives us a list of 13 data frames, one for each table on the webpage. Now we cycle through each of these and stack them into one data frame. Unfortunately, the tables aren’t consistent in the number of columns. So, the approach is a bit messy and we process each one in a loop.</p>
<pre class="r"><code># Setup empty dataframe to store results
candidate_parties <- tibble::as_tibble(
electoral_district_name = NULL,
party = NULL,
candidate = NULL
)

for(i in seq_along(1:length(candidates))) { # Messy, but works
this_table <- candidates[[i]]
# The header spans mess up the header row, so renaming
names(this_table) <- c(this_table[1,-c(3,4)], "NA", "Incumbent")
# Get rid of the blank spacer columns
this_table <- this_table[-1, ]
# Drop the NA columns by keeping only odd columns
this_table <- this_table[,seq(from = 1, to = dim(this_table)[2], by = 2)]
this_table %<>%
  tidyr::gather(party, candidate, -`Electoral District`) %>%
  dplyr::rename(electoral_district_name = `Electoral District`) %>%
  dplyr::filter(party != "Incumbent")
candidate_parties <- dplyr::bind_rows(candidate_parties, this_table)
}
candidate_parties</code></pre>
<pre>

# A tibble: 649 x 3

electoral_district_name party candidate

1 Carleton—Mississippi Mills Liberal Rosalyn Stevens

2 Nepean—Carleton Liberal Jack Uppal

3 Ottawa Centre Liberal Yasir Naqvi

4 Ottawa—Orléans Liberal Marie-France Lalonde

5 Ottawa South Liberal John Fraser

6 Ottawa—Vanier Liberal Madeleine Meilleur

7 Ottawa West—Nepean Liberal Bob Chiarelli

8 Carleton—Mississippi Mills PC Jack MacLaren

9 Nepean—Carleton PC Lisa MacLeod

10 Ottawa Centre PC Rob Dekker

# … with 639 more rows

</pre>
</div>
<div id="electoral-district-names" class="section level2">
<h2>Electoral district names</h2>
<p>One issue with pulling party affiliations from Wikipedia is that candidates are organized by Electoral District <em>names</em>. But the voting results are organized by Electoral District <em>number</em>. I couldn’t find an appropriate resource on the Elections Ontario site. Rather, here we pull the names and numbers of the Electoral Districts from the <a href="https://www3.elections.on.ca/internetapp/FYED_Error.aspx?lang=en-ca">Find My Electoral District</a> website. The xpath selector is a bit tricky for this one. The <code>ed_xpath</code> object below actually pulls content from the drop down list that appears when you choose an Electoral District. One nuisance with these data is that Elections Ontario uses <code>--</code> in the Electoral District names, instead of the — used on Wikipedia. We use <code>str_replace_all</code> to fix this below.</p>
<pre class="r"><code>ed_webpage <- "https://www3.elections.on.ca/internetapp/FYED_Error.aspx?lang=en-ca"
ed_xpath <- "//*[(@id = \"ddlElectoralDistricts\")]" # Use an xpath selector to get the drop down list by ID

electoral_districts <- xml2::read_html(ed_webpage) %>%
  rvest::html_node(xpath = ed_xpath) %>%
  rvest::html_nodes("option") %>%
  rvest::html_text() %>%
  .[-1] %>% # Drop the first item on the list ("Select...")
  tibble::as.tibble() %>% # Convert to a data frame and split into ID number and name
  tidyr::separate(value, c("electoral_district", "electoral_district_name"),
                  sep = " ",
                  extra = "merge") %>%
  # Clean up district names for later matching and presentation
  dplyr::mutate(electoral_district_name = stringr::str_to_title(
    stringr::str_replace_all(electoral_district_name, "--", "—")))
electoral_districts</code></pre>

<pre>

# A tibble: 107 x 2

electoral_district electoral_district_name

1 001 Ajax—Pickering

2 002 Algoma—Manitoulin

3 003 Ancaster—Dundas—Flamborough—Westdale

4 004 Barrie

5 005 Beaches—East York

6 006 Bramalea—Gore—Malton

7 007 Brampton—Springdale

8 008 Brampton West

9 009 Brant

10 010 Bruce—Grey—Owen Sound

# … with 97 more rows

</pre>

<p>Next, we can join the party affiliations to the Electoral District names to join candidates to parties and district numbers.</p>
<pre class="r"><code>candidate_parties %<>%
  # These three lines are cleaning up hyphens and dashes, seems overly complicated
  dplyr::mutate(electoral_district_name = stringr::str_replace_all(electoral_district_name, "—\n", "—")) %>%
  dplyr::mutate(electoral_district_name = stringr::str_replace_all(electoral_district_name,
                                                                   "Chatham-Kent—Essex",
                                                                   "Chatham—Kent—Essex")) %>%
  dplyr::mutate(electoral_district_name = stringr::str_to_title(electoral_district_name)) %>%
  dplyr::left_join(electoral_districts) %>%
  dplyr::filter(!candidate == "") %>%
  # Since the vote data are identified by last names, we split candidate's names into first and last
  tidyr::separate(candidate, into = c("first","candidate"), extra = "merge", remove = TRUE) %>% 
  dplyr::select(-first)</code></pre>
<pre><code>## Joining, by = "electoral_district_name"</code></pre>
<pre class="r"><code>candidate_parties</code></pre>
<pre>

# A tibble: 578 x 4

electoral_district_name party candidate electoral_district

*

1 Carleton—Mississippi Mills Liberal Stevens 013

2 Nepean—Carleton Liberal Uppal 052

3 Ottawa Centre Liberal Naqvi 062

4 Ottawa—Orléans Liberal France Lalonde 063

5 Ottawa South Liberal Fraser 064

6 Ottawa—Vanier Liberal Meilleur 065

7 Ottawa West—Nepean Liberal Chiarelli 066

8 Carleton—Mississippi Mills PC MacLaren 013

9 Nepean—Carleton PC MacLeod 052

10 Ottawa Centre PC Dekker 062

# … with 568 more rows

</pre>
<p>All that work just to get the name of each candiate for each Electoral District name and number, plus their party affiliation.</p>
</div>
<div id="votes" class="section level2">
<h2>Votes</h2>
<p>Now we can finally get to the actual voting data. These are made available as a collection of Excel files in a compressed folder. To avoid downloading it more than once, we wrap the call in an <code>if</code> statement that first checks to see if we already have the file. We also rename the file to something more manageable.</p>
<pre class="r"><code>raw_results_file <- "[www.elections.on.ca/content/d...](http://www.elections.on.ca/content/dam/NGW/sitecontent/2017/results/Poll%20by%20Poll%20Results%20-%20Excel.zip)"

zip_file <- "data-raw/Poll%20by%20Poll%20Results%20-%20Excel.zip"
if(file.exists(zip_file)) { # Only download the data once
  # File exists, so nothing to do
}  else {
  download.file(raw_results_file,
                destfile = zip_file)
  unzip(zip_file, exdir="data-raw") # Extract the data into data-raw
  file.rename("data-raw/GE Results - 2014 (unconverted)", "data-raw/pollresults")
}</code></pre>
<pre><code>## NULL</code></pre>
<p>Now we need to extract the votes out of 107 Excel files. The combination of <code>purrr</code> and <code>readxl</code> packages is great for this. In case we want to filter to just a few of the files (perhaps to target a range of Electoral Districts), we declare a <code>file_pattern</code>. For now, we just set it to any xls file that ends with three digits preceeded by a “_“.</p>
<p>As we read in the Excel files, we clean up lots of blank columns and headers. Then we convert to a long table and drop total and blank rows. Also, rather than try to align the Electoral District name rows with their polls, we use the name of the Excel file to pull out the Electoral District number. Then we join with the <code>electoral_districts</code> table to pull in the Electoral District names.</p>
<pre class="r">

file_pattern <- “*_[[:digit:]]{3}.xls” # Can use this to filter down to specific files poll_data <- list.files(path = “data-raw/pollresults”, pattern = file_pattern, full.names = TRUE) %>% # Find all files that match the pattern purrr::set_names() %>% purrr::map_df(readxl::read_excel, sheet = 1, col_types = “text”, .id = “file”) %>% # Import each file and merge into a dataframe

Specifying sheet = 1 just to be clear we’re ignoring the rest of the sheets

Declare col_types since there are duplicate surnames and map_df can’t recast column types in the rbind

For example, Bell is in both district 014 and 063

dplyr::select(-starts_with(“X__")) %>% # Drop all of the blank columns dplyr::select(1:2,4:8,15:dim(.)[2]) %>% # Reorganize a bit and drop unneeded columns dplyr::rename(poll_number = POLL NO.) %>% tidyr::gather(candidate, votes, -file, -poll_number) %>% # Convert to a long table dplyr::filter(!is.na(votes), poll_number != “Totals”) %>% dplyr::mutate(electoral_district = stringr::str_extract(file, “[[:digit:]]{3}"), votes = as.numeric(votes)) %>% dplyr::select(-file) %>% dplyr::left_join(electoral_districts) poll_data

</pre>
<pre>

# A tibble: 143,455 x 5

poll_number candidate votes electoral_district electoral_district_name

1 001 DICKSON 73 001 Ajax—Pickering

2 002 DICKSON 144 001 Ajax—Pickering

3 003 DICKSON 68 001 Ajax—Pickering

4 006 DICKSON 120 001 Ajax—Pickering

5 007 DICKSON 74 001 Ajax—Pickering

6 008A DICKSON 65 001 Ajax—Pickering

7 008B DICKSON 81 001 Ajax—Pickering

8 009 DICKSON 112 001 Ajax—Pickering

9 010 DICKSON 115 001 Ajax—Pickering

10 011 DICKSON 74 001 Ajax—Pickering

# … with 143,445 more rows

</pre>
<p>The only thing left to do is to join <code>poll_data</code> with <code>candidate_parties</code> to add party affiliation to each candidate. Because the names don’t always exactly match between these two tables, we use the <code>fuzzyjoin</code> package to join by closest spelling.</p>
<pre class="r"><code>poll_data_party_match_table <- poll_data %>%
  group_by(candidate, electoral_district_name) %>%
  summarise() %>%
  fuzzyjoin::stringdist_left_join(candidate_parties,
                                  ignore_case = TRUE) %>%
  dplyr::select(candidate = candidate.x,
                party = party,
                electoral_district = electoral_district) %>%
  dplyr::filter(!is.na(party))
poll_data %<>%
  dplyr::left_join(poll_data_party_match_table) %>% 
  dplyr::group_by(electoral_district, party)
tibble::glimpse(poll_data)</code></pre>
<pre>

Observations: 144,323

Variables: 6

$ poll_number “001”, “002”, “003”, “006”, “007”, “00…

$ candidate “DICKSON”, “DICKSON”, “DICKSON”, “DICK…

$ votes 73, 144, 68, 120, 74, 65, 81, 112, 115…

$ electoral_district “001”, “001”, “001”, “001”, “001”, “00…

$ electoral_district_name “Ajax—Pickering”, “Ajax—Pickering”, “A…

$ party “Liberal”, “Liberal”, “Liberal”, “Libe…

</pre>
<p>And, there we go. One table with a row for the votes received by each candidate at each poll. It would have been great if Elections Ontario released data in this format and we could have avoided all of this work.</p>
</div>

Charity donations by province

This tweet about the charitable donations by Albertans showed up in my timeline and caused a ruckus.

Many people took issue with the fact that these values weren’t adjusted for income. Seems to me that whether this is a good idea or not depends on what kind of question you’re trying to answer. Regardless, the CANSIM table includes this value. So, it is straightforward to calculate. Plus CANSIM tables have a pretty standard structure and showing how to manipulate this one serves as a good template for others.

library(tidyverse)
# Download and extract
url <- "[www20.statcan.gc.ca/tables-ta...](http://www20.statcan.gc.ca/tables-tableaux/cansim/csv/01110001-eng.zip)"
zip_file <- "01110001-eng.zip"
download.file(url,
              destfile = zip_file)
unzip(zip_file) 
# We only want two of the columns. Specifying them here.
keep_data <- c("Median donations (dollars)",
               "Median total income of donors (dollars)")
cansim <- read_csv("01110001-eng.csv") %>% 
  filter(DON %in% keep_data,
         is.na(`Geographical classification`)) %>% # This second filter removes anything that isn't a province or territory
  select(Ref_Date, DON, Value, GEO) %>%
  spread(DON, Value) %>% 
  rename(year = Ref_Date,
         donation = `Median donations (dollars)`,
         income = `Median total income of donors (dollars)`) %>% 
  mutate(donation_per_income = donation / income) %>% 
  filter(year == 2015) %>% 
  select(GEO, donation, donation_per_income)
cansim
## # A tibble: 16 x 3
##                                  GEO donation donation_per_income
##                                                   
##  1                           Alberta      450         0.006378455
##  2                  British Columbia      430         0.007412515
##  3                            Canada      300         0.005119454
##  4                          Manitoba      420         0.008032129
##  5                     New Brunswick      310         0.006187625
##  6         Newfoundland and Labrador      360         0.007001167
##  7 Non CMA-CA, Northwest Territories      480         0.004768528
##  8                 Non CMA-CA, Yukon      310         0.004643499
##  9             Northwest Territories      400         0.003940887
## 10                       Nova Scotia      340         0.006505932
## 11                           Nunavut      570         0.005651398
## 12                           Ontario      360         0.005856515
## 13              Prince Edward Island      400         0.008221994
## 14                            Quebec      130         0.002452830
## 15                      Saskatchewan      410         0.006910501
## 16                             Yukon      420         0.005695688

Curious that they dropped the territories from their chart, given that Nunavut has such a high donation amount.

Now we can plot the normalized data to find how the rank order changes. We’ll add the Canadian average as a blue line for comparison.

I’m not comfortable with using median donations (adjusted for income or not) to say anything in particular about the residents of a province. But, I’m always happy to look more closely at data and provide some context for public debates.

One major gap with this type of analysis is that we’re only looking at the median donations of people that donated anything at all. In other words, we aren’t considering anyone who donates nothing. We should really compare these median donations to the total population or the size of the economy. This Stats Can study is a much more thorough look at the issue.

For me the interesting result here is the dramatic difference between Quebec and the rest of the provinces. But, I don’t interpret this to mean that Quebecers are less generous than the rest of Canada. Seems more likely that there are material differences in how the Quebec economy and social safety nets are structured.


Canada LEED projects

The CaGBC maintains a list of all the registered LEED projects in Canada. This is a great resource, but rather awkward for analyses. I’ve copied these data into a DabbleDB application with some of the maps and tabulations that I frequently need to reference.

Here for example is a map of the density of LEED projects in each province. While here is a rather detailed view of the kinds of projects across provinces. There are several other views available. Are there any others that might be useful?