Connect with us

Internet of Things

Human Programming Interface | beepb00p

My life in a Python package TLDR: I’m using HPI (Human Programming Interface) package as a means of unifying, accessing and interacting with all of my personal data. If you’re in a hurry, feel free to jump straight to the demos. see SETUP for the installation/configuration guide see DEVELOPMENT for the development/extension guide HPI is…




Human Programming Interface | beepb00p

My lifestyles in a Python equipment

TLDR: I’m the usage of HPI (Human Programming Interface) equipment as a approach of unifying, gaining access to and interacting with all of my non-public info.

Must you are in a crawl, if truth be told feel free to leap straight to the demos.

  • sight SETUP for the installation/configuration e-book
  • sight DEVELOPMENT for the enhance/extension e-book

HPI is a Python equipment (named my), a series of modules for:

  • social networks: posts, comments, favorites
  • reading: e-books and pdfs
  • annotations: highlights and comments
  • todos and notes
  • well being info: sleep, sigh, weight, heart rate, and other physique metrics
  • plan
  • photos & movies
  • browser history
  • instantaneous messaging

The equipment hides the gory well-known components of finding info, parsing, error going by and caching.
You simply ‘import’ your info and salvage to work with familiar Python forms and info structures.

  • Right here is a brief example to give you a theory: “which subreddits I rating the most interesting?”

    import my.reddit
    from collections import Counter
    return Counter(s.subreddit for s in my.reddit.saved()).most_common(4)
    orgmode 62
    emacs 60
    selfhosted 51
    QuantifiedSelf 46

I again in mind my digital ticket crucial allotment of my identity.
The truth that the tips is siloed, and gaining access to it is inconvenient and borderline frustrating feels very sinful.

As soon as the tips is available as Python objects, I’m able to with out utter lag it into present instruments, libraries and frameworks.
It makes building current instruments considerably more uncomplicated and permits creating current ways of interacting with the tips.

I tried varied things over time and I reflect I’m attending to the level where folks can moreover befriend from my code by ‘correct’ plugging in their info,
and that is clarification why I’m sharing this.

Imagine if all of your lifestyles became mirrored digitally and available at your fingertips.
This library is my strive and influence this imaginative and prescient.

1 Why?

The main reason that led me to form right here’s the dissatisfaction of the present utter:

  • Our non-public info is siloed and trapped across cloud services and products and somewhat an excellent deal of devices

    Even when or no longer it is likely to salvage admission to it by approach to the API, or no longer it is hardly ever ever priceless, except you are an experienced programmer, fascinating to speculate your time and infrastructure.

  • We get insane amounts of info scattered across the cloud, but we’re left at the mercy of these who rating it to offer one thing priceless based mostly on it

    Integrations of info across silo boundaries are nearly non-existent. There is so critical capacity and or no longer it is all wasted.

  • I’m no longer fascinating to again till some vaporwave venture reinvents your entire computing mannequin from scratch

    As a programmer, I’m in capability to scheme one thing gorgeous now, even even supposing or no longer it is no longer essentially ideal and consistent.

I’ve written a lot about it right here, so enable me to easily quote:

  • search and info salvage admission to
    • Why can’t I search over all of my non-public chat history with a friend, whether or no longer it is ICQ logs from 2005 or Whatsapp logs from 2019?
    • Why can’t I if truth be told get incremental search over my tweets? Or browser bookmarks? Or over the entire lot I’ve ever typed/be taught on the Internet?
    • Why can’t I search across my watched youtube movies, even even supposing most of them get subtitles hence pondering elephantine text search?
    • Why can’t I sight the locations my chums if truth be told handy me on Google maps (or every other maps app)?
  • productiveness
    • Why can’t my Google Dwelling add looking out record items to Google Rating? Let by myself other todo-record apps.
    • Why can’t I create a role in my todo record or calendar from a conversation on Facebook Messenger/Whatsapp/
  • journaling and history
    • Why scheme I if truth be told have to lose all my browser history if I mediate to switch browsers?
    • Why can’t I sight your entire locations I traveled to on a single arrangement and photos alongside?
    • Why can’t I sight what my heart rate (i.e. pleasure) and tempo were facet by facet with the video I recorded on GoPro while skiing?
    • Why can’t I with out utter switch all my books and metadata if I mediate to switch from Kindle to PocketBook or vice versa?
  • drinking digital yell
    • Why can’t I sight stuff I highlighted on Instapaper as an overlay on high of web yell?
    • Why can’t I if truth be told get single ‘be taught it later’ record, unifying all things saved on Reddit/Hackernews/Pocket?
    • Why can’t I exploit my todo app in desire to ‘See later’ playlist on youtube?
    • Why can’t I ‘observe’ some individual on Hackernews?
    • Why can’t I sight if I’ve crawl across a Youtube video because my friend sent me a link months ago?
    • Why can’t I if truth be told get uniform song listening stats based mostly on my
    • Why am I compelled to use Spotify’s song advice algorithm and scheme no longer get an likelihood to strive one thing else?
    • Why can’t I with out utter sight what were the books/song/artwork if truth be told handy by my chums or some suppose Twitter/Reddit/Hackernews users?
    • Why my in every other case ideal hackernews app for Android doesn’t fragment saved posts/comments with the web yell?
  • well being and physique upkeep
    • Why can’t I reveal if I became extra sedentary than unheard of for the duration of the previous week and whether I have to compensate by doing pretty extra sigh?
    • Why can’t I sight what is the influence of cardio sigh on my resting HR?
    • Why can’t I if truth be told get a dashboard for all of my well being: meals, sigh and sleep to gawk baselines and traits?
    • Why can’t I sight the influence of temperature or CO2 focus in room on my sleep?
    • Why can’t I sight how holidays (as in, no longer going to work) influence my stress phases?
    • Why can’t I take my Headspace app info and sight how/if meditation impacts my sleep?
    • Why can’t I crawl a brief snippet of code and check some random well being advice on the Internet in opposition to my well being info.
  • non-public finance
    • Why am I compelled to manually copy transactions from varied banking apps into a spreadsheet?
    • Why can’t I with out utter match my Amazon/Ebay orders with my monetary institution transactions?
  • why I’m able to’t scheme the relaxation after I’m offline or get a wonky connection?
  • instruments for pondering and studying
    • Why when one thing admire ‘mind palace’ is actually likely with VR expertise, we scheme no longer sight any in use?
    • Why can’t I with out utter convert decide Instapaper highlights or current international words I encountered on my Kindle into Anki flashcards?
  • mediocre interfaces
    • Why scheme I if truth be told have to undergo from depressed management and invent choices in UI adjustments, although the interface is no longer the major reason I’m the usage of the product?
    • Why can’t I leave priorities and notes on my saved Reddit/Hackernews items?
    • Why can’t I leave non-public notes on Deliveroo restaurants/dishes, so I’d be conscious what to reveal/now to no longer reveal subsequent time?
    • Why scheme folks have to undergo from Google Inbox shutdown?
  • communique and collaboration
    • Why can’t I with out utter fragment my web or e book highlights with a friend? Or correct salvage highlights in decide books public?
    • Why can’t I with out utter rating out other individual’s trip with out interrogating them, correct by looking out what they be taught as a replace?
  • backups
    • Why scheme I if truth be told have to think it and actively invest time and energy?
  • I’m drained of having to use multiple varied messengers and social networks
  • I’m drained of shitty bloated interfaces

    Why will we have to be at mercy of their developers, designers and product managers? If we had our info at hand, we may perchance perchance per chance magnificent-tune interfaces for our wants.

  • I’m drained of mediocre search trip

    Textual yell search is one thing computers scheme exceptionally well.
    But, customarily or no longer it is no longer available offline, or no longer it is no longer incremental, all and sundry reinvents their very have faith ask language, etc.

  • I’m pissed off by depressed info exploring and processing trip

    While for many folks, services and products admire Reddit or Twitter are simply time killers (and I scheme no longer mediate), some want to use them successfully, as a source of info/research.
    Up-to-the-minute bookmarking trip makes it removed from ideal.

You may perchance perchance perchance per chance perchance brush aside this as a record of first-world issues, and also you may perchance perchance perchance per chance perchance be gorgeous, they’re.
However the major reason I want to resolve these issues is to be greater at studying and dealing with info,
so I’d be greater at solving the true issues.

2 How does a Python equipment abet?

After I started solving a majority of these issues for myself, I’ve noticed a general pattern: the hardest bit is definitely getting your info in the first set up.
Or no longer it is inherently error-prone and frustrating.

However whilst you may perchance perchance perchance well get the tips in a handy representation, working with it is relaxing – you salvage to stumble on and invent in desire to fighting with but every other slow REST API.

This equipment is conscious of rating info for your filesystem, deserialize it and normalize it to a handy representation.
You get the elephantine energy of the programming language to remodel the tips and scheme whatever involves your mind.

Why scheme no longer you correct set up the entire lot in a gigantic database?

Happy you may perchance perchance perchance well get asked! I wrote a entire submit about it.

Briefly: while databases are ambiance pleasant and straightforward to be taught from, customarily they scheme no longer seem like versatile ample to swimsuit your info.
You are doubtless going to extinguish up writing code anyway.

While working with your info, you are going to inevitably gawk general patterns and code repetition, which you are going to doubtless want to extract someplace.
That’s where a Python equipment is available in.

3 What’s interior?

Right here is the (incomplete) record of the modules:

Some modules are non-public, and wish pretty of cleanup earlier than merging:

my.workout routines Exercise project, from Endomondo and handbook logs
my.sleep.handbook Subjective sleep info, manually logged
my.vitamin Meals and drink consumption info, logged manually from varied sources Charges and looking out info
my.webhistory Looking out history (allotment of promnesia)

4 How scheme you sigh it?

Mainly I exploit it as an info provider for my scripts, instruments, and dashboards.

Additionally, strive my infrastructure arrangement.
Or no longer it is a draft in the intervening time, however it can perchance per chance very well be well-known for working out what’s my imaginative and prescient on HPI.


orger is a instrument that helps you generate an org-mode representation of your info.

It lets you befriend from the present tooling and infrastructure around org-mode, the most infamous being Emacs.

I’m the usage of it for:

  • looking out, overviewing and navigating the tips
  • creating duties straight from the apps (e.g. Reddit/Telegram)
  • spaced repetition by approach to org-drill

Orger comes with some present modules, however it wants to be easy to adapt your have faith info source if you may perchance perchance perchance per chance perchance admire one thing else.

I write about it intimately right here and right here.


promnesia is a browser extension I’m engaged on to flee silos by unifying annotations and looking out history from varied info sources.

I have been the usage of it for additional than a year now and engaged on remaining touches to well birth it for folks.


As a mountainous fan of #quantified-self, I’m engaged on non-public well being, sleep and sigh dashboard, constructed from somewhat an excellent deal of info sources.

I’m engaged on making it public, you are going to sight some screenshots right here.


Timeline is a #lifelogging venture I’m engaged on.

I want to gawk all my digital history, search in it, filter, with out utter jump at a particular level in time and sight the context when it took set up.
That method it if truth be told works as a form of exterior memory.

Ideally, it would gawk such as Andrew Louis’s Memex, or may perchance perchance well even reuse his interface if
he birth sources it. I highly counsel staring at his discuss for inspiration.

5 Ad-hoc and interactive

What were my song listening stats for 2018?

Single import a ways flung from getting tracks you listened to:

from my.lastfm import get_scrobbles
scrobbles = get_scrobbles()
scrobbles[200: 205]
[Scrobble(raw={'album': 'Nevermind', 'artist': 'Nirvana', 'date': '1282488504', 'name': 'Drain You'}),
 Scrobble(raw={'album': 'Dirt', 'artist': 'Alice in Chains', 'date': '1282489764', 'name': 'Would?'}),
 Scrobble(raw={'album': 'Bob Dylan: The Collection', 'artist': 'Bob Dylan', 'date': '1282493517', 'name': 'Like a Rolling Stone'}),
 Scrobble(raw={'album': 'Dark Passion Play', 'artist': 'Nightwish', 'date': '1282493819', 'name': 'Amaranth'}),
 Scrobble(raw={'album': 'Rolled Gold +', 'artist': 'The Rolling Stones', 'date': '1282494161', 'name': "You Can't Always Get What You Want"})]

Or, as a pandas physique to salvage it moderately:

import pandas as pd
df = pd.DataFrame([{
    'dt': s.dt,
    'track': s.track,
} for s in scrobbles])
cdf = df.set_index('dt')
cdf[200: 205]
2010-08-22 14: 48: 24+00: 00                                Nirvana — Drain You
2010-08-22 15: 09: 24+00: 00                           Alice in Chains — Would?
2010-08-22 16: 11: 57+00: 00                   Bob Dylan — Enjoy a Rolling Stone
2010-08-22 16: 16: 59+00: 00                               Nightwish — Amaranth
2010-08-22 16: 22: 41+00: 00  The Rolling Stones — You Can no longer Repeatedly Rating What...

We are able to use calmap library to set up a github-vogue song listening project heatmap:

import matplotlib.pyplot as plt
plt.decide(figsize=(10, 2.3))

import calmap
cdf = cdf.set_index(cdf.index.tz_localize(None)) 
calmap.yearplot(cdf['track'], how='depend', year=2018)

plt.title('My song listening project for 2018')
plot_file = ""

This is no longer essentially very insightful info, however relaxing to gawk at now after which!

What are the most interesting Slate Well-known individual Codex posts I’ve be taught?

My friend asked me if I may perchance perchance per chance counsel them posts I found interesting on Slate Well-known individual Codex.
With few lines of Python I’m able to like a flash counsel them posts I engaged most with, i.e. the ones I annotated most on Hypothesis.

from my.hypothesis import get_pages
from collections import Counter
cc = Counter({(p.title + ' ' + p.url): len(p.highlights) for p in get_pages() if 'slatestarcodex' in p.url})
return cc.most_common(10)

Gaining access to sigh info

E.g. sight use of my.workout routines right here.

E book reading development

I submit my reading stats on Goodreads so folks can sight what I’m reading/get be taught, however Kobo lacks integration with Goodreads.
I’m the usage of kobuddy to salvage admission to my my Kobo info, and I’ve bought a customary project that jogs my memory to sync my development once a month.

The duty seems to be to be like admire this:

With a single Enter keypress on the inlined eshell: reveal I’m able to print the development and like in the achieved books on Goodreads, e.g.:

A_Mathematician's_Apology by G. H. Hardy
Started : 21 Aug 2018 11: 44
Completed: 22 Aug 2018 12: 32

Dread and Loathing in Las Vegas: A Savage Gallop to the Coronary heart of the American Dream (Vintage) by Thompson, Hunter S.
Started : 06 Sep 2018 05: 54
Completed: 09 Sep 2018 12: 21

Sapiens: A Immediate Historical previous of Humankind by Yuval Noah Harari
Started : 09 Sep 2018 12: 22
Completed: 16 Sep 2018 07: 25

Insufficient Equilibria: Where and How Civilizations Rating Stuck by Eliezer Yudkowsky
Started : 31 Jul 2018 22: 54
Completed: 16 Sep 2018 07: 25

Albion Dreaming by Andy Roberts
Started : 20 Aug 2018 21: 16
Completed: 16 Sep 2018 07: 26

Messenger stats

How critical scheme I chat on Facebook Messenger?

from my.fbmessenger import messages

import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame({'dt': m.dt, 'messages': 1} for m in messages())
df.set_index('dt', inplace=Ravishing)

df = df.resample('M').sum() 
df = df.loc['2016-01-01':'2019-01-01'] 

fig, ax = plt.subplots(figsize=(15, 5))
df.set up(form='bar', ax=ax)

x_labels = df.index.strftime('%Y %b')

plot_file = ""
return plot_file

Querying Scuttle Reasearch database

I’ve bought some code examples right here.

6 How does it salvage input info?

Must you are fresh about any suppose info sources I’m the usage of, I’ve written it up intimately.


  • The information is periodically synchronized from the services and products (cloud or no longer) locally, on the filesystem

    This implies that, you salvage JSONs/sqlite (or other codecs, reckoning on the service) for your disk.

    Whenever you may perchance perchance perchance well get it, or no longer it is trivial to relief it up and synchronize to other computers/telephones, if necessary.

    To time desk periodic sync, I’m the usage of cron.

  • my. equipment supreme accesses the tips on the filesystem

    That makes it extraordinarily fleet, unswerving, and fully offline succesful.

As you are going to sight, in this kind of setup, the tips is lagging in the assist of the ‘realtime’.
I again in mind it a necessary sacrifice to salvage the entire lot fleet and resilient.

In belief, or no longer it is likely to salvage the machine nearly realtime by having a service that sucks in info constantly (pretty than periodically), however or no longer it is more durable apart from.

7 Q & A

Why Python?

I scheme no longer again in mind Python fresh as a language correct for this kind of venture.
It correct happens to be the one I’m most relaxed with.
I scheme get some causes that I reflect salvage it specifically correct, however explaining them is out of this submit’s scope.

As well to, Python offers a very well to attach ecosystem for info analysis, which we can use to our befriend.

That acknowledged, I’ve by no technique seen the relaxation the same in other programming languages, and I’d be if truth be told drawn to, so please ship me links if some.
I’ve heard LISPs are titanic for info? 😉

General, I wish FFIs were pretty extra passe, so we didn’t have to think suppose programming languages in any admire.

Can any individual use it?


  • you are going to lag in your have faith info
  • most modules are remoted, so you are going to supreme use the ones that you need to
  • the entire lot is with out utter extensible

    Initiating from simply adding current modules to any dynamic hackery you are going to perchance factor in interior Python.

How easy is it to use?

Your entire setup requires some general programmer literacy:

  • installing/working and doubtlessly modifying Python code
  • the usage of symlinks
  • doubtlessly working Cron jobs

Must you may perchance perchance perchance well get any tips on making the setup extra efficient, please let me know!

What about privacy?

The modules get no info, supreme code to feature on the tips.

Every little thing is local first, the input info is for your filesystem.
Must you are if truth be told paranoid, you are going to even wrap it in a Docker container.

There is soundless a ask of whether you have faith yourself at even conserving your entire info for your disk, however it is out of the scope of this submit.

Must you are going to pretty again some code non-public too, or no longer it is moreover trivial to influence with a interior most subpackage.

However have to soundless I exploit it?

Certain, perchance you are going to influence a ideal machine where you are going to straight rating and take the relaxation that you may perchance perchance perchance well get achieved. Attain you if truth be told desire it?
Wouldn’t that, admire, salvage you less human?

I’m no longer a gatekeeper of what it technique to be human, however I scheme no longer reflect that the shortcomings of the human brain are what makes us such.

So I’m able to’t answer that for you. I with out a doubt desire it even supposing.
I’m somewhat birth about my targets – I’d fortunately salvage merged/augmented with a laptop to toughen my pondering and analytical skills.

While in the intervening time we scheme no longer even remotely mark what would such merging or “mind importing” entail exactly,
I’m able to clearly delegate some duties, admire prolonged period of time memory, info look up, and info processing to a laptop. They’ll already take care of it if truth be told well.

What about these these who get ideal take and desire they hadn’t.

Certain, perchance it sucks. For the time being even supposing, my take is critical from ideal, and this supreme annoys me.
I want to get a replace no longer decrease than, and digital instruments give me this replace.

Would it swimsuit me?

Perhaps, no longer decrease than to a pair degree.

First, our lives are varied, so our APIs may perchance perchance per chance very well be varied too.
Right here’s extra of a demonstration of what’s I’m the usage of, even supposing I did exhaust effort in opposition to making it as modular and extensible as likely, so folks may perchance perchance per chance use it too.
Or no longer it is a ways simple to switch code, add extra strategies and modules. You may perchance perchance perchance per chance perchance even again all of your changes non-public.

However in spite of the entire lot, we now get all sharing many the same activities and the usage of the same merchandise, so there’s a mountainous overlap.
I’m no longer obvious how a ways we can stretch it and again modules generic ample to be worn by multiple folks. However let’s give it a strive perchance? 🙂

2d, interacting with your info by the code is the central belief of the venture.
That extra or less cuts off folks with out technical skills, and even many folks succesful of coding,
who hate the foundation of writing code outdoor of work.

It may perchance per chance very well be likely to reveal some no-code interfaces,
however I soundless if truth be told feel that wouldn’t be ample.

I’m no longer obvious whether or no longer it is a solvable utter at this level, however joyful to hear any strategies!

What it is no longer?

  • Or no longer it is no longer vaporwave

    The venture is a miniature coarse, however or no longer it is true and dealing. I have been the usage of it for a truly prolonged time now, and rating it moderately sustainable to again the usage of for the foreseeable future.

  • Or no longer it is no longer going to be every other silo

    While I scheme no longer get the relaxation in opposition to commercial use (and I factor in any work on this role will befriend all of us), I’m no longer planning to invent a product out of it.

    I if truth be told hope it would grow into or inspire some passe birth source machine.

    Please take my tips and code and invent one thing cool from it!


Birth to any strategies and tips!

Additionally, scheme no longer hesitate to select a utter, or attain me individually if you may perchance perchance perchance per chance perchance admire to strive the usage of it, and rating the instructions confusing. Your questions would abet me to salvage it extra efficient!

In some shut to future I will write extra about:

  • suppose technical choices and patterns
  • challenges I had so solve
  • extra use-cases and demos – or no longer it is very no longer going to suit the entire lot in one submit!

, however joyful to reply to any questions on these matters now!


Read More

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published.

seventeen + 2 =