Details for quantified-flu-fitbit-intraday.ipynb

Published by Konstantin (This notebook is an edited copy of gedankenstuecke's notebook)

Description

A notebook that visualizes your resting heart rate before & after flu/cold events. For more experimental use as the data collection isn't as easy and the runtime of the notebook is underwhelming at this point.

0

Tags & Data Sources

quantifiedflu Fitbit Intraday Fitbit Connection

Other Notebook Versions

Comments

Please log in to comment.

Notebook
Last updated 3 years, 12 months ago

Heart rate variations when sick

A Fitbit allows you to measure vital signs during sleep such as resting heart rate. Here we will use this data to see if there are any patterns that emerge before falling sick. This is part of the #quantifiedflu project that came out of one of the Open Humans community calls!

This notebook is a bit messier to use, as you will need to use both the regular Fitbit connection on Open Humans as well as the Intraday Fitbit connection. We will need to use both as the intraday one lacks the sleep times, while the regular one does not provide high resolution heart rate data. Long story short, you'll need to set up both connections through the links above to make this work.

There are three parameters you can easily change for this notebook in the cell below:

  • The first one, sick_dates, is a list of dates on which you fell sick. Enter dates as YYYY-MM-DD format, e.g. 2019-11-29 for the 29th of November 2019. You can provide as many sick dates as you remember as a comma-separated ist.
  • The second one, weeks_before_sick, decides how many weeks before the actual falling sick you want to have shown in your graphs. By default it is 3.
  • The third one, weeks_after_sick, decides how many weeks after the event you want to see in your graph. By default it's 1.

The code below will now get your data from Open Humans and create a table with all the data points in the date ranges you specified above:

Getting the actual HR data

There's no way around it. This step feels like it takes forever because of how the data is structured. There's probably more clever ways to access the data, but for now it needs to check most of the records to find out where the actual data of our data ranges is by looping over it. Please be patient with this step

Visualizing heart rate changes around the sick dates:

Heart rate changes

Each sick-period will be it's own little sub-graph, with the date of falling sick being highlighted by a vertical, red bar. Each sick-period shows the resting heart rate changes, as well as a fitted line for the data.

/opt/conda/lib/python3.6/site-packages/rpy2/robjects/pandas2ri.py:57: UserWarning: Error while trying to convert the column "timestamp". Fall back to string conversion. The error is: Cannot pass numpy arrays with non-native byte orders at the moment.
  (name, str(e)))

Can you observe clear patterns? Is there something unexpected in your data? You can publish your own notebook after running it via the menu above: Click File -> Share Notebook -> Upload to Open Humans to share it with others.

Notebook
Last updated 3 years, 12 months ago

Heart rate variations when sick

A Fitbit allows you to measure vital signs during sleep such as resting heart rate. Here we will use this data to see if there are any patterns that emerge before falling sick. This is part of the #quantifiedflu project that came out of one of the Open Humans community calls!

This notebook is a bit messier to use, as you will need to use both the regular Fitbit connection on Open Humans as well as the Intraday Fitbit connection. We will need to use both as the intraday one lacks the sleep times, while the regular one does not provide high resolution heart rate data. Long story short, you'll need to set up both connections through the links above to make this work.

There are three parameters you can easily change for this notebook in the cell below:

  • The first one, sick_dates, is a list of dates on which you fell sick. Enter dates as YYYY-MM-DD format, e.g. 2019-11-29 for the 29th of November 2019. You can provide as many sick dates as you remember as a comma-separated ist.
  • The second one, weeks_before_sick, decides how many weeks before the actual falling sick you want to have shown in your graphs. By default it is 3.
  • The third one, weeks_after_sick, decides how many weeks after the event you want to see in your graph. By default it's 1.
In [8]:
import os
import requests
import tempfile
import json
import arrow
import json
import glob
import pandas as pd
In [3]:
sick_dates = ['2018-12-31']
weeks_before_sick = 2
weeks_after_sick = 1
In [4]:
fitbit_data = requests.get('http://195.201.146.207/STATIC/qf/fitbit-data.json').json()

The code below will now get your data from Open Humans and create a table with all the data points in the date ranges you specified above:

In [5]:
from ohapi import api
import os
import requests
import tempfile
import arrow

"""
user_details = api.exchange_oauth2_member(os.environ.get('OH_ACCESS_TOKEN'))
for i in user_details['data']:
    if i['basename'] == 'fitbit-data.json' and i['source'] == 'direct-sharing-102':
        fitbit_data = requests.get(i['download_url']).json()
"""
sd_dict = {}

for sd in sick_dates:
    sdd = arrow.get(sd)
    period_start = sdd.shift(weeks=weeks_before_sick*-1).format('YYYY-MM-DD')
    period_end = sdd.shift(weeks=weeks_after_sick).format('YYYY-MM-DD')
    sd_dict[sd] = {'period_start': period_start, 'period_end': period_end, 'sleep_start_dates': [], 'sleep_amount_minutes': []}


### YEP, THIS IS PROBABLY ALL AS COMPLICATED AS I COULD MAKE IT, BUT I FEEL A BIT BRAIN DEAD TODAY. 
### PLEASE CLEAN UP THE MESS BELOW IF YOU FEEL CAPABLE OF IT!

for p in sd_dict.keys():
    for year in fitbit_data['sleep-start-time']:
        for entry in fitbit_data['sleep-start-time'][year]['sleep-startTime']:
            sdate = arrow.get(entry['dateTime'])
            if (sdate >= arrow.get(sd_dict[p]['period_start'])) and (sdate <= arrow.get(sd_dict[p]['period_end'])):
                sd_dict[p]['sleep_start_dates'].append(entry)
                
for p in sd_dict.keys():
    for year in fitbit_data['sleep-minutes']:
        for entry in fitbit_data['sleep-minutes'][year]['sleep-minutesAsleep']:
            sdate = arrow.get(entry['dateTime'])
            if (sdate >= arrow.get(sd_dict[p]['period_start'])) and (sdate <= arrow.get(sd_dict[p]['period_end'])):
                sd_dict[p]['sleep_amount_minutes'].append(entry)
                
for p in sd_dict.keys():
    sd_dict[p]['entries'] = {}
    for entry in sd_dict[p]['sleep_start_dates']:
        sd_dict[p]['entries'][entry['dateTime']] = {'start': arrow.get(entry['dateTime'] + " " + entry['value'])}
    for entry in sd_dict[p]['sleep_amount_minutes']:
        sd_dict[p]['entries'][entry['dateTime']]['end'] = sd_dict[p]['entries'][entry['dateTime']]['start'].shift(minutes=int(entry['value']))
        
for k in sd_dict.keys():
    del sd_dict[k]['sleep_start_dates']
    del sd_dict[k]['sleep_amount_minutes']

Getting the actual HR data

There's no way around it. This step feels like it takes forever because of how the data is structured. There's probably more clever ways to access the data, but for now it needs to check most of the records to find out where the actual data of our data ranges is by looping over it. Please be patient with this step

In [6]:
files = ['fitbit-intraday-2018-12.json', 'fitbit-intraday-2019-01.json']
hr_data = []
for file in files:
    data = requests.get(f'http://195.201.146.207/STATIC/qf/{file}').json()
    hr_data += data['activities-heart-intraday']
In [12]:
import json
period = []
timestamp = []
heart_rate = []

for p in sd_dict.keys():
    """
    for i in user_details['data']:
        if i['source'] == 'direct-sharing-191' and i['basename'] == 'fitbit-intraday-{}.json'.format(sd_dict[p]['period_start'][:7]):
            data = json.loads(requests.get(i['download_url']).content)
            hr_data = data['activities-heart-intraday']
    if sd_dict[p]['period_start'][:7] != sd_dict[p]['period_end'][:7]:
        for i in user_details['data']:
            if i['source'] == 'direct-sharing-191' and i['basename'] == 'fitbit-intraday-{}.json'.format(sd_dict[p]['period_end'][:7]):
                data = json.loads(requests.get(i['download_url']).content)
                hr_data = hr_data + data['activities-heart-intraday']
    """
    for entry_date,entry in sd_dict[p]['entries'].items():
        for hr_point in hr_data:
            hr_date = hr_point['date']
            if (arrow.get(hr_date) >= arrow.get(sd_dict[p]['period_start'])) & (arrow.get(hr_date) <= arrow.get(sd_dict[p]['period_end'])):
                for hr_datapoint in hr_point['dataset']:
                    hr_timestamp = hr_date + ' ' + hr_datapoint['time']
                    hr_timestamp = arrow.get(hr_timestamp)
                    if (hr_timestamp >= entry['start']) and (hr_timestamp <= entry['end']):
                        period.append(p)
                        timestamp.append(hr_timestamp)
                        heart_rate.append(hr_datapoint['value'])


dataframe = pd.DataFrame(
    data = {
        'period': period,
        'timestamp': timestamp,
        'heart_rate': heart_rate,
    }
)
In [29]:
# Convert hr_data to pandas dataframe
rows = []
for hr_point in hr_data:
    hr_date = hr_point['date']
    for hr_datapoint in hr_point['dataset']:
        rows.append((hr_date + ' ' + hr_datapoint['time'], hr_datapoint['value']))
hr_dataframe = pd.DataFrame(rows, columns=['timestamp', 'heart_rate'])
hr_dataframe['timestamp'] = pd.to_datetime(hr_dataframe['timestamp'], utc=True)

# combine with sick days data
dataframes = []
for p in sd_dict.keys():
    for entry_date,entry in sd_dict[p]['entries'].items():
        #
        start = entry['start'].datetime
        end = entry['end'].datetime
        filtered = hr_dataframe[(hr_dataframe['timestamp'] >= start) & (hr_dataframe['timestamp'] <= end)].copy()
        filtered['period'] = p
        dataframes.append(filtered)

dataframe_new = pd.concat(dataframes).reset_index(drop=True)

Visualizing heart rate changes around the sick dates:

Heart rate changes

Each sick-period will be it's own little sub-graph, with the date of falling sick being highlighted by a vertical, red bar. Each sick-period shows the resting heart rate changes, as well as a fitted line for the data.

In [17]:
%load_ext rpy2.ipython
In [ ]:
%%R -i dataframe -w 15 -h 4 --units in -r 200
library(ggplot2)

ggplot(subset(dataframe,dataframe$heart_rate > 0), aes(x=as.Date(timestamp),y=heart_rate,color=period)) + 
    geom_vline(aes(xintercept=as.Date(period)),alpha=0.6,color='red') + 
    geom_point(alpha=0.1) + 
    geom_smooth(method='loess') + theme_minimal() + scale_x_date('timestamp') + 
    facet_grid(. ~ period, scales='free')
/opt/conda/lib/python3.6/site-packages/rpy2/robjects/pandas2ri.py:57: UserWarning: Error while trying to convert the column "timestamp". Fall back to string conversion. The error is: Cannot pass numpy arrays with non-native byte orders at the moment.
  (name, str(e)))

Can you observe clear patterns? Is there something unexpected in your data? You can publish your own notebook after running it via the menu above: Click File -> Share Notebook -> Upload to Open Humans to share it with others.