Tremor analysis multi FFT.ipynb
Analyze a set of Gforce datasets (e.g. tremor tracking data) with fourier transform to get the peak frequency and associated magnitude. Output a table for frequency & magnitude information at each timepoint.
You can use this notebook to analyze tremor frequency and strength using accelerometer data for a set of files.
See instructions below on how to collect this data & run this code for yourself.
Note: you will an Open Humans account to use this. Open Humans is a nonprofit platform enabling individuals to manage personal data and privately conduct personal analyses. Click here to sign up.
This notebook expects to analyze a data file from the "Physics Toolbox Sensor Suite" phone app.
Upload this data type here: https://upload.openhumans.org/upload?datatype_id=23
Note: this will prompt a log in Open Humans.
If you're viewing this notebook in the exploratory, click the button "Open in Personal Data Notebooks" to analyze your data privately on a personal virtual machine provided by Open Humans.
Note: this will prompt a log in Open Humans.
There are several ways to do run this notebook code, including...
NOTE: at one point this notebook will ask for user input, to select a file for analysis.
A code block (or "cell") will display [*] while running, and a number (e.g. [3]) once it's done.
It's possible to load multiple accelerometer data files into your account.
USER INPUT: Which data set do you want to analyze? Hit ENTER after selecting.
The code below prompts you to select which data to analyze.
Your data files are listed according to the date they were recorded. Select the number identifying the file you want to analyze. If you leave this blank, it will analyze the most recent recording.
Before doing a fourier transform, data should be a uniform sampling – there should be an equal amount of time between each data timepoint. This isn't always true in the raw data. Use an interpolation to get uniform data.
You can use this notebook to analyze tremor frequency and strength using accelerometer data for a set of files.
See instructions below on how to collect this data & run this code for yourself.
Note: you will an Open Humans account to use this. Open Humans is a nonprofit platform enabling individuals to manage personal data and privately conduct personal analyses. Click here to sign up.
This notebook expects to analyze a data file from the "Physics Toolbox Sensor Suite" phone app.
Upload this data type here: https://upload.openhumans.org/upload?datatype_id=23
Note: this will prompt a log in Open Humans.
If you're viewing this notebook in the exploratory, click the button "Open in Personal Data Notebooks" to analyze your data privately on a personal virtual machine provided by Open Humans.
Note: this will prompt a log in Open Humans.
There are several ways to do run this notebook code, including...
NOTE: at one point this notebook will ask for user input, to select a file for analysis.
A code block (or "cell") will display [*] while running, and a number (e.g. [3]) once it's done.
# This code sets up a connection to your data files in Open Humans.
import os
import ohapi
token = os.environ.get('OH_ACCESS_TOKEN')
user = ohapi.api.exchange_oauth2_member(token)
# Load raw data and select the one you want to analyze.
import csv, io, re
import arrow, requests
loaded_data = []
# Fix malformed datetimes – the toolkit fails to zero-pad minutes!
def fix_datetime(datetime_string):
regex = re.compile(r'([0-9]{4}-[0-9]{2}-[0-9]{2} )([0-9]{1,2})\:([0-9]{1,2})')
matched = regex.search(datetime_string)
if matched:
return regex.sub(matched.groups()[0] + matched.groups()[1].zfill(2) +
':' + matched.groups()[2].zfill(2), datetime_string)
return datetime_string
for x in user['data']:
if 'https://www.openhumans.org/api/public/datatype/23/' in x['datatypes']:
try:
raw_data = requests.get(x['download_url']).content
datafile = io.StringIO(raw_data.decode('utf-8'))
csv_in = csv.reader(datafile)
csv_in.__next__()
datetime = arrow.get(fix_datetime(csv_in.__next__()[0])).datetime
stored = x.copy()
stored.update({'raw_data': raw_data, 'datetime': datetime})
loaded_data.append(stored)
except Exception:
continue
if not loaded_data:
raise Exception('No valid data found! Check "Getting Data" instructions above.\n'
'Files should be visible as "G-force accelerometer data" on the '
'dashboard at: https://upload.openhumans.org/')
else:
loaded_data.sort(key=lambda ld: ld['datetime'])
It's possible to load multiple accelerometer data files into your account.
USER INPUT: Which data set do you want to analyze? Hit ENTER after selecting.
The code below prompts you to select which data to analyze.
Your data files are listed according to the date they were recorded. Select the number identifying the file you want to analyze. If you leave this blank, it will analyze the most recent recording.
# Choose a data set?
print("Pick which dataset to analyze.")
for i in range(0, len(loaded_data)):
print("{}. {}".format(i+1, loaded_data[i]['datetime']))
dataset_id = input("Number of dataset (hit ENTER, leave blank for most recent): ")
dataset = loaded_data[-1]
if dataset_id:
try:
dataset = loaded_data[int(dataset_id) - 1]
except Exception:
print("\nPlease run this code block again and pick a valid number.")
raw_data_asfile = io.StringIO(dataset['raw_data'].decode('utf-8'))
plot_title = "FFT tremor analysis for {}".format(dataset['datetime'])
import math
import statistics
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.fftpack import fft
from scipy.interpolate import interp1d
import seaborn
file_data_set = [io.StringIO(ds['raw_data'].decode('utf-8')) for ds in loaded_data]
data_raw_set = [pd.read_csv(fd, parse_dates=['time']) for fd in file_data_set]
peak_freq_set = [None] * len(data_raw_set)
peak_mag_set = [None] * len(data_raw_set)
datetime_set = [None] * len(data_raw_set)
Before doing a fourier transform, data should be a uniform sampling – there should be an equal amount of time between each data timepoint. This isn't always true in the raw data. Use an interpolation to get uniform data.
for i in range(len(data_raw_set)):
data_raw = data_raw_set[i]
# Get the median interval time for this data (i.e. typical interval)
intervals = [data_raw.index[i] - data_raw.index[i-1] for
i in range(1, len(data_raw.index))]
interval = statistics.median(intervals)
# Convert to unix epoch timestamp for easier math.
ts_raw = [t.timestamp() for t in data_raw.time]
# Set up interpolation for total gForce.
gf_raw = list(data_raw.gFTotal)
interpolate = interp1d(ts_raw, gf_raw)
# Create uniform timepoints, derive interpolated gForce values.
ts_uniform = np.linspace(ts_raw[0], ts_raw[-1], len(ts_raw))
avg_delta = (ts_raw[-1] - ts_raw[0]) / len(ts_raw)
gf_uniform = interpolate(ts_uniform)
gf_fft_all = np.fft.fft(gf_uniform)
freqs_all = np.fft.fftfreq(len(ts_uniform), d=avg_delta)
# discard complex conjugate
target_len = int(len(freqs_all)/2)
freqs = freqs_all[1:target_len]
gf_fft = gf_fft_all[1:target_len]
peak_index = np.argmax(gf_fft)
peak_freq_set[i] = freqs[peak_index]
peak_mag_set[i] = abs(gf_fft[peak_index])
datetime_set[i] = data_raw['time'][0]
set_df = pd.DataFrame({'Peak magnitude': peak_mag_set, 'Peak frequency': peak_freq_set}, index=datetime_set)
print(set_df)