Wearable Polygraph Update

This post is an update on the Wearable Polygraph project that I undertook as part of the Wearable Technologies class at CU Boulder. Naturally this post will build on multiple previous posts in the wearables thread on this blog so if you are looking for some context be sure to check out Fantasy Wearables and Wearable Polygraph posts.

Below I will cover the system design, integration, and shortcomings of the device and the interface for this v0.0.1 release. I would like to note up front that the project is not entirely functional and I came up short on many goals, but the system design that I have developed up to this point is solidifying and the different choke-points have brought up interesting new questions that are prudent to document with the current context of this project.

Reading forward you can expect to find documentation on the current state of the device and the interface that I have developed as well as discussion on an interesting future wearables project using this project as motivation.

System Design

The construction of this system has a couple of pieces to it that I would like to highlight here to draw attention to some of the design choices that I made.

Collection & Actuation - The Wearable Polygraph consists of a few parts addressed here generically. The collection device is a wearable technology that makes a tangible measurement and shares that data in aggregate with the server. The server stores the data, and in future iterations will make more meaningful aggregations including things like activity classification. This can then be accessed through the web portal or API.

I have designed the system outlined above to underlay the Wearable Polygraph project to make it as generic and extensible as possible. This came naturally out of the variety of the data sources that I am working with, but it is important because I want to extend this project further in the future.

The current vision and reality of the function of this model works as follow. A user who voluntarily participates wears the measurement device (with a designed low profile) which collects information and shares aggregated data with the server.

A controller, possibly the user them-self, can interact with the website to start a control period which establishes a relative normal. The control period allows us to overcome changes in measurement based on the current context which is extrinsic to the experiment that the controller wishes to test. The controller can then terminate the control period on the website or through the API and initiate a test period. This transition ideally spans the change that is to be measured.

From here the controller can compare the sensor behaviors from the control Kolmogorov–Smirnov test) to empirically measure whether the control and test samples are drawn from the same distribution.
and test period to establish significance. This is currently done visually (when the Grafana integration on the website is working), however I hope to develop this to use statistics (i.e.

From here we can make inferences about the impression that the catalyst in the test either imposes or fails to impose on the person wearing the device.

Usage scenarios

Given the above description of the function of the Wearable Polygraph I imagine a couple usage scenarios to demonstrate my vision of the audience.

1. A public speaking class maintains multiple devices that students wear when practicing public speaking to measure the progress throughout the semester of their involuntary response to speaking publicly.

2. An individual wears this device when interacting at a social gathering, and surreptitiously measures themselves throughout the event creating experiments about their performance as they interact with others in their environment. This can allow them to quantify their involuntary responses to possibly uncomfortable scenarios.

3. A dating app wants to match individuals who have a significant interest in each other so they ask users to use the wearable polygraph device conducting experiments as users browse photos and read descriptions of others. The involuntary response of the application users can be folded into the inference about possible matches.

These scenarios demonstrate the breadth of applicability of this quantified-self project when extended. We generate all kinds of data relevant to social interaction and day to day life that can be captured by wearables in new ways.

Design Process

The materials that were used for the device component of this project are listed here and shown in the photo to the right.

Polar Heart Rate Arduino Sensor
Accelerometer
GSR sensor
Arduino Temperature sensor
Arduino MKR WiFi 1010
Battery
Conductive Thread
~8 inches of 1.5" elastic strap
3D printer & filament
Needle & Thread
Clear Nail Polish

Check out the code!

3d MKR WiFi 1010 enclosure
This project went rough a couple of stages to make it to the point where it currently sits. I will not try to document some of the more important decisions that were made along throughout that process.

1. Circuit design

The circuit in it's current form uses analog readings for all of it's sensor data. This project makes use of 5 out of 6 of the analog pins on the MKR WIFI 1010 board. Originally I had hoped to include blood pressure as a data point collected as well, however this would have significantly complicated the wearable device. Also the frequency at which this sensor would be read is significantly different than the others that I used in the final version. The Galvanic Skin Response probes and the temperature sensor must all be seated directly against the skin. All other components can be mounted elsewhere in the enclosure.

2. Device layout

Originally I had envisioned the supplementary sensors being sewed into the fabric strap that holds the Polar heart sensor across the chest. However, this provided a pretty limited interface for adding sensors, and sewing across the (intentionally) elastic strap would be difficult with the inelastic conductive thread.
As you can see in the image at the top of the post, the final version of the device was integrated with the arduino (and many of the sensors) integrated in a 3d printed enclosure seated over the hard plastic of the polar heart rate sensor.

3. Service operation

To optimize for personal privacy I had hoped to perform all of the aggregation and computation locally on the device, which would subsequently be interfaced with froma phone or local device directly. For this iteration of the project this was obviously infeasible early on as the MKR WiFi 1010 has very limited storage, computation power, and battery life. To overcome this I transitioned to a model requiring a remote aggregation server. I image that this could still be run with reasonable privacy if a user is willing to host this service for them-self. I intend to provide more thought to this in the future.

4. Current device

The final implementation of the v.0.0.1 Wearable Polygraph uses a MKR WiFi 1010 device that has had the headers de-soldered so that the connections can be sewn through. This Arduino sits close to the middle of an 8 inch section of 1.5" elastic strap. The sensors which do not rely on skin contact are mounted either directly below the Arduino on the strap or otherwise within the 3d printed enclosure. The connections as defined by the circuit diagram above are sewn in using conductive thread. The holes in the MKR WiFi 1010 board were pretty small so shared connections (like GND and VCC) often forked from common paths in the thread rather than all connecting directly to the board. While this
simplified the circuit it also raises the probability of a faulty connection.

Challenges

This project was investigatory in nature and along the way certain elements didn't work out as I had intended them to.

A small setback in the integration process was fractures in the 3d printed enclosure. Small pieces came off and I had to adapt to supplement the structure of the enclosure. This could be rectified if a smoother print setting was used. Overall the construction performed as intended for a testing implementation. You can see on the right side of the image here that a small piece of an eraser was sewn in place to support the 3d printed enclosure where a portion broke off.

Another challenge that I faced when implementing the service that went with teh Wearable Polygraph was slow development of API authentication. I originally hoped to implement full authentication for the API such that individual devices could manage their own data. This infrastructure developed slowly so in certain places I included the device_id and api_key as url arguments for ease of implementation.

A more significant shortcoming of the current version of this project is the challenge of sharing plots through a reverse proxy so that the Grafana interface is available on the front-end web-server. This is certainly possible but the full visualization took too much time to implement properly.

The final significant challenge that I faced when implementing this project was the naturally low signal from the data sources collected in natural environments. I believe that these levels would increase and a more noticeable distinction would be visible in test / train for higher stress environments. Alternatively the data collected could be supplemented with other sensors that quantify involuntary response in more immediately measurable ways.

Discussion & Future Direction

From the challenges discussed above I have established numerous ideas for improvement, which I will try to quickly outline in this section.

First, noting that the signal for emotional response from sensors implemented here is faint to non-existent for natural interactions I think that the idea of naive polygraph for arbitrary speech isn't practical. However, there is still valuable information in the significant emotional responses that can be detected. If these reactions could be captured in real time through a normalized interface we could implement a real time User Experience (UX) feedback. For example, this could be used to modify the interface of an app or record the response of an individual to a directed advertising campaign measuring engagement.

But can this be done with current consumer grade wearable technologies? This is a significant question relevant to the theoretical implementation of Real Time UX (RTUX) feedback at scale. This question should be evaluated using 1) the device designed in this post and throughout this series, 2) commercially available devices such as fitbit, smartwatches, and other quantified-self devices, and finally 3) new sources such as attention monitors (eye tracking) and more advanced emotional detectors such as brain computer interfaces. This breakdown would give an answer to both the current applicability of a RTUX model as well as the extent of the accuracy this could reasonably adopt with non-standard wearable devices.

This integration of involuntary user response into application design (and possibly domains like advertising) is clearly a privacy concern, and as such this would be best implemented as locally as possible. At the very least this project will establish the reliability of such a response collection system before such a system exists -- allowing for privacy to be built in as a primary concern before monetization is considered. For reading on WHY this is important see this article, this book, and this report on journalist safety as starter material.

The next thing to expect from this effort is a technical write-up of the proposal for Real Time IoT UX Feedback for advertising and application design.

Search This Blog

rm -rf /