Introduction
The Microsoft Hololens technology provides a myriad of possibilities, from its speech recognition to the gaze and gesture controls, all in an untethered environment. But there is one glaring omission, the lack of a GPS sensor onboard the device. This article provides a way to transmit a Bluetooth GPS signal from your smartphone (in this article, an Android OS) to the Hololens.Explaining the Concept
This technique leverages Bluetooth LE advertiser/watcher technologies available on the Android and UWP platforms. Similar advertiser code is available on iOS, which has subsequently coined the use of the term "iBeacons" which speaks to all Bluetooth advertiser type devices. Unfortunately, this technology is not available on all devices and should be checked for during runtime. The benefit of the advertiser pattern on the smartphone is that a socket server (RFCOMM) connection does not need to be maintained on either the Hololens or Android side, reducing code complexity and making your application more crash proof. Another major benefit of the Beacon approach is lower overall power consumption on both the advertising and watching devices. I felt the easiest way to explain my technique was to divide the implementation into four sections, the first two are generic to all Bluetooth Beacon applications, and the next two sections describe the GPS specific implementation.Implementation
BLE Advertiser on Android Java
Starting off with the content serving device, or Advertiser, the code below explains the initial setup of the data advertiser. GPS specific details are explained in other section. One thing to note the max size of an advertisement is 31 bytes. If the amount is exceeded, the advertisement callback will fail with a result of 1 or ADVERTISE_FAILED_DATA_TOO_LARGE. A check is highly recommended in both the application manifest and Java code to ensure the device has the permission and capability to produce Bluetooth LE advertisements. The user will be kicked out of the application to turn on Bluetooth if Bluetooth is off during activation. Add the following to your Android Manifest file:
Add the following to your Android MainActivity file:
public class MainActivity extends AppCompatActivity {
BluetoothAdapter mBAdapter;
BluetoothManager mBManager;
BluetoothLeAdvertiser mBLEAdvertiser;
static final int BEACON_ID = 1775;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
mBManager = (BluetoothManager)getSystemService(BLUETOOTH_SERVICE);
mBAdapter = mBManager.getAdapter();
mBLEAdvertiser = mBAdapter.getBluetoothLeAdvertiser();
}
@Override
protected void onResume()
{
super.onResume();
if(mBAdapter == null || !mBAdapter.isEnabled())
{
Intent enableBtIntent = new Intent(BluetoothAdapter.ACTION_REQUEST_ENABLE);
startActivity(enableBtIntent);
finish();
return;
}
if(!getPackageManager().hasSystemFeature(PackageManager.FEATURE_BLUETOOTH_LE))
{
Toast.makeText(this,"No LE support on this device", Toast.LENGTH_SHORT).show();
finish();
return;
}
if(!mBAdapter.isMultipleAdvertisementSupported())
{
Toast.makeText(this,"No advertising support on this device", Toast.LENGTH_SHORT).show();
finish();
return;
}
startAdvertising();
}
@Override
protected void onPause()
{
super.onPause();
stopAdvertising();
}
private void startAdvertising()
{
if(mBLEAdvertiser == null) return;
AdvertiseSettings settings = new AdvertiseSettings.Builder()
.setAdvertiseMode(AdvertiseSettings.ADVERTISE_MODE_BALANCED)
.setConnectable(false)
.setTimeout(0)
.setTxPowerLevel(AdvertiseSettings.ADVERTISE_TX_POWER_MEDIUM)
.build();
AdvertiseData data = new AdvertiseData.Builder()
.addManufacturerData(BEACON_ID,buildGPSPacket())
.build();
mBLEAdvertiser.startAdvertising(settings, data, mAdvertiseCallback);
}
private void stopAdvertising()
{
if(mBLEAdvertiser == null) return;
mBLEAdvertiser.stopAdvertising(mAdvertiseCallback);
}
private void restartAdvertising()
{
stopAdvertising();
startAdvertising();
}
private AdvertiseCallback mAdvertiseCallback = new AdvertiseCallback() {
@Override
public void onStartSuccess(AdvertiseSettings settingsInEffect) {
super.onStartSuccess(settingsInEffect);
String msg = "Service Running";
mHandler.sendMessage(Message.obtain(null,0,msg));
}
@Override
public void onStartFailure(int errorCode)
{
if(errorCode != ADVERTISE_FAILED_ALREADY_STARTED)
{
String msg = "Service failed to start: " + errorCode;
mHandler.sendMessage(Message.obtain(null,0,msg));
}
else
{
restartAdvertising();
}
}
};
private Handler mHandler = new Handler()
{
@Override
public void handleMessage(Message msg) {
super.handleMessage(msg);
/*
UI feedback to the user would go here.
*/
}
};
private byte[] buildGPSPacket()
{
byte[] packet = new byte[24];
/* GPS code packet generation goes here */
return packet;
}
BLE Watcher in UWP C#
The C# code for the Bluetooth advertisement is straight forward. The watcher listens for a manufacturer data id the same as in the Android application (in this example, 1775). As the watcher runs in another thread, you will need an application or event dispatcher to roll it back into the UWP / Unity application. You also need to add Bluetooth as a capability in the UWP Package Manifest. General UWP (Windows 10) code:
public sealed partial class MainPage : Page
{
BluetoothLEAdvertisementWatcher watcher;
public static ushort BEACON_ID = 1775;
public MainPage()
{
this.InitializeComponent();
}
private void Button_Click(object sender, RoutedEventArgs e)
{
if(watcher != null)
watcher.Stop();
watcher = new BluetoothLEAdvertisementWatcher();
var manufacturerData = new BluetoothLEManufacturerData
{
CompanyId = BEACON_ID
};
watcher.AdvertisementFilter.Advertisement.ManufacturerData.Add(manufacturerData);
watcher.Received += Watcher_Received;
watcher.Start();
}
private async void Watcher_Received(BluetoothLEAdvertisementWatcher sender, BluetoothLEAdvertisementReceivedEventArgs args)
{
ushort identifier = args.Advertisement.ManufacturerData.First().CompanyId;
byte[] data = args.Advertisement.ManufacturerData.First().Data.ToArray();
var ignore = Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
() =>
{
/* GPS Data Parsing / UI integration goes here */
}
);
}
}
Adapted to ingest into Unity, do not forget to surround with #if NETFX_CORE preprocessor statements so the runtime editor does not complain.
First, the role of the event dispatcher gameobject that acts like the Dispatcher in UWP for Unity (Also attach to a Game Manager object in the scene):
using System;
using System.Collections.Generic;
using UnityEngine;
public class EventProcessor : MonoBehaviour
{
public void QueueEvent(Action action)
{
lock (m_queueLock)
{
m_queuedEvents.Add(action);
}
}
void Update()
{
MoveQueuedEventsToExecuting();
while (m_executingEvents.Count > 0)
{
Action e = m_executingEvents[0];
m_executingEvents.RemoveAt(0);
e();
}
}
private void MoveQueuedEventsToExecuting()
{
lock (m_queueLock)
{
while (m_queuedEvents.Count > 0)
{
Action e = m_queuedEvents[0];
m_executingEvents.Add(e);
m_queuedEvents.RemoveAt(0);
}
}
}
private System.Object m_queueLock = new System.Object();
private List m_queuedEvents = new List();
private List m_executingEvents = new List();
}
Now the Unity GPS Watcher class:
using UnityEngine;
#if NETFX_CORE
using Windows.Devices.Bluetooth.Advertisement;
using System.Runtime.InteropServices.WindowsRuntime;
#endif
public class GPS_Receiver : MonoBehaviour
{
#if NETFX_CORE
BluetoothLEAdvertisementWatcher watcher;
public static ushort BEACON_ID = 1775;
#endif
private EventProcessor eventProcessor;
void Awake()
{
eventProcessor = GameObject.FindObjectOfType();
#if NETFX_CORE
watcher = new BluetoothLEAdvertisementWatcher();
var manufacturerData = new BluetoothLEManufacturerData
{
CompanyId = BEACON_ID
};
watcher.AdvertisementFilter.Advertisement.ManufacturerData.Add(manufacturerData);
watcher.Received += Watcher_Received;
watcher.Start();
#endif
}
#if NETFX_CORE
private async void Watcher_Received(BluetoothLEAdvertisementWatcher sender, BluetoothLEAdvertisementReceivedEventArgs args)
{
ushort identifier = args.Advertisement.ManufacturerData[0].CompanyId;
byte[] data = args.Advertisement.ManufacturerData[0].Data.ToArray();
eventProcessor.QueueEvent(() =>
{
/* Unity UI ingestion here
});
}
#endif
}
At this point, you can use an Android device to pass any information that you would like to the UWP application, but for the sake of this article, GPS data code is provided below.
GPS Sensor on Android Java
The hard part of setting up a Bluetooth advertiser/watcher is over, all that is left is to read the GPS sensor on your smartphone and serialize the information for an advertisement to the Hololens. Mentioned above, the advertisement will fail if the size of it is over 31 bytes. Luckily for me, I only required two doubles (8 bytes per double for the Latitude and Longitude) and two floats for my GPS data (4 bytes per float for the heading and speed data) for a total of 24 bytes. You also need to explicitly ask for permission to use location services in Android 6.0+ from the user, and it cannot just be in the application manifest. Add this to your Android manifest:
Add this to your MainActivity file:
LocationManager mLocationManager;
static final int PERMISSION_RESULT_CODE = 1;
Location currentLocation;
@Override
protected void onCreate(Bundle savedInstanceState) {
/*
* code from above section is same here
*/
int permissionCheck = ContextCompat.checkSelfPermission((Context)this, Manifest.permission.ACCESS_FINE_LOCATION);
if (permissionCheck == PackageManager.PERMISSION_GRANTED) {
startGPS();
}
else
{
// No explanation needed, we can request the permission.
ActivityCompat.requestPermissions(this,
new String[]{Manifest.permission.ACCESS_FINE_LOCATION},
PERMISSION_RESULT_CODE);
}
}
public void startGPS()
{
//Execute location service call if user has explicitly granted ACCESS_FINE_LOCATION..
mLocationManager = (LocationManager)getSystemService(Context.LOCATION_SERVICE);
LocationListener listener = new LocationListener() {
@Override
public void onLocationChanged(Location location) {
UpdatePosition(location);
}
@Override
public void onStatusChanged(String provider, int status, Bundle extras) {
}
@Override
public void onProviderEnabled(String provider) {
}
@Override
public void onProviderDisabled(String provider) {
}
};
try
{
mLocationManager.requestLocationUpdates(LocationManager.GPS_PROVIDER, 0, 0, listener);
mLocationManager.requestLocationUpdates(LocationManager.NETWORK_PROVIDER, 0, 0, listener);
currentLocation = mLocationManager.getLastKnownLocation(LocationManager.GPS_PROVIDER);
Log.i("GPS_Receiver", "startGPS: GPS Started..");
}
catch(SecurityException e)
{
}
}
public void UpdatePosition(Location location)
{
currentLocation = location;
restartAdvertising();
}
/* Updated method */
private byte[] buildGPSPacket()
{
Location location = mLocationManager.getLastKnownLocation(LocationManager.GPS_PROVIDER);
byte[] packet = new byte[24];
if(location != null) {
try {
double latitude = location.getLatitude();
byte[] buffer = ByteBuffer.allocate(8).putDouble(latitude).array();
for (int i = 0, j =7; i < 8; i++, j--) packet = buffer[j];
double longitude = location.getLongitude();
buffer = ByteBuffer.allocate(8).putDouble(longitude).array();
for (int i = 8, j =7; i < 16; i++, j--) packet = buffer[j];
float bearing = 0;
bearing = location.getBearing();
buffer = ByteBuffer.allocate(4).putFloat(bearing).array();
for (int i = 16, j =3; i < 20; i++, j--) packet = buffer[j];
float speed = 0;
speed = location.getSpeed();
buffer = ByteBuffer.allocate(4).putFloat(speed).array();
for (int i = 20, j =3; i < 24; i++, j--) packet = buffer[j];
} catch (NumberFormatException e) {
packet = new byte[24];
}
}
return packet;
}
GPS Decoder in UWP C#
The last section of the decoder is the easiest since all that is required is deserialization of the data contained in the advertisement manufacturer data. I created a specific data class to handle the Data.
using System;
public class GPS_DataPacket
{
public double Latitude;
public double Longitude;
public float Heading;
public float Speed;
public static GPS_DataPacket ParseDataPacket(byte[] data)
{
GPS_DataPacket gps_Data = new GPS_DataPacket();
gps_Data.Latitude = BitConverter.ToDouble(data, 0);
gps_Data.Longitude = BitConverter.ToDouble(data, 8);
gps_Data.Heading = BitConverter.ToSingle(data, 16);
gps_Data.Speed = BitConverter.ToSingle(data, 20);
return gps_Data;
}
public override string ToString()
{
string lat, lng;
if (Latitude > 0)
{
lat = string.Format("{0:0.00} ???????????????????????????????????'??N", Latitude);
}
else
{
lat = string.Format("{0:0.00} ???????????????????????????????????'??S", -Latitude);
}
if (Longitude > 0)
{
lng = string.Format("{0:0.00} ???????????????????????????????????'??E", Longitude);
}
else
{
lng = string.Format("{0:0.00} ???????????????????????????????????'??W", -Longitude);
}
return string.Format("Latitude: {0}, Longitude: {1}, Heading: {2:0.00}???????????????????????????????????'??, Speed: {3:0.00} knots", lat, lng, Heading, Speed);
}
}
Interesting Points
This technique provides the raw data to your GPS Android device; you still should follow some of the positioning filtering/business logic covered in the Android developer's guidance at: https://developer.android.com/guide/topics/location/strategies.html After you have started receiving the data you want to receive. Heading and Speed data can be unreliable if the user in not moving. Another gotcha is the explicit request permission calls that need to be made in the MainActivity file for GPS position access. I had added permission statements in the manifest but still had the application crash on security exceptions before adding this to the MainActivity file (new security feature of the Android 6.0+ I guess).Conclusion
The Youtube video:
was used as a reference for the Bluetooth advertiser, it also has information on other Android Bluetooth LE techniques.
I hope this article helps others with getting GPS position data into their Hololens (or any other UWP devices like tablets, XboxOne, etc.). I think this way of sending data is more robust (stable) than the GATT socket server alternatives to passing information across devices. I did not include the frontend code the GPS information in either the UWP or Android because I felt it was out of scope. Sample projects including this are available on request.
Please feel free to comment or work on an iOS advertiser version that I can roll into the article later. Otherwise, thanks for the read and pass on if it is useful. And if I see this in the Unity store for anything other than free, I will be pretty upset.
cool