Damian Oslebo

  • Content count

  • Joined

  • Last visited

Community Reputation

505 Good

About Damian Oslebo

  • Rank

Personal Information

  • Interests
  1. GPS on the Microsoft Hololens

      Thanks, original article corrected for change.
  2. Introduction The Microsoft Hololens technology provides a myriad of possibilities, from its speech recognition to the gaze and gesture controls, all in an untethered environment. But there is one glaring omission, the lack of a GPS sensor onboard the device. This article provides a way to transmit a Bluetooth GPS signal from your smartphone (in this article, an Android OS) to the Hololens. Explaining the Concept This technique leverages Bluetooth LE advertiser/watcher technologies available on the Android and UWP platforms. Similar advertiser code is available on iOS, which has subsequently coined the use of the term "iBeacons" which speaks to all Bluetooth advertiser type devices. Unfortunately, this technology is not available on all devices and should be checked for during runtime. The benefit of the advertiser pattern on the smartphone is that a socket server (RFCOMM) connection does not need to be maintained on either the Hololens or Android side, reducing code complexity and making your application more crash proof. Another major benefit of the Beacon approach is lower overall power consumption on both the advertising and watching devices. I felt the easiest way to explain my technique was to divide the implementation into four sections, the first two are generic to all Bluetooth Beacon applications, and the next two sections describe the GPS specific implementation. Implementation BLE Advertiser on Android Java Starting off with the content serving device, or Advertiser, the code below explains the initial setup of the data advertiser. GPS specific details are explained in other section. One thing to note the max size of an advertisement is 31 bytes. If the amount is exceeded, the advertisement callback will fail with a result of 1 or ADVERTISE_FAILED_DATA_TOO_LARGE. A check is highly recommended in both the application manifest and Java code to ensure the device has the permission and capability to produce Bluetooth LE advertisements. The user will be kicked out of the application to turn on Bluetooth if Bluetooth is off during activation. Add the following to your Android Manifest file: Add the following to your Android MainActivity file: public class MainActivity extends AppCompatActivity { BluetoothAdapter mBAdapter; BluetoothManager mBManager; BluetoothLeAdvertiser mBLEAdvertiser; static final int BEACON_ID = 1775; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); mBManager = (BluetoothManager)getSystemService(BLUETOOTH_SERVICE); mBAdapter = mBManager.getAdapter(); mBLEAdvertiser = mBAdapter.getBluetoothLeAdvertiser(); } @Override protected void onResume() { super.onResume(); if(mBAdapter == null || !mBAdapter.isEnabled()) { Intent enableBtIntent = new Intent(BluetoothAdapter.ACTION_REQUEST_ENABLE); startActivity(enableBtIntent); finish(); return; } if(!getPackageManager().hasSystemFeature(PackageManager.FEATURE_BLUETOOTH_LE)) { Toast.makeText(this,"No LE support on this device", Toast.LENGTH_SHORT).show(); finish(); return; } if(!mBAdapter.isMultipleAdvertisementSupported()) { Toast.makeText(this,"No advertising support on this device", Toast.LENGTH_SHORT).show(); finish(); return; } startAdvertising(); } @Override protected void onPause() { super.onPause(); stopAdvertising(); } private void startAdvertising() { if(mBLEAdvertiser == null) return; AdvertiseSettings settings = new AdvertiseSettings.Builder() .setAdvertiseMode(AdvertiseSettings.ADVERTISE_MODE_BALANCED) .setConnectable(false) .setTimeout(0) .setTxPowerLevel(AdvertiseSettings.ADVERTISE_TX_POWER_MEDIUM) .build(); AdvertiseData data = new AdvertiseData.Builder() .addManufacturerData(BEACON_ID,buildGPSPacket()) .build(); mBLEAdvertiser.startAdvertising(settings, data, mAdvertiseCallback); } private void stopAdvertising() { if(mBLEAdvertiser == null) return; mBLEAdvertiser.stopAdvertising(mAdvertiseCallback); } private void restartAdvertising() { stopAdvertising(); startAdvertising(); } private AdvertiseCallback mAdvertiseCallback = new AdvertiseCallback() { @Override public void onStartSuccess(AdvertiseSettings settingsInEffect) { super.onStartSuccess(settingsInEffect); String msg = "Service Running"; mHandler.sendMessage(Message.obtain(null,0,msg)); } @Override public void onStartFailure(int errorCode) { if(errorCode != ADVERTISE_FAILED_ALREADY_STARTED) { String msg = "Service failed to start: " + errorCode; mHandler.sendMessage(Message.obtain(null,0,msg)); } else { restartAdvertising(); } } }; private Handler mHandler = new Handler() { @Override public void handleMessage(Message msg) { super.handleMessage(msg); /* UI feedback to the user would go here. */ } }; private byte[] buildGPSPacket() { byte[] packet = new byte[24]; /* GPS code packet generation goes here */ return packet; } BLE Watcher in UWP C# The C# code for the Bluetooth advertisement is straight forward. The watcher listens for a manufacturer data id the same as in the Android application (in this example, 1775). As the watcher runs in another thread, you will need an application or event dispatcher to roll it back into the UWP / Unity application. You also need to add Bluetooth as a capability in the UWP Package Manifest. General UWP (Windows 10) code: public sealed partial class MainPage : Page { BluetoothLEAdvertisementWatcher watcher; public static ushort BEACON_ID = 1775; public MainPage() { this.InitializeComponent(); } private void Button_Click(object sender, RoutedEventArgs e) { if(watcher != null) watcher.Stop(); watcher = new BluetoothLEAdvertisementWatcher(); var manufacturerData = new BluetoothLEManufacturerData { CompanyId = BEACON_ID }; watcher.AdvertisementFilter.Advertisement.ManufacturerData.Add(manufacturerData); watcher.Received += Watcher_Received; watcher.Start(); } private async void Watcher_Received(BluetoothLEAdvertisementWatcher sender, BluetoothLEAdvertisementReceivedEventArgs args) { ushort identifier = args.Advertisement.ManufacturerData.First().CompanyId; byte[] data = args.Advertisement.ManufacturerData.First().Data.ToArray(); var ignore = Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => { /* GPS Data Parsing / UI integration goes here */ } ); } } Adapted to ingest into Unity, do not forget to surround with #if NETFX_CORE preprocessor statements so the runtime editor does not complain. First, the role of the event dispatcher gameobject that acts like the Dispatcher in UWP for Unity (Also attach to a Game Manager object in the scene): using System; using System.Collections.Generic; using UnityEngine; public class EventProcessor : MonoBehaviour { public void QueueEvent(Action action) { lock (m_queueLock) { m_queuedEvents.Add(action); } } void Update() { MoveQueuedEventsToExecuting(); while (m_executingEvents.Count > 0) { Action e = m_executingEvents[0]; m_executingEvents.RemoveAt(0); e(); } } private void MoveQueuedEventsToExecuting() { lock (m_queueLock) { while (m_queuedEvents.Count > 0) { Action e = m_queuedEvents[0]; m_executingEvents.Add(e); m_queuedEvents.RemoveAt(0); } } } private System.Object m_queueLock = new System.Object(); private List m_queuedEvents = new List(); private List m_executingEvents = new List(); } Now the Unity GPS Watcher class: using UnityEngine; #if NETFX_CORE using Windows.Devices.Bluetooth.Advertisement; using System.Runtime.InteropServices.WindowsRuntime; #endif public class GPS_Receiver : MonoBehaviour { #if NETFX_CORE BluetoothLEAdvertisementWatcher watcher; public static ushort BEACON_ID = 1775; #endif private EventProcessor eventProcessor; void Awake() { eventProcessor = GameObject.FindObjectOfType(); #if NETFX_CORE watcher = new BluetoothLEAdvertisementWatcher(); var manufacturerData = new BluetoothLEManufacturerData { CompanyId = BEACON_ID }; watcher.AdvertisementFilter.Advertisement.ManufacturerData.Add(manufacturerData); watcher.Received += Watcher_Received; watcher.Start(); #endif } #if NETFX_CORE private async void Watcher_Received(BluetoothLEAdvertisementWatcher sender, BluetoothLEAdvertisementReceivedEventArgs args) { ushort identifier = args.Advertisement.ManufacturerData[0].CompanyId; byte[] data = args.Advertisement.ManufacturerData[0].Data.ToArray(); eventProcessor.QueueEvent(() => { /* Unity UI ingestion here }); } #endif } At this point, you can use an Android device to pass any information that you would like to the UWP application, but for the sake of this article, GPS data code is provided below. GPS Sensor on Android Java The hard part of setting up a Bluetooth advertiser/watcher is over, all that is left is to read the GPS sensor on your smartphone and serialize the information for an advertisement to the Hololens. Mentioned above, the advertisement will fail if the size of it is over 31 bytes. Luckily for me, I only required two doubles (8 bytes per double for the Latitude and Longitude) and two floats for my GPS data (4 bytes per float for the heading and speed data) for a total of 24 bytes. You also need to explicitly ask for permission to use location services in Android 6.0+ from the user, and it cannot just be in the application manifest. Add this to your Android manifest: Add this to your MainActivity file: LocationManager mLocationManager; static final int PERMISSION_RESULT_CODE = 1; Location currentLocation; @Override protected void onCreate(Bundle savedInstanceState) { /* * code from above section is same here */ int permissionCheck = ContextCompat.checkSelfPermission((Context)this, Manifest.permission.ACCESS_FINE_LOCATION); if (permissionCheck == PackageManager.PERMISSION_GRANTED) { startGPS(); } else { // No explanation needed, we can request the permission. ActivityCompat.requestPermissions(this, new String[]{Manifest.permission.ACCESS_FINE_LOCATION}, PERMISSION_RESULT_CODE); } } public void startGPS() { //Execute location service call if user has explicitly granted ACCESS_FINE_LOCATION.. mLocationManager = (LocationManager)getSystemService(Context.LOCATION_SERVICE); LocationListener listener = new LocationListener() { @Override public void onLocationChanged(Location location) { UpdatePosition(location); } @Override public void onStatusChanged(String provider, int status, Bundle extras) { } @Override public void onProviderEnabled(String provider) { } @Override public void onProviderDisabled(String provider) { } }; try { mLocationManager.requestLocationUpdates(LocationManager.GPS_PROVIDER, 0, 0, listener); mLocationManager.requestLocationUpdates(LocationManager.NETWORK_PROVIDER, 0, 0, listener); currentLocation = mLocationManager.getLastKnownLocation(LocationManager.GPS_PROVIDER); Log.i("GPS_Receiver", "startGPS: GPS Started.."); } catch(SecurityException e) { } } public void UpdatePosition(Location location) { currentLocation = location; restartAdvertising(); } /* Updated method */ private byte[] buildGPSPacket() { Location location = mLocationManager.getLastKnownLocation(LocationManager.GPS_PROVIDER); byte[] packet = new byte[24]; if(location != null) { try { double latitude = location.getLatitude(); byte[] buffer = ByteBuffer.allocate(8).putDouble(latitude).array(); for (int i = 0, j =7; i < 8; i++, j--) packet = buffer[j]; double longitude = location.getLongitude(); buffer = ByteBuffer.allocate(8).putDouble(longitude).array(); for (int i = 8, j =7; i < 16; i++, j--) packet = buffer[j]; float bearing = 0; bearing = location.getBearing(); buffer = ByteBuffer.allocate(4).putFloat(bearing).array(); for (int i = 16, j =3; i < 20; i++, j--) packet = buffer[j]; float speed = 0; speed = location.getSpeed(); buffer = ByteBuffer.allocate(4).putFloat(speed).array(); for (int i = 20, j =3; i < 24; i++, j--) packet = buffer[j]; } catch (NumberFormatException e) { packet = new byte[24]; } } return packet; } GPS Decoder in UWP C# The last section of the decoder is the easiest since all that is required is deserialization of the data contained in the advertisement manufacturer data. I created a specific data class to handle the Data. using System; public class GPS_DataPacket { public double Latitude; public double Longitude; public float Heading; public float Speed; public static GPS_DataPacket ParseDataPacket(byte[] data) { GPS_DataPacket gps_Data = new GPS_DataPacket(); gps_Data.Latitude = BitConverter.ToDouble(data, 0); gps_Data.Longitude = BitConverter.ToDouble(data, 8); gps_Data.Heading = BitConverter.ToSingle(data, 16); gps_Data.Speed = BitConverter.ToSingle(data, 20); return gps_Data; } public override string ToString() { string lat, lng; if (Latitude > 0) { lat = string.Format("{0:0.00} AfAfAfAfAfAfAfAfAfAfAfAfAfAfAfAfAfA'A?N", Latitude); } else { lat = string.Format("{0:0.00} AfAfAfAfAfAfAfAfAfAfAfAfAfAfAfAfAfA'A?S", -Latitude); } if (Longitude > 0) { lng = string.Format("{0:0.00} AfAfAfAfAfAfAfAfAfAfAfAfAfAfAfAfAfA'A?E", Longitude); } else { lng = string.Format("{0:0.00} AfAfAfAfAfAfAfAfAfAfAfAfAfAfAfAfAfA'A?W", -Longitude); } return string.Format("Latitude: {0}, Longitude: {1}, Heading: {2:0.00}AfAfAfAfAfAfAfAfAfAfAfAfAfAfAfAfAfA'A?, Speed: {3:0.00} knots", lat, lng, Heading, Speed); } } Interesting Points This technique provides the raw data to your GPS Android device; you still should follow some of the positioning filtering/business logic covered in the Android developer's guidance at: https://developer.android.com/guide/topics/location/strategies.html After you have started receiving the data you want to receive. Heading and Speed data can be unreliable if the user in not moving. Another gotcha is the explicit request permission calls that need to be made in the MainActivity file for GPS position access. I had added permission statements in the manifest but still had the application crash on security exceptions before adding this to the MainActivity file (new security feature of the Android 6.0+ I guess). Conclusion The Youtube video: was used as a reference for the Bluetooth advertiser, it also has information on other Android Bluetooth LE techniques. I hope this article helps others with getting GPS position data into their Hololens (or any other UWP devices like tablets, XboxOne, etc.). I think this way of sending data is more robust (stable) than the GATT socket server alternatives to passing information across devices. I did not include the frontend code the GPS information in either the UWP or Android because I felt it was out of scope. Sample projects including this are available on request. Please feel free to comment or work on an iOS advertiser version that I can roll into the article later. Otherwise, thanks for the read and pass on if it is useful. And if I see this in the Unity store for anything other than free, I will be pretty upset. Article Update Log 9 Sep 2016: Initial release 28 Mar 2017: Corrected typo in code for "buildGPSPacket()" under "GPS Sensor on Android Java"
  3. Panning or centering location on a sphere/globe

    Thanks for the reply.  I want to apply both longitude and latitude rotation.   What if the globe had a pre existing rotation applied?  I want to animate from one rotation to the new center.  Can the target angles calculated be added (or multiplied) to successive rotations?    I will try to implement what you said assuming the steps are: 1) Store current sphere rotation in temporary variable 2) Zero rotation on the globe 3) Raycast hit on the globe using view vector and camera position vector 4) Calculate longitudinal angles for both view and target angles (atan2(p.z, p.x)) 5) Calculate goal angle 6) Calculate vertical angle delta using asin(v.y / v.Length()) 7) Restore original rotation to globe 8) Apply longitudinal delta to globe 9) Apply latitude delta to globe on the view/up cross product axis   but I think I will still have the issue of vertical rotations on the back side of the globe not being the same as the front side.  My solution right now results in latitude requiring to be inverted when camera is looking on the backside.  For example starting with a view rotation looking at the US and recentering after moving the camera (not the look direction) towards the eastern hemisphere.  
  4. Hey all,   Was wondering if I code get some assistance implementing a snap-to function that snaps a users gaze point projection on a sphere to the user's/camera's projection transform onto the world.  I want to maintain the world-up as to say I do not want the sphere rotation to twist/roll when rotating from one point to another / using a latitude and longitude coordinate system.    I have been using Unity so the code below is what I have been working with.  I was storing the world rotation in a current latitude and longitude variable and subtracting it from the projection of the user projection ray onto the sphere.  This way, the user can walk around the object and center/rotate the sphere to face the user/camera transform projection on the sphere.  However, the result does not work as intended because as the user goes behind the sphere, the offsets should slowly be reversed.  I suspect a Sin/Cos based on longitude but still can't make it work.   Intended behavior: When user is in front of globe and requests "Center" on a location such as San Diego the globe moves San Diego from the San Diego location to the intersection of the (user camera position/ globe) ray on the globe. When user is behind globe and requests "Center" on a location such as Tokyo the globe moves Tokyo from the Tokyo location to the intersection of the (user camera position/ globe) ray on the globe.   Code I have so far in C# Unity: void Update() { if(rotationInProgress) { // slowly move the currentLat / Lon towards our targetLat / Lon currentLat = Mathf.LerpAngle(currentLat, targetLat, Time.deltaTime * 5); currentLon = Mathf.LerpAngle(currentLon, targetLon, Time.deltaTime * 5); // build our rotation from the two angles transform.localRotation = Quaternion.AngleAxis(-currentLat, Vector3.right) * Quaternion.AngleAxis(currentLon, Vector3.up); if (Mathf.Abs(currentLat - targetLat) < 0.05f && Mathf.Abs(currentLon - targetLon) < 0.05f) rotationInProgress = false; } } void Center() { // Do a raycast into the world based on the user's // head position and orientation. var headPosition = Camera.main.transform.position; var gazeDirection = Camera.main.transform.forward; var headGlobeDirection = transform.position - Camera.main.transform.position; RaycastHit hitInfo, posInfo; if (Physics.Raycast(headPosition, gazeDirection, out hitInfo) && Physics.Raycast(headPosition,headGlobeDirection,out posInfo)) { #region Determine offset location // convert the hit point into local coordinates Vector3 localPos = transform.InverseTransformPoint(posInfo.point); Vector3 longDir = localPos; // zero y to project the vector to the x-z-plane longDir.y = 0; //calculate the angle between our reference and the hitpoint float offsetLon = Vector3.Angle(-Vector3.forward, longDir); // if our point is on the western hemisphere negate the angle if (longDir.x < 0) offsetLon = -offsetLon; offsetLon -= currentLon; // calculate the latitude in degree float offsetLat = Mathf.Asin(localPos.normalized.y) * Mathf.Rad2Deg - currentLat; #endregion #region Determine Globe Pan-To Location // convert the hit point into local coordinates localPos = transform.InverseTransformPoint(hitInfo.point); longDir = localPos; // zero y to project the vector to the x-z-plane longDir.y = 0; //calculate the angle between our reference and the hitpoint targetLon = Vector3.Angle(-Vector3.forward, longDir); // if our point is on the western hemisphere negate the angle if (longDir.x < 0) targetLon = -targetLon; // calculate the latitude in degree targetLat = Mathf.Asin(localPos.normalized.y) * Mathf.Rad2Deg; targetLon -= offsetLon; targetLat -= offsetLat; #endregion rotationInProgress = true; } } Thanks in advance if this brings about a solution to my problem!  If I am not clear I can provide explain more in a further post.
  5. Introduction WebSockets are a great choice to implement networking in your Unity WebGL games. WebSockets provide two-way persistent connection to a server. Unfortunately, WebSockets in Unity WebGL is limited at this time. There is a sample source library package in the Unity Store from the Unity Team, however, websocket functionality is not fully implemented. This article proposes the use of Javascript Socket.io to implement persistent p2p communication through a web server. A particular useful aspect of Socket.io is that the server code can be written in Javascript. This allows for the code to be analogous on both the server and client sides for better clarity and maintenance. Explaining the Concept There are three major parts to this approach in order to implement Socket.io with your WebGL game: Setting up a network communication script within Unity Creating the javascript client code Creating the javascript server code Data is then passed back and forth to the Unity Game by converting the data to/from JSON objects and strings. Implementation Project Setup In order to try this method out for yourself Node.js must be installed on your server. Once installed, open up a command line terminal and change directory to your Unity Project WebGL Build. Then create a JSON file titled package.json in that directory with the following contents: { "dependencies": { "express": "4.13.4", "socket.io": "1.4.5" } } The actual latest version can be obtained from the command "npm info express version". After the file is created: Run "npm install" to download node modules for express and socket.io into your build directory. Create a folder "public" in the root of your unity build directory Create a blank "client.js" script inside the "public" folder Unity Specific Code The following is an example template that you would use to interact with the external client-side javascript that would in turn interact with the server script. The JSONUtility class is leveraged in the example, since only string data can be passed via Application.externalCall and its corresponding receiver method on the client javascript side. Data can be passed for execution in the browser using: Application.ExternalCall (string functionName, string dataParameter); Ideally, we should set some data classes first before coding the nnetwork manager to aid in conversion to/from Json Objects using JSONUtility: // Sample Data Classes that could be stringified by JSONUtility public class User { public string uid; public string displayname; public User(string u,string d) { uid = u; displayname = d; } } public class MatchJoinResponse { public bool result; } Next create a C# script named "NetworkManager" and attach to a GameObject in your scene: using UnityEngine; public class NetworkManager : MonoBehaviour { public void ConnectToServer(User user) { Application.ExternalCall ("Logon", JsonUtility.ToJson (user)); } public void OnMatchJoined(string jsonresponse) { MatchJoinResponse mjr = JsonUtility.FromJson (jsonresponse); if(mjr.result) { Debug.Log("Logon successful"); } else { Debug.Log("Logon failed"); } } public void BroadcastQuit() { Application.ExternalCall ("QuitMatch"); } } When you rebuild your Unity project. Make sure you now add the following lines to the "index.html" file: Client Javascript The client javascript accepts calls from the Unity Game running in the previous section and connects with the server in the next section. Calls are forwarded back to the Unity Game with the following call: SendMessage(string GameObjectName, string MethodName, string data); The example uses a timeout to prevent locking up your game in the event the server is down. In the client.js file: var connection; var logonTimeout; var logonCallback = function (res) { clearTimeout(logonTimeout); // Send Message back to Unity GameObject with name 'XXX' that has NetworkManager script attached SendMessage('XXX','OnMatchJoined',JSON.stringify(response)); }; function Logon(str) { var data = JSON.parse(str); connection = io.connect(); // Setup receiver client side function callback connection.on('JoinMatchResult', logonCallback); // Attempt to contact server with user data connection.emit('JoinMatchQueue', data); // Disconnect after 30 seconds if no response from server logonTimeout = setTimeout(function(){ connection.disconnect(); connection = null; var response ={result:false}; // Send Message back to Unity GameObject with name 'XXX' that has NetworkManager script attached SendMessage('XXX','OnMatchJoined',JSON.stringify(response)); },30000); } function QuitMatch() { if(connection) { connection.disconnect(); connection = null; } } Server Javascript The example server is setup using express for routing and file delivery simplicity. The server and client functions are analogous in this example. // Variable Initialization var express = require('express'), app = express(), server = require('http').createServer(app), port = process.env.PORT || 3000, io = require('socket.io')(server); // Store Client list var clients = {}; // Allow express to serve static files in folder structure set by Unity Build app.use("/TemplateData",express.static(__dirname + "/TemplateData")); app.use("/Release",express.static(__dirname + "/Release")); app.use(express.static('public')); // Start server server.listen(port); // Redirect response to serve index.html app.get('/',function(req, res) { res.sendfile(__dirname + '/index.html'); }); // Implement socket functionality io.on('connection', function(socket){ socket.on('JoinMatchQueue', function(user){ socket.user = user; clients[user.uid] = socket; var response ={result:true}; socket.emit('JoinMatchResult', response); console.log(user.uid + " connected"); }); socket.on('disconnect', function() { delete clients[socket.user.uid]; console.log(socket.user.uid + " disconnected"); }); }); Once saved, the server can then be started on your home computer with Node.js with the command line "node server.js". Other Notes There is most likely a performance consideration in your game. Data is converted to and from JSON objects and strings from the browser and back in the game; each conversion is necessary overhead in this approach. I would be interested in hearing if anyone implementing this method has any information regarding any performance hit experienced. Also, if implementing Socket.io on an Azure server, make sure the following line is also added in the website's web.config file: This disables the IIS WebSockets module, which includes its own implementation of WebSockets and conflicts with Node.js specific WebSocket modules such as Socket.IO. Conclusion This article presented a way to network with Socket.io and WebGL builds of Unity. Obviously a better way to execute Socket.io and Unity interoperability would be a plugin solely hosted in the Unity environment. The solution presented in this article is a extensible alternative until such time that Unity implements websocket functionality natively into its game engine. Finally, there is added benefit in that the javascript client and server codes will be very similar. Article Update Log 10 Apr 2016: Initial release