(2007-2012)
|
Swiss Experiment
|
STILL UNDER CONSTRUCTION
My involvement: programme coordination, construction of wiki metadata system, construction of SLF data system with GSN, data management, construction of long-distance wifi links, initiation and field maintenance of Sensorscope deployments and other field deployments in Davos.
Swiss Experiment was an initiative of the ETH Competence Centre for Environment and Sustainability (CCES) to provide cross funding for environmental infrastructure projects and enable the application of new measurement techniques and technologies to environmental research, combining technological and environmental research in one place. Swiss Experiment particularly aimed to increase the availability of data at high temporal and spatial resolutions. This was a collaboration of 12 departments from across the ETH domain and UniZH. The funding from Swiss Experiment mainly contributed to existing projects with the aim of performing additional cross-disciplinary activities.
Data Infrastructure
Swiss Experiment aimed to create an infrastructure for storing data and providing the interfaces (data discovery interfaces and APIs) to enable interdisciplinary data discovery and sharing. Such data sharing enables many collaborations, but is also of benefit for enabling contact between more diverse groups who happen to have common data requirements (or a potential provider-consumer relationship). e.g. computer science or robotics and environmental science.
GSN Data middleware - TIMESERIES DATA STORAGE AND ACCESS
My involvement: construction of SLF data system with GSN, data management, product owner role
The data infrastructure used the Global Sensor Networks (GSN) middleware to create a distributed network of live data repositories. This was an open source project of EPFL.
The repositories were located at the research institutes of the respective data owners, ensuring that data ownership concerns were addressed by keeping control with the original data owners. GSN simplified data acquisition through the use of data acquisition wrappers, which could be configured for a specific data type. By chaining multiple wrapper types, further data cleaning could be achieved, either allowing both raw and processed data to be saved, or just processed data.
A model interface took this wrapper integration one step further by allowing models to be configured and run on real-time data from within the GUI. Processing was dispatched to dedicated model servers and the results returned to the web platform.
Research also took place to try to automate data quality calculations for live data.
Both a web interface and an API were available for querying the data and notifications were configurable for events occurring in the data.
http://sourceforge.net/apps/trac/gsn
The data infrastructure used the Global Sensor Networks (GSN) middleware to create a distributed network of live data repositories. This was an open source project of EPFL.
The repositories were located at the research institutes of the respective data owners, ensuring that data ownership concerns were addressed by keeping control with the original data owners. GSN simplified data acquisition through the use of data acquisition wrappers, which could be configured for a specific data type. By chaining multiple wrapper types, further data cleaning could be achieved, either allowing both raw and processed data to be saved, or just processed data.
A model interface took this wrapper integration one step further by allowing models to be configured and run on real-time data from within the GUI. Processing was dispatched to dedicated model servers and the results returned to the web platform.
Research also took place to try to automate data quality calculations for live data.
Both a web interface and an API were available for querying the data and notifications were configurable for events occurring in the data.
http://sourceforge.net/apps/trac/gsn
The data portal of WSL/SLF using GSN middleware
The GSN data portal of EPFL-EFLUM showing the La Fouly experiment
|
The control interface of for model integration in GSN. This shows a GeoTop simulation.
|
- Bonvin N., Papaioannou T.G., Aberer K., Autonomic SLA-driven Provisioning for Cloud Applications, August 2011
- Popescu A.D., Dash D., Kantere V. and Ailamaki A., Adaptive query execution for data management in the cloud, Proceedings of the second international workshop on Cloud data management (CloudDB), October 2010
- Jurca O.A., Michel S., Herrmann A., Aberer K., Continuous Query Evaluation over Distributed Sensor Networks, Proc. 26th IEEE International Conference on Data Engineering (ICDE 2010); Long Beach, California, USA, March 2010
- Jeung H., Yiu M.L., Zhou X. and Jensen C.S., Path Prediction and Predictive Range Querying in Spatial Network Databases, The VLDB Journal, August 2010
- Haghani P., Michel S., Aberer K., The gist of everything new - personalized top-k processing over web 2.0 streams, Proceedings of the 19th ACM Conference on Information and Knowledge Management (CIKM), November 2010
- Haghani P. , Michel S., Aberer K., Evaluating Top-k Queries over Incomplete Data Streams, Proc. 18th ACM conference on Information and Knowledge Management, November 2009
- Zhou Y., Salehi A., Aberer K., Scalable Delivery of Stream Query Result, Proc. 35th International Conference on Very Large Data Bases (VLDB), July 2009
- Salehi A., Riahi M., Michel S., Aberer K., Knowing When to Slide - Efficient Scheduling for Sliding Window Processing, Proc. 10th International Conference on Mobile Data Management (MDM), May 2009
- Salehi A., Riahi M., Michel S., Aberer K., GSN, Middleware for Stream World, Proc. 10th International Conference on Mobile Data Management (MDM) (demonstration track), May 2009
- Michel S., Salehi A., Luo L., Dawes N., Aberer K., Barrenetxea G., Bavay M., Kansal A., Kumar K.A., Nath S. , Parlange M.B., Tansley S. , Van Ingen C. , Zhao F. , Zhou Y., Environmental Monitoring 2.0, Proceedings of ICDE (2009), March 2009
- Jurca O.A., Michel S., Herrmann A. , Aberer K., Processing Publish/Subscribe Queries over Distributed Data Streams, Proc. 3rd ACM International Conference on Distributed Event-Based Systems, 2009
- Jurca O.A., Michel S., Herrmann A., Aberer K., Query Driven Operator Placement for Complex Event Detection over Data Streams, Proc. 3rd IEEE European Conference on Smart Sensing and Context, 2008
- Sathe, S., Hoyoung Jeung, Aberer, K.; Creating probabilistic databases from imprecise time-series data; Data Engineering (ICDE), IEEE 27th International Conference pp.327 – 338, doi:10.1109/ICDE.2011.5767838 IEEE CONFERENCE PUBLICATIONS, 2011
SEMANTIC MEDIAWIKI - CENTRAL dATA DISCOVERY and data entry INTERFACE
My involvement: construction of wiki metadata system, implementation of joint data/metadata queries and visualization, metadata entry and management
The semantic mediawiki served as a central platform for entering data and metadata and for querying this metadata together with the automated data collections from GSN including the results of any automated modeling.
The semantic mediawiki supported a web interface on top of a database, which could be queried using SPARQL. In addition to this, extensions were added to allow the use of the R programming language within the wiki. This allowed us to query both metadata and data simultaneously and plot the annotated data in the Wiki interface. Combined, this data source search functionality together with the data query interface, provided a centralised interface to the distributed data system.
The semantic mediawiki served as a central platform for entering data and metadata and for querying this metadata together with the automated data collections from GSN including the results of any automated modeling.
The semantic mediawiki supported a web interface on top of a database, which could be queried using SPARQL. In addition to this, extensions were added to allow the use of the R programming language within the wiki. This allowed us to query both metadata and data simultaneously and plot the annotated data in the Wiki interface. Combined, this data source search functionality together with the data query interface, provided a centralised interface to the distributed data system.
The metadata discovery user interface, showing a timeline of events and a map of locations, with filters according to data types etc.
An example showing the availability of metadata in the system (left and middle) and the availability of data of a specific type (right)
|
An interface in the semantic mediawiki, mixing data queried from GSN with metadata from the mediawiki to highlight events in the data.
|
- Paparrizos I., Jeung H., Aberer K., Visualization and Tagging of Sensor Metadata, Proceedings of the IEEE International Conference on Data Engineering (ICDE), 3 April 2011
- Paparrizos I., Jeung H., Aberer K., Advanced Search, Visualization and Tagging of Sensor Metadata, Proceedings of the IEEE International Conference on Data Engineering (ICDE), March 2011
- Jeung H., Sarni S., Paparrizos I., Sathe S., Aberer K., Dawes N., Papaioannou T.G. , Lehning M., Effective Metadata Management in Federated Sensor Networks, 3rd IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing (SUTC2010); California, USA, 7 June 2010
- Wombacher A., Schneider P., Observation Centric Data Model, Proc. 21st DEXA Conference, Bilbao, Spain, 2010
- Michel S., Salehi A., Luo L., Dawes N., Aberer K., Barrenetxea G., Bavay M., Kansal A., Kumar K.A., Nath S. , Parlange M.B., Tansley S. , Van Ingen C. , Zhao F. , Zhou Y., Environmental Monitoring 2.0, Proceedings of ICDE (2009), March 2009
- Dawes N., Kumar K.A. , Michel S. , Aberer K. , Lehning M., Sensor Metadata Management and Its Application in Collaborative Environmental Research, escience, pp.143-150, 2008 Fourth IEEE International Conference on eScience, December 2008
Microsoft SENSORMAP - sensor data VISUALISATION
My involvement: meetings as customer with Microsoft Research, provision of data
WIthin SwissEx, we collaborated with Microsoft research to push the boundaries of what was available for data discovery and visualisation. SensorMap allowed the display of both point data time-series in a plot as well as time-series raster data e.g. from simulations with a time control. Data was queried from the SwissEx GSN instances and visualised on top of Bing Maps, which allowed multiple base maps and even display of the overlayed raster on the 3D topology (in 2008! Google maps was only launched in 2005). The experience of SensorMap was ahead of its time, even adding 3D visualisations of global cloud cover data. Unfortunately, SensorMap was discontinued by Microsoft Research before it had a chance to make an impact on the community.
WIthin SwissEx, we collaborated with Microsoft research to push the boundaries of what was available for data discovery and visualisation. SensorMap allowed the display of both point data time-series in a plot as well as time-series raster data e.g. from simulations with a time control. Data was queried from the SwissEx GSN instances and visualised on top of Bing Maps, which allowed multiple base maps and even display of the overlayed raster on the 3D topology (in 2008! Google maps was only launched in 2005). The experience of SensorMap was ahead of its time, even adding 3D visualisations of global cloud cover data. Unfortunately, SensorMap was discontinued by Microsoft Research before it had a chance to make an impact on the community.
The Microsoft Research SensorMap system showing some of the capabilities
Lehning, M., Dawes, N., Bavay, M., Parlange, M.B., Nath, S., Zhao, F.: Instrumenting the earth: next-generation sensor networks and environmental science; Chapter in The Fourth Paradigm: Data Intensive Scientific Discovery, 2009, doi: https://www.microsoft.com/en-us/research/publication/fourth-paradigm-data-intensive-scientific-discovery/, 10.1109/jproc.2011.2155130. http://research.microsoft.com/en-us/collaboration/fourthparadigm/
GEovite - geospatial data discovery Interface
My involvement: collaboration with IKG to design the features required. Provision of data.
At the same time as we were working on SensorMap, we also had a collaboration with ETHZ-IKG on the GeoVITe platform. The focus of GeoVITe was on sharing Swiss spatial data, which meant that it provided SwissEx users with an interface for accessing Swisstopo data as well as other national data sources (vector and raster). Unfortunately, due to content licencing restrictions at the time, this also meant that usage was restricted to the ETH domain. GeoVITe had much more of a cartographical focus than SensorMap, and conformed to web standards, meaning that it was able to access many more cartographical data sources.
Later (in the OSPER project), this interface was also extended to enable access to the GSN-based sensor data.
At the same time as we were working on SensorMap, we also had a collaboration with ETHZ-IKG on the GeoVITe platform. The focus of GeoVITe was on sharing Swiss spatial data, which meant that it provided SwissEx users with an interface for accessing Swisstopo data as well as other national data sources (vector and raster). Unfortunately, due to content licencing restrictions at the time, this also meant that usage was restricted to the ETH domain. GeoVITe had much more of a cartographical focus than SensorMap, and conformed to web standards, meaning that it was able to access many more cartographical data sources.
Later (in the OSPER project), this interface was also extended to enable access to the GSN-based sensor data.
The ETHZ-IKG GeoVITe user interface
|
An example of the 3D visualization capabilities of GeoVITe
|
Iosifescu I. and Hurni L. (2010). GIS Platform for Interdisciplinary Environmental Research, in Proceedings of the 7th ICA Mountain Cartography Workshop, Borsa, Romania
Sensor Networks
Wireless sensor networks enable data acquisition at high spatial and temporal scales whilst keeping devices low power and low cost (particularly with the data transmission technologies available in 2007). One aim of Swiss Experiment was to develop the technologies behind these networks further to enable the collection of the desired data resolutions.
SensorScope 2nd and 3rd Generation Networks
My involvement: feedback from deployments
SensorScope was a successful attempt to build low energy, plug and play, wireless self-configuring local networks of stations. This meant that the stations could communicate data locally at low power, and only needed one higher powered base station with a mobile connection to communicate the data back to the server. SensorScope became a successful spinoff of EPFL. The SwissEx collaboration provided significant funding both directly and in terms of orders to develop the SensorScope system.
SensorScope was a successful attempt to build low energy, plug and play, wireless self-configuring local networks of stations. This meant that the stations could communicate data locally at low power, and only needed one higher powered base station with a mobile connection to communicate the data back to the server. SensorScope became a successful spinoff of EPFL. The SwissEx collaboration provided significant funding both directly and in terms of orders to develop the SensorScope system.
Hydromon self organising wireless network
My involvement: none
Hydromon aimed to create a sensor network for monitoring the drinking water distribution system of a major agglomeration in southern Switzerland. A sensor network was deployed where several tens of parameters (physical and chemical) and five pathogens are constantly monitored 24/7 at each node of the network. SwissEx provided academic funding in a collaboration between EPFL and a commercial company, which became an operational system.
Hydromon aimed to create a sensor network for monitoring the drinking water distribution system of a major agglomeration in southern Switzerland. A sensor network was deployed where several tens of parameters (physical and chemical) and five pathogens are constantly monitored 24/7 at each node of the network. SwissEx provided academic funding in a collaboration between EPFL and a commercial company, which became an operational system.
Permasense 2nd Generation Network
My involvement: management of SwissEx and hence collaboration on shared data acquisition technologies
In the Permasense 2nd generation network, the SwissEx collaboration provided Permasense with the GSN data acquisition system as well as long distance wifi links to communicate the data from the Matterhorn, across to the Kleine Matterhorn. As of 2025, these are still in operation.
http://data.permasense.ch/data.html#data
In turn, Permasense technologies were also deployed to SwissEx partner experiments, RECORD for groundwater monitoring.
Images taken during a SwissEx excursion to the permasense site on the Matterhorn:
In the Permasense 2nd generation network, the SwissEx collaboration provided Permasense with the GSN data acquisition system as well as long distance wifi links to communicate the data from the Matterhorn, across to the Kleine Matterhorn. As of 2025, these are still in operation.
http://data.permasense.ch/data.html#data
In turn, Permasense technologies were also deployed to SwissEx partner experiments, RECORD for groundwater monitoring.
Images taken during a SwissEx excursion to the permasense site on the Matterhorn:
Long distance communication links
My involvement: research, implementation and testing of long distance links (incl. custom router builds)
Data communications in 2008 were different to today. Mobile data communication was slow and expensive and coverage, though good, was still limited in comparison to today. In order to overcome these hurdles, I built WiFi communications links with directional WiFi antennae. In the case shown below, the router was given GSM capabilities and provided mobile hotspot access to a valley where mobile coverage was unavailable.
Data communications in 2008 were different to today. Mobile data communication was slow and expensive and coverage, though good, was still limited in comparison to today. In order to overcome these hurdles, I built WiFi communications links with directional WiFi antennae. In the case shown below, the router was given GSM capabilities and provided mobile hotspot access to a valley where mobile coverage was unavailable.
Creation of a wifi hotspot across the valley floor using a directional antenna
The mobile links were used for many years in Davos in a multi-hop scenario, providing high-bandwidth data communications to several field-sites, one of which was not in a direct line-of-sight. These links were used to acquire acoustic (geophone) data for the TRAMM project and to return metadata from the EPFL-LTE radar.
|
|
Measurement Technologies
Huge data volumes may be collected during the development of new technologies. But these technologies are tested in the same environments that the eventual customers use for research. By connecting the two parties during the development of the technology, SwissEx aimed to speed up the technology development by enabling direct feedback, and allow the environmental scientists early access to the equipment and hence data that it could produce.
X-Band Radar Precipitation Measurements
My involvement: assistance in deployment and maintenance
SwissEx was the start of a long-term collaboration between the group of EPFL LTE and WSL/SLF for joint precipitation experiments using X-Band radars and intensive ground-based measurement campaigns. During SwissEx, the newly acquired dual-polarized X-Band radar was installed on Jakobshorn in Davos, across the valley from the Wannengrat field site and Sensorscope measurement campaign. Since this time, similar experiments have taken place both in Davos and elsewhere.
The dual-polarized X-Band radar allowed new finer resolution measurement of precipitation and indication of particulate dimensions in two axes.
SwissEx was the start of a long-term collaboration between the group of EPFL LTE and WSL/SLF for joint precipitation experiments using X-Band radars and intensive ground-based measurement campaigns. During SwissEx, the newly acquired dual-polarized X-Band radar was installed on Jakobshorn in Davos, across the valley from the Wannengrat field site and Sensorscope measurement campaign. Since this time, similar experiments have taken place both in Davos and elsewhere.
The dual-polarized X-Band radar allowed new finer resolution measurement of precipitation and indication of particulate dimensions in two axes.
The dual-polarized X-Band radar installed at Brämabüehl, Jakobshorn, Davos, on the opposite side of the valley to the Wannengrat field site, for precipitation experiments
Optical disdrometer development
My involvement: none
EPFL-ISIM together with EPFL-LTE pushed the boundaries of disdrometer development by manufacturing a 1k pixel single photon camera which could operate at 6kfps continuously. This meant that individual water droplets could be analyzed for size and shape, creating an optical disdrometer.
EPFL-ISIM together with EPFL-LTE pushed the boundaries of disdrometer development by manufacturing a 1k pixel single photon camera which could operate at 6kfps continuously. This meant that individual water droplets could be analyzed for size and shape, creating an optical disdrometer.
An initial prototype of the disdrometer hardware
|
A snapshot of the disdrometer output, showing a single droplet measured in real-time
The disdrometer main board showing the single-photon optical detector in the middle
|
Low-cost SENSIBLE HEAT-FLUX Sensors
My involvement: none
Thermistor based sensing technologies developed for other projects were compared to sonic anemometer measurements to analyze the feasibility of cheap sensible heat-flux measurements.
Thermistor based sensing technologies developed for other projects were compared to sonic anemometer measurements to analyze the feasibility of cheap sensible heat-flux measurements.
A thermistor-based detector built for a different experiment was tested for its potential as an anemometer.
|
Thermocouple-based wind-speed calculation was compared to a 3D anemometer
|
- Bahr, A., Evans, C., Martinoli, A., Huwald, H., Higgins, C., & Parlange, M. (2012). Measuring sensible heat flux with high spatial density. Ecole Polytechnique Fédérale de Lausanne (EPFL), Distributed Intelligent Systems and Algorithms Laboratory & Environmental Fluid Mechanics and Hydrology.
Monitoring of snow height and snow water equivalent with GPS
My involvement: assistance in field deployment and provision of data
10.13140/RG.2.2.11896.29442
10.13140/RG.2.2.11896.29442
|
|
Development of Distributed Temperature Sensor (DTS) calibration techniques for field deployment
My involvement: assistance in logistics
DTS allows continuous temperature measurements at discrete locations along an optical fibre. To do this, the temperature at the incoming and outgoing ends of the fibre must be known in order to infer the temperature at locations between these calibration points. Researchers at EAWAG developed calibration techniques for DTS equipment which could be deployed in the field.
DTS allows continuous temperature measurements at discrete locations along an optical fibre. To do this, the temperature at the incoming and outgoing ends of the fibre must be known in order to infer the temperature at locations between these calibration points. Researchers at EAWAG developed calibration techniques for DTS equipment which could be deployed in the field.
Intensive Deployments
Combining the technology and environmental research, an colocating experiments to improve cost-efficiency enables data to be collected at high spatial and temporal resolutions without being prohibitively costly. SwissEx aimed to act on this principle and set up such large scale experiments.
SensorScope extreme environment test deployments
My involvement: management of field deployments in Davos
In order to field test the SensorScope stations in extreme environments whilst also contributing to scientific experiments, two experiments were set up, one in La Fouly, Valais, and one on Wannengrat, Davos. These experiments both ran for many years and provided vital feedback to the development of SensorScope, resulting in more robust and power efficient hardware.
In order to field test the SensorScope stations in extreme environments whilst also contributing to scientific experiments, two experiments were set up, one in La Fouly, Valais, and one on Wannengrat, Davos. These experiments both ran for many years and provided vital feedback to the development of SensorScope, resulting in more robust and power efficient hardware.
SensorScope device pool
My involvement: acquisition, management and deployment of the SwissEx equipment pool
A pool of SensorScope devices was purchased and used for short-term (e.g. 1 year) deployments in various CCES projects
A pool of SensorScope devices was purchased and used for short-term (e.g. 1 year) deployments in various CCES projects
LIDAR field campaign and development
My involvement: none
Precipitation measurement campaign in alpine terrain
My involvement: none
Vispa catchment measurement campaign
My involvement: none
Scaling of data
Downscaling of wind fields
My involvement: field deployments and data management
|
In the SwissEx Science project, a large scale experiment was performed using the Wannengrat SensorScope deployment, meteo stations from around Davos, Lidars, the EPFL-LTE X-Band radar and a wind tower with 3D anemometers. This data was used to verify experiments in the downscaling of wind fields.
|
|
Scaling of precipitation data
My involvement: none
ETHZ-HRWM carried out temporal scaling studies on precipitation data and determined that whilst precipitation cannot be perfectly scaled, there are distinct regimes where scaling can be successful, connected to atmospheric processes.
Paschalis, A., P. Molnar, and P. Burlando, (2012) Temporal dependence structure in weights in a multiplicative cascade model for precipitation, Water Resour. Res., 48, 1, doi:10.1029/2011WR010679.
ETHZ-HRWM carried out temporal scaling studies on precipitation data and determined that whilst precipitation cannot be perfectly scaled, there are distinct regimes where scaling can be successful, connected to atmospheric processes.
Paschalis, A., P. Molnar, and P. Burlando, (2012) Temporal dependence structure in weights in a multiplicative cascade model for precipitation, Water Resour. Res., 48, 1, doi:10.1029/2011WR010679.
Education
Educational campaigns were regularly carried out in the areas of climate and seismology, with cross-funding provided by SwissEx. These campaigns took place in schools across Switzerland as well as in Burkina Faso.
ClimATScope
My involvement: no direct involvement
|
|
|
Seismo@School
|
My involvement: no direct involvement
http://www.seismo.ethz.ch/en/research-and-teaching/projects/seismo-at-school/ |
|