Commit 0e49805b authored by Adam Kučera's avatar Adam Kučera
Browse files

chages to the testbench documentation

parent 1a6a3a98
......@@ -7,8 +7,11 @@ http://link.springer.com/chapter/10.1007/978-3-319-31165-4_5
The Source files in this repository are proof-of-concept implementations of following
components of the middleware layer:
* **SemanticBMSServer** - SemanticAPI & Ontology repository
* **Semantic API** - SemanticAPI & Ontology repository, currently in relatively stable version 1.0
* **Data Access API** - Data Access API, in a phase of development snapshots
* **Use-case Plugins** - Data providers for the BACnet automation protocol and Delta Historian archive server
* **SemanticBMSClient** - Client for querying the Semantic API
* **TestBench** - Tools for generating the sample data set and testing the query performance + pre-generated sample data
The project is developed and by maintained by Lasaris lab at Faculty of
Informatics, Masaryk University, Brno, Czech Republic.
......@@ -17,7 +20,12 @@ The main developer is Adam Kučera. Contact: **akucera@mail.muni.cz**
ResearchGate profile: https://www.researchgate.net/profile/Adam_Kucera/
## How To
## Sample Semantic BMS Ontology data
The sample data can be found in the [TestBench/NT-Data/sample.nt](https://gitlab.fi.muni.cz/xkucer16/semanticBMS/raw/master/TestBench/NT-Data/sample.nt) file.
For more information about the data set, see [TestBench readme](https://gitlab.fi.muni.cz/xkucer16/semanticBMS/raw/master/TestBench/README.md).
## How To Install SemanticBMS
This section describes one of the ways to deploy the Semantic BMS APIs. There are certainly many other ways, however this one
is tested and pre-configured.
......
$env:Path = $env:Path + ";C:\apache-jena-3.0.1\bin;C:\apache-jena-3.0.1\bat"
tdbloader --loc=.\tdb\ data.nt
tdbstats --loc=.\tdb\
\ No newline at end of file
......@@ -7,6 +7,10 @@ Param(
[String]$OutFolder = ".\CSV-Data"
)
if(-not(Test-Path $OutFolder)) {
mkdir $OutFolder
}
.\CSV-Generators\Generate-Rooms.ps1 -sites $Sites -buildings $Buildings -floors $Floors -rooms $Rooms | Out-File -Encoding utf8 $OutFolder\rooms.csv
.\CSV-Generators\Generate-Devices.ps1 -sites $Sites -buildings $Buildings -floors $Floors -rooms $Rooms | Out-File -Encoding utf8 $OutFolder\devices.csv
.\CSV-Generators\Generate-DataPoints.ps1 -sites $Sites -buildings $Buildings -floors $Floors -rooms $Rooms | Out-File -Encoding utf8 $OutFolder\datapoints.csv
......
......@@ -51,15 +51,15 @@ Request "0: Init" $auth ($Server + "/sbms/semantics/types/type")
$auth = Get-Auth $User $Pass
Request "1: All information about a DP" $auth ($Server + "/sbms/semantics/datapoints/?fields=bmsId%2Ctype%2Csource.bimId%2Csource.type%2Csource.location%2Cscope.bimId%2Cscope.type%2Cscope.location%2Csensing.type%2Csensing.window%2Cproperty.domain%2Cproperty.quality%2Cpublisher.bimId&bmsId=bacnet%3A%2F%2F04010306.AV1")
#Request "1: All information about a DP" $auth ($Server + "/sbms/semantics/datapoints/?fields=bmsId%2Ctype%2Csource.bimId%2Csource.type%2Csource.location%2Cscope.bimId%2Cscope.type%2Cscope.location%2Csensing.type%2Csensing.window%2Cproperty.domain%2Cproperty.quality%2Cpublisher.bimId&bmsId=bacnet%3A%2F%2F04010306.AV1")
Request "2a: All DPs according to strict criteria + grouping" $auth ($Server + "/sbms/semantics/datapoints/?fields=bmsId&grouping=scope.floor&type=Input&source.type=TemperatureSensor&source.location=S01B04&scope.type=Room&property.domain=Air&property.quality=temperature")
#Request "2a: All DPs according to strict criteria + grouping" $auth ($Server + "/sbms/semantics/datapoints/?fields=bmsId&grouping=scope.floor&type=Input&source.type=TemperatureSensor&source.location=S01B04&scope.type=Room&property.domain=Air&property.quality=temperature")
Request "2b: The same query as above, different building" $auth ($Server + "/sbms/semantics/datapoints/?fields=bmsId&grouping=scope.floor&type=Input&source.type=TemperatureSensor&source.location=S02B04&scope.type=Room&property.domain=Air&property.quality=temperature")
#Request "2b: The same query as above, different building" $auth ($Server + "/sbms/semantics/datapoints/?fields=bmsId&grouping=scope.floor&type=Input&source.type=TemperatureSensor&source.location=S02B04&scope.type=Room&property.domain=Air&property.quality=temperature")
Request "3: Generic query with large number of results + grouping" $auth ($Server + "/sbms/semantics/datapoints/?fields=bmsId&grouping=scope.building&source.type=HumiditySensor&property.domain=Air")
#Request "3: Generic query with large number of results + grouping" $auth ($Server + "/sbms/semantics/datapoints/?fields=bmsId&grouping=scope.building&source.type=HumiditySensor&property.domain=Air")
Request "4: Generic query with large number of results + datapoint type + grouping" $auth ($Server + "/sbms/semantics/datapoints/?fields=bmsId%2Ctype%2Cscope.bimId&grouping=scope.building&source.type=TemperatureSensor&property.domain=Air&property.quality=temperature")
#Request "4: Generic query with large number of results + datapoint type + grouping" $auth ($Server + "/sbms/semantics/datapoints/?fields=bmsId%2Ctype%2Cscope.bimId&grouping=scope.building&source.type=TemperatureSensor&property.domain=Air&property.quality=temperature")
Request "5: Generic query with large number of results + all available information + grouping" $auth ($Server + "/sbms/semantics/datapoints/?fields=bmsId%2Ctype%2Csource.bimId%2Csource.type%2Csource.location%2Cscope.bimId%2Cscope.type%2Cscope.location%2Csensing.type%2Csensing.window%2Cproperty.domain%2Cproperty.quality%2Cpublisher.bimId&grouping=scope.building&source.type=TemperatureSensor&property.domain=Air&property.quality=temperature")
......
......@@ -13,9 +13,9 @@ line utility. For the performance testing, other tools, such as curl or wget, mu
The generated data represent mock-up facility of an organization.
They consist of several sites and buildings. Each building is equipped with an PLC, an energy meter and two data points representing energy consumption - overall and last month.
Each room is equipped with a temperature sensor, a humidity sensor, a motion sensor and a PLC that publishes following data points:
* Air temeprature
* Air temperature
* Air humidity
* Setpoint temperature
* Set-point temperature
* Occupancy
For each of the data points, there are two trends defined - one in the PLC itself, the other in the archive historian database.
......@@ -28,7 +28,7 @@ The benchmarking procedure consist of several API calls to the datapoints endpoi
5. Selecting large number of data points based on loose criteria with all the available information retrieved (all temperature sensors in the database + all attributes)
The aim of the benchmarking is to prove that the qeury performance is sufficent to intended purposes, as the performance was not
The aim of the benchmarking is to prove that the query performance is sufficient for intended purposes, as the performance was not
a primary design goal during the development.
The prerequisites to run this benchmark are:
......@@ -39,10 +39,12 @@ The prerequisites to run this benchmark are:
Generally, the optional parameters are not needed when following the expected workflow - they are used only for special use cases.
The scripts mentioned in the following manual HAVE TO be executed from the directory they are placed in.
If you intend to use the provided pre-generated data, proceed directly to step 3 and use the optional parameter to specify the correct input file.
1. [Optional] Run the script Generate-CSVs.ps1
1. [Optional] Run the script .\Generate-CSVs.ps1
The script generates CSV files into the CSV-Data folder.
Optional parameters:
-Sites [Default: 5] - number of sites in the dataset
......@@ -51,7 +53,7 @@ If you intend to use the provided pre-generated data, proceed directly to step 3
-Rooms [Default: 10] - number of rooms in each floor
-OutFolder [Default: ".\CSV-Data"] - output folder for the generated CSV
2. [Optional] Run the script Generate-Triples.ps1
2. [Optional] Run the script .\Generate-Triples.ps1
The script generates an N-Triples file into the NT-Data folder, based on the CSV files located in the .\CSV-Data folder.
Optional parameters:
-InFolder [Default: .\CSV-Data] - location of the source CSV files generated by Generate-CSVs script
......@@ -61,7 +63,7 @@ If you intend to use the provided pre-generated data, proceed directly to step 3
Pre-generated NT file for the test mock-up with the default size is available in the NT-Data folder as sample.nt.
3. Run the script Create-TDB.ps1
3. Run the script .\Create-TDB.ps1
The script will create the TDB triplestore using the Apache Jena framework.
Optional parameters:
-TDBPath [Default: \apache-tomcat\webapps\sbms\WEB-INF\tdb] - location of the resulting TDB directory
......
AMD Athlon II X4 641 Quad-Core 2.8GHz
Apache Jena 3.2.0
Consumed: 500 MB of RAM
One processor core at 100% during the query execution
82000 triples
sbim:EnergyMeter 25
sbim:Site 5
sbim:HumiditySensor 750
sbms:Input 2300
sbms:Observation 3050
sbim:TemperatureSensor 750
sbim:MovementSensor 750
sbms:ObservedProperty 3050
sbms:StatelessDirectSensing 3000
sbim:Building 25
sbim:ProgrammableController 775
sbms:UserDefined 750
sbms:Trend 6100
sbms:AggregateSumSensing 50
sbim:Room 750
sbim:Floor 75
sbim:hasRoom 750
sbms:hasBMSId 9150
sbms:observedProperty 3050
sbms:featureOfInterest 3050
sbim:hasBIMId 3905
sbim:isInstallationLocationOfDevice 3050
sbms:hasAggregationTimeWindow 50
sbim:isFloorOf 75
sbms:expressesObservation 3050
sbms:isPropertyOf 3050
sbim:isRoomOf 750
sbim:hasBuilding 25
sbms:publishedByDevice 3050
sbms:hasPropertyDomain 3050
rdfs:label 6100
sbim:hasInstallationInRoom 3050
rdf:22205
sbms:trends 6100
sbms:observedBy 3050
sbim:isBuildingOf 25
sbms:hasPhysicalQuality 3050
sbim:hasFloor 75
sbms:sensingMethodUsed 3050
#datapoint clause at the end - fresh start
duration;testName;statusCode;contentLength
06,19;0: Init;200;68
02,17;1: All information about a DP;200;387
02,16;2a: All DPs according to strict criteria + grouping;200;1122
01,89;2b: Same query as above, different building;200;1122
02,06;3: Generic query with large number of results + grouping;200;26024
34,33;4: Generic query with large number of results + datapoint + grouping;200;62024
147,43;5: Generic query with large number of results + all available information + grouping;200;249524
# all clauses at the end - fresh start
duration;testName;statusCode;contentLength
06,64;0: Init;200;68
02,54;1: All information about a DP;200;387
02,20;2a: All DPs according to strict criteria + grouping;200;1122
02,11;2b: Same query as above, different building;200;1122
01,96;3: Generic query with large number of results + grouping;200;26024
67,96;4: Generic query with large number of results + datapoint + grouping;200;62024
180,77;5: Generic query with large number of results + all available information + grouping;200;249524
#datapoint clause at the end - fresh start
duration;testName;statusCode;contentLength
06,86;0: Init;200;68
02,18;1: All information about a DP;200;387
02,22;2a: All DPs according to strict criteria + grouping;200;1122
01,97;2b: Same query as above, different building;200;1122
02,09;3: Generic query with large number of results + grouping;200;26024
37,36;4: Generic query with large number of results + datapoint + grouping;200;62024
151,19;5: Generic query with large number of results + all available information + grouping;200;249524
#datapoint clause at the end - repeated
duration;testName;statusCode;contentLength
00,31;0: Init;200;68
01,03;1: All information about a DP;200;387
01,87;2a: All DPs according to strict criteria + grouping;200;1122
02,32;2b: Same query as above, different building;200;1122
01,94;3: Generic query with large number of results + grouping;200;26024
36,27;4: Generic query with large number of results + datapoint + grouping;200;62024
155,71;5: Generic query with large number of results + all available information + grouping;200;249524
00,05;0: Init;200;0
00,49;1: All information about a DP;200;1
01,62;2a: All DPs according to strict criteria + grouping;200;30
01,91;2b: Same query as above, different building;200;30
01,72;3: Generic query with large number of results + grouping;200;750
37,54;4: Generic query with large number of results + datapoint + grouping;200;750
145,99;5: Generic query with large number of results + all available information + grouping;200;750
---------------------------------------------------------------------------------------------------------
1 000 000 triples
sbim:EnergyMeter 100
sbim:Site 10
sbim:HumiditySensor 10000
sbms:Input 30200
sbms:Observation 40200
sbim:TemperatureSensor 10000
sbim:MovementSensor 10000
sbms:ObservedProperty 40200
sbms:StatelessDirectSensing 40000
sbim:Building 100
sbim:ProgrammableController 10100
sbms:UserDefined 10000
sbms:Trend 80400
sbms:AggregateSumSensing 200
sbim:Room 10000
sbim:Floor 500
sbim:hasRoom 10000
sbms:hasBMSId 120600
sbms:observedProperty 40200
sbms:featureOfInterest 40200
sbim:hasBIMId 50810
sbim:isInstallationLocationOfDevice 40200
sbms:hasAggregationTimeWindow 200
sbim:isFloorOf 500
sbms:expressesObservation 40200
sbms:isPropertyOf 40200
sbim:isRoomOf 10000
sbim:hasBuilding 100
sbms:publishedByDevice 40200
sbms:hasPropertyDomain 40200
rdfs:label 80400
sbim:hasInstallationInRoom 40200
rdf:type 292010
sbms:trends 80400
sbms:observedBy 40200
sbim:isBuildingOf 100
sbms:hasPhysicalQuality 40200
sbim:hasFloor 500
sbms:sensingMethodUsed 40200
other 0
760 MB of RAM
58,97;0: Init;200;0
12,60;1: All information about a DP;200;1
26,80;2a: All DPs according to strict criteria + grouping;200;100
27,47;2b: Same query as above, different building;200;100
27,08;3: Generic query with large number of results + grouping;200;10000
453,22;4: Generic query with large number of results + datapoint + grouping;200;10000
-;5: Generic query with large number of results + all available information + grouping;-;-
# Benchmark results
This document describes the benchmark results.
## Software used
* Apache Jena 3.2.0
* Apache Tomcat 9.0.0.M19 64bit Windows distribution
* Jersey 2.25.1
* Windows 10
## Hardware used
The tests were performed on two hardware configurations:
### Configuration A
* Desktop PC
* 5 years old
* AMD Athlon II X4 641 Quad-Core 2.8GHz (PassMark CPU result: 983/3399)
* 4 GB RAM
* HDD
### Configuration B
* Laptop PC
* <1 year old
* Intel COre i5-6200U 2.4GHz/2.8GHz Turbo (PassMark CPU result: 1494/3951)
* 8 GB RAM
* SSD
## Data sets
Two data sets were used during the benchmarking:
### Data set A
over 82000 triples
No of individuals:
sbim:EnergyMeter 25
sbim:Site 5
sbim:HumiditySensor 750
sbms:Input 2300
sbms:Observation 3050
sbim:TemperatureSensor 750
sbim:MovementSensor 750
sbms:ObservedProperty 3050
sbms:StatelessDirectSensing 3000
sbim:Building 25
sbim:ProgrammableController 775
sbms:UserDefined 750
sbms:Trend 6100
sbms:AggregateSumSensing 50
sbim:Room 750
sbim:Floor 75
No of property assertions:
sbim:hasRoom 750
sbms:hasBMSId 9150
sbms:observedProperty 3050
sbms:featureOfInterest 3050
sbim:hasBIMId 3905
sbim:isInstallationLocationOfDevice 3050
sbms:hasAggregationTimeWindow 50
sbim:isFloorOf 75
sbms:expressesObservation 3050
sbms:isPropertyOf 3050
sbim:isRoomOf 750
sbim:hasBuilding 25
sbms:publishedByDevice 3050
sbms:hasPropertyDomain 3050
rdfs:label 6100
sbim:hasInstallationInRoom 3050
rdf:22205
sbms:trends 6100
sbms:observedBy 3050
sbim:isBuildingOf 25
sbms:hasPhysicalQuality 3050
sbim:hasFloor 75
sbms:sensingMethodUsed 3050
### Data set B
over 1 000 000 triples
No of individuals:
sbim:EnergyMeter 100
sbim:Site 10
sbim:HumiditySensor 10000
sbms:Input 30200
sbms:Observation 40200
sbim:TemperatureSensor 10000
sbim:MovementSensor 10000
sbms:ObservedProperty 40200
sbms:StatelessDirectSensing 40000
sbim:Building 100
sbim:ProgrammableController 10100
sbms:UserDefined 10000
sbms:Trend 80400
sbms:AggregateSumSensing 200
sbim:Room 10000
sbim:Floor 500
No of property assertions:
sbim:hasRoom 10000
sbms:hasBMSId 120600
sbms:observedProperty 40200
sbms:featureOfInterest 40200
sbim:hasBIMId 50810
sbim:isInstallationLocationOfDevice 40200
sbms:hasAggregationTimeWindow 200
sbim:isFloorOf 500
sbms:expressesObservation 40200
sbms:isPropertyOf 40200
sbim:isRoomOf 10000
sbim:hasBuilding 100
sbms:publishedByDevice 40200
sbms:hasPropertyDomain 40200
rdfs:label 80400
sbim:hasInstallationInRoom 40200
rdf:type 292010
sbms:trends 80400
sbms:observedBy 40200
sbim:isBuildingOf 100
sbms:hasPhysicalQuality 40200
sbim:hasFloor 500
sbms:sensingMethodUsed 40200
## Test runs
### Configuration A -- Data set A
RAM consumption during querying: 500 MB of RAM
Processor time consumption during querying: ~30%
duration;testName;statusCode;datapoints
02,17;1: All information about a DP;200;1
02,16;2a: All DPs according to strict criteria + grouping;200;30
01,89;2b: Same query as above, different building;200;30
02,06;3: Generic query with large number of results + grouping;200;750
34,33;4: Generic query with large number of results + datapoint + grouping;200;750
147,43;5: Generic query with large number of results + all available information + grouping;200;750
02,18;1: All information about a DP;200;1
02,22;2a: All DPs according to strict criteria + grouping;200;30
01,97;2b: Same query as above, different building;200;30
02,09;3: Generic query with large number of results + grouping;200;750
37,36;4: Generic query with large number of results + datapoint + grouping;200;750
151,19;5: Generic query with large number of results + all available information + grouping;200;750
01,03;1: All information about a DP;200;1
01,87;2a: All DPs according to strict criteria + grouping;200;30
02,32;2b: Same query as above, different building;200;30
01,94;3: Generic query with large number of results + grouping;200;750
36,27;4: Generic query with large number of results + datapoint + grouping;200;750
155,71;5: Generic query with large number of results + all available information + grouping;200;750
00,49;1: All information about a DP;200;1
01,62;2a: All DPs according to strict criteria + grouping;200;30
01,91;2b: Same query as above, different building;200;30
01,72;3: Generic query with large number of results + grouping;200;750
37,54;4: Generic query with large number of results + datapoint + grouping;200;750
145,99;5: Generic query with large number of results + all available information + grouping;200;750
### Configuration A -- Data set B
RAM consumption during querying: 760 MB of RAM
Processor time consumption during querying: ~30%
duration;testName;statusCode;dataPoints
12,60;1: All information about a DP;200;1
26,80;2a: All DPs according to strict criteria + grouping;200;100
27,47;2b: Same query as above, different building;200;100
27,08;3: Generic query with large number of results + grouping;200;10000
453,22;4: Generic query with large number of results + datapoint + grouping;200;10000
-;5: Generic query with large number of results + all available information + grouping;-;-
### Configuration B -- Data set A
RAM consumption during querying: 860 MB of RAM
Processor time consumption during querying: ~30%
duration;testName;statusCode;dataPoints
01,17;1: All information about a DP;200;1
01,57;2a: All DPs according to strict criteria + grouping;200;30
00,98;2b: The same query as above, different building;200;30
01,30;3: Generic query with large number of results + grouping;200;750
18,28;4: Generic query with large number of results + datapoint type + grouping;200;750
74,48;5: Generic query with large number of results + all available information + grouping;200;750
00,48;1: All information about a DP;200;1
00,85;2a: All DPs according to strict criteria + grouping;200;30
00,87;2b: The same query as above, different building;200;30
00,83;3: Generic query with large number of results + grouping;200;750
18,82;4: Generic query with large number of results + datapoint type + grouping;200;750
77,67;5: Generic query with large number of results + all available information + grouping;200;750
00,34;1: All information about a DP;200;1
00,88;2a: All DPs according to strict criteria + grouping;200;30
00,87;2b: The same query as above, different building;200;30
00,96;3: Generic query with large number of results + grouping;200;750
17,95;4: Generic query with large number of results + datapoint type + grouping;200;750
84,64;5: Generic query with large number of results + all available information + grouping;200;750
00,76;1: All information about a DP;200;1
01,19;2a: All DPs according to strict criteria + grouping;200;30
00,99;2b: The same query as above, different building;200;30
01,05;3: Generic query with large number of results + grouping;200;750
18,44;4: Generic query with large number of results + datapoint type + grouping;200;750
76,12;5: Generic query with large number of results + all available information + grouping;200;750
00,34;1: All information about a DP;200;1
00,92;2a: All DPs according to strict criteria + grouping;200;30
00,94;2b: The same query as above, different building;200;30
01,13;3: Generic query with large number of results + grouping;200;750
27,34;4: Generic query with large number of results + datapoint type + grouping;200;750
84,66;5: Generic query with large number of results + all available information + grouping;200;750
### Configuration B -- Data set B
duration;testName;statusCode;dataPoints
06,94;1: All information about a DP;200;1
13,60;2a: All DPs according to strict criteria + grouping;200;100
14,06;2b: The same query as above, different building;200;100
16,97;3: Generic query with large number of results + grouping;200;10000
247,06;4: Generic query with large number of results + datapoint type + grouping;200;10000
\ No newline at end of file
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment