BenchFlow
A Benchmark for Workflow Management Systems

The Project

The goal of the BenchFlow project is to design the first benchmark for assessing and comparing the performance of workflow management systems. Due to the inherent complexity of workflow engine architectures and the very large number of parameters affecting their performance, benchmarking workflow systems poses a number of scientific research challenges that require investigating a novel set of performance evaluation techniques. These will complement existing ones used to benchmark database management systems (e.g., TPC) or programming language compilers (e.g., SPEC), which are already well understood and have contributed to drive a very large performance improvement within the database, compiler and processor architecture industry. Workflow systems have become the platform to build composite service-oriented applications, whose performance depends on two factors: the performance of the workflow system itself and the performance of the composed services (which could lie outside of the control of the workflow). We plan to use a model-driven, self-/recursive testing approach to eliminate the impact of the external services by having them implemented as processes. The processes selected for the benchmark will be synthesized out of real-world processes. Given the very large number of metrics that can be used to observe the performance of a workflow system, we aim at distilling a reduced set of performance indicators to compare different engines as well as different configurations of a given engine.


Also see: BenchFlow - Universitá della Svizzera Italiana

European Partners

Seek Input from Companies (Vendors and Users of Workflow Technologies)

The goal of the BenchFlow project is to design the first benchmark for assessing and comparing the performance of workflow management systems. Therefore we need to collect a set of real-world scenarios, which form a fair representation of the applications that use workflow management systems as their base. We ask for companies to contribute some of the process models that they are running. This will help us understand, test and evaluate the workflow engines, as well as build a benchmark that reflects the usage of workflow management systems in the real-world and respects customer needs.

Frequently Asked Questions (FAQ)

  • Over what time frame is our company's involvement needed?

    The company should provide Process Models sometime among October 2013 to March 2014.

  • Do I need to give the actual Process Models?

    No. Realistic scenarios or more abstractive process models would also be helpful to our research.

  • Will the data provided be protected?

    All the data are kept private and will only be used for the purposes of the BenchFlow project.

  • What type of benchmark are you looking at?

    We are looking at a non-functional benchmark mostly focused on performance. However aspects such as fault-tolerance, recoverability and scalability may also come into play.

  • Will the involvement of our company be visible?

    Only if you allow us, otherwise we will protect your identity.

  • Can I contribute something other than process models?

    Yes, we are interested in workload traces and performance/QoS targets.

  • I would like to contribute to the project some of our company's data. Who should I contact?

    For these purposes please contact marigianna.skouradaki(at)iaas.uni-stuttgart.de


Publications

Conference Publications

  1. Skouradaki, Marigianna; Andrikopoulos, Vasilios; Kopp, Oliver and Leymann, Frank: "RoSE: Reoccurring Structures Detection in BPMN 2.0 Process Models Collections". In: On the Move to Meaningful Internet Systems: OTM 2016 Conferences, Springer, Rhodes, Greece, October 24-28, 2016. (to appear)
  2. Skouradaki, Marigianna; Azad, Tayyaba; Breitenbücher, Uwe; Kopp, Oliver and Leymann, Frank: "A Decision Support System for the Performance Benchmarking of Workflow Management Systems". In: Proceedings of In: Proceedings of the 10th Symposium and Summer School On Service-Oriented Computing, (SummerSOC'16); Heraklion, Greece, June 27 - July 01, 2016.
  3. Ferme, Vincenzo; Ivanchikj, Ana and Pautasso, Cesare"Estimating the Cost for Executing Business Processes in the Cloud" BPM Forum, Rio de Janeiro, Brazil, Springer, September, 2016.
  4. Skouradaki, Marigianna; Andrikopoulos, Vasilios; Leymann, Frank: "Representative BPMN 2.0 Process Model Generation from Recurring Structures". In: Proceedings of the 23rd IEEE International Conference on Web Services, (ICWS 2016), June 27- July 02, 2016, San Francisco, USA. pp 468-475. ( pdf).
  5. Skouradaki, Marigianna; Ferme, Vincenzo; Pautasso, Cesare; Leymann, Frank and van Hoorn Andre; "Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns". In: Proceedings of the 28th International Conference on Advanced Information Systems Engineering, (CAiSE 2016), Ljubljana, Slovenia. pp 67-82.( pdf).
  6. Ferme, Vincenzo; Ivanckikj Ana; Pautasso, Cesare; Skouradaki, Marigianna and Leymann, Frank: "A Container-centric Methodology for Benchmarking Workflow Management Systems". In: Proceedings of the 6th International Conference on Cloud Computing and Service Science, (CLOSER 2016), 22-24 April, 2016, Rome, Italy. pp 74-84. ( pdf).
  7. Ferme, Vincenzo and Pautasso Cesare: "Integrating Faban with Docker for Performance Benchmarking", 7th ACM/SPEC International Conference on Performance Engineering, Delft, The Netherlands, March, 2016. (to appear).
  8. Ivanchikj, Ana; Ferme, Vincenzo and Pautasso Cesare: "BPMeter: Web Service and Application for Static Analysis of BPMN 2.0 Collections", 13th International Conference on Business Process Management (BPM 2015), Innsbruck Austria, Springer, pp. 30-34, August, 2015.
  9. Skouradaki, Marigianna and Leymann, Frank: Detecting Frequently Recurring Structures in BPMN 2.0 Process Models. In: Proceedings of the 9th Symposium and Summer School On Service-Oriented Computing: SummerSOC'15; Heraklion, Greece, June 28 - July 04, 2015.( pdf) ( presentation)
  10. Ferme, Vincenzo; Ivanchikj, Ana and Pautasso Cesare: "A Framework for Benchmarking BPMN 2.0 Workflow Management Systems", 13th International Conference on Business Process Management (BPM 2015), Innsbruck, Austria, Springer, August, 2015. (pdf).
  11. Skouradaki, Marigianna; Roller, Dieter H.; Leymann, Frank; Vincenzo, Ferme and Pautasso, Cesare: "BPELanon": Protect Business Processes on the Cloud. In: Proceedings of the 5nd International Conference on Cloud Computing and Service Science, CLOSER 2015, 20-22 May 2015, Lisbon, Portugal. ( pdf)
  12. Skouradaki, Marigianna; Görlach, Katharina; Hahn, Michael and Frank, Leymann; Application of Sub-Graph Isomorphism to Extract Reoccurring Structures from BPMN 2.0 Process Models. In: Proceedings of the 9th International IEEE Symposium on Service Oriented Systems Engineering, SOSE 2015, 30 March – 3 April, 2015, San Francisco, USA. ( pdf) ( presentation)
  13. Pautasso, Cesare; Roller, Dieter H.; Ferme, Vincenzo; Leymann, Frank and Skouradaki, Marigianna: Towards Workflow Benchmarking: Open Research Challenges. In: Proceedings of the 16th Conference on Database Systems for Business, Technology, and Web: Hamburg, Germany, March 2-6, 2015.pp. 331-350 ( pdf) (presentation)
  14. Skouradaki, Marigianna; Roller, Dieter H.; Leymann, Frank; Vincenzo, Ferme and Pautasso, Cesare: On the Road to Benchmarking BPMN 2.0 Workflow Engines. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering ICPE 2015.pp.301-304 ( pdf) (presentation).

Workshopbeiträge

  1. Skouradaki, Marigianna; Roller, Dieter H.; Pautasso, Cesare; Leymann, Frank: BPELanon: Anonymizing BPEL Processes. In: Kopp, Oliver (Hrsg); Lohmann, Niels (Hrsg): Proceedings of the 6th Central European Workshop on Services and their Composition (ZEUS 2014), Potsdam Germany, February 20-21, 2014 ( pdf).

Technische Berichte

  1. Skouradaki, Marigianna; Roller, Dieter H.; Leymann, Frank; Vincenzo, Ferme; Pautasso, Cesare; Technical Open Challenges on Benchmarking Workflow Management Systems Proceedings of Symposium on Software Performance 2014. ( pdf).

Staff Members

Contact

You can contact the BenchFlow Team at: marigianna.skouradaki(@)iaas.uni-stuttgart.de