The goal of the BenchFlow project is to design the first benchmark for assessing and comparing the performance of workflow management systems. Due to the inherent complexity of workflow engine architectures and the very large number of parameters affecting their performance, benchmarking workflow systems poses a number of scientific research challenges that require investigating a novel set of performance evaluation techniques. These will complement existing ones used to benchmark database management systems (e.g., TPC) or programming language compilers (e.g., SPEC), which are already well understood and have contributed to driving a very large performance improvement within the database, compiler, and processor architecture industry. Workflow systems have become the platform to build composite service-oriented applications, whose performance depends on two factors: the performance of the workflow system itself and the performance of the composed services (which could lie outside of the control of the workflow). We plan to use a model-driven, self-/recursive testing approach to eliminate the impact of the external services by having them implemented as processes. The processes selected for the benchmark will be synthesized out of real-world processes. Given the very large number of metrics that can be used to observe the performance of a workflow system, we aim at distilling a reduced set of performance indicators to compare different engines as well as different configurations of a given engine.
- Universitá della Svizzera Italiana
- University of Stuttgart
Seek Input from Companies (Vendors and Users of Workflow Technologies)
The goal of the BenchFlow project is to design the first benchmark for assessing and comparing the performance of workflow management systems. Therefore we need to collect a set of real-world scenarios, which form a fair representation of the applications that use workflow management systems as their base. We ask for companies to contribute some of the process models that they are running. This will help us understand, test and evaluate the workflow engines, as well as build a benchmark that reflects the usage of workflow management systems in the real-world and respects customer needs.
Frequently Asked Questions (FAQ)
- Over what time frame is our company's involvement needed?
The company should provide Process Models sometime among October 2013 to March 2014.
- Do I need to give the actual Process Models?
No. Realistic scenarios or more abstractive process models would also be helpful to our research.
- Will the data provided be protected?
All the data are kept private and will only be used for the purposes of the BenchFlow project.
- What type of benchmark are you looking at?
We are looking at a non-functional benchmark mostly focused on performance. However, aspects such as fault-tolerance, recoverability, and scalability may also come into play.
- Will the involvement of our company be visible?
Only if you allow us, otherwise we will protect your identity.
- Can I contribute something other than process models?
Yes, we are interested in workload traces and performance/QoS targets.
- I would like to contribute to the project some of our company's data. Who should I contact?
For these purposes please contact marigianna.skouradaki(at)iaas.uni-stuttgart.de