SLA Assessment

Security metrics’ importance for the decision making in ICT systems with respect to security has been recognised by organizations such as ENISA [1], CIS [2] and NIST [3]. Security metrics have been applied for quantifying the security of network systems though several attack graph-based security metrics (e.g., Idika et al. [4], Lippman et al. [5], Ortalo et al. [6], Pamula et al [7]), or to measure the degree of trustworthiness of software-intensive systems (e.g., Manadhata et al. [8], Savola [9], Wang et al. [10]).

Functionality & Security assessment

Multiple approaches are emerging to assess the functionality and security of CSPs. Li [11] presents a framework to compare Cloud providers according to performance indicators. Garg et al [12] use the Analytic Hierarchy Process (AHP) to rank providers based also on performance data to measure various Quality of Service (QoS) attributes. A framework of critical characteristics and measures that enable comparison of Cloud services is also presented in [13] by Siegel at al. With respect to security assessment presented by Almorsy et al. [14], the authors propose the notion of evaluating Cloud secSLAs, by introducing a metric to benchmark the security of a CSP based on categories. However, the resulting security categorization is purely qualitative. In Casola et al ([15]) a methodology for evaluating and comparing security SLAs expressed through the use of standard policy languages is presented. Luna [16] uses a similar approach to quantify the security of a Public Key Infrastructure, based on its Certificate Policy. In [17] Ghani et al. present a metric-based approach for assessing the security level of Critical Infrastructures.

Security metrics in SLAs

Security metrics are also used for the definition of SLAs. In [18], Luna et al. point to the need of developing a security metrics framework for the Cloud. Other security-metrics based approach propose mechanisms to describe and quantify security are given in [19] by Breier et al. Few works focus on security metrics aggregation in order to enable the quantification of the security level in the end-to-end of all collaborators of the supply chain. Unfortunately, metrics aggregation mostly remains a research challenge as acknowledged by NIST [3] and ENISA [20]. Very few frameworks have been proposed to aggregate security metrics. Among them are the works done by Massacci et al. [21], Frankova et al. [22], Seamons et al. [23] and Smith et al. [24]. Predictive approaches for anticipating how security metrics will develop have been also studied and applied (e.g., Trust Economics system modelling paradigm [25], [26] and the ADVISE modelling approach of [27].

[1] P. Trimintzios, "Measurement Frameworks and Metrics for Resilient Networks and Services," European Network and Information Security Agency (ENISA), Technical report, 2011. D2.2 Requirements emerging from a state-of-the-art analysis – Final Report Page 92

[2] "The CIS security metrics V1.1.0.". Center for Internet Security (CIS), Technical Report, 2010.

[3] W. Jansen, "Directions in security metrics research." National Institute of Standards and Technology (NIST), DIANE Publishing, U.S. 2010.

[4] N. Idika and B. Bhargava, "Extending Attack Graph-Based Security Metrics and Aggregating Their Application," IEEE Transactions on Dependable and Secure Computing (TDSC), vol. 9, n. 1, pp. 75-85, 2012.

[5] R. Lippmann, K. Ingols, C. Scott, K. Piwowarski, K. Kratkiewicz, M. Artz and R. Cunningham, "Validating and Restoring Defense in Depth Using Attack Graph," in Proc. of Military Communications Conference (MILCOMM 2006), 2006, pp. 1-10.

[6] R. Ortalo, Y. Deswarte and M. Kaâniche, "Experimenting with Quantitative Evaluation Tools for Monitoring Operational Security," IEEE Transactions on Software Engineering (TSE), vol. 25, n.5, pp. 633-650, 1999

[7] J. Pamula, S. Jajodia, P. Ammann and V. Swarup, "A weakest-adversary security metric for network configuration," in Proc. of the 2nd ACM Workshop on Quality of Protection (QOP 2006), 2006, pp. 31-38.

[8] P.K. Manadhata and J.M. Wing, "An Attack Surface Metric," IEEE Transactions on Software Engineering (TSE), vol. 37, no. 3, pp. 371-386, 2011

[9] R.M. Savola, "On the Feasibility of Utilizing Security Metrics in SoftwareIntensive Systems," in International Journal of Computer Science and Network Security (IJCSNS), vol. 10, no. 1, pp. 230-239, 2010.

[10] J.A. Wang, H. Wang, M. Guo and M. Xia, "Security Metrics for Software Systems," in Proc. of the 47th ACM Southeast Conference (ACMSE 2009), 2009.

[11] A. Li, X. Yang, S. Kandula, and M. Zhang, "Cloudcmp: Comparing public cloud providers," in Proc. Conf. Appl., Technol., Archit., Protocols Comput. Commun., 2010, pp. 1–14.

[12] S. K. Garg, S. Versteeg, and R. Buyya, "A framework for comparing and ranking cloud services," J. Future Generation Comput. Syst., vol. 29, no. 4, pp. 1012–1023, 2013.

[13] J. Siegel and J. Perdue, "Cloud services measures for global use: The service measurement index," in Proc. Annu. SRII Global. Conf., 2012, pp. 411–415.

[14] M. Almorsy, J. Grundy, and A. Ibrahim, "Collaboration-based cloud computing security management framework," in Proc. IEEE Int. Conf. Cloud Comput., 2011, pp. 364–371.

[15] V. Casola, et.al. "A SLA evaluation methodology in Service Oriented Architectures,"Quality of Protection of Springer Advances in Information Security, vol. 23, pp. 119 – 130, 2006.

[16] J. Luna, H. Ghani, D. Germanus, and N. Suri, "A security Metrics Framework for the Cloud," in Proc. of the 6th International Conference on Security and Cryptography (SECRYPT 2011), 2011, pp. 245-250.

[17] H. Ghani, A. Khelil, N. Suri, G. Csertan, L. Gonczy, G. Urbanics, and J. Clarke, "Assessing the Security of Internet Connected Critical Infrastructures (The CoMiFin Project Approach)," in Proc. of the 1st Workshop on the Security of the Internet of Things (SecIoT 2010), 2010.

[18] J. Luna, R. Langenberg, N. Suri, "Benchmarking cloud security level agreements using quantitative policy trees," ACM Workshop on Cloud computing security workshop, pp. 103-112, 2012.

[19] J. Breier and L. Hudec, "Risk Analysis Supported by Information Security Metrics,". In Proc. of the 12th International Conference on Computer Systems and D2.2 Requirements emerging from a state-of-the-art analysis – Final Report Page 93 Technologies (CompSysTech 2011), 2011, pp. 393-398.

[20] P. Trimintzios, "Measurement Frameworks and Metrics for Resilient Networks and Services," European Network and Information Security Agency (ENISA), Technical report, 2011.

[21] F. Massacci and A. Yautsiukhin, "An algorithm for the appraisal of assurance indicators for complex business processes," in Proceedings of the 2007 ACM workshop on Quality of protection, 2007, pp. 22-27.

[22] G. Frankova and A. Yautsiukhin, "Service and protection level agreements for business processes," In The 2nd European Young Researchers Workshop on Service Oriented Computing, 2007.

[23] K.E. Seamons, T. Chan, E. Child, M. Halcrow, A. Hess, J. Holt, J. Jacobson, R. Jarvis, A. Patty, B. Smith, T. Sundelin, and L. Yu, "TrustBuilder: negotiating trust in dynamic coalitions," in Proceedings DARPA Information Survivability Conference and Exposition, 2003, pp. 49–51.

[24] B. Smith, K.E. Seamons, and M.D. Jones, "Responding to policies at runtime in TrustBuilder," in IEEE International Workshop on Policies for Distributed Systems and Networks (POLICY’04), 2004, pp. 149–158.

[25] A. Beautement, R. Coles, J. Griffin, B. Monahan, D. Pym, M.A. Sasse and M. Wonham, "Modelling the Human and Technological Costs and Benefits of USB Memory Stick Security," Workshop on Economics in Information Security. 2008, pp. 141-163.

[26] R. Coles, J. Griffin, H. Johnson, B. Monahan, S. Parkin, D. Pym, A. Sasse, and A. van Moorsel, "Trust Economics Feasibility Study," in Workshop on Resilience Assessment and Dependability Benchmarking, IEEE Computer Press, 2008.

[27] E. LeMay, M.D. Ford, K. Keefe, W.H. Sanders and C. Muehrcke, "Model-based Security Metrics using ADversary VIew Security Evaluation (ADVISE)," in Proceedings of the 8th International Conference on Quantitative Evaluation of SysTems (QEST 2011), 2011, pp. 191-200.