Evaluation

Background 

1.1. Potato Corporation

Potato Corp (a pseudonym) is a global food and agribusiness manufacturer with a dominant presence in the frozen potato market. The company operates across five continents and eleven countries, generating over $10 billion in annual revenue. Potato Corp’s team implemented SAP Ariba in 2019 with the expectation that the platform would modernize sourcing practices, strengthen compliance, and provide strategic visibility across indirect spending. After more than five years, system adoption remains significantly below expectations. Ariba is used primarily as a contract repository while its strategic sourcing, workflow automation, and savings-tracking capabilities remain largely underutilized. In response to these concerns, the evaluation team conducted a formative evaluation to understand the root causes of low adoption, assess the system’s current effectiveness, and identify practical opportunities for improvement.

1.2. Program Design 

The team is evaluating the implementation of Ariba, a procurement operations management system, which is owned by the parent company SAP. The enablement team is the system owner; they determine functionality, process integration, and template design. The enablement team also owns the less technical maintenance. The app development team owns technical integrations like APIs. The two teams work together on system configurations, but enablement mostly takes the lead.

1.3. Stakeholders

As presented there are three distinct groups of stakeholders; the upstream stakeholders that design, configure, and integrate the software, the downstream direct stakeholders that use the software daily, and downstream indirect stakeholders which are typically customers of GIP that benefit from the execution of the system.

1.4. Evaluation Feasibility and Risk Factors

Upon the client’s request to conduct an evaluation of the program, the evaluation team assessed the evaluation feasibility and risk factors. The team’s assessment of evaluation feasibility revealed that it was a feasible evaluation project to implement and complete within the available timeframe. The evaluation team also identified and monitored the project risk factors throughout the project

2.2. Dimensions, Evaluation Questions, and Importance Weighting

Dimensions were determined by 4 things stakeholders needs, program logic model (PLM), theoretical frameworks and professional and ethical standards. 

Importance weighting was based on the anticipated use of findings by leadership (with a high focus on usability and adoption), input from the Enablement and App Development teams, and organizational priorities (including auditability, compliance, and cost savings). Weighting reflects both immediate usability gaps and long-term strategic impacts.

As a result, usability and strategic impact were weighted as “extremely important,” while system functionality, compliance/auditability, and training/support were weighted as “very important” to “important,” depending on their relative influence on user adoption and organizational outcomes.

• Dimension 1 – System Functionality & Integration
  PLM Category: Resources / Activities
  Dimensional Question: How well do the system configurations support real-world procurement needs and processes?
  Importance Weighting: Very Important
  Rationale: Even if users are willing to adopt the system, functionality and integration issues could prevent consistent use. Technical gaps (e.g., broken workflows, poor API connections) were flagged by the App Development team as major risks. This dimension ensures the evaluation addresses whether the tool is technically aligned with business needs.

• Dimension 2 – Training & Support Effectiveness
  PLM Category: Resources / Activities
  Dimensional Question: Are training and support mechanisms sufficient to drive adoption?
  Importance Weighting: Important
  Rationale: Training and support drive adoption indirectly by building confidence and reducing user resistance. While important, this dimension is weighted slightly lower than functionality and compliance because training fixes alone will not solve systemic usability gaps. Still, addressing this area can significantly improve user engagement and long-term system value

• Dimension 3– Compliance, Auditability, and Data Integrity
  PLM Category: Outputs / Outcomes
  Dimensional Question: Does the system ensure contracts, sourcing, and savings are traceable and audit-ready?
  Importance Weighting: Very Important
  Rationale: Compliance and auditability are core reasons Ariba was implemented in the first place. Procurement Enablement stressed the importance of reliable, auditable records (Data Integrity) to reduce risk and meet regulatory obligations. While not as immediate a barrier as usability, this dimension carries significant organizational weight because it protects the company against financial, legal, and reputational risks. Ultimately, compliance leads to improved data integrity, which leads to strategic decision- making. 

Dimension 4– Strategic Impact

PLM Category: Outcomes / Impact

Dimensional Question: Does Ariba improve procurement efficiency, savings visibility, and position procurement as a strategic partner?

Importance Weighting: Extremely Important

• Rationale: Beyond usability and compliance, the ultimate test of Ariba’s success is whether it delivers strategic value. Leadership views procurement as a lever for cost savings, risk management, and organizational efficiency. This efficacy delivers on the need to not only execute at a high level but also to do so quickly  This dimension was given the highest weighting alongside usability because it captures long-term benefits that directly tie to enterprise-wide priorities (efficiency, savings, competitiveness). 

Evaluation Results

Scoring Rubric:

● Superior: Two or three data sources indicate superior, and no data source indicates Did not meet expectations. 

● Met Expectations: No more than one data source indicates Did not meet expectations

● Did not meet expectations: More than one data source indicates Did not meet expectations.

3.1. Dimension 1: System Functionality & Integration

●      Template/Workflow Review Checklist—to determine workflow and template existence, and alignment with Ariba best practices to determine if users have the tools they need to be successful.

o   Review found that though few templates have been created, there were no known process maps supporting the flows resulting in a score of 0%. This result falls into the Did not meet expectations category.

●      Interview Protocol—includes introduction, consent script, and guiding questions.

o   System functionality and & integration questions and statements resulted in 10% positive feedback. This falls into the Did not meet expectations category.

●      Survey Instrument—Likert-scale items assessing usability, system speed, and perceived integration quality (based on interview themes).

o   Analysis showed an average of 3.1 out of 5 on survey results. Though survey responses are better than results from the other two instruments, this result also falls into the Did not meet expectations category.

●  Overall Dimension Grade - Did not meet expectations.

The system integration and functionality dimension has objective hurdles built into the system. The system was built with a fit-to-process approach. The lack of defined processes and low count of supporting templates stems from unclear goals and outcomes for the system. All three data types show a consistent view that the system is not optimized for operational support.

3.2. Dimension 2: Training & Support Effectiveness

●      Training Review Checklist - Indicates the existence and timely review of training documentation.

o   Review found that only 18% of common activities are supported with proper and current training, falling into the Did not meet expectations category.

●      Interview Protocol—includes introduction, consent script, and guiding questions.

o   Interview questions, responses, and statements yielded a rate of 14% positive feedback. It is important to note that support for current templates and category manager activities is seen as more favorable than training support for suppliers and stakeholders.

●      Survey Instrument—Likert-scale items assessing availability and quality (based on interview themes).

o   The survey score for training and support effectiveness was a 2.9 out of 5, the lowest of all dimensional scores for survey results.

●  Overall Dimension Grade - Did not meet expectations.

Training and support nearly made it out of the Did not meet expectations category, but there is a very strong pain point among category managers needing to train a revolving cast of suppliers and stakeholders to interact with Ariba, which has a reputation for not being user friendly. 

3.3. Dimension 3: Compliance, Auditability, and Data Integrity

●      Compliance and Audit Checklist for System Data-to determine if reporting exists to support Ariba governance and compliance.

o   Review found that there are reports to show compliance for 43% of fit-to-process outputs and outcomes, leading to an instrument categorization of Did not meet expectations.

●      Interview Protocol—includes introduction, consent script, and guiding questions.

o   Interview statements fell into the Did not meet expectations with 55% positive feedback, just falling short of meeting expectations.

●      Survey Instrument—Likert-scale items assessing the perceived value Ariba can provide when the system is used correctly.

o   Survey results show that users do see the value of using Ariba from a governance and auditability perspective with a Likert score of 3.8 out of 5, which meets expectations.

●  Overall Dimension Grade - Did not meet expectations.

There is a bit of a perception shift between high level leadership and category managers. The category managers report a higher level of perceived value for using Ariba to track and manage effort. Management tends to lean toward the idea that the extra time spent managing projects in Ariba may degrade customer service or be better spent on other activities, but survey results indicate there is a desire to increase visibility, traceability, and auditability even if it means a few extra clicks in Ariba.

3.4. Dimension 4: Strategic Impact

●      Reporting Review Checklist: assesses depth, accuracy, and business alignment of current Ariba reports.

o   The checklist results show that only 25% of logic model outcomes have supporting reports which results in an instrument categorization of Did not meet expectations.

●      Interview Protocol—includes introduction, consent script, and guiding questions.

o   Interview scores had 33% positive feedback in Ariba’s perceived value of supporting strategic decisions, yielding an instrumental categorization of Did not meet expectations.

●      Survey Instrument—Likert-scale items assessing the perceived value of Ariba’s ability to provide strategic value.

o   Survey results met expectations with a score of 3.8 out of 5, indicating that Category managers believe Ariba utilization can yield strategic insights.

●  Overall Dimension Grade - Did not meet expectations.

Potato Corp has some unique organizational barriers to adopting Ariba's strategic functionality. The organization does not have strong budget scrutiny when compared to similar organizations. There is more of a desire to move quickly than to source efficiently and reduce costs. Most leaders believe that even if they had access to data providing strategic value, the stakeholders wouldn’t be interested in using it. This creates a paradox that leaders won’t encourage buy in because they don’t know if the buy in will amount to anything fruitful.

Conclusions

4.1. Overall Results

●      The evaluation shows that Ariba’s low adoption is not the result of a single failure but a systemic misalignment between the system’s fit-to-process implementation and Potato Corp’s culture, structure, and ways of working. The original rollout lacked clear goals, defined outcomes, and mapped processes, leaving Ariba configured for a level of governance and compliance the organization does not enforce. As a result, all four evaluated dimensions—System Functionality & Integration, Training & Support Effectiveness, Compliance/Auditability/Data Integrity, and Strategic Impact—were rated Did Not Meet Expectations. This outcome reflects broader ERP research showing that rigid, compliance-driven implementations struggle in organizations with adhocratic or relationship-driven cultures.

●      Potato Corp prioritizes speed, flexibility, and stakeholder relationships, the current fit-to-process model creates friction and reinforces user workarounds. Improvements in training or documentation alone will not shift behavior in an environment where governance is unlikely to be mandated. Instead, meaningful progress requires redesigning Ariba to better align with the organization’s culture, user needs, and operational realities.

●      A key opportunity exists for the Enablement Team to adopt a hybrid fit-to-process/fit-to-culture approach. By defining desired outcomes, conducting process inventories, mapping high-value workflows, improving templates, and iterating through agile, user-centered design cycles, the system can begin supporting, rather than impeding how work actually gets done. As processes are clarified and aligned with cultural norms, Ariba can more effectively reduce effort, generate reliable data, improve traceability, and elevate procurement’s strategic influence across the business.

4.2. Recommendations

The evaluation team has come to the conclusion that Potato Corp should shift to a more agile approach (Guo & Wang, 2014) that blends fit-to-process with a fit-to-culture approach. Research shows that organizations with lower levels of compliance have more successful ERP implementations when users are involved in decision making, fit-to-process rigidity is abandoned when the organization knows it will not force compliance and it becomes a process barrier rather than supporting process, and outcomes are defined by the organization, not the ERP developer (Ke & Wei, 2008).

Meta-Evaluations and Limitations

5.1. Meta-Evaluations

In this report the evaluation team has presented a comprehensive document that includes all necessary information to follow Chyung’s (2019) 10-step evaluation framework. Multiple meta-evaluations were performed during this evaluation throughout this 15 week period. This also included external meta-evaluations with an evaluation expert Dr. Chyung. This ensured we not only presented all of the information accurately but also ethically by protecting client anonymity.

5.2. Limitations

The one thing we could have done better was, since there was a project member in the organization, he may have come up with a way to capture water cooler talk and things stated in passing rather than just formally capturing interview and survey results.

We discovered the lack of goals for what the software should accomplish later than we should have. We could have used interview questions and survey statements to uncover conflicting personal and organizational goals. This could have led to more concise recommendations rather than high level observations.

The business has been adamant that Potato Corp should not award higher scores to suppliers for RFx compliance, and the downstream stakeholders tend to be an ever-changing group of individuals that is in constant need of training.

REFERENCES

Chyung, S. Y. (2019). 10-step evaluation for training and performance improvement. Thousand Oaks, CA: Sage.

Guo, Z., Feng, J., & Wang, T. (2014). The impact mechanism of organizational culture on ERP assimilation. In 2014 47th Hawaii International Conference on System Sciences (pp. 4263–4272). IEEE. https://doi.org/10.1109/HICSS.2014.524

Ke, W., & Wei, K. K. (2008). Organizational culture and leadership in ERP implementation. Decision Support Systems, 45(2), 208–218. https://doi.org/10.1016/j.dss.2007.02.002