CONIECTO

ECITE took place this year in Amsterdam on the 11 and 12 November, so I was lucky to be able to attend two important events in the same week and same city. The Conference is at its 11th edition, but for me it was the first time when I succeeded to attend. Both for the 5th and the 6th edition, I sent papers( one of them was also published in EJISE) that were accepted, but with all the efforts, I couldn't find the necessary funding to go there.

Actually, IT Evaluation was the topic of my PhD, and prof. Dan Remenyi, who's leading Academic Conferences International - the conference organiser, and prof.Egon Berghout, the host of this edition, helped me a lot at the time to find my own way in this very controversial field. Meeting them and another couple of big names and reconnecting to what was going on in the field was my primary motivation - even if in the last few years my research interests went more into the direction of knowledge management. And my first paper on KM was an attempt to build a model for evaluating a company's KM strategy - trying to marry the two of them, KM and IT Evaluation ;-) !

The conference events started one evening before with a get-together at the IBIS hotel. Dan Remenyi and Sue Nugus, together with Egon Berghout were already there to welcome us. I had the chance to meet few interesting people from the Organising Committee (Ann Brown, David Barnes) and to get acquainted to a group of participants (Shailey Minocha, Mohini Singh, Zuzana Kucerova). The organisers also draw my attention to a Romanian PhD student coming from Finland, Dorina Marghescu and it was nice to exchange few words in Romanian with someone.
It was a long day and I was exhausted, and I decided to go back to my hotel, hunting the hot spot. No chance - the signal was too weak!

Thursday, 11 November
Our host, Egon Berghout welcomed everybody and told us a few words about the building hosting the conference, the Trippenhuis. Built by a rich Dutch family, the building hosted the Royal Netherlands Academy of Sciences since 1887. For a period of 70 years, it also hosted the famous Night Watch of Rembrandt, and that's why one of the rooms hosting the conference's sections was called Rembrandt. It was really impressive, the upper part of the building was probably looking the same way as 1-200 years before, while the ground floor was transformed and redecorated in order to adapt it to the current destination: few offices, a big conference room, a cosy lobby and a cafeteria.

The first speaker in the A track was Frank Bannister. I found the title of his paper appealing(IT Value, meaning and identity: An exploration of things to come) and I was acquainted to some of his previous papers(see 1, 2 ,3 and 4). But I think Frank Bannister surpassed anyone's expectations: his paper was a sort of radiography of what IT Evaluation will have to deal with in the future. Starting with a classification of technologies in incremental, transformative and disruptive, Frank reviewed the possible developments together with their ethical implications.

The problems he foresees for IT Evaluation in the years to come have to do with:
-metric shifting- the more transformative an IT induced change, the more difficult to evaluate;
- subjectivity- the more transformative the change, the more likely evaluation will depend on subjective metrics;
-uncertainty- the more transformative a technology, the greater the degree of uncertainty about the outcome; the law of unintended consequences;

The conclusion was that we probably need brand new techniques for solving this kind of problems and a new critical approach, and we need them yesterday.

The comments after the presentation proved most of the audience was in a state of perplexity:
- Egon Berghout- your story was like a roller coaster!
- Dan Remenyi- paraphrasing Disraeli (there are three types of lies: lies, damn lies and statistics)- there are crackpots, super-crackpots and ICT futurologists!
The speech presented a rather scary perspective, involving cyborgs and RFIDs. But just refusing to think about will not make these possible alternatives disappear, I guess!

Another interesting paper was that of Dan Remenyi & Paul Griffiths & Ed Dinsz: Manager in the field & IT Success. The authors maintained that business management is by itself challenging, even without taking into account ICT. ICT can help or destroy a business, and in our days using ICT is so common, that it doesn't provide any competitive advantage anymore. "The manager in the field" concept proposed by the authors is a decentralised approach meant to overcome the ICT paradox. The key to ICT investment success is, according to the authors, to place all the responsibility and decisions at the manager-in-the-field level, who will be the owner and the user of this technology.

Frank Bannister opposed his perspective arguing that not everyone has business and IT competences at the same time-not everyone can do everything, and that for the sake of creating synergy, the decisions on IT investments have to be centralised. He advocated for decentralised use and centralised standards.

Another interesting paper - I was more attracted by the title (my blogger friends were at the time attending the Howard Rheingold seminar in Umea, Sweden) Smart Mobs' and technology probes: Evaluating texting at work- authored by Mark Rouncefield, Keith Cheverst, Dan Fitton, Conner Graham, from the University of Lancaster. Well, it was not exactly what I was expecting, but it was interesting. Taking Rheingold's claims on the social impact of texting as starting point, their studies are mainly concerned to understand the use of SMS texting as a means of enabling people to send messages to displays situated in the fabric of a setting rather than to another mobile device. The setting was provided by UK Mental Care Houses, where the personnel has to face various coordination problems. This was a concrete example of how common technology can become an organisational success. A logging system was embedded in the technology - as a device for understanding technology in use.

Beyond the “Average” User – A General Framework for Understanding the Determinants of IS Success Judgement at the Personal Level From Information on Individual Thinking and Reasoning About IS - Tuan Yu, Canterbury Business School, University of Kent- an interesting approach, suggesting an individual centred approach to IS evaluation. The interesting part of the approach is that it provides an insight into how the IS user reaches a judgment of whether and to what extent an IS is a success, and it tries to merge individual assessments into a generalised one. Tuan proposed a new definition for IS work effectiveness - the match of the information system to the task, and four "fits": User-Work, User-IS, IS-Work and User-IS Organisation, in an attempt to diffract the IS user's concept of IS work effectiveness into four components.

From Tuan's perspective, reflection and reasoning should play a bigger role in IS Evaluation, and the field itself should be considered part of the Organisational Learning and Knowledge Management processes - perspective that I am endorsing 100%.

The Knowledge Cafe in the afternoon was dedicated to an attempt of finding some answers to few important questions for the future of IT Evaluation research:

1. What are, in your opinion, the key research issues that academics and practitioners should focus on in order to help organisations achieve their most benefit from their investments in ITC? 2. What are the most appropriated research paradigms for these research issues?
3. What we should do as a research community to disseminate our ideas better?

Our group, formed spontaneously, included Mohini Singh (AU), Reima Suomi (FI), Jan Bots (NL), Peter Wendorff(DE) and myself. Four academics and a consultant(Peter) who declared he came there to learn what IT Evaluation really was and if it is worth it- what an explosive mixture!
After a short table tour for introducing ourselves and our experience in IT Evaluation to each other, we spent most of the time trying to provide Peter with good reasons for businesses to commission IT/IS Evaluation studies to academia and consulting companies. I must confess I was disappointed myself by our flawed performance - while KM and blogging were concepts I was prepared to explain to my mom, I never thought of IT Evaluation this way. Maybe because there are so many possible approaches, ways of doing it and situations when it proves necessary...

Of course we got to the sensitive topic of academic studies being more unbiased than those provided by consulting companies - it should be like this, but is it really?

Some interesting conclusions that were shared in the plenary discussion:
1. main research issues to be addressed in the next 5 years:
- while the field of IT Evaluation is very broad and context-dependent, there are some issues that deserve our attention: benefits management, value systems, change management, organisational culture, agile systems, technology acceptance and dissemination;
- the most important matters for the next 5 years: mobility, IT-business linkage, IT processes improvement
- people are only interested in what should they do to increase benefits
- we should look for new models; we also need better ways to present our studies;
- we will have to reconsider what we mean by success, taking into account different drivers(financial, human, health etc);
- not IT investments in particular, but complex investments including IT components will have to be evaluated;
- we will need to redefine what is value , both for profit and for social organisations;

2. research paradigms to be employed:
- positivist, interpretative, critical, post-structuralist research paradigmes should be used adequately while we are thinking ahead to new approaches;

3. better dissemination of our ideas:
- the lack of cohesiveness in the field is a barrier for dissemination;
- IT evaluation researchers should become change agents rather than "trusted third parties";
- better visibility in MISQ;
- producing a web site;
- a really significant book in the field, that would appeal to a wider audience, written in a simple language;
- presentation in Wikipedia ;

Friday, 12 November 2004 - Day 2
Keynote Speech: Daniel Hartert, CIO Philips - IT Governance and Value at Philips

Introducing the speaker, Egon Berghout spoke about 4 categories of businesses:
1. companies for whom IT Governance does not exist;
2. companies where the IT department tried to define it, but the managers don't care;
3. companies where IT Governance was defined, agreed upon, but it is not applied;
4. companies where it is defined, agreed upon, and lived every day.

Of course, the Philips case belonged to the 4th category. And we listened with astonishment to the presentation of a case where most of the things we talked about a day before were applied into practice, and real benefits were realised.

Knowing the Knower: A Systems Development Case Study- a paper by Ciara Heavin, John McAvoy and Karen Neville from University College Cork, Ireland, describing the efforts for designing and implementing an Expert Development Project Forum. Its aim was to enhance communication between learners and experts(both from inside and from outside the college). Because the forum had not been yet implemented, we couldn't find much about it's success.
I dared to ask if they ever thought of encouraging their experts to blog, as a direct way of communicating their experience and of interacting with students. The answer was negative.

My own presentation on the Empirical Study on Knowledge Based Systems went all right. Well, the story of this paper is an unusual one. At Fraunhofer, I was commissioned a study on this topic (do KBSs still exist as a research topic? who's involved in building them? what methods do they employ?). Well, my first reaction was to answer I haven't heard speaking about KBS in the last 10 years. But after an extensive literature and Internet research, I had to change my opinion: it was still there, but different research teams were focusing on different things, labelling them as KBS. Abandoned after the end of my German stage, the study received a new focus in my current host institute: which are the domains where KBSs can be useful for doing KM, what KM functions do they merely support, what methods could be used for selecting the appropriate tools for KM in every situation.

While I was pretty sure in my presentation where was I coming from, I wasn't decided about where am I going to - and this was noticed by the audience (few people I remember being there:Frank Bannister, Tuan Yu, Dirk Deschoolmeester(the session's chair), Peter Wendorff, Shailey Minocha). There was an interesting discussion afterwards, and some of the questions helped me a lot to see the flaws in my paper and its possible continuation.

Thank you all, your help was extremely valuable and I will use your suggestions for re-writing it, since my paper was selected for being published in the EJISE!

Later on, I had to chair another session. Three papers were presented:

Using Causal Mapping methods to analyse risk in an implemented Information System project as a post-evaluation process.- Abdullah Al-Shehab, Robert T. Hughes and Graham Winstanley, University of Brighton, UK - graphical representation of various risks A

Structurational Perspective on Technology: An Evaluation of why the same Technical Solution is Adopted Differently Across an Organisation- Arild Jansen, University of Oslo and Nes Terje, The national archives of Norway, Norway - an evaluation of a complex system for case processing based on the Giddens & Orlikowski structuration concept and on the Actor Network Theory.

Building evaluation into REALITY Telecare- Ewart Carson and Abdul Roudsari, City University, UK - the results of a European project on caring for patients in home settings - a summative evaluation and the difficulties encountered in quantifying different attributes.

Afternoon- Panel discussion led by Egon Berghout
Questions to be answered:
1.Do organisations really take ICT evaluation sufficiently seriously?
2.Should they take ICT evaluation more seriously and if so why?

The panelists were Dan Remenyi, Frank Bannister, John Cormack and Shaun Pather.
Well, the panelists mainly answered "yes" to both questions, so Egon Berghout as a moderator had to play the devile's advocate trying to balance the discussion. The way he did it was excellent: he unvailed a lot of weak points of ICT evaluation we were all aware about - in a sort of enlightening mockery.

Some of the arguments brought into discussion:
1.Do organisations really take ICT evaluation sufficiently seriously?
- yes, but unfortunatedly most of the times the benefits figures are twisted to suit people's concerns;
- yes, at least in Ireland in the public sector, where almost no public service computer disaster ever happened. The secret- a set of accounting type guidelines, focused on costs and benefits.
-yes; management is ready to listen, but not ready to act upon. All they want to know is how to use ICT evaluation to increase benefits. While cost reduction reached a limit, other alternatives such as option pricing and built-in risk are rising.
-no; 1) it isn't the first priority 2) it's a mess anyhow 3)it works without it.
-the tools (BSC, ROI) are not good enough - yes they are, it is not a tool issue;
-studies are usually initiated by the IS/IT departments - biased;
-business people are not interested in evaluation - we, the academics, are;

2.Should they take ICT evaluation more seriously and if so why?
- there are specific domains, like banking, where they don't have any other choice;
- for other situations, evaluation is expensive, time consuming, nobody's pressing us to do it - so forget about it!
-yes, they should, and non-monetary evaluations should be given more attention;
-yes, but putting the emphasize on more strategic matters (not discounted cash-flows)
-not for every investment; think of the IT embedded today in most of the machines; in the public sector, benefits could be pretty nebulous;
-yes, because companies should become more aware of what ICT means for their growth.

After the panel, I had to run and catch my train.
Few other things:
A couple of interesting papers found in the Proceedings (when there are 4 parallel session, you always miss something important :-( ):

A managerial release-decision model for IT -applications- J.A. Sassenburg and Egon Berghout, University of Groningen

Narratives: Missing Link in IS Evaluation - Andreas Borell, Tetra Pak Global IM, Sweden, Jonas Hedman, Lund University

Evaluation of open source software: comparative perspectives between propriety and open code development - Andrew Schofield, Amit Mitra, Salford University

IT Governance Disclosure of Web100 Companies - Gail Ridley and Qiang Liu, University of Tasmania


Networking was great:
- long coffee breaks and unsofisticated lunches gave us the chance to speak to each other; the organisers did a great job by introducing people to each other and by bringing together people sharing the same interests; I had the chance to meet Ann Brown (who's name and face were so familiar to me from the conference site ever since I started to study the field) and to spend quite some time with her, Mohini Singh, Tuan Yu, Dirk Deschoolmeester and a lot of others.
- the conference gala dinner was held at the Sea Palace, a floating Chinese restaurant. Didn't have the chance to circulate much around, but I found great company at my own table.
- I didn't lose any chance to convert people to blogs, wikis and social networking; definitely, IT Evaluation people are far away from these tools, but why should they?
- I found several people who never heard about Wikipedia; am I a kind of weird?
- a young PhD candidate, Dace Apshvalka, from Riga Technical University, presented a paper on Personal Knowledge Management; I didn't have the chance to attend the presentation, but I had the chance to talk to her during the coffee break. She wasn't aware of the blogging phenomenon, never heard of Lilia Efimova, Denham Grey , Steve Barth or David Gurteen. Well, I think this is the risk when you stick to the books and paper magazines, and avoid Internet sources - not to get the real touch of things. Funny, right now I tried to Schoogle PKM - I got lousy results. I think I'll stick to what Google says.

Getting back to your first love doesn't prove to be a good idea in most of the cases. Well, IT Evaluation was "my first love", but I suppose I changed a lot in these last 5 years, I do not feel the same anymore. I feel much better now when I meet people from the Knowledge Management field, and even better around bloggers. Anyhow, it was an excellent conference and re-connecting to the field gave me some great ideas for future research!