Report Card and Dashboard
July 22, 2004
Wenmey Hsi:
Does anyone ever has done anything like dashboard stuff? I would be much appreciated if you
can share your experience with.
I have been asked from my boss (IR Director) about this info from OCAIR group.
Shiji Shen: Wenmey:
At the poster session of this year's AIR conference in Boston,
Bruce Morrison of Grand Rapids Community College did a poster presentatino on Dashboard.
I believe this presentation won an award at the award luncheon, if I remember it right.
I consulted the AIR Roster and found his email: bmorriso@grcc.edu.
You may want to try to contact him. Good luck!
Henry Y. Zheng Wenmey,
I assume that the Dashboard stuff you asked is the Dashboard of
Indicators for the Balanced Scorecard. There are tons of materials out there
on the internet and at AIR and AAHE Assessment Conferences. Do a
"google" search with the following key words: dashboard scorecard college.
Here are a few links from the first 10 hits in the search that may be
helpful:
Biao Zhang (Bill):
I have just started doing something similar on this campus, having come back from a seminar on
balanced score card in higher education, which is on how to construct a dashboard of
performance indicators. So I have got some thoughts and have collected some materials.
If you can specify your question, I may be able to help.
Wenmey: Dear Biao:
We learned dash-board as a new name and also have some print out from the Web sites as a beginner.
We are trying to figure out what we can do on our campus about dash-board.
Do you have any good experience from runing this?
Meihua Zhai: Thanks a lot, Henry,
Biao and Shiji for the searching and the sharing.
Sometime ago, when I mentioned the design and
implementation of a campus portal, I heard people
say:"You mean something like a dash board." I wasn't
quite certain if I could draw an equation sign in
between the two, but I figured that regardless of the
names, the functions would be the same: for IR offices
to turn data into information/knowledge and use data
for some kind of measurement and decision-making
support.
Based on what I read from the URLs below, I think dash
board is more like a report card. Using the definition
by St. Charles CC, somehow, I think any performance
profiles can function as a dash board. Am I right?
Constructing such a board would require lots of
thinking at forefront.
Biao: Would you mind sharing your collections with us
and we add "Instititution Profiles and Dash Board" to
our knowledge Base?
Meihua Zhai: I think the most difficult part for the dash
board would be to determine when to use what color by whose
criteria.
In going over the Unit Profiles that my office
constructed, one of Mason's Board members and my boss,
the Senior Finance and Planning VP commented that the
profiles were still reporting (true). They suggested
that I use different colors to indicate how healthy
the numbers were. My immediate mental/inner reaction
(didn't have the guts to disagree or disobey)
was:"what should I use to measure against?"
Internally, we cannot compare one school with another
(Nursing is for sure more expensive than Arts &
Sciences) and externally, where can I get reliable and
objective data?" Measuring against ourselves is
meaningless. Even for the school of computer
sciences, different institutions might have different
majors...
My thought right now is to use different schools'
strategic plans/goals as the benchmark in determining
the color system (how am I going to plug those into
any programming codes to automate the process). This
might work well with enrollment goals. How about cost
and effectiveness? Ratio? This is not an one-office
project. It must come from the top, higher than IR,
even higher than the Provost and the President.
Just my $.02 to share. No intention of being a Ms.
Mission Impossible.
Biao Zhang: Yes, the idea of dashboard is not new in higher education or in
management/business administration. It is simply a collection of
performance indicators, arranged for executives such as P, VP's, or
Deans. It looks like there can be dashboards at all levels of
administration cascaded from the P down to the department. Apparently
the dashboard indicators may be different for the P than for the chief
academic officer, as for the former the indicators will have to measure
all areas of the institution, and for the latter the focus will be on
effectiveness and efficiency of instructional delivery and faculty
performance. For example, the indicators for the P may include
enrollment, retention, graduation, credit hour production, fiscal
viability, fund raising, public relations, space utilization, strategic
initiatives, etc. For the chief academic officer, the indicators may
include course evaluation results, outcomes of gen. edu, results from
major field tests, faculty qualities etc.
Indeed, there are some challenges in implementing the dashboard.
To begin with, it is not something that IR alone can handle. It has to
start with the top. It takes a mindset of the top management team. In
other words, unless the top dogs want to do it, you can not do it.
After
all it is going to be a 'dashboard' for the P and VP's. They are going
to be in the driver's seat looking at the meters.
How much of it is art, and how much is science? Hard to tell.
For instance, there are dozens of, if not more, potential performance
indicators out there that could be used for dashboard indicators. Just
think of the variables collected in IPEDS surveys and US News surveys,
College Board, CDS. What variables would you select to put on the P's
dashboard? You will have to review theories in higher edu
administration, the strategic plan of your institution, the strategic
initiatives, the mission statement. You may need to build a link
between any one of these to the dashboard indicator, which can be difficult at
times. Then, the executive for which the dashboard is intended will
have to buy it.
Another challenge, like Meihua mentioned, is how to determine
the color coding, you may or may not want to calculate the %95
confidence intervals for all the indicators. In some cases, you will
have to use an arbitrary decision. Without question, the executive will
have to bite into it.
Benchmarking may use the averages of your own past records or
the averages of your institutional comparison group, depending on the
specific indicators.
Do I have any good experience from running this? I should say
that I have got considerable experience doing the performance
indicators at different institutions. Is it 'good' experience? Depending on what
you call 'good'. The dashboard provides a good management tool, and a
good frame of thinking about institutional effectiveness. It is a good
exercise for IR professionals to prepare the dishes and serve in
different plates to the diners (executives), this GongBao Chicken is
for the P, that Sweet & Sour Pork is for the VP. It is also good in the
sense that IR has a job to do, and lots of work. This work may be
easily recognized by the bosses, as they are going to use the dashboard
(hopefully). Does the dashboard guarantee a managerial success at your
institution? Who knows? What we do know is perhaps that those who do it
are more likely to succeed than those who do not.
Sorry that I ran so long, did not mean it.
Datatel's Link on dashboard:
http://info.nwmissouri.edu/~oehler/oehler.html
Jion Yen: Although I was not involved in developing dashboard indicators for my
institution, I've gained some knowledge from reading our IR historical
document on this subject. Below are my findings:
- What are them?
Dashboard indicators are strategic measures designed to track
institutional progress. They summarize strategic information that
gauges the relative progress and effectiveness of your institution.
Simply put, they are performance indicators that are vital to your
institution's vitality. For example, listed below are four indicators
form finance aspect:
- Indicator #1: Revenue-Operating Income
- Indicator #2: Expenditures
- Indicator #3: Endowment Fund
- Indicator #4: Annual Giving
- How to measure?
A strategic measuring in relatively term on the above indicators, which
means comparisons are decided by the focus group that comprising VP
level executives. Their membership is institution specific and this
group decides on every aspect that goes into the dashboard project.
- How to present?
Depending on your audiences, the data layout could be in word, excel,
or/and PowerPoint format. The imperatives are the visual display on
measures against your baseline data which again are arbitrarily
determined by the focus group. The color code could be in Green,
Yellow, and Red to signify the same notions as those from the traffic
lights.
- When to measure?
This is also the focus group's decision, ranged from quarterly to
annually.
- How is it be used?
My institution completely abandoned this project before its full
operation two years ago. The decision made before my tenure here so
there is no info on why such a tool to gauge institutional relative
progress did not pass beyond its initiative stage.
Brian: Bill:
It is a wonderful piece about the Dashboard project. I quite agree with you on the
determining of indicators. There are too many indicators out there, but it all depends on what your
leader wants to look at. I think for IRers, what we need to do is to provide research results, theories and
actual data analysis for our boss to buy in.
The dashboard is actually a transformation of the balanced scorecard in the business world.
There is a Balanced Scorecard Institute in San Diego, which provides a lot of studies, articles and
advice on balanced scorecard. You may go to
http://www.balancedscorecard.org/ to check those articles.
I have been working with our President Cabinet for a U. Scorecard Project benchmarking project.
I have found their interest is really different from IR’s, although I have tried to persuade them to buy something in.
Our VP of Student Affairs who is in charge of the project wants to keep the indicators under 30. The
attached is what we finally come up with. Here you can see
what our President Cabinet is interested in. We have selected 8 compatible institutions and now are
collecting data from those institutions (most of them actually available in CDS or other sources except
research grants for faculty and the finance part). I would like to have some feedback from you, if possible.
Bill: Jion,
I guess your institution is not the only one that abandoned it. Echoing
your case, the first question that needs to ask when you do a score
card is how long has the current P in office and how long will (s)he be?
Again the top dog will have to buy into the idea in order for it to
take off.
Jion: Biao,
So the indicators on the dashboard project would be those you described
below. Its success rate all depends upon the stability and mentality of
those higher up. I agree with you 100%. Thank you.
Henry Zheng:Hi, Brian,
Thanks for sharing your knowledge and informatin.
I have been working on balanced scorecard projects for more than 6
years. So, let me offer my opinions on the example that you shared with
us. Let's pretend that we are doing this as a case study.
In my opinion, the excel table that you sent out represents a typical
IR laundry list of data items. They are data items, not performance
information. Yes, from this list of data items, you know the enrollment
level, the faculty size, financial aid situation, or other data about
the current state of affairs at your college. Now, after knowing all
these data points, so what?
What do these data points tell us about where this college is in
relation to its competitors? And who are its competitors anyway? What about
changes over time? Are you better off than what you were 5 or 10 years
ago? What are your college's strategic growth areas and what are your
priorities? Can we tell from those data?
Many colleges are doing scorecard for scorecard's sake. They are not
very conscious about the rationale behind it. In performance
evaluation, we keep saying, "what gets measured gets done." This means that you
measure the things that you want to change. Tracking organizational
data is important. But not all data should be in the scorecard.
Scorecard items should always reflect the organization's strategic focuses and
operational priorities.
In the balanced scorecard literature, a key principle is that strategic
planning must be data driven, but once the planning is completed, the
strategic focuses and priorities should drive the selection of
performance indicators. In other words, you measure what you want to change.
Strategic planning is a dynamic process and we contanstly have to
examine our data and baseline to evaluate whether the strategic directions
that we set earlier need to be revised. That is why the data points are
important. But those data points cannot be the scorecard itself. We
need to decide what are the important things to measure, and measure them
against our benchmark schools, and over time. We need to set
performance targets and stick with them. Assign individual responsibility (at
the VP level) for meeting those performance targets and reward those who
meet the targets.
I may sound like lecturing but really this is how it should done. Like
you, Biao and Meihua suggested, a meaningful scorecard project must
have the highest level buy-in. If not, you may not have the necessary
information to know what are the most important things to measure.
Just my two cents.
Brian: Henry:
Well said! I said similar things to our leaders but your words weigh
more than mine. I agree with you 100% that high-ranking administrators'
buy-in is the critical factor for such a project; otherwise there is no
point of doing this with much trouble in collecting and analyzing data.
Our leaders finally agree that this data collection will set up a
baseline for a five-year/10-year strategic plan for the administrators
to set up some measurable goals. But, to tell you the truth, to me this
is still a game of planning and nobody can really control all the
factors to predict the future. Moreover, how can benchmarking control
all different variables that each individual institution is situated
in?
Anyway, administration is administration; it often needs to look good
in planning, organizing and administration. So, our role here is to
provide some measurable indicators and data to make administrators think how to
look good! (Politics is politics all the time.) Therefore, in my
opinion, assessment is more valuable than planning; at least it can
give you some sense about the history and current performance status.
Dashboard is basically used for planning and quality control, some kind
of non-sense business after all :-). Thanks.
Shuling Chen: I'm joining this discussion a little late, but wanted to share with you
a pretty useful source I have consulted when presenting performance
indicators to the board of trustees at my institution. The book is
Taylor, Barbara E. and William F. Massy. 1996. Strategic Indicators
for
Higher Education: Vital Benchmarks and Information to Help You Evaluate
and Improve Your Institution's Performance. Princeton, NJ: Peterson's.
For each indicator, the book explains the significance of it,
interpretation and questions that policy makers should ask. At my
college, we had created a Board Information System that provided our
Board of Trustees with information about different performance
indicators. Since not all trustees are familiar with the higher
education "business", I focused on explaining the significance of each
indicator as well as how we are doing in comparison to our peer
institutions. The cabinet also used it as an opportunity to share with
the trustees current institutional priorities.
Huiming Wang: Thank you for the resource.
I was trying to find the exact reference to share.
At the U of MO, I participated in the DB project (The System's
President's project, of course). We invited Willaim Massy over to
speak to the Univ. presidents. I remembered he was especailly instrumental
in suggesting how the subjective scores and the colors of the DB should be
determined, or rather agreed upon. There were lots of discussions
involved. Everyone has to agree upon each measure, each comparator.
In the end, only the general portion of the indicators were finalized.
What I learned from this experience are:
- Educate the decision-makers of the theoretical model;
- Inform the decision-makers of the rationale of using each data
element, and how the data are derived and used;
- Keep the all the windows open, nothing is final, or frozen,
goals and objectives can be revised every year
In the beginning, the president intended to tie the indicators to
budget allocation which was too intimidating to all. With the money in mind,
no discussion was even possible. In the end, the consensus reached
that the DB project will be an on-going effort with all the changes as we
go.
Hope this is not too late to catch the Ocair DB train :-)
