Bb Performance Problems after Upgrade to 7.1

The Bb upgrade from 7.0 to 7.1 involved a lot more than simply upgrading the Bb application. Many infrastructure changes were also done during the upgrade (5/22/07 - 5/23/07). To better handle the number of small files in Blackboard the infrasture team recommended that the back end file server be changed from the current configuration to a new file server using the Veritas file system. Testing showed that throughput to the Veritas file system was much better than to the old file system. Backups from the Veritas file system fininshed in about half of the time as the old file system.

However, after upgrading to the Veritas file system Bb performance slowed to a crawl as the Bb application seemed to wait for files from the file system. After much research, and with key technical help from Jeremy at Bb, it was found that while regular file IO is faster on the Veritas file system, when Java access files for Bb that file system was much slower responding to file attritibute requests (does the file exist, does Bb have access to the file, etc).

Testing confirmed that when a Bb 7.1 web/app server was pointed at an "old" file server with current data that Bb performance problems went away. To resolve the issue a plan was developed the move the current 7.1 data to a new LUN on the EVA that would be accessed through the "old" file system (but with a larger LUN). This plan was executed the night of 5/30 and into 5/31. The morning of 5/31 when Bb was pointed at the old file server with the current data in the larger volumne, the probem was resolved.

This problem may have been avoided if the test environments for Bb more closely mirrored the production environment at the infrastructure level. Due to the limitations of the "old" file system this issue still must be addressed. The old file system takes too long to backup and there is the danger that there will simply be too many files for the file system to support (especially in the "courses" folder). It is likely that the file server configuration will be changed at the end of the Summer I semester, the night of June 28th.