File Server Upgrade Completed

The Blackboard file server was replaced with a new file server running the Veritas file system. The new file system will give us better backups and the ability to expand the EVA LUN easily. Blackboard was performance tested with Empirix both before and after the file server was replaced. The new file server performed as good or better in all tests.

Also during this maintenance time the PAE/High Mem switch was enabled on the SQL server. This should give SQL access to more memory and reduce OS paging. This could help SQL performance.

LO Campus Pack Building Blocks For All Courses

Today the problems with the license keys from Learning Objects has been resolved and the new keys have been installed in the production environment. The new license keys will expand the availablilty of the Learning Objects Campus Pack Building Blocks to all courses in Maricopa's Blackboard.

The new license keys will take effect Saturday morning (07/21/07) when the production Blackboard environment does its daily reboot.

Bb Announces 7.3 - May Upgrade to 7.3 and Content System

With the announcement today by Blackboard Inc. that Bb7.3 is now available it should be the assumption that the May 2008 upgrade will include Bb7.3 and the Bb Content System.

Given the normal development/patch cycle of Blackboard I expect at least one service pack to be released between now and Spring 2008. Maricopa will still need to test both 7.3 and the Content System but unless major issues are found in this combination of products we should plan for the May 2008 upgrade to include Bb7.3 and the Content System.

Blackboard Grade Export Building Block

Maricopa continues to work with the vendor that is developing the Grade Export building block. Testing is going well and every indication is that it will be ready to be installed in the Blackboard production environment sometime in July.

The performance impact of the Grade Export Building Block is being rigorously tested using the Empirix tool to simulate students on the Test system while the export is being executed.

Initial Fall 2007 Course Creation Completed

Over the weekend (6/9 & 6/10) the initial Fall 2007 course creation completed. The initial phase of course creation was done while Maricopa's Blackboard was experiencing issues with the file server (see post below for details). This weekend run completes the creation of all courses that are currently flagged, bringing the number of Fall courses to just over 5700.

Beginning 6/11/2007 the Fall 2007 course creation will now be included in the nightly inteface run, any new courses for Fall 2007 that are flagged during the day will be created that night.

Bb Performance Problems after Upgrade to 7.1

The Bb upgrade from 7.0 to 7.1 involved a lot more than simply upgrading the Bb application. Many infrastructure changes were also done during the upgrade (5/22/07 - 5/23/07). To better handle the number of small files in Blackboard the infrasture team recommended that the back end file server be changed from the current configuration to a new file server using the Veritas file system. Testing showed that throughput to the Veritas file system was much better than to the old file system. Backups from the Veritas file system fininshed in about half of the time as the old file system.

However, after upgrading to the Veritas file system Bb performance slowed to a crawl as the Bb application seemed to wait for files from the file system. After much research, and with key technical help from Jeremy at Bb, it was found that while regular file IO is faster on the Veritas file system, when Java access files for Bb that file system was much slower responding to file attritibute requests (does the file exist, does Bb have access to the file, etc).

Testing confirmed that when a Bb 7.1 web/app server was pointed at an "old" file server with current data that Bb performance problems went away. To resolve the issue a plan was developed the move the current 7.1 data to a new LUN on the EVA that would be accessed through the "old" file system (but with a larger LUN). This plan was executed the night of 5/30 and into 5/31. The morning of 5/31 when Bb was pointed at the old file server with the current data in the larger volumne, the probem was resolved.

This problem may have been avoided if the test environments for Bb more closely mirrored the production environment at the infrastructure level. Due to the limitations of the "old" file system this issue still must be addressed. The old file system takes too long to backup and there is the danger that there will simply be too many files for the file system to support (especially in the "courses" folder). It is likely that the file server configuration will be changed at the end of the Summer I semester, the night of June 28th.

Fall 2007 Courses Created

The Fall 2007 courses will be created over the Memorial Day weekend. The process will start on Saturday morning (05/26/2007). This initial course creation for Fall 2007 will create just over 5,600 courses in Blackboard.

Bb Upgrade to 7.1 Complete

On Tuesday May 22nd and Wednesday May 23rd Maricopa's Blackboard was upgraded from Bb7.0 to Bb7.1. This upgrade was completed ahead of schedule.

The upgrade includes backend infrastructure changes in additon to the upgrade of Bb itself.

• Installed new hardware for the file servers
• Installed a new more robust file system (allows the file store to be grown more easily)
• Setup file replication to almost instantly copy all uploaded file to a different EVA
• Increased the size of the file store on the EVA by about 200GB
• Updated the firmware in the EVA disk drives
• Applied OS and security patches to all Blackboard servers
• Moved some of the Bb web/app servers to a different switch (adds redundancy & reduces risk of downtime)
• Upgraded Bb to the latest build of Version 7.1 (7.1.467.28)
• Configured Bb to use a new e-mail relay server (sysmail2)

MCC Interested in Using Bb for OE/OE Courses

In the past couple of days both Clyde and Randy have received calls from Mesa Community College saying that they are interested in using Blackboard for their Open Entry/Open Exit courses that had been delivered using MIDAS. This information was passed onto Steve and Darrel. Darrel is negoiating the new license agreement with Bb that would have been for all of the colleges except Rio and Mesa. Both Mesa and PC are WebCt users, PC is just beginning their migration from WebCt to Maricopa's Blackboard.

MIDAS Update - Rio off of MIDAS

On May 21, 2007 Clyde Resendez e-mailed Angela Ambrosia (and others) at Rio because he noticed that Rio had not been sending grade interface data from MIDAS to SIS. Later that day Angela responded that Rio is now completly off of MIDAS.

Bb 7.3 Rumored to be Released Summer 2007

Today our Blackboard Technical Service Engineer told Dave Fraedrich that he has heard that Bb 7.3 will be released in Summer 2007. If true, the timing of this release should allow Maricopa enough time to evaluate Bb7.3 in preparation of the May 2008 upgrade. If Blackboard follows their normal maintenance schedule, SP1 for Bb7.3 would be released near the end of 2007 or early in 2008. After testing Bb7.3 the Blackboard Alliance will then be able to decide what to include in the May 2008 upgrade: Bb7.2 or Bb7.3 and possibly the Content System.

At this time we do not know what new features will be included in Bb7.3. I expect that there will be a lot of discussion and possibly demos of Bb7.3 at the Blackboard "Bb World" user conference in July 2007.

BbTest Environment "Dry Run" of May Upgrade Complete

The BbTest environment, which contains the recent restore of Bb Production data, has been upgraded from Bb7 SP1 HotFix 4 (7.0.404.41) to Bb 7.1 SP1 HotFix 3 (7.1.467.28). This upgrade using the restored production data is a test or "Dry Run" of the May upgrade scheduled for May 22nd & May 23rd.

Bb Production Data Restored to BbTest Environment

In preparation for the Bb upgrade in May production databases and course content files have been resored to the BbTest environment. This environment has been tested and appears that the "recovery" has been successful.

BbDev2 Environment Upgraded

The BbDev2 devlopment environment has now been upgraded to: Bb 7.1 SP2 Hotfix 3 (7.1.467.28)

Bb Outage

A bad data center switch caused Bb to be unavailable this morning. At 5:14 AM CMS was notified by PreceptIS (our helpdesk vendor) that Bb was unavailable. Shortly thereafter ITS reported that a network switch had malfunctioned. ITS replaced the switch and Bb was back on-line at approximately 9:12 AM.

Bb 7.1 Test Server on BbDev2

25 April 2007
"BBDev2" test environment with Bb 7.1 only was released to community for testing and training purposes. This environment is a new build on a temporary server and does not include the Content System.