The STAR experiment

Minutes 

Attendees: Wayne B., Leve H., Jérôme L., Gene V.B.

ShiftLog [Leve]:

• New server:
  • Ready:
    • App copied to new server (Wayne)
      • Successfully passed stability tests (pounding on it)
    • All WARs deployed
    • Scripts ready to flush cache and re-copy WARs
  • To do:
    • Will re-check links to and from RunLog
    • Watcher script not on yet
    • Verify Offline QA Shift Report connections

Online infrastructure [Wayne]:

• New web server:
  • Links on homepage checked
    • Some issues with DB access and privilege from current node name (dean2)
  • New jpgraph version installed in same location as the old one
  • Disk exports are not in place, so content isn't being updated currently
  • Will do another sync over the weekend or early next week, in addition to the final one when we swap servers
  • Ganglia to be installed
  • User accounts to be added
• New web server backup:
  • Configuration kept the same as main new web server
  • ...but content isn't sync-ed
• Password rotation completed
• mysql-devel installed on online linux nodes (RT 2068)
• Two online disk failures (on onl09 (non-critical) RT 2065 and on onlldap (critical) RT 2067)
  • RAID still providing for the file systems in the interim
• ITD network backups failing (RT 2069), but resolution from ITD may have just come through
• Machine replacements:
  • emcsc ready (needs either deployed by tomorrow or ITD block needs postponed)
  • emcspin ready (Monday)
  • starutilities awaiting C-AD software

FastOffline [Jerome]:

• Many files coming into the system, but all are tests, pulsers, etc., so not being processed
• Some missing detector setup, beam energy, etc. information: could mean misconfiguration, or potentially not properly propagated data

QA [Gene]:

• OnlinePlots switched back to newer EVP server
  • Switching steps still undocumented
• Pre-combining of FastOffline files still needs tested and configured to be the default

 Minutes

Attendees: Kefeng, Wayne B., Leve H., Gene V.B., Dmitry A.

ShiftLog [Leve]:

• Manual changes have been reviewed (by Jerome) and committed
• Re-deployed

Databases [Dmitry]:

• cdev enabled (by C-AD), so parameter propagation has begun
  • Some data is still empty (e.g. beam species) and may not be filled for several weeks (when beams actually start)
    • Might be possible to enter some default values for now
  • Numbers are meaningless, so not useful for FastOffline testing
• Testing new hardware nodes now (2 are available)
  • Configuration of new nodes in discussion

Online Infrastructure [Wayne]:

• STAR login environment needs to be able to handle SL5.5 (which was installed on some nodes)
  • Will follow up with Jerome
• Online linux pool is losing roughly 1 node a day (rebooting)
  • Hit a few with network scans, but response was OK, so no clues
• Spin request (for Pibero):
  • User has access to the nodes and is waiting for directory structures and mount points
    • Expect to complete within a few days
• Condor installation on for rterm-like access to online pool from gateway
  • Installed on the gateway
  • Components for the pool nodes is to be done
• Slowness on evp
  • Only occurred yesterday (Dec. 16)
  • Nothing obvious, but suspicious of AFS, and coincided with mock data transfer challenge from counting house to HPSS
  • Paths forward discussed:
    • Remove outside AFS dependence using an online repository
    • Remove AFS dependence using local codes on evp
    • Write a new tool better identify AFS issues (i.e. more proactive than 'fs checkservers')
  • Problem doesn't seem to persist, not a priority for now (no action)
• Yury G. requested networking support for the east FPD/FMS rack
  • Just extends the starp network geographically
  • Needs to be a fiber connection for proper grounding
• Webserver replacement
  • Two redhat 5 machines given the ITD thumbs up
  • Shared filespaces in progress
  • No services started yet
  • Request for an account to test tomcat (Leve)

QA [Gene]:

• Demo of the Offline QA with reference histogram comparison
  • Missing some features for flexibly defining reference histogram sets (will work on over the holidays)
  • Minor suggestion made for improving the "waiting..." display

 Minutes

Attendees: Leve H., Dmitry A., Jerome L., Wayne B., Gene V.B.

Databases [Dmitry]:

• All DB collectors running now
  • Only TPC voltages absent due to no data yet
• Slow Controls Archiver not yet running to avoid collecting value-less data
  • With Run start imminent, this needs to get going
  •  Dmitry will follow up with Yury G.
• Potential to benefit from an unused EPICS feature to read multiple channels in one request (we have been reading one channel at a time across the board), which could cut down the overhead in obtaining data
  • Unknown why this wasn't previously used, so testing with caution to learn (perhaps the multiple channel data comes in a burst which could fail for some reason)
• New shift sign-up release is imminent (possibly today)
  • Only minor features getting final adjustments (features have been [node:19902 "previously presented"])
• RunLog working fine: new runs appearing, but all being marked bad
  • Jerome notes that bad runs won't come through FastOffline, which has been turned on already
• New DB nodes
  • Tested and ready for use
  • New configuration (relevant for FastOffline use) not yet in place (Jerome and Dmitry will work this out)
• Online DB plots working and logs show usage
  • Only unavailable quantities are those collected from the Slow Controls Archiver

ShiftLog [Leve]:

• New manual is printed out and in place at the counting house
• ShiftLeader desktop computer has all the necessary icons, correctly linked and tested, including making a ShiftLog entry
• New web server not critical (stable operation on old web server)

Online infrastructure [Wayne & Jerome]:

• OS upgrade
  • One critical: a replacement node for usage in monitoring chilled water (old machine is Windows 2000) is awaiting ITD processing.
    • Follow-up with ITD on Monday if nothing transpires by then
  • Other machines in the queue ("bond", "beatrice", "l3disp")
• Online web server replacement
  • Currently replicating old machine on new one: file copying and version checking
    • ShiftLog request to continue with same Tomcat version (a known quantity) instead of upgrading in hopes of improved stability
  • Spin resource request involves storage to be delivered on new server
• Uncertainty in status of UPS services for computers at the experiment
  • Testing today before the Run starts
• Network request for east FMS/FPD racks completed
• Online linux pool has been stable for the past ~3 weeks (after a couple weeks of apparently random reboots)

FastOffline [Jerome]:

• Ready, running, and waiting for data

QA [Gene]:

• OnlinePlots still running on older evp machine (but stable)
  • Will look into moving it back over to the new node
  • Need Jeff L. to flip some switches (and document it)
• Offline QA
  • Finished implementing tools to update just specific histograms in reference set
  • QA Shift set of histograms will be flushed next week and started anew
    • Histograms only go into the set if given a description, and only stay in the set if a reference is set

We are expecting some collider operations imminently, so data will likely flow through the entire system over the next week.