SAM Updates

I have changed the main SAM tests from SRM to SRMv2. I have also added ATLAS specific SAM tests (to join CMS and LHCb ones). It is not clear if these are the right tests. Each SAM summary page now has a list of the tests being polled under the results table. If the experiments want different tests please let me know.

Proxy problems

Most tests have been red for the last 24 hrs or so due to failure to submit any jobs. This was due to trying to use an ATLAS production role which seems to stop anything working. The tests should be OK again now I've stopped trying to do this.

New Server

I have moved everything on to one dedicated server. There should be no noticeable change but if there is please let me know.

CMS SAM Tests added

I've added CMS specific SAM tests to the suite. See
http://pprc.qmul.ac.uk/~lloyd/gridpp/cms_samtest.html.

Network Tests Updated

I have revamped the network tests as agreed at a dteam meeting a couple of weeks ago. The tests now involve copying files from the Tier-1 to the local SE, copying the files from the local SE to the WN and copying them back to all SEs. There is quite a high failure rate. Some of this is transfer/catalogue/info failures and some of it is attempts to get more reliable results by using different timings and asking for consistency between them. Sometimes the times vary wildly with smaller files taking longer etc. I am sure there is more to be done.

LHCb Tests

I have added LHCb tests to the collection. These are actually SAM tests tailored to LHCb. They are kept separate from the normal (Ops Critical) SAM tests to avoid distorting the availability/reliability statistics. See http://pprc.qmul.ac.uk/~lloyd/gridpp/lhcb_samtest.html.

Local problems

The bad news: The raid array I store my outputs and log files on developed an error over the weekend. It is being rebuilt now but may take some time before it comes back.

The good news: I've instigated a 'heartbeat' system so that if all the tests stop running and the pages look green there is something to tell you that in fact it hasn't been updated for many hours.

Network Tests

I have introduced some network tests. Test jobs are sent to every UK site once an hour which attempt to copy a number of differently sized test files (1MB, 5MB, 10MB, 50MB and 100MB) from the Storage Element at each of the sites. The transfer times are then used to calculate the average throughput in MB/s. The results are averaged over the last 24 hours to produce a matrix of site to site transfer throughputs. There is an average over the whole matrix on the main UK Grid page. See http://pprc.qmul.ac.uk/~lloyd/gridpp/nettest.html.

New test

I have enabled a new test which sends an ATLAS analysis job to ".ac.uk" every 10 minutes and wherever it ends up reads data from the local SE and does some analysis. This should give a much better estimate of success rate as seen by users (as jobs won't go to sites that are down). There is still the problem for users whose data is on the site that is down of course). See http://pprc.qmul.ac.uk/~lloyd/gridpp/uktest.html for a summary.

New views

There are a new set of results pages showing all the results for a particular site or Tier-2 on one page. Go to the main page
http://pprc.qmul.ac.uk/~lloyd/gridpp/ukgrid.html and click on one of the institute names down the left hand side. Or click on a Tier-2 just above the big table.

FCR History

I have been collecting the FCR history since just before Christmas. I have now put up a new page that shows the percentage of time each site is blacklisted by each VO using the FCR. This can be found at http://pprc.qmul.ac.uk/~lloyd/gridpp/fcrtest.html and is linked from the other test pages as "FCR".

Late Spring Clean

I've done some tidying up so that the ATLAS tests are more resilient to changing versions of the ATLAS code and this seems to mean that sites with only Release 13 and not 14 are passing again. I have also tried to make all the times consistent - some were previously GMT and some BST. They are now all supposed to be BST in Summer and GMT in Winter.

Release 14 Complete

All three ATLAS tests are now running on ATLAS Release 14. If you want to pass you need to install 14.1.0 or higher.

Release 14 User Analysis Success

Release 14 User Analysis seems to work at some sites at least. The Z mass has gone up from 96.661 to 97.203!

Upgrading to ATLAS Release 14

I am currently upgrading my tests to ATLAS Release 14 starting with User Analysis. All sites without Release 14, which requires SL4, will fail the test. All sites will probably fail while I get the bugs out on my side! For the time being the other tests will use Release 13 (even if 14 is installed).

New SE Tests

There are now some new tests of the UK SEs. These run every hour (for the moment) and attempt to copy a (2.8MB) file to an SE (lcg-cr), check it is there (lcg-lr), read it back (lcg-cp), delete it (lcg-del) and check it is no longer there (lcg-lr). The log files from these operations are available by clicking on the test result. You can see that the results are not 100% correlated with the SAM test results (which is why I did this). See:
http://hepwww.ph.qmul.ac.uk/~lloyd/gridpp/setest.html.

Auto RB

On the basis of my RB tests I now define an 'Auto RB' which is one of the 'Good' RBs or failing that one of the 'Fair' ones. This RB is then used for my ATLAS tests. This should avoid having to manually switch RBs every time the RAL ones go down.